Software Engineer · advanced

Distributed Job Queue & Worker System

An async task processing system with persistent storage, auto-retries, dead-letter queues, and job prioritization.

Project overview

This project mirrors the core infrastructure of almost every major tech company (Uber, Airbnb). It shows you can handle complex data integrity issues and build systems that don't lose data when things crash.

Real-world use case

E-commerce platforms processing thousands of order confirmation emails, invoice generation, or image resizing tasks in the background without slowing down checkout.

Tools and technologies

• Go
• PostgreSQL
• Docker

Skills you can demonstrate

Distributed Systems, Concurrency, Fault Tolerance

Build roadmap

1. Design the jobs table
   Columns for id, payload (JSONB), priority, status, attempts, max_attempts, scheduled_at, locked_at, and locked_by. Index (status, priority, scheduled_at) — the worker query depends on it.
2. Build the enqueue API
   POST /jobs inserts a pending job and supports delayed execution through scheduled_at. Validate payloads so a bad job fails at the door, not inside a worker.
3. Write the worker loop
   Claim jobs with SELECT ... FOR UPDATE SKIP LOCKED so many workers run in parallel without double-processing. Run three workers and prove no job executes twice.
4. Add retry with exponential backoff
   On failure, increment attempts and reschedule after 2^attempts seconds with jitter, up to max_attempts. Log every retry with the error that caused it.
5. Build the dead-letter queue
   When max_attempts is reached, move the job to dead_jobs with the final error, and add an endpoint to inspect and requeue it. Dead jobs are a feature, not a bug.
6. Test the failure modes
   Kill a worker mid-job, restart Postgres, enqueue a poison payload. Document what happened in each case; that write-up is what proves you understand distributed systems.

Ways to extend the project

Add an AI "Job Priority Predictor" that analyzes job metadata to automatically move urgent tasks (like "Password Reset") to a high-priority queue.

How to showcase your work

• GitHub: Document the "failure modes" you tested (what happens if a worker dies mid-job?).
• LinkedIn: Share a diagram of the job lifecycle from "Pending" → "Completed" or "DLQ".
• Resume: Highlight the use of "FOR UPDATE SKIP LOCKED" for high-concurrency job claiming.