Project Description
Design and build a scalable distributed task execution engine similar to Celery. Support job serialization, task routing, worker process pools, exponential backoff retries, dead-letter queues, and event stream monitoring.
Learning Objectives
- Architect message broker task abstractions for decoupled execution.
- Implement multi-process worker pools processing concurrent job queues.
- Build automatic retry strategies with exponential jitter backoff.
- Route failed jobs to Dead-Letter Queues (DLQ) for inspection.
Technical Requirements & Specifications
### Project Overview
Engineer a fault-tolerant task queue and multi-worker execution engine handling background workloads at scale.
