Software Engineer · advanced

Event-Driven Microservices with the Outbox Pattern and Distributed Tracing

Three services that communicate through Kafka — orders, inventory, notifications — where every database write and its event are committed atomically through a transactional outbox, and a single trace follows one order across every hop so you can see exactly where time goes.

Project overview

The dual-write problem — a database commit succeeding while the event publish fails — is the backend interview question of the year, and the outbox pattern is the accepted answer. Tracing a single request across services is how senior engineers debug production, and doing both in one project reads as staff-level thinking.

Real-world use case

An order placed at checkout must reserve inventory and send a confirmation, reliably, even if Kafka is briefly down or a service crashes mid-request. Nothing is lost and nothing is duplicated.

Tools and technologies

• Go or Java
• Kafka (Redpanda)
• PostgreSQL Outbox
• Debezium
• OpenTelemetry
• Jaeger or Tempo

Skills you can demonstrate

Event-Driven Architecture, Transactional Consistency, Change Data Capture, Distributed Tracing, Idempotent Consumers

Build roadmap

1. Build the three services
   Orders, inventory, and notifications, each with its own Postgres database, wired to Redpanda in Docker Compose. Publish events naively first — you need to see the dual-write problem before you fix it.
2. Reproduce the dual-write bug
   Kill Kafka between the database commit and the publish. Show the order that exists but was never announced. Screenshot it for the README.
3. Implement the outbox
   Add an outbox table written in the same transaction as the order. Relay it to Kafka with Debezium or a poller. Repeat the kill test and show nothing is lost.
4. Make consumers idempotent
   Store processed event ids per consumer and skip duplicates. Replay a topic from the beginning and confirm inventory counts stay correct.
5. Add distributed tracing
   Instrument all three services with OpenTelemetry, propagate context through Kafka headers, and confirm a single order appears as one trace in Jaeger.
6. Load, break, and document
   Run load, kill services mid-flow, and verify consistency. Write a README with the architecture diagram, the failure tests, and the trace screenshot.

Ways to extend the project

Add a saga for order cancellation with compensating events, and a dead-letter topic with a small admin page for replaying failed messages.

How to showcase your work

• GitHub: Docker Compose for the whole system, the services, and a Jaeger screenshot of one order’s trace spanning three services with the Kafka hops visible.
• LinkedIn: Show the trace waterfall for one order, then kill the inventory service mid-flow, restart it, and show the order completing with no duplicate.
• Resume: Name the patterns: "Designed event-driven services with a transactional outbox, idempotent consumers, and end-to-end OpenTelemetry tracing across Kafka."