Data Analyst · advanced

Streaming Lakehouse: Redpanda to Iceberg to DuckDB

A real-time analytics pipeline — events produced into Redpanda, landed continuously into Apache Iceberg tables with schema evolution and time travel, and queried seconds later from DuckDB and a live dashboard. The open lakehouse architecture that data platforms are standardizing on.

Project overview

Choosing an open table format is described as the architecture decision of the decade, and Iceberg has become the default. A working stream-to-Iceberg pipeline with time travel is a senior-level talking point that most analysts never get to build on the job.

Real-world use case

A retailer wants inventory dashboards that update within seconds of a sale, plus the ability to query "what did stock look like at 3 p.m. yesterday" without a separate snapshot process.

Tools and technologies

• Redpanda (Kafka API)
• Apache Iceberg
• DuckDB or Trino
• Python Producer
• MinIO (S3-compatible)
• Grafana

Skills you can demonstrate

Stream Processing, Open Table Formats, Lakehouse Architecture, Schema Evolution, Real-Time Analytics

Build roadmap

1. Stand up the stack
   Docker Compose with Redpanda, MinIO, and a catalog. Produce one test message and read it back before touching Iceberg.
2. Write the producer
   A Python producer emitting order events at a controllable rate, with idempotency keys and a small percentage of late and duplicate events.
3. Land events into Iceberg
   Configure the Iceberg sink to write the topic into a partitioned table in MinIO. Confirm files and metadata appear and that DuckDB can read the table.
4. Query in near real time
   Write the current-inventory query and measure the lag from produce to visible. Tune the commit interval and record the trade-off.
5. Use time travel and evolve the schema
   Query a past snapshot. Add a column to the events, keep producing, and show both old and new snapshots query correctly.
6. Ship the dashboard and write-up
   A Grafana panel on the live query, plus a README explaining the metadata layers, the freshness numbers, and how duplicates are handled.

Ways to extend the project

Add a schema change mid-stream — a new column in the events — and show queries against old and new snapshots both succeeding, then add compaction to keep small files under control.

How to showcase your work

• GitHub: Docker Compose for the whole stack, the producer, table definitions, queries, and a dashboard screenshot with a timestamp proving sub-minute freshness.
• LinkedIn: Record a split screen: producer emitting events on the left, dashboard number ticking up on the right, then a time-travel query showing the past.
• Resume: Use the architecture terms: "Built a streaming lakehouse with Redpanda and Apache Iceberg supporting time travel and schema evolution, queried via DuckDB."