Computer Science · beginner
Key-Value Store from Scratch with a Write-Ahead Log
A persistent key-value database written from nothing — an in-memory index, a write-ahead log so data survives a crash, and a compaction step that keeps the file from growing forever. The foundations every real database is built on.
Project overview
Storage-engine fundamentals are the most common systems interview topic, and building one makes you the person who can explain why a database is slow instead of guessing. It is a weekend project that reads as far deeper than a CRUD app.
Real-world use case
An embedded device needs to store settings and counters reliably across power loss without the weight of SQLite. Your engine is the answer, and you can explain every byte on disk.
Tools and technologies
- Python or Go
- Write-Ahead Log
- Binary File Formats
- pytest
- Crash Tests
Skills you can demonstrate
Storage Engine Internals, Durability and Crash Safety, File I/O, Data Structures, Test Design
Build roadmap
- Start in memory
Implement get, set, and delete on a dictionary with a small CLI. Write the tests first — they will run unchanged against every later version. - Add the write-ahead log
Append every write to a file with a length prefix and checksum before updating memory. Flush and sync so the data is really on disk. - Implement recovery
On startup, replay the log to rebuild the index. Handle a truncated last record without crashing — that is the torn-write case. - Write crash tests
Spawn the store in a subprocess, kill it during a batch of writes, restart, and verify. Run it a hundred times. This step is what makes the project credible. - Add compaction
Rewrite the log with only live keys into a new file and atomically swap it in. Make sure a crash during compaction leaves the old log intact. - Benchmark and write it up
Measure writes per second and recovery time at 10k, 100k, and 1M keys. Document the format, the guarantees, and the trade-offs you chose.
Ways to extend the project
Replace the hash index with a sorted structure and add range queries, which puts you on the road to an LSM tree.
How to showcase your work
- GitHub: The engine, a small CLI, the crash-test suite, and a README with a diagram of the file format and the recovery sequence.
- LinkedIn: Show the crash test: write a thousand keys, kill the process mid-batch, restart, and every acknowledged key comes back.
- Resume: Describe the internals: "Implemented a persistent key-value store with write-ahead logging, checksummed records, crash recovery, and log compaction."