Computer Science · advanced
Raft Consensus Key-Value Cluster with Chaos Tests
A replicated key-value store where three to five nodes agree on every write using the Raft consensus algorithm — leader election, log replication, and safety under partitions — with a chaos suite that kills nodes and splits the network to prove it never loses a committed write.
Project overview
Distributed consensus is the gold-standard advanced systems project — the same algorithm runs inside etcd, Consul, and CockroachDB. Adding chaos tests turns it from a course exercise into an SRE story about proving correctness under failure.
Real-world use case
A configuration service must stay available and consistent when one datacenter drops. Clients keep writing, a new leader is elected in under a second, and nothing that was acknowledged is ever lost.
Tools and technologies
- Go
- gRPC
- Raft (leader election + log replication)
- Docker Compose
- Fault-injection Test Harness
Skills you can demonstrate
Distributed Consensus, Fault Tolerance, Concurrency, Network Programming, Systems Testing
Build roadmap
- Read the paper and build the skeleton
Read the Raft paper, then set up nodes that talk over gRPC with persistent state on disk. No consensus yet — just RPC plumbing and a node lifecycle. - Implement leader election
Randomized timeouts, RequestVote, term handling. Start three nodes and confirm exactly one leader emerges and re-elects when killed. - Implement log replication
AppendEntries with consistency checks, commit index advancement, and applying entries to the key-value state machine. Verify all nodes converge. - Build the chaos harness
A test controller that can partition, delay, and crash nodes on demand. Wire it so the whole cluster runs in one Docker Compose file. - Run the safety scenarios
Partition the leader, write to both sides, heal, and verify only one history survives. Repeat with crash-restarts mid-replication. Fix what breaks. - Measure and document
Record election time and write throughput under load, then write a README explaining the guarantees, the scenarios, and one bug the harness caught.
Ways to extend the project
Add log compaction with snapshots and dynamic membership changes, the two features that separate a textbook Raft from a production one.
How to showcase your work
- GitHub: The implementation, the chaos harness, and a results table showing every scenario passing, plus a diagram of a leader election under partition.
- LinkedIn: Record a terminal split: kill the leader on the left, watch the election and continued writes on the right.
- Resume: Name the properties: "Implemented Raft consensus with leader election and log replication, verified against partitions and crashes with a fault-injection suite."