M
You don't implement consensus, but you operate it every day — etcd, Consul, CockroachDB, Kafka's KRaft. This is the mechanism behind the elections, the read-only clusters, and the fsync obsession: enough Raft to turn its surprises into diagnoses.
Both read like bugs. Neither is. In both cases the system is doing exactly what its algorithm promises — and the only reason they feel like mysteries is that the algorithm is the one piece most of us never got around to learning.
That's a fair thing to have skipped. Almost nobody implements consensus from scratch; the papers are dense, and the systems built on it are reliable enough that you can run them for years without opening the hood. But implement and operate are different verbs.
You almost certainly operate consensus right now: etcd under Kubernetes, Consul under service discovery, the replication cores of CockroachDB and TiKV, Kafka's newer KRaft mode. A couple of close relatives sit next door — ZooKeeper runs Zab, and MongoDB's replica sets use a Raft-like protocol of their own.
When one of them does something surprising, the incident channel fills with people reverse-engineering a mechanism they'd filed under "someday."
This is the someday.
Not a proof walkthrough — a working model of Raft at the depth an operator actually needs: what it's for, how it moves, and mostly what it predicts about how your systems behave when things go wrong.
State the problem precisely, because a fuzzy statement produces fuzzy intuition:
Several machines have to agree on a single, ordered sequence of decisions — and keep agreeing through crashes, restarts, and lost or delayed messages — so that no two of them ever believe different things were decided.
The word doing the work is sequence.
It's tempting to picture consensus as machines voting on one value, but the useful picture is a replicated log: an append-only list of entries that every replica eventually holds in the same order.
Get the log right and the rest is free — feed the entries in order to any deterministic state machine and every replica computes identical state.
A key-value store, a lock service, a database's replication stream: same trick underneath.
Agree on the log, replay it, derive the state.
What stands in the way isn't failure, exactly. It's ambiguity.
When node A can't reach node B, A cannot tell whether B has crashed, is merely slow, or is alive and well behind a broken switch. From the outside those cases are identical, and they call for opposite responses.
Guess wrong in the optimistic direction and you get the worst outcome in the discipline: two halves of a cluster each conclude the other is gone, each start taking writes, and later somebody has to reconcile two histories that both claim to be the truth.
Split-brain.
Everything intricate in Raft exists to make that one outcome impossible — and it buys the guarantee with availability.
A system that must never fork its history has to be willing to stop serving someone during a partition.
Raft is three commitments. It's worth holding them together, because every operational behavior later is a consequence of one of them.
One leader at a time; all writes flow through it.
Rather than symmetric peers negotiating each entry — the Paxos style that famously resists intuition — Raft elects a single leader that orders every write into the log.
The hard problem of agreeing on many things collapses into the smaller one of agreeing, now and then, on a single thing: who is in charge.
That's the real difference from Paxos. They target the same guarantees; Raft's contribution was decomposing the problem into leader election, log replication, and a safety rule you can actually hold in your head.
For operating a system — and for most interviews — deep Raft plus "Paxos is the older, harder-to-explain relative" is the right allocation of attention.
A majority decides everything.
You become leader with votes from a majority; an entry commits once a majority has stored it.
And the single fact that makes all of it safe is almost embarrassingly small:
Three of five over here, three of five over there — they cannot avoid meeting in at least one node.
That overlap is the whole safety story, and it earns its own section below, because it's the part most explanations wave past.
Time is cut into numbered terms.
Each election bumps a term counter; every message carries its sender's term; any node that hears a term higher than its own steps down to follower immediately.
Terms are Raft's logical clock — a monotonic epoch that lets nodes discard stale leaders and stale messages without trusting a wall clock.
When the leader-change counter on an etcd dashboard ticks up, you're watching this: each increment is a new term.
A follower's life is simple.
It expects to hear from the leader on a steady heartbeat and keeps a countdown timer that it resets on each one.
If that timer ever expires — no heartbeat for a full election timeout — the follower assumes the leader is gone, increments the term, votes for itself, and asks everyone else for their vote.
The voting rules are short, and one of them carries the whole safety argument:
Two lines of policy — one vote per term, and never vote for a candidate whose log is behind yours — plus a timer.
Hold onto that second rule.
The genuinely clever part is that the timeout is randomized per node.
If every follower used the same value, a dead leader would make them all time out together, all become candidates together, split the vote, and — nobody holding a majority — try again, and again.
A livelock wearing an election costume.
Randomizing each node's timeout means one usually fires first, gathers votes before its peers wake, and wins in a single round.
A coordination problem solved with a dice roll.
Now the operational reflex worth building, because it's the highest-leverage idea here:
Elections are triggered by silence, and silence has many causes besides death.
A crashed leader is one.
So is a stop-the-world GC pause on the leader, a saturated NIC, a disk that has gone slow to persist the log, or a timeout tuned so tight that ordinary jitter trips it.
A cluster that "elects constantly" almost never has a bug in its election code.
It's slow somewhere, and the election storm is the smoke, not the fire.
Which is why the instinct to shorten the timeout to "recover faster" usually backfires — it makes the cluster quicker to abandon a leader that was only briefly late.
One note on those numbers, since it's a common confusion: the original Raft paper uses 150–300 ms as an illustrative range, but production systems pick their own.
etcd, for example, defaults to a 1000 ms election timeout with a 100 ms heartbeat interval, and its guidance is to keep the election timeout comfortably above the round-trip time between members.
All of it is configurable, and the right values depend on your network and disks — which is exactly why copying someone else's numbers is a mistake.
With a leader in place, the steady state is a pipeline.
A client's write lands at the leader, which appends it to its own log as uncommitted and ships it to the followers.
Each replication message carries a small consistency check — the index and term of the entry immediately before it — so a follower whose log has diverged rejects the message, and the leader walks backward until the two logs line up, overwriting the follower's stray tail.
Once a majority hold the entry, the leader marks it committed, applies it, answers the client, and advertises the new commit point on later heartbeats so the followers apply it too.
Three consequences fall out of that pipeline, and all three show up in production.
Write latency is a majority round-trip — not an all-nodes round-trip.
The leader waits only for enough followers to form a majority, so a five-node cluster shrugs off one slow member: the majority forms without it.
The flip side is unforgiving — stretch that cluster across regions and you've put the speed of light under every write.
Your slowest necessary member sets the pace, and it's usually a disk.
Appending to the log means an
This is why etcd's documentation is preoccupied with disk latency, and why putting its log on network-attached storage — where an
Followers are allowed to be behind.
A follower that hasn't yet received or applied the latest entries is stale by design, not by malfunction.
So a read from a follower can return old data — and a read from even the leader needs care, because a leader that was just deposed may not know it yet.
That's why "linearizable" reads in Raft systems either pay for a round-trip to confirm the node still leads a majority (the ReadIndex approach) or lean on a time-based lease.
It's also why the read-consistency setting in your client library has real latency on the other side of it, and why its default is often the cheaper, possibly-stale option.
Worth knowing which one you're running.
Here's the scenario that should worry you, stated plainly:
The leader commits some entries, then dies.
A follower that never received those entries wins the next election.
Does it now overwrite committed history with its own shorter log?
It can't — and the reason is the overlap fact from earlier, finally doing its work.
Walk it through.
"Committed" means a majority stored the entry.
Winning an election means a majority voted for you.
Those two majorities have to share at least one node — pigeonhole, no way around it.
That shared node holds the committed entry, and by the voting rule from a moment ago it will refuse any candidate whose log is less complete than its own.
So a candidate missing the committed entry cannot assemble a majority; the overlap node stands in the way.
Every leader that can win already carries the full committed history.
The same overlap kills split-brain from the other side.
Two simultaneous leaders would each need their own majority, and two majorities of one cluster can't be disjoint — they'd have to share a node, and a node backs only one leader per term.
So two leaders can't coexist.
Notice what this does and doesn't promise.
Committed entries survive any leader change — that part is ironclad.
Uncommitted entries on a dead leader may simply vanish, and that's acceptable, because the client was never told they succeeded.
Which gives you the rule every client of a consensus store has to internalize:
A write is durable when it's acknowledged, and not a moment sooner.
A timeout on a write drops you into the same ambiguous place as any distributed call — maybe it committed, maybe it didn't — and the only correct response is an idempotent retry.
Once the mechanism is in your head, a lot of on-call mysteries decode themselves.
A few shapes you'll start to recognize:
The pattern under all five is identical:
The system did exactly what the algorithm guarantees, and the surprise lived entirely in the operator's mental model.
Closing that gap is the whole return on learning this.
A handful of operational rules that look like trivia until you notice they're just the algorithm seen from the side.
Clusters come in odd numbers because tolerance is
To survive
An even count buys nothing — four nodes still tolerate only one, since a majority of four is three — and it makes partitions behave worse.
That's why 3 and 5 are everywhere and 7 is a practical ceiling.
Every node you add is another replication stream and a larger majority to wait on: fault tolerance you may not need, at a latency cost you certainly pay.
Adding nodes does not scale writes.
This one catches people who reach for the usual horizontal-scaling reflex.
Every write still funnels through one leader and still has to reach a majority; more nodes mean more work per write, not less.
Consensus scales fault tolerance, not throughput.
You scale writes by sharding — running many independent Raft groups, each owning a slice of the keyspace, which is exactly the "multi-Raft" design inside systems like CockroachDB and TiKV.
Losing quorum is unavailability on purpose.
Lose two of three nodes and the survivor — holding all your data, perfectly healthy — refuses writes.
With no majority it cannot rule out that it's the partitioned minority while the others carry on without it.
The runbook is restore a peer or carefully re-bootstrap, never force the survivor writable.
The "force" flags various tools expose are split-brain generators with a help page; reach for one only when you have genuinely accepted that data will diverge.
Deployments cause elections.
Restarting the leader triggers an election and a few seconds of write unavailability; restarting members too quickly can briefly cost you quorum.
Mature deploy tooling restarts followers first, transfers leadership deliberately before touching the leader (most implementations support a leadership transfer), and spaces restarts beyond the election-settle time.
If your Kubernetes control plane hiccups during an etcd upgrade, this is usually why.
Geography is a quorum decision made in advance.
Two datacenters can't split a majority safely — whichever site holds the minority goes dark during a partition — so a two-site "HA" consensus setup is really an outage waiting for a cut fiber.
Three sites let you place members so no single site holds a majority alone (or deliberately so that one does), and that placement is a conscious choice about which failures leave you writable.
Every multi-region consensus deployment decides, ahead of time, who goes dark in a partition.
Better to decide it on a whiteboard than to discover it at 3 a.m.
Strip away the operational detail and Raft is small enough to hold in one hand.
One leader sequences a log.
Majorities commit entries and elect leaders.
Any two majorities overlap, so committed history can neither fork nor disappear.
Numbered terms let every node reject the past.
That's the entire machine — and everything that felt like lore (the odd-sized clusters, the obsession with disk latency, the deliberate darkness of a partitioned minority) is just those four facts showing through.
Consensus has a reputation for being inscrutable, and it's half-earned: the proofs are genuinely hard.
But operating one of these systems doesn't require the proofs.
It requires the model — enough of it that when the cluster goes read-only, or the writes freeze for nine seconds, you aren't reverse-engineering the algorithm under incident pressure.
You already know what it's telling you.
Both read like bugs. Neither is. In both cases the system is doing exactly what its algorithm promises — and the only reason they feel like mysteries is that the algorithm is the one piece most of us never got around to learning.
That's a fair thing to have skipped. Almost nobody implements consensus from scratch; the papers are dense, and the systems built on it are reliable enough that you can run them for years without opening the hood. But implement and operate are different verbs.
You almost certainly operate consensus right now: etcd under Kubernetes, Consul under service discovery, the replication cores of CockroachDB and TiKV, Kafka's newer KRaft mode. A couple of close relatives sit next door — ZooKeeper runs Zab, and MongoDB's replica sets use a Raft-like protocol of their own.
When one of them does something surprising, the incident channel fills with people reverse-engineering a mechanism they'd filed under "someday."
This is the someday.
Not a proof walkthrough — a working model of Raft at the depth an operator actually needs: what it's for, how it moves, and mostly what it predicts about how your systems behave when things go wrong.
The Only Job Consensus Has
State the problem precisely, because a fuzzy statement produces fuzzy intuition:
Several machines have to agree on a single, ordered sequence of decisions — and keep agreeing through crashes, restarts, and lost or delayed messages — so that no two of them ever believe different things were decided.
The word doing the work is sequence.
It's tempting to picture consensus as machines voting on one value, but the useful picture is a replicated log: an append-only list of entries that every replica eventually holds in the same order.
Get the log right and the rest is free — feed the entries in order to any deterministic state machine and every replica computes identical state.
A key-value store, a lock service, a database's replication stream: same trick underneath.
Agree on the log, replay it, derive the state.
What stands in the way isn't failure, exactly. It's ambiguity.
When node A can't reach node B, A cannot tell whether B has crashed, is merely slow, or is alive and well behind a broken switch. From the outside those cases are identical, and they call for opposite responses.
Guess wrong in the optimistic direction and you get the worst outcome in the discipline: two halves of a cluster each conclude the other is gone, each start taking writes, and later somebody has to reconcile two histories that both claim to be the truth.
Split-brain.
Everything intricate in Raft exists to make that one outcome impossible — and it buys the guarantee with availability.
A system that must never fork its history has to be willing to stop serving someone during a partition.
One Leader, a Majority, and a Number
Raft is three commitments. It's worth holding them together, because every operational behavior later is a consequence of one of them.
One leader at a time; all writes flow through it.
Rather than symmetric peers negotiating each entry — the Paxos style that famously resists intuition — Raft elects a single leader that orders every write into the log.
The hard problem of agreeing on many things collapses into the smaller one of agreeing, now and then, on a single thing: who is in charge.
That's the real difference from Paxos. They target the same guarantees; Raft's contribution was decomposing the problem into leader election, log replication, and a safety rule you can actually hold in your head.
For operating a system — and for most interviews — deep Raft plus "Paxos is the older, harder-to-explain relative" is the right allocation of attention.
A majority decides everything.
You become leader with votes from a majority; an entry commits once a majority has stored it.
And the single fact that makes all of it safe is almost embarrassingly small:
Three of five over here, three of five over there — they cannot avoid meeting in at least one node.
That overlap is the whole safety story, and it earns its own section below, because it's the part most explanations wave past.
Time is cut into numbered terms.
Each election bumps a term counter; every message carries its sender's term; any node that hears a term higher than its own steps down to follower immediately.
Terms are Raft's logical clock — a monotonic epoch that lets nodes discard stale leaders and stale messages without trusting a wall clock.
When the leader-change counter on an etcd dashboard ticks up, you're watching this: each increment is a new term.
Leader Election Is Triggered by Silence
A follower's life is simple.
It expects to hear from the leader on a steady heartbeat and keeps a countdown timer that it resets on each one.
If that timer ever expires — no heartbeat for a full election timeout — the follower assumes the leader is gone, increments the term, votes for itself, and asks everyone else for their vote.
The voting rules are short, and one of them carries the whole safety argument:
Code:
// A node deciding whether to grant its vote (simplified)
function handleRequestVote(req: RequestVote): VoteResult {
if (req.term < currentTerm) {
return "REJECT"; // stale candidate
}
if (alreadyVotedThisTerm(req.term)) {
return "REJECT"; // one vote per term
}
// The safety rule: never vote for a log less complete than your own.
const mine = lastLogEntry();
const candidateUpToDate =
req.lastLogTerm > mine.term ||
(req.lastLogTerm === mine.term &&
req.lastLogIndex >= mine.index);
if (!candidateUpToDate) {
return "REJECT";
}
return "GRANT";
}Two lines of policy — one vote per term, and never vote for a candidate whose log is behind yours — plus a timer.
Hold onto that second rule.
The genuinely clever part is that the timeout is randomized per node.
If every follower used the same value, a dead leader would make them all time out together, all become candidates together, split the vote, and — nobody holding a majority — try again, and again.
A livelock wearing an election costume.
Randomizing each node's timeout means one usually fires first, gathers votes before its peers wake, and wins in a single round.
A coordination problem solved with a dice roll.
Now the operational reflex worth building, because it's the highest-leverage idea here:
Elections are triggered by silence, and silence has many causes besides death.
A crashed leader is one.
So is a stop-the-world GC pause on the leader, a saturated NIC, a disk that has gone slow to persist the log, or a timeout tuned so tight that ordinary jitter trips it.
A cluster that "elects constantly" almost never has a bug in its election code.
It's slow somewhere, and the election storm is the smoke, not the fire.
Which is why the instinct to shorten the timeout to "recover faster" usually backfires — it makes the cluster quicker to abandon a leader that was only briefly late.
One note on those numbers, since it's a common confusion: the original Raft paper uses 150–300 ms as an illustrative range, but production systems pick their own.
etcd, for example, defaults to a 1000 ms election timeout with a 100 ms heartbeat interval, and its guidance is to keep the election timeout comfortably above the round-trip time between members.
All of it is configurable, and the right values depend on your network and disks — which is exactly why copying someone else's numbers is a mistake.
Replication: A Write Is a Majority Round-Trip
With a leader in place, the steady state is a pipeline.
A client's write lands at the leader, which appends it to its own log as uncommitted and ships it to the followers.
Each replication message carries a small consistency check — the index and term of the entry immediately before it — so a follower whose log has diverged rejects the message, and the leader walks backward until the two logs line up, overwriting the follower's stray tail.
Once a majority hold the entry, the leader marks it committed, applies it, answers the client, and advertises the new commit point on later heartbeats so the followers apply it too.
Three consequences fall out of that pipeline, and all three show up in production.
Write latency is a majority round-trip — not an all-nodes round-trip.
The leader waits only for enough followers to form a majority, so a five-node cluster shrugs off one slow member: the majority forms without it.
The flip side is unforgiving — stretch that cluster across regions and you've put the speed of light under every write.
Your slowest necessary member sets the pace, and it's usually a disk.
Appending to the log means an
fsync, and consensus systems are bound by fsync latency long before they are bound by CPU.This is why etcd's documentation is preoccupied with disk latency, and why putting its log on network-attached storage — where an
fsync now includes a network hop — is a classic self-inflicted wound.Followers are allowed to be behind.
A follower that hasn't yet received or applied the latest entries is stale by design, not by malfunction.
So a read from a follower can return old data — and a read from even the leader needs care, because a leader that was just deposed may not know it yet.
That's why "linearizable" reads in Raft systems either pay for a round-trip to confirm the node still leads a majority (the ReadIndex approach) or lean on a time-based lease.
It's also why the read-consistency setting in your client library has real latency on the other side of it, and why its default is often the cheaper, possibly-stale option.
Worth knowing which one you're running.
Why a New Leader Can't Lose Your Data
Here's the scenario that should worry you, stated plainly:
The leader commits some entries, then dies.
A follower that never received those entries wins the next election.
Does it now overwrite committed history with its own shorter log?
It can't — and the reason is the overlap fact from earlier, finally doing its work.
Walk it through.
"Committed" means a majority stored the entry.
Winning an election means a majority voted for you.
Those two majorities have to share at least one node — pigeonhole, no way around it.
That shared node holds the committed entry, and by the voting rule from a moment ago it will refuse any candidate whose log is less complete than its own.
So a candidate missing the committed entry cannot assemble a majority; the overlap node stands in the way.
Every leader that can win already carries the full committed history.
The same overlap kills split-brain from the other side.
Two simultaneous leaders would each need their own majority, and two majorities of one cluster can't be disjoint — they'd have to share a node, and a node backs only one leader per term.
So two leaders can't coexist.
Notice what this does and doesn't promise.
Committed entries survive any leader change — that part is ironclad.
Uncommitted entries on a dead leader may simply vanish, and that's acceptable, because the client was never told they succeeded.
Which gives you the rule every client of a consensus store has to internalize:
A write is durable when it's acknowledged, and not a moment sooner.
A timeout on a write drops you into the same ambiguous place as any distributed call — maybe it committed, maybe it didn't — and the only correct response is an idempotent retry.
Reading Your Own Outages
Once the mechanism is in your head, a lot of on-call mysteries decode themselves.
A few shapes you'll start to recognize:
- Writes stalled for several seconds, then recovered on their own. An election happened. Confirm it with the term counter, then go find the silence that caused it — a GC pause, a disk stall, a network blip. The election worked; the thing that made the old leader go quiet is what you're chasing.
- The cluster is up but read-only. Quorum loss. Count the healthy members before touching anything, and understand that the lone survivor refusing writes is behaving correctly — it can't safely tell "my peers are dead" from "I've been partitioned away from them."
- Everything got slower after we moved etcd onto bigger network volumes. You moved
fsynconto the network. Consensus write latency is majority-fsynclatency, and you just added a hop to everyfsync. - We added two nodes for reliability and p99 writes got worse. Bigger majority, more replication fan-out per write. Which failure were the extra nodes protecting against, exactly?
- After a network flap, some clients read stale config for a minute. Follower reads, or a lease that hadn't expired yet. Linearizable reads cost a round-trip, and something in the path was configured for the cheap one.
The pattern under all five is identical:
The system did exactly what the algorithm guarantees, and the surprise lived entirely in the operator's mental model.
Closing that gap is the whole return on learning this.
The Shape Shows Through in Operations
A handful of operational rules that look like trivia until you notice they're just the algorithm seen from the side.
Clusters come in odd numbers because tolerance is
2f + 1.To survive
f failures you need 2f + 1 nodes: three tolerate one loss, five tolerate two.An even count buys nothing — four nodes still tolerate only one, since a majority of four is three — and it makes partitions behave worse.
That's why 3 and 5 are everywhere and 7 is a practical ceiling.
Every node you add is another replication stream and a larger majority to wait on: fault tolerance you may not need, at a latency cost you certainly pay.
Adding nodes does not scale writes.
This one catches people who reach for the usual horizontal-scaling reflex.
Every write still funnels through one leader and still has to reach a majority; more nodes mean more work per write, not less.
Consensus scales fault tolerance, not throughput.
You scale writes by sharding — running many independent Raft groups, each owning a slice of the keyspace, which is exactly the "multi-Raft" design inside systems like CockroachDB and TiKV.
Losing quorum is unavailability on purpose.
Lose two of three nodes and the survivor — holding all your data, perfectly healthy — refuses writes.
With no majority it cannot rule out that it's the partitioned minority while the others carry on without it.
The runbook is restore a peer or carefully re-bootstrap, never force the survivor writable.
The "force" flags various tools expose are split-brain generators with a help page; reach for one only when you have genuinely accepted that data will diverge.
Deployments cause elections.
Restarting the leader triggers an election and a few seconds of write unavailability; restarting members too quickly can briefly cost you quorum.
Mature deploy tooling restarts followers first, transfers leadership deliberately before touching the leader (most implementations support a leadership transfer), and spaces restarts beyond the election-settle time.
If your Kubernetes control plane hiccups during an etcd upgrade, this is usually why.
Geography is a quorum decision made in advance.
Two datacenters can't split a majority safely — whichever site holds the minority goes dark during a partition — so a two-site "HA" consensus setup is really an outage waiting for a cut fiber.
Three sites let you place members so no single site holds a majority alone (or deliberately so that one does), and that placement is a conscious choice about which failures leave you writable.
Every multi-region consensus deployment decides, ahead of time, who goes dark in a partition.
Better to decide it on a whiteboard than to discover it at 3 a.m.
The Core Fits in a Paragraph
Strip away the operational detail and Raft is small enough to hold in one hand.
One leader sequences a log.
Majorities commit entries and elect leaders.
Any two majorities overlap, so committed history can neither fork nor disappear.
Numbered terms let every node reject the past.
That's the entire machine — and everything that felt like lore (the odd-sized clusters, the obsession with disk latency, the deliberate darkness of a partitioned minority) is just those four facts showing through.
Consensus has a reputation for being inscrutable, and it's half-earned: the proofs are genuinely hard.
But operating one of these systems doesn't require the proofs.
It requires the model — enough of it that when the cluster goes read-only, or the writes freeze for nine seconds, you aren't reverse-engineering the algorithm under incident pressure.
You already know what it's telling you.