Leader and Follower

📘 Learn Leader and Follower from zero

Start from the problem. You have one piece of data that many clients want to read and write at once. If every node accepts writes independently, two clients can change the same record concurrently and the copies disagree — there is no single source of truth. You also want the data to survive a machine dying and to serve fast reads under heavy traffic.

The analogy. Picture a classroom with one teacher (the leader) and several teaching assistants (the followers). Only the teacher writes in the master notebook — every change flows through the teacher, so there is never a conflict over the "official" value, and the order changes happen in is fixed. The teacher then dictates each change to the TAs, who copy it into their notebooks. Students may ask any TA a question (a read), spreading the load. If the teacher disappears, the TAs hold a quick vote and promote one of themselves (leader election / failover).

Worked example. A photo app uses one leader DB and two followers. A user posts a caption — that write goes only to the leader, which appends it to its replication log at a definite position and applies it. The leader streams that log to both followers (replication). Millions of viewers then read captions from the followers, so reads scale out. If the leader's disk fails, the system notices the missing heartbeat, the followers elect the one with the most up-to-date log, and writes resume on it. The catch: a follower may lag a few hundred milliseconds, so a viewer can briefly see the old caption — that is replication lag.

Key insight: funnel all writes through ONE leader to get a single, conflict-free order, then replicate that ordered log to followers to scale reads and tolerate failure — trading away write scalability and perfect read freshness in exchange for a simple consistency model and high read availability.

✨ Added by the guide to build intuition — not from the source course.

🎯 Guided practice

1. Easy — "Where does each operation go?"

   You run one leader and three followers for a user-profile service. A client (a) updates their display name, then (b) loads their profile page. Route each operation.

   Reasoning: Step 1 — classify each op. (a) is a write; writes must go through the single leader to preserve the total order, so it goes to the leader. (b) is a read; reads can be served by any follower to spread load. Step 2 — spot the trap: if you route (b) to a follower immediately after (a), replication may not have caught up, so the user sees their OLD name — a read-your-writes (read-after-write) consistency violation. Step 3 — fix it: for a short window after a user's own write, serve that user's reads from the leader (or from a follower you know has applied that write, e.g. by tracking the log position the write reached). Core takeaway: writes → leader (ordering), reads → followers (scale), but a user's reads of their own recent writes need special routing to beat replication lag.

2. Medium — "The leader just died. What happens?"

   Leader L and followers F1, F2 use asynchronous replication. L crashes after acknowledging write W to the client but before W reached any follower. The system must stay available. Walk through the recovery and the risk.

   Reasoning: Step 1 — detection: followers miss L's heartbeat past a timeout and declare L dead. Step 2 — election: the followers run a leader-election protocol (e.g. Raft) requiring a majority quorum to agree, which prevents two leaders from being elected at once. Say F1 has the highest replication position (Raft only elects a candidate whose log is at least as up-to-date as the voter's) and wins. Step 3 — the loss: because W was async and never replicated, F1's log lacks W; when F1 becomes leader, W is silently lost even though the client was told "success." Step 4 — the second danger: if the old L revives and still believes it is leader, you get split-brain; the quorum rule stops a second leader from being elected, and fencing tokens stop the stale L's late writes from being accepted by the storage layer. Step 5 — the design fix: switch to semi-synchronous replication so the leader acks W only after at least one follower has stored it, bounding worst-case loss to in-flight writes (assuming the leader and that synchronous follower don't fail together). Core takeaway: single-leader gives clean ordering, but failover forces a choice on the durability-vs-availability spectrum — quorum elections prevent two leaders, fencing tokens neutralize a stale one, and the sync level sets how much acknowledged data a failover can lose.

✨ Added by the guide — work these before the full problem set.