Consensus Gaps - Systems - Jordi Villar
ClickHouse system design · consensus follow-up lab

Consensus / Gaps

The cases interviewers probe after you've shown you know 2PC, 3PC and Raft: who blocks, who may decide, what breaks under partitions, and what ClickHouse actually leans on Keeper for. Every lab opens on a failure worth explaining out loud.

1 · Two-phase commit failure drills

Who blocks, who may decide, and what the log says on restart

One transaction, coordinator C, participants P1–P3. Click any arrow to drop that message. Set crash points per node. The verdict cards assume timeouts eventually fire, a live coordinator answers inquiries, and crashed nodes lose every log record that was not forced.

Drills
Presumption
Votes
Drop messages (or click an arrow)
delivereddropped ✕receiver downnot sent / pending forced write lazy write (lost on crash)
Trace
Presumed abort (PA)Presumed commit (PC)
Coordinator before PREPARENothingForce COLLECTING with participant list
Commit pathC forces COMMIT · P forces COMMIT · P sends ACK · C writes END lazilyC forces COMMIT · P writes COMMIT lazily · no ACKs · C forgets at once
Abort pathNo forced writes, no ACKs; C forgets at onceP forces ABORT · ACKs required · then END
Inquiry with no recordAnswer ABORTAnswer COMMIT (safe only because COLLECTING is forced first)
Wins whenAborts and read-only participants are common. Default in most XA-style systems.Commits dominate and the per-transaction COLLECTING force is cheaper than commit ACKs.
PREPARED is a promiseA participant that voted YES has given away its right to abort. That is the whole source of blocking. A participant that has not voted can always abort on timeout.
Cooperative termination is partialAn in-doubt participant can ask peers. It resolves only if some reachable peer already knows the outcome or never voted. If every reachable peer is also PREPARED, everyone waits for C.
Lost ACK is harmlessC keeps the record and resends the decision; participants must treat a repeated decision as idempotent. A lost DECISION is resolved by inquiry or retransmission. Only a dead coordinator turns a lost message into blocking.
Heuristic decisionsReal XA systems let operators force a "heuristic commit/rollback" on an in-doubt branch to release locks. That trades atomicity for availability and must be reconciled by hand.