Distributed Fundamentals - Systems - Jordi Villar
System design prep · ClickHouse · Distributed systems fundamentals

Acknowledged ≠ durable.

An ack is a promise about which failures a write will survive. Most questions in this interview reduce to three words: ack after what, held where, against which failure?

00

Summary in ten lines

Ack
A promise scoped to a failure model. Say which failures it covers, never just "it's saved".
fsync vs page cache
write() survives a process crash. Only fsync (data + parent dir) survives power loss.
WAL + group commit
Append intent sequentially, fsync once per batch of commits, apply later. Recovery replays the log.
Replication vs disk
Replication covers node and disk loss; fsync covers power loss. Correlated failures beat replication alone.
RPO
The window of acknowledged data you can lose: replication lag, writeback interval, or backup interval.
Idempotency
Timeouts leave outcomes unknown, so retries are mandatory. Dedup tokens make them safe.
Exactly-once
Not a transport property. It is at-least-once delivery plus idempotent or transactional apply.
Failure models
Crash-stop ⊂ crash-recovery, omission, Byzantine. Gray failures are the ones your health check cannot see.
CAP / PACELC
Partitions force A or C. The rest of the time you still pay Latency for Consistency.
Backpressure
Bounded queues plus a signal upstream. Unbounded queues only move the failure to latency and OOM.
01

Durability

What "acknowledged" really means

When a server returns success it has placed the data somewhere: in its own heap, in the kernel page cache, on a device, or on other machines. Each location dies with a different failure. A precise answer names the location and the failures it covers.

fsync vs the page cache

  • write() returns once bytes are in the kernel page cache. The pages are dirty and get written back later (Linux expires dirty pages after about 30 s by default). A process crash keeps them; a kernel panic or power loss does not.
  • fsync(fd) pushes the file's data and metadata to the device and asks it to flush its volatile write cache. A new file also needs an fsync on its parent directory, or the directory entry itself can vanish.
  • fsync can lie: consumer SSDs without power-loss protection, or RAID controllers with volatile caches, may acknowledge flushes they have not made durable.
  • fsync errors are not retryable. After an EIO, Linux may mark the dirty pages clean, so a second fsync "succeeds" with data gone (the 2018 PostgreSQL "fsyncgate"). The safe response is to crash and recover from the log or a replica.

Write-ahead log and group commit

A WAL turns random in-place updates into one sequential append. The commit protocol is: append the record, fsync the log, ack, apply to the main structures later. After a crash, replay the log from the last checkpoint. Group commit makes this affordable: an fsync costs from tens of microseconds (NVMe with PLP) to milliseconds, so the log writer waits briefly, gathers every commit that arrived, and pays one fsync for all of them. Per-commit latency rises slightly; throughput rises a lot.

time → txn 1append txn 2append txn 3append one fsync(log) covers all three records ack 1ack 2ack 3 Without grouping: 3 fsyncs, one per commit. With grouping: 1 fsync, and txn 1 waits a little longer. ClickHouse analogue: async inserts gather many small INSERTs into one part, one write, one (optional) fsync.
Group commit amortizes the expensive durable step across concurrent writers.

Durability by replication vs durability by disk

They protect against different things. A local fsync survives power loss but not a dead disk or a lost VM. Copies on several machines survive losing one machine, but if every copy sits in volatile memory, a correlated event (a rack power cut, a kernel bug that panics every node, a bad deploy) loses all of them at once. Kafka with default settings and ClickHouse with default settings both lean on replication and the page cache rather than per-write fsync. The strongest guarantee combines both: quorum of replicas, each with an fsync, ideally in different failure domains.

RPO and RTO

RPO (recovery point objective) is how much acknowledged data you can lose, measured in time. RTO is how long recovery takes. With async replication, RPO equals replication lag at the moment of failure. With page-cache-only writes and a correlated power loss, it equals the dirty-page writeback window. With backups alone, it equals the backup interval. RPO = 0 requires synchronous replication (or synchronous fsync for single-node faults).

Write path: pick an ack point and a crash

Simulator 1

Two replicas, A (receives the insert) and B. Pick when the server acknowledges, what fails, and when. Background work is modeled: async buffers flush within 200 ms, replicas fetch new parts within about 1 s, dirty pages reach disk within about 30 s.

Ack after
Failure
Failure happens
holds the data in flight wiped by the failure

02

The ClickHouse insert path

A synchronous INSERT into a MergeTree table sorts the block, writes it as a new immutable part in a temporary directory, renames the directory into place, and then acknowledges. On ReplicatedMergeTree the part is also registered in ClickHouse Keeper (with the block's hash, for dedup) before the ack, and the other replicas fetch the part asynchronously. So the default ack means: one replica's page cache, plus metadata in Keeper. No fsync, no second copy yet.

Write part tmp_insert_…/ columns, marks, idx fsync files fsync_after_insert=1 rename(2) → 202610_7_7_0 atomic: all or nothing fsync dir fsync_part_directory=1 Keeper commit log entry + block hash (dedup) ACK or wait for quorum replicas fetch Dashed boxes are off by default. With insert_quorum=N the ACK waits until N replicas have the part; otherwise fetching happens after the ACK.
One INSERT block becomes one immutable part. Visibility flips at the rename; durability depends on the dashed steps and on replication.
SettingDefaultWhat it changes about the ack
async_insert0Server buffers small inserts in memory and flushes them as one part. Flush on whichever comes first: async_insert_max_data_size (100 MiB), async_insert_busy_timeout_ms (200 ms, 1 s on Cloud, adaptive since 24.2), or async_insert_max_query_number (450).
wait_for_async_insert11: ack after the buffer is flushed to a part; flush errors reach the client. 0: ack once buffered. Fire-and-forget: lost on crash, errors only in logs and system.asynchronous_inserts, and no backpressure reaches the client.
fsync_after_insert0MergeTree setting. fsync every file of each inserted part. Large insert-throughput cost.
fsync_part_directory0MergeTree setting. fsync the part directory after writes and renames, so the rename itself is durable.
insert_quorum0N or 'auto' (majority). Ack only after N replicas have the part. Fails if the quorum isn't reached within insert_quorum_timeout (600 s).
insert_quorum_parallel1Allows concurrent quorum inserts. Set to 0 if you need sequential consistency with select_sequential_consistency=1.
select_sequential_consistency0With quorum inserts, a SELECT on a replica that lacks quorum-acknowledged data fails instead of returning stale results.
insert_deduplication_tokenemptyExplicit idempotency key that replaces the content hash. Retrying with the same token is deduplicated.
replicated_deduplication_windowboundedHow many recent block hashes per table Keeper remembers (plus _seconds). A retry after the window has rolled past can duplicate.
Version noteDeduplication defaults moved recently: since 26.2 sync and async inserts are deduplicated by default under deduplicate_insert, and plain MergeTree needs non_replicated_deduplication_window > 0. In an interview, say "Replicated tables dedup by block hash in Keeper within a window" and mention that you would check system.settings on the version in front of you.
-- Strong, retry-safe insert: two replicas before ack, explicit idempotency key
INSERT INTO events
SETTINGS insert_quorum = 2,
         insert_deduplication_token = 'ingest-7f3a:batch-000123'
VALUES (...);

-- High-volume small writes: server-side batching, still waits for the flush
INSERT INTO events SETTINGS async_insert = 1, wait_for_async_insert = 1 VALUES (...);
03

Atomicity and idempotency

Atomic part renames

rename(2) within one filesystem is atomic: a reader sees the old name or the new one, never a half-written directory. ClickHouse uses this so every part appears whole or not at all. Parts are immutable; merges write a new part, swap it in atomically, and delete the inputs later. The unit of atomicity is the part, not the INSERT: one INSERT that spans several partitions, or exceeds max_insert_block_size, becomes several parts, and a crash can leave some of them committed.

Atomic is not durable. Pillai et al. (OSDI 2014) showed that file systems differ in what they persist and in what order after a crash, and that many applications, including databases, got this wrong. A rename only survives power loss if the directory is fsynced after it.

Retries and dedup tokens

A timeout tells the client nothing. The request may have been lost, applied with the response lost, or still be running. The only way to guarantee the data lands is to retry, and the only way to make retries safe is idempotency: the server recognizes the second attempt and skips it. ClickHouse does this for Replicated tables by hashing each inserted block and remembering recent hashes in Keeper. That only works if the retry sends the identical block. If the client rebuilds the batch (adds new rows, reorders, changes block size), the hash changes and the retry duplicates. An explicit insert_deduplication_token fixes the identity to something the client controls.

Retries with and without idempotency

Simulator 2

24 batches of 1,000 rows. Each attempt can be lost on the way in (not applied), or applied with the ack lost on the way back. On timeout the client retries.

Server deduplication
On retry, the client
lostapplied, no ackdeduplicatedduplicate writtenapplied + ack
04

Delivery semantics

GuaranteeMechanismFailure outcomeClickHouse example
At-most-onceSend once, no retry. Or commit the consumer offset before processing.Loss, never duplicateswait_for_async_insert=0 with no retry; a client that drops a batch on error.
At-least-onceRetry until acked. Commit offset after processing.Duplicates, never loss (within retry budget)Kafka table engine; any retrying client without dedup.
Exactly-once (effectively)At-least-once delivery + idempotent apply (dedup keys), or atomic commit of output and input offset.Neither, within the dedup windowRetry with the same insert_deduplication_token; or ReplacingMergeTree + FINAL for eventual dedup.
Client ClickHouse INSERT … token=T applied; T recorded ✕ ack lost timeout: unknown retry, same token=T T seen: skip write ok (deduplicated)
The Two Generals problem means the client can never be sure a single attempt landed. Effectively-once moves the problem to the receiver, which can remember what it has seen.
05

Availability and failure models

A failure model is the contract your algorithm assumes about how components break. Pick one explicitly; every guarantee you state is only valid inside it.

ModelWhat can happenWhat it demandsExample
Crash-stopNode halts and never returns.Detect and replace. Simplest model; f+1 replicas tolerate f crashes for durability.Terminated VM that is never restarted.
Crash-recoveryNode halts, later restarts with stable storage intact and memory lost.Persist state before replying (Raft persists term, vote and log before acking). Recovery must handle torn writes.clickhouse-server OOM-killed, restarted by systemd.
OmissionMessages are dropped on send or receive; the node otherwise runs.Retries, timeouts, idempotency. Indistinguishable from slowness.Overloaded NIC, full socket buffer, firewall flap.
ByzantineArbitrary behavior: wrong answers, corruption, lies.3f+1 nodes and signed messages for full tolerance. Datacenter systems usually cover only a subset with checksums.Bit flip in RAM, firmware bug corrupting blocks, compromised node.
Gray failurePartial failure the system's own detector misses while clients suffer.Detect from the client's view: end-to-end probes, latency outlier detection, multiple observers.Replica passes Keeper heartbeats but its disk does 5 MB/s, so merges and queries crawl.

Availability arithmetic

99.9% is 8.8 hours of downtime a year; 99.99% is 53 minutes; 99.999% is 5.3 minutes. Components in series multiply (three 99.9% hops give about 99.7%). Independent replicas in parallel give 1 − (1 − a)n, but replicas share power, network, config and code, so real gains are smaller. Measure availability as the fraction of successful requests, not process uptime: a gray failure keeps uptime at 100% while users see errors.

06

Timeouts and failure detection

In an asynchronous network, a slow node and a dead node look identical. A timeout is a guess, and you choose which error to make. Too short: healthy nodes are declared dead during GC pauses or bursts, which triggers failovers, duplicate work and flapping, often under exactly the load that caused the slowness. Too long: real failures stay undetected and requests hang. Practical rules: base the timeout on measured tail latency (p99.9 plus headroom), require several consecutive misses, prefer adaptive detectors such as phi accrual (Cassandra, Akka), and fence deposed leaders with leases plus monotonically increasing tokens so a node that was only paused cannot act on stale authority.

ClickHouse examples: a replica that loses its Keeper session turns its replicated tables read-only until it reconnects; connect_timeout_with_failover_ms and use_hedged_requests let distributed queries route around a slow replica; max_replica_delay_for_distributed_queries avoids replicas that lag too far behind.

Choosing a timeout

Simulator 3

Healthy heartbeat latency: log-normal with a 2 ms median, plus 0.5% of probes hitting a 20–800 ms pause (GC, compaction, noisy neighbor). 100 nodes, one probe per second each. A node is declared dead after k consecutive timeouts.

Independence between consecutive probes is assumed, which flatters k > 1: real pauses are correlated, so one long GC can cause several misses in a row.

07

CAP and PACELC

CAP: during a network partition, a replicated system must either refuse some requests (keep linearizable Consistency) or answer them from possibly stale data (keep Availability). It is not "pick two of three": partitions are not optional. CAP's definitions are also narrow: C means linearizability and A means every non-failed node answers. Most real systems are neither strictly CP nor AP.

PACELC (Abadi, 2012) adds the case that matters every day: if there is a Partition, choose A or C; Else, choose Latency or Consistency. Synchronous replication costs a round trip on every write even when nothing is broken.

PACELC chooser

Simulator 4
During a partition, the minority side should
In normal operation, a read should
Else → LatencyElse → Consistency P → A
PA / ELDynamo, Cassandra and Riak with default settings. ClickHouse reads by default.
PA / ECMongoDB, in Abadi's 2012 classification.
P → C
PC / ELYahoo PNUTS: timeline consistency, local reads.
PC / ECVoltDB/H-Store, Megastore, BigTable/HBase, Spanner-style systems.

Where does a ClickHouse configuration land?

Keeper (Raft) is the consistency anchor. The side of a partition that can reach a Keeper majority keeps accepting inserts; replicas cut off from Keeper turn their replicated tables read-only. So ClickHouse is not one letter: writes lean PC, and reads lean PA unless you opt in to sequential consistency.

08

Backpressure

When producers outpace consumers, something has to give. An unbounded queue hides the problem until memory runs out, and on the way there latency grows without limit (Little's law: queue length = arrival rate × wait time). The options are to slow the producer (block, delay, return "retry later"), to shed load (drop or reject), or to add capacity. Whatever you choose must propagate upstream; a bounded queue that silently drops is just loss. Retries amplify overload, so pair them with exponential backoff, jitter and a retry budget.

ClickHouse applies backpressure on the number of active parts in a partition: above parts_to_delay_insert it slows inserts down, above parts_to_throw_insert it rejects them with "Too many parts". The fix is upstream: bigger batches, async inserts, fewer partitions. wait_for_async_insert=0 removes this signal entirely, which is one reason the docs call that mode risky.

Queue under load

Simulator 5
Policy when the queue is full
queue depthwait time (Little's law)dropped or rejected per second
09

Likely interviewer follow-ups

Q1The INSERT returned success. Can the data still be lost?

Yes. By default the part sits in one replica's page cache with no fsync, and replication is asynchronous. Power loss on that node before writeback, or losing the node before a replica fetches the part, loses acknowledged data. insert_quorum=2 closes the single-node window; adding fsync_after_insert and fsync_part_directory closes the correlated power-loss window. Each costs latency and throughput.

Q2Why doesn't ClickHouse fsync by default?

It is built for high-throughput batch ingestion, where an fsync per part would add milliseconds and extra write amplification on every insert. It relies on replication for durability, and node loss is the common failure in practice. The operator can opt in per table when the failure model includes correlated power loss.

Q3The client timed out on an insert. What should it do?

Treat the outcome as unknown and retry with exactly the same block, or with the same insert_deduplication_token. Do not re-batch the rows, because a different block hashes differently and duplicates. Retry within the dedup window, with exponential backoff and jitter.

Q4wait_for_async_insert=0 vs 1?

With 1 the ack comes after the buffer flushes to a part, so errors reach the client and slow flushes push back on it. With 0 the ack comes as soon as the data is in server memory: lower latency, but a crash loses the buffer, errors only appear in logs, and the client keeps writing during overload. Use 1 unless the data is genuinely disposable.

Q5How do async inserts relate to group commit?

Same idea: amortize an expensive step across many writers by waiting briefly to batch. In a database the expensive step is the log fsync; in ClickHouse it is creating a part, which costs file writes now and merge work later. Both trade a little latency per write for much higher throughput and fewer small parts.

Q6Is exactly-once delivery possible?

Not over a lossy network: the sender can never know whether the last attempt arrived (Two Generals). What systems offer is effectively-once processing: at-least-once delivery combined with idempotent apply, or an atomic commit of output and input offset (Kafka transactions). In ClickHouse that means dedup tokens, or ReplacingMergeTree with a version column for eventual dedup.

Q7Is rename atomic? Is it durable?

Atomic within one filesystem: readers see the old or new name, never a mix. Not durable on its own: the new directory entry can be lost on power failure until the parent directory is fsynced. That is the gap fsync_part_directory closes. Pillai et al. show file systems differ on persistence ordering, so applications should not assume it.

Q8How would you set a failure-detection timeout?

From data: measure tail latency of heartbeats under load, set the timeout above p99.9 with headroom, require a few consecutive misses, and prefer an adaptive detector. Then check the cost of a false positive. If a false failover is expensive (data movement, split brain risk), lean longer and use fencing tokens so a paused old leader cannot cause damage.

Q9Give a gray failure in a ClickHouse cluster and how you'd catch it.

One replica's disk degrades. It keeps its Keeper session, so it looks healthy, but merges fall behind, parts pile up toward "Too many parts", and distributed queries that touch it are slow. Catch it from the client's side: per-replica query latency, merge backlog and part counts, replication delay. Mitigate with hedged requests and max_replica_delay_for_distributed_queries, then drain the node.

Q10Where does ClickHouse sit in PACELC?

It depends on the operation. Default reads are PA/EL: any replica answers, possibly stale, because replication is asynchronous. Writes need Keeper, so replicas cut off from the Keeper majority become read-only (PC). With insert_quorum plus select_sequential_consistency=1 you pay latency in normal operation for consistency (EC).

Q11A customer hits "Too many parts". What is it and what do you do?

It is backpressure: active parts per partition exceeded parts_to_throw_insert because inserts create parts faster than merges combine them. Fix the cause upstream: larger, less frequent inserts (thousands to hundreds of thousands of rows), async inserts for many small clients, and a coarser partition key. Raising the threshold only delays the failure.

Q12Replication or fsync: which gives better durability?

Neither in general; it depends on the failure model. Independent node or disk loss is covered by replication; correlated power loss is covered by fsync. For RPO = 0 against both, use synchronous replication to separate failure domains with fsync on each, and accept the latency.

Q13fsync returned EIO. Retry it?

No. The kernel may already have dropped or marked clean the failed pages, so a retried fsync can report success with data missing. Treat it as fatal for that file: crash, then recover from the WAL or from a replica.

Q14Is an INSERT atomic in ClickHouse?

Each part is atomic. An INSERT that writes several parts (multiple partitions, or more rows than max_insert_block_size) is not atomic as a whole; a failure can leave some parts committed. Retrying the same INSERT with dedup in place makes the outcome converge. Keep one INSERT to one partition when atomicity matters.

10

Further reading