Acknowledged ≠ durable.
An ack is a promise about which failures a write will survive. Most questions in this interview reduce to three words: ack after what, held where, against which failure?
Summary in ten lines
- Ack
- A promise scoped to a failure model. Say which failures it covers, never just "it's saved".
- fsync vs page cache
write()survives a process crash. Onlyfsync(data + parent dir) survives power loss.- WAL + group commit
- Append intent sequentially, fsync once per batch of commits, apply later. Recovery replays the log.
- Replication vs disk
- Replication covers node and disk loss; fsync covers power loss. Correlated failures beat replication alone.
- RPO
- The window of acknowledged data you can lose: replication lag, writeback interval, or backup interval.
- Idempotency
- Timeouts leave outcomes unknown, so retries are mandatory. Dedup tokens make them safe.
- Exactly-once
- Not a transport property. It is at-least-once delivery plus idempotent or transactional apply.
- Failure models
- Crash-stop ⊂ crash-recovery, omission, Byzantine. Gray failures are the ones your health check cannot see.
- CAP / PACELC
- Partitions force A or C. The rest of the time you still pay Latency for Consistency.
- Backpressure
- Bounded queues plus a signal upstream. Unbounded queues only move the failure to latency and OOM.
Durability
What "acknowledged" really means
When a server returns success it has placed the data somewhere: in its own heap, in the kernel page cache, on a device, or on other machines. Each location dies with a different failure. A precise answer names the location and the failures it covers.
fsync vs the page cache
write()returns once bytes are in the kernel page cache. The pages are dirty and get written back later (Linux expires dirty pages after about 30 s by default). A process crash keeps them; a kernel panic or power loss does not.fsync(fd)pushes the file's data and metadata to the device and asks it to flush its volatile write cache. A new file also needs anfsyncon its parent directory, or the directory entry itself can vanish.- fsync can lie: consumer SSDs without power-loss protection, or RAID controllers with volatile caches, may acknowledge flushes they have not made durable.
- fsync errors are not retryable. After an
EIO, Linux may mark the dirty pages clean, so a second fsync "succeeds" with data gone (the 2018 PostgreSQL "fsyncgate"). The safe response is to crash and recover from the log or a replica.
Write-ahead log and group commit
A WAL turns random in-place updates into one sequential append. The commit protocol is: append the record, fsync the log, ack, apply to the main structures later. After a crash, replay the log from the last checkpoint. Group commit makes this affordable: an fsync costs from tens of microseconds (NVMe with PLP) to milliseconds, so the log writer waits briefly, gathers every commit that arrived, and pays one fsync for all of them. Per-commit latency rises slightly; throughput rises a lot.
Durability by replication vs durability by disk
They protect against different things. A local fsync survives power loss but not a dead disk or a lost VM. Copies on several machines survive losing one machine, but if every copy sits in volatile memory, a correlated event (a rack power cut, a kernel bug that panics every node, a bad deploy) loses all of them at once. Kafka with default settings and ClickHouse with default settings both lean on replication and the page cache rather than per-write fsync. The strongest guarantee combines both: quorum of replicas, each with an fsync, ideally in different failure domains.
RPO and RTO
RPO (recovery point objective) is how much acknowledged data you can lose, measured in time. RTO is how long recovery takes. With async replication, RPO equals replication lag at the moment of failure. With page-cache-only writes and a correlated power loss, it equals the dirty-page writeback window. With backups alone, it equals the backup interval. RPO = 0 requires synchronous replication (or synchronous fsync for single-node faults).
Write path: pick an ack point and a crash
Simulator 1Two replicas, A (receives the insert) and B. Pick when the server acknowledges, what fails, and when. Background work is modeled: async buffers flush within 200 ms, replicas fetch new parts within about 1 s, dirty pages reach disk within about 30 s.
The ClickHouse insert path
A synchronous INSERT into a MergeTree table sorts the block, writes it as a new immutable part in a temporary directory, renames the directory into place, and then acknowledges. On ReplicatedMergeTree the part is also registered in ClickHouse Keeper (with the block's hash, for dedup) before the ack, and the other replicas fetch the part asynchronously. So the default ack means: one replica's page cache, plus metadata in Keeper. No fsync, no second copy yet.
| Setting | Default | What it changes about the ack |
|---|---|---|
async_insert | 0 | Server buffers small inserts in memory and flushes them as one part. Flush on whichever comes first: async_insert_max_data_size (100 MiB), async_insert_busy_timeout_ms (200 ms, 1 s on Cloud, adaptive since 24.2), or async_insert_max_query_number (450). |
wait_for_async_insert | 1 | 1: ack after the buffer is flushed to a part; flush errors reach the client. 0: ack once buffered. Fire-and-forget: lost on crash, errors only in logs and system.asynchronous_inserts, and no backpressure reaches the client. |
fsync_after_insert | 0 | MergeTree setting. fsync every file of each inserted part. Large insert-throughput cost. |
fsync_part_directory | 0 | MergeTree setting. fsync the part directory after writes and renames, so the rename itself is durable. |
insert_quorum | 0 | N or 'auto' (majority). Ack only after N replicas have the part. Fails if the quorum isn't reached within insert_quorum_timeout (600 s). |
insert_quorum_parallel | 1 | Allows concurrent quorum inserts. Set to 0 if you need sequential consistency with select_sequential_consistency=1. |
select_sequential_consistency | 0 | With quorum inserts, a SELECT on a replica that lacks quorum-acknowledged data fails instead of returning stale results. |
insert_deduplication_token | empty | Explicit idempotency key that replaces the content hash. Retrying with the same token is deduplicated. |
replicated_deduplication_window | bounded | How many recent block hashes per table Keeper remembers (plus _seconds). A retry after the window has rolled past can duplicate. |
deduplicate_insert, and plain MergeTree needs non_replicated_deduplication_window > 0. In an interview, say "Replicated tables dedup by block hash in Keeper within a window" and mention that you would check system.settings on the version in front of you.-- Strong, retry-safe insert: two replicas before ack, explicit idempotency key INSERT INTO events SETTINGS insert_quorum = 2, insert_deduplication_token = 'ingest-7f3a:batch-000123' VALUES (...); -- High-volume small writes: server-side batching, still waits for the flush INSERT INTO events SETTINGS async_insert = 1, wait_for_async_insert = 1 VALUES (...);
Atomicity and idempotency
Atomic part renames
rename(2) within one filesystem is atomic: a reader sees the old name or the new one, never a half-written directory. ClickHouse uses this so every part appears whole or not at all. Parts are immutable; merges write a new part, swap it in atomically, and delete the inputs later. The unit of atomicity is the part, not the INSERT: one INSERT that spans several partitions, or exceeds max_insert_block_size, becomes several parts, and a crash can leave some of them committed.
Atomic is not durable. Pillai et al. (OSDI 2014) showed that file systems differ in what they persist and in what order after a crash, and that many applications, including databases, got this wrong. A rename only survives power loss if the directory is fsynced after it.
Retries and dedup tokens
A timeout tells the client nothing. The request may have been lost, applied with the response lost, or still be running. The only way to guarantee the data lands is to retry, and the only way to make retries safe is idempotency: the server recognizes the second attempt and skips it. ClickHouse does this for Replicated tables by hashing each inserted block and remembering recent hashes in Keeper. That only works if the retry sends the identical block. If the client rebuilds the batch (adds new rows, reorders, changes block size), the hash changes and the retry duplicates. An explicit insert_deduplication_token fixes the identity to something the client controls.
Retries with and without idempotency
Simulator 224 batches of 1,000 rows. Each attempt can be lost on the way in (not applied), or applied with the ack lost on the way back. On timeout the client retries.
Delivery semantics
| Guarantee | Mechanism | Failure outcome | ClickHouse example |
|---|---|---|---|
| At-most-once | Send once, no retry. Or commit the consumer offset before processing. | Loss, never duplicates | wait_for_async_insert=0 with no retry; a client that drops a batch on error. |
| At-least-once | Retry until acked. Commit offset after processing. | Duplicates, never loss (within retry budget) | Kafka table engine; any retrying client without dedup. |
| Exactly-once (effectively) | At-least-once delivery + idempotent apply (dedup keys), or atomic commit of output and input offset. | Neither, within the dedup window | Retry with the same insert_deduplication_token; or ReplacingMergeTree + FINAL for eventual dedup. |
Availability and failure models
A failure model is the contract your algorithm assumes about how components break. Pick one explicitly; every guarantee you state is only valid inside it.
| Model | What can happen | What it demands | Example |
|---|---|---|---|
| Crash-stop | Node halts and never returns. | Detect and replace. Simplest model; f+1 replicas tolerate f crashes for durability. | Terminated VM that is never restarted. |
| Crash-recovery | Node halts, later restarts with stable storage intact and memory lost. | Persist state before replying (Raft persists term, vote and log before acking). Recovery must handle torn writes. | clickhouse-server OOM-killed, restarted by systemd. |
| Omission | Messages are dropped on send or receive; the node otherwise runs. | Retries, timeouts, idempotency. Indistinguishable from slowness. | Overloaded NIC, full socket buffer, firewall flap. |
| Byzantine | Arbitrary behavior: wrong answers, corruption, lies. | 3f+1 nodes and signed messages for full tolerance. Datacenter systems usually cover only a subset with checksums. | Bit flip in RAM, firmware bug corrupting blocks, compromised node. |
| Gray failure | Partial failure the system's own detector misses while clients suffer. | Detect from the client's view: end-to-end probes, latency outlier detection, multiple observers. | Replica passes Keeper heartbeats but its disk does 5 MB/s, so merges and queries crawl. |
Availability arithmetic
99.9% is 8.8 hours of downtime a year; 99.99% is 53 minutes; 99.999% is 5.3 minutes. Components in series multiply (three 99.9% hops give about 99.7%). Independent replicas in parallel give 1 − (1 − a)n, but replicas share power, network, config and code, so real gains are smaller. Measure availability as the fraction of successful requests, not process uptime: a gray failure keeps uptime at 100% while users see errors.
Timeouts and failure detection
In an asynchronous network, a slow node and a dead node look identical. A timeout is a guess, and you choose which error to make. Too short: healthy nodes are declared dead during GC pauses or bursts, which triggers failovers, duplicate work and flapping, often under exactly the load that caused the slowness. Too long: real failures stay undetected and requests hang. Practical rules: base the timeout on measured tail latency (p99.9 plus headroom), require several consecutive misses, prefer adaptive detectors such as phi accrual (Cassandra, Akka), and fence deposed leaders with leases plus monotonically increasing tokens so a node that was only paused cannot act on stale authority.
ClickHouse examples: a replica that loses its Keeper session turns its replicated tables read-only until it reconnects; connect_timeout_with_failover_ms and use_hedged_requests let distributed queries route around a slow replica; max_replica_delay_for_distributed_queries avoids replicas that lag too far behind.
Choosing a timeout
Simulator 3Healthy heartbeat latency: log-normal with a 2 ms median, plus 0.5% of probes hitting a 20–800 ms pause (GC, compaction, noisy neighbor). 100 nodes, one probe per second each. A node is declared dead after k consecutive timeouts.
Independence between consecutive probes is assumed, which flatters k > 1: real pauses are correlated, so one long GC can cause several misses in a row.
CAP and PACELC
CAP: during a network partition, a replicated system must either refuse some requests (keep linearizable Consistency) or answer them from possibly stale data (keep Availability). It is not "pick two of three": partitions are not optional. CAP's definitions are also narrow: C means linearizability and A means every non-failed node answers. Most real systems are neither strictly CP nor AP.
PACELC (Abadi, 2012) adds the case that matters every day: if there is a Partition, choose A or C; Else, choose Latency or Consistency. Synchronous replication costs a round trip on every write even when nothing is broken.
PACELC chooser
Simulator 4Where does a ClickHouse configuration land?
Keeper (Raft) is the consistency anchor. The side of a partition that can reach a Keeper majority keeps accepting inserts; replicas cut off from Keeper turn their replicated tables read-only. So ClickHouse is not one letter: writes lean PC, and reads lean PA unless you opt in to sequential consistency.
Backpressure
When producers outpace consumers, something has to give. An unbounded queue hides the problem until memory runs out, and on the way there latency grows without limit (Little's law: queue length = arrival rate × wait time). The options are to slow the producer (block, delay, return "retry later"), to shed load (drop or reject), or to add capacity. Whatever you choose must propagate upstream; a bounded queue that silently drops is just loss. Retries amplify overload, so pair them with exponential backoff, jitter and a retry budget.
ClickHouse applies backpressure on the number of active parts in a partition: above parts_to_delay_insert it slows inserts down, above parts_to_throw_insert it rejects them with "Too many parts". The fix is upstream: bigger batches, async inserts, fewer partitions. wait_for_async_insert=0 removes this signal entirely, which is one reason the docs call that mode risky.
Queue under load
Simulator 5Likely interviewer follow-ups
Q1The INSERT returned success. Can the data still be lost?
Yes. By default the part sits in one replica's page cache with no fsync, and replication is asynchronous. Power loss on that node before writeback, or losing the node before a replica fetches the part, loses acknowledged data. insert_quorum=2 closes the single-node window; adding fsync_after_insert and fsync_part_directory closes the correlated power-loss window. Each costs latency and throughput.
Q2Why doesn't ClickHouse fsync by default?
It is built for high-throughput batch ingestion, where an fsync per part would add milliseconds and extra write amplification on every insert. It relies on replication for durability, and node loss is the common failure in practice. The operator can opt in per table when the failure model includes correlated power loss.
Q3The client timed out on an insert. What should it do?
Treat the outcome as unknown and retry with exactly the same block, or with the same insert_deduplication_token. Do not re-batch the rows, because a different block hashes differently and duplicates. Retry within the dedup window, with exponential backoff and jitter.
Q4wait_for_async_insert=0 vs 1?
With 1 the ack comes after the buffer flushes to a part, so errors reach the client and slow flushes push back on it. With 0 the ack comes as soon as the data is in server memory: lower latency, but a crash loses the buffer, errors only appear in logs, and the client keeps writing during overload. Use 1 unless the data is genuinely disposable.
Q5How do async inserts relate to group commit?
Same idea: amortize an expensive step across many writers by waiting briefly to batch. In a database the expensive step is the log fsync; in ClickHouse it is creating a part, which costs file writes now and merge work later. Both trade a little latency per write for much higher throughput and fewer small parts.
Q6Is exactly-once delivery possible?
Not over a lossy network: the sender can never know whether the last attempt arrived (Two Generals). What systems offer is effectively-once processing: at-least-once delivery combined with idempotent apply, or an atomic commit of output and input offset (Kafka transactions). In ClickHouse that means dedup tokens, or ReplacingMergeTree with a version column for eventual dedup.
Q7Is rename atomic? Is it durable?
Atomic within one filesystem: readers see the old or new name, never a mix. Not durable on its own: the new directory entry can be lost on power failure until the parent directory is fsynced. That is the gap fsync_part_directory closes. Pillai et al. show file systems differ on persistence ordering, so applications should not assume it.
Q8How would you set a failure-detection timeout?
From data: measure tail latency of heartbeats under load, set the timeout above p99.9 with headroom, require a few consecutive misses, and prefer an adaptive detector. Then check the cost of a false positive. If a false failover is expensive (data movement, split brain risk), lean longer and use fencing tokens so a paused old leader cannot cause damage.
Q9Give a gray failure in a ClickHouse cluster and how you'd catch it.
One replica's disk degrades. It keeps its Keeper session, so it looks healthy, but merges fall behind, parts pile up toward "Too many parts", and distributed queries that touch it are slow. Catch it from the client's side: per-replica query latency, merge backlog and part counts, replication delay. Mitigate with hedged requests and max_replica_delay_for_distributed_queries, then drain the node.
Q10Where does ClickHouse sit in PACELC?
It depends on the operation. Default reads are PA/EL: any replica answers, possibly stale, because replication is asynchronous. Writes need Keeper, so replicas cut off from the Keeper majority become read-only (PC). With insert_quorum plus select_sequential_consistency=1 you pay latency in normal operation for consistency (EC).
Q11A customer hits "Too many parts". What is it and what do you do?
It is backpressure: active parts per partition exceeded parts_to_throw_insert because inserts create parts faster than merges combine them. Fix the cause upstream: larger, less frequent inserts (thousands to hundreds of thousands of rows), async inserts for many small clients, and a coarser partition key. Raising the threshold only delays the failure.
Q12Replication or fsync: which gives better durability?
Neither in general; it depends on the failure model. Independent node or disk loss is covered by replication; correlated power loss is covered by fsync. For RPO = 0 against both, use synchronous replication to separate failure domains with fsync on each, and accept the latency.
Q13fsync returned EIO. Retry it?
No. The kernel may already have dropped or marked clean the failed pages, so a retried fsync can report success with data missing. Treat it as fatal for that file: crash, then recover from the WAL or from a replica.
Q14Is an INSERT atomic in ClickHouse?
Each part is atomic. An INSERT that writes several parts (multiple partitions, or more rows than max_insert_block_size) is not atomic as a whole; a failure can leave some parts committed. Retrying the same INSERT with dedup in place makes the outcome converge. Keep one INSERT to one partition when atomicity matters.
Further reading
- Designing Data-Intensive Applications, chapters 1, 7 and 8Ch. 1 Reliable, Scalable and Maintainable Applications: faults vs failures, the vocabulary for SLOs. Ch. 7 Transactions: atomicity, isolation levels, what "durable" means. Ch. 8 The Trouble with Distributed Systems: unreliable networks and clocks, process pauses, timeouts, fencing tokens, Byzantine faults.
- Consistency Tradeoffs in Modern Distributed Database System DesignThe PACELC paper. Short and readable; argues that the latency-consistency tradeoff in normal operation shapes designs more than CAP does.
- All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent ApplicationsShows how persistence ordering differs across file systems and how many applications rely on behavior that is not guaranteed. The source for "rename is atomic, not durable".
- ClickHouse docs: Asynchronous insertsFlush conditions,
wait_for_async_insertsemantics, deduplication of async inserts, and the warnings about fire-and-forget mode.