Storage and Fault Tolerance - Systems - Jordi Villar
System design prep · Storage & fault tolerance

ClickHouse
Storage Lab

ClickHouse storage is a log of immutable parts. Inserts create parts, background merges combine them, and replication moves either the parts themselves (ReplicatedMergeTree) or only their metadata (SharedMergeTree). Almost every storage and fault-tolerance question in the interview comes back to one of those three moves.

The 60-second answer
  1. Parts are immutable. A write builds a new part in a temp directory and publishes it with one atomic rename. Nothing is updated in place, so crash recovery is "ignore the temp dirs".
  2. Merges are the price of immutability. A background pool merges adjacent parts using a size-balanced heuristic. If inserts outrun merges, ClickHouse first slows inserts, then rejects them with TOO_MANY_PARTS.
  3. ReplicatedMergeTree is multi-master and asynchronous. Keeper holds an ordered log of operations; each replica replays it and fetches part data from peers. Keeper allocates block numbers and stores insert hashes for deduplication.
  4. SharedMergeTree keeps one copy of data in object storage and the part list in Keeper. Compute nodes do not talk to each other, so adding or losing a replica moves no data.
  5. Object storage trades latency and request cost for elasticity. Caches (marks, filesystem cache on local SSD, page cache) and fewer, larger parts are what make it usable.
§1 · MergeTree parts

Immutable parts, granules, sparse index, marks

A MergeTree table is a set of parts, each a directory holding data sorted by the table's ORDER BY key. Rows are grouped into granules (default index_granularity = 8192 rows, capped by index_granularity_bytes, 10 MiB). The granule is the smallest unit ClickHouse reads.

The primary index (primary.idx) is sparse: one key per granule, small enough to sit in memory. For each column, a marks file maps granule i to a pair of offsets: where its compressed block starts in the .bin file, and where the granule starts inside the decompressed block. Query planning is a binary search over the index, then a lookup in the marks for each selected granule range.

/var/lib/clickhouse/store/…/hits/
├─ 202410_1_1_0/      partition_minBlock_maxBlock_level
├─ 202410_2_9_2/      merged twice, covers blocks 2..9
│  ├─ primary.idx       sparse index, 1 key / granule
│  ├─ UserID.bin        compressed column blocks
│  ├─ UserID.cmrk2      marks: granule → (file off, block off)
│  ├─ URL.bin  URL.cmrk2
│  ├─ minmax_EventDate.idx  partition.dat
│  ├─ skp_idx_url.idx   data-skipping index
│  └─ checksums.txt count.txt columns.txt
├─ 202410_10_10_0_14/ …_14 = mutation version
├─ tmp_insert_202410_11_11_0/ invisible until renamed
└─ detached/          broken_, ignored_, unexpected_ …
Part directory layout. Wide parts store one file per column; Compact parts (used below min_bytes_for_wide_part / min_rows_for_wide_part) pack all columns into one file. That matters a lot on S3 (§6).
primary.idx granules (8192 rows) UserID.cmrk2 UserID.bin
One index key per granule. Marks are (offset of compressed block in .bin, offset inside the decompressed block), so several granules can share one compressed block.

Mini-lab: what the sparse index skips

ORDER BY (UserID, EventTime)

Atomic publish: why a crash never leaves half a part

  1. The insert block is sorted by the ORDER BY key and split by partition.
  2. Each partition's rows are written to tmp_insert_<name>/: column files, marks, index, checksums.txt.
  3. The directory is renamed to its final name. A rename within one filesystem is atomic, so readers see the whole part or nothing.
  4. The part joins the in-memory set of active parts. Queries take a snapshot of that set, so they never see a part appear mid-query.
  5. After a restart, leftover tmp_* directories are deleted, and parts covered by a newer merged part are removed after old_parts_lifetime.
Say it in the room

"ClickHouse gets crash consistency the way an LSM tree does: write new immutable files, publish them atomically, never modify in place. Durability on a single node is weaker by default because fsync_after_insert is off; replication is what makes an acknowledged insert durable." (More in §10.)

§2 · Merges and merge scheduling

Merges, the merge selector and TOO_MANY_PARTS

Each insert creates at least one part per partition it touches. Background merges combine adjacent parts (contiguous block ranges in the same partition) into a larger part one level higher. Merges run in a shared pool (background_pool_size, default 16) together with mutations.

The default SimpleMergeSelector scans ranges of adjacent parts and favours ranges that are balanced (no single part dominates the sum), have many parts, and have a small total size. It relaxes the balance rule as parts age or as the partition gets crowded. When the pool is busy, it caps merge size so large merges cannot starve small ones; the maximum is max_bytes_to_merge_at_max_space_in_pool (150 GiB).

Backpressure is per partition. Above parts_to_delay_insert (default 1000) inserts are slowed; above parts_to_throw_insert (default 3000) they fail with TOO_MANY_PARTS. The simulator uses lab-sized thresholds so you can watch it happen.

Simulator A · inserts vs merges

one partition · lab thresholds
Scenarios
accepting
level 0 (fresh insert)higher level (merged more)being mergedwidth ∝ log(rows)

Model: each merge costs a fixed 0.6 s (open files, write, fsync, rename) plus rows ÷ 4M rows/s. Fan-in is capped at 10 parts. Write amplification = rows written by inserts and merges ÷ rows inserted. Try Tiny inserts, then the same load with async_insert: identical rows per second, far fewer parts.

Why tiny inserts hurt

  • Every part costs files, inodes, Keeper entries (replicated) and S3 PUTs (§6), whatever its size.
  • Merge throughput is bounded by fixed per-merge overhead, not just bytes. Thousands of tiny parts saturate the pool.
  • Queries open every active part, so read latency grows with part count.
  • Fixes, in order: batch on the client (10k–100k+ rows per insert), async_insert=1 to let the server batch, fewer partitions (merges never cross partitions), then more merge threads. Raising parts_to_throw_insert only buys time.

Mutations ride the same pool

OperationWhat happens
ALTER … UPDATE/DELETEAsynchronous. Rewrites every affected part into a new part with a mutation-version suffix (…_14). Tracked in system.mutations, stoppable with KILL MUTATION. Heavy: competes with merges.
DELETE FROM (lightweight)Writes a hidden _row_exists mask; rows are filtered at read time and physically dropped by later merges.
Lightweight UPDATENewer releases write small patch parts applied at read time and folded in by merges, instead of rewriting whole parts.
DROP PARTITIONMetadata-only: removes whole parts. The cheapest delete; a strong argument for time-based partitions with TTL.
Say it in the room

"Merges in ClickHouse are size-tiered compaction restricted to adjacent block ranges. Adjacency keeps a part a contiguous range of insert history, which is what makes block-number-based deduplication, DROP PARTITION and replica-identical merges simple."

§3 · ReplicatedMergeTree

A replicated log of part operations in Keeper

ReplicatedMergeTree replicates operations on parts, not rows or SQL. Every replica has a full local copy of the data. Keeper (or ZooKeeper) holds the table's shared state. Replication is asynchronous and multi-master: any replica accepts inserts.

/clickhouse/tables/{shard}/hits/
├─ log/            ordered entries: GET_PART, MERGE_PARTS,
│                   MUTATE_PART, DROP_RANGE, ATTACH_PART…
├─ block_numbers/   per-partition sequential counters
├─ blocks/          hashes of recent inserts (dedup window)
├─ quorum/          status of in-flight quorum inserts
├─ mutations/
└─ replicas/
   └─ r1/
      ├─ log_pointer  next /log entry to copy
      ├─ queue/       entries still to execute
      ├─ parts/       parts this replica has
      ├─ is_lost      1 → must clone on restart
      └─ host         where peers fetch from (:9009)

Insert path

  1. Client inserts into any replica. The block's hash is checked against /blocks; a hit means a retried insert, which is silently skipped.
  2. The replica takes the next block number for the partition from Keeper, which gives the part its name, e.g. all_17_17_0.
  3. It writes the part locally, then commits in one Keeper multi-op: the block hash, the part under its parts/, and a GET_PART entry in /log.
  4. With insert_quorum = N the insert waits until N replicas have the part before acknowledging.
  5. Other replicas copy new /log entries into their queue and fetch the part over interserver HTTP from a replica that has it.

Merges stay identical

One replica selects a merge and writes MERGE_PARTS to the log. Each replica runs the same merge on the same inputs, and the result is checked by checksum. A replica that lacks the inputs, or is configured to (always_fetch_merged_part, execute_merges_on_single_replica_time_threshold), fetches the merged part instead.

Block numbers

Monotonic per partition, so parts are named by contiguous ranges. Merges only combine adjacent ranges; DROP_RANGE and mutations target a block range; a mutation applies to all blocks below its own number.

Quorum inserts

insert_quorum trades latency for durability at ack time. insert_quorum_parallel allows concurrent quorum inserts. With select_sequential_consistency=1, reads only see quorum-confirmed parts.

Deduplication

Retrying an insert is safe if the retry sends the same block (same rows, same order, same split). insert_deduplication_token lets the client supply the key. Only the last replicated_deduplication_window hashes are kept, so a very late retry can duplicate.

§4 · SharedMergeTree

Stateless compute, metadata in Keeper, data in object storage

SharedMergeTree is ClickHouse Cloud's engine (not in open source). It keeps the MergeTree part format but moves both halves of state off the servers: part data lives once in object storage, and the authoritative part list lives in Keeper. Servers keep only a cache of metadata and a filesystem cache of data, and learn about new parts through Keeper watches. Replicas never fetch from each other.

ReplicatedMergeTree · shared-nothing SharedMergeTree · shared storage Keeper /log /blocks /block_numbers replica 1queue · merges replica 2queue · merges replica 3queue · merges all parts (copy 1) all parts (copy 2) all parts (copy 3) pull log fetchfetch Each replica: full data copy on local disk + local part metadata. Keeper part list = source of truth server 1metadata + fs cache server 2metadata + fs cache server Nmetadata + fs cache watch / commit Object storage (S3 / GCS / Azure) one copy of every part GET / PUT No server-to-server data traffic. Add a server: load metadata, start serving.
ConcernReplicatedMergeTreeSharedMergeTree
Data copiesOne per replica on local diskOne in object storage; durability delegated to it
Part metadataLocal per replica, plus Keeper replication stateKeeper is authoritative; servers cache a subset
Insert ackLocal disk; optionally insert_quorumAfter PUT to object storage and Keeper commit; quorum not needed
MergesLogged once, executed on every replica (or fetched)Executed once, result visible to every server
New / replaced replicaClone: fetch every part from peers (hours for TBs)Fetch metadata, serve immediately, cache warms over time
Scaling limitsData copies × replicas; log contention with many replicasKeeper metadata throughput, object-store latency and request rates
Main failure riskDivergent or lagging replicas, lost replicasKeeper availability; cold caches after scale-out
Say it in the room

"SharedMergeTree is the same move Snowflake made in 2016: immutable files in object storage, metadata in a strongly consistent store (FoundationDB there, Keeper here), stateless compute with local SSD caches. The design question becomes cache locality and metadata throughput instead of replica consistency."

§5 · Simulator B

Replica failure and recovery

Three replicas of one table. Kill a replica, keep inserting until Keeper trims the log, then restart it. Corrupt a part, replace a disk, or wipe Keeper. Switch engines to see which failures stop mattering when compute is stateless. The log is trimmed to 10 entries here; production uses min/max_replicated_logs_to_keep (10 / 1000).

Guided
§6 · Object-storage-backed tables

S3 disks: latency, cost and request amplification

A MergeTree table can live on an s3 (or Azure/GCS) disk via a storage policy, with local disks as a hot tier and TTL … TO VOLUME moving old parts out. Each file in a part becomes one or more objects; the local disk keeps small metadata files that point to object keys (SharedMergeTree moves that mapping into Keeper).

  • Latency. A local NVMe read takes ~100 µs. An S3 GET takes tens of milliseconds to first byte. ClickHouse hides it with parallel and prefetched reads, large read buffers, and the filesystem cache (§8).
  • Request amplification. A Wide part has about two files per column. One insert into a 200-column table is ~400 PUTs, and merges write the same data again several times. A query touching many small parts and columns turns into thousands of GETs.
  • Cost. Storage is cheap; requests are not. At S3 Standard list prices (about $0.005 per 1,000 PUTs, $0.0004 per 1,000 GETs) the request bill can exceed the storage bill for a small-insert workload.

Request calculator

rough model, list prices

Assumes ~7 metadata files per part, merges written in Wide format, and cold reads (no filesystem cache). Merged parts are larger, so the merge PUT estimate is an upper bound.

Say it in the room

"On object storage I optimise for fewer, larger objects: batch or async inserts, Compact parts for small inserts, coarse partitions, and a filesystem cache sized to the working set. Then I measure S3GetObject/S3PutObject in system.events rather than guessing."

§7 · Zero-copy replication

Why zero-copy replication was a dead end

Zero-copy replication (allow_remote_fs_zero_copy_replication) was the first attempt to put ReplicatedMergeTree on S3 without storing the data N times. Replicas still replayed the replication log, but instead of fetching part data they fetched only the local metadata that points to shared objects. It has been off by default since 22.8 and is documented as not ready for production.

Shared objects, separate owners

Each replica believed it owned its part metadata, but the objects underneath were shared. Deleting an old part required knowing no other replica still referenced it, which meant reference locks in Keeper per part per replica.

Races and leaks

Concurrent merges, mutations, fetches and drops on different replicas raced on those locks. Lost Keeper sessions left orphaned objects (storage leaks) or, worse, objects deleted while still referenced (data loss). Exclusive locks also blocked each other.

Three things to keep durable

Durability now depended on object storage, Keeper, and each server's local disk holding metadata. Losing a local disk still meant a replica rebuild. Lock overhead grew with replica count, so it did not scale past a handful of replicas.

The lesson

If data is shared, its metadata must be shared too, with one atomic commit point. SharedMergeTree drops the per-replica log and per-replica metadata entirely and makes Keeper the single source of truth.

§8 · Caches

The read path, cache by cache

A cold query on object storage pays S3 latency for every mark file and column range it touches. Each cache removes one layer of that. The bars use a log scale of typical access latency, as orders of magnitude.

Mark cache

Holds decompressed marks so planning does not re-read .mrk files. Sized by mark_cache_size (default 5 GiB). Too small and every query re-reads marks; on S3 that is a GET per mark file.

Uncompressed cache

Holds decompressed column blocks. Off by default (use_uncompressed_cache = 0). Useful for many short repeated queries on a small hot set; harmful for large scans, which evict it.

Page cache

The OS caches compressed .bin pages of local files for free. That is why ClickHouse nodes need RAM headroom beyond the server's own memory use. It does not help remote disks.

Filesystem cache

A cache disk wraps a remote disk and stores object segments on local SSD. Settings: max_size, cache_on_write_operations (cache new parts at insert), enable_filesystem_cache per query. After a restart it is still warm; after a scale-out the new node starts cold.

§9 · Recovery

Lost replicas, broken parts, detached parts

Stale replica

Down for a while but its log_pointer is still within the retained log. On restart it copies the missing entries into its queue and fetches or merges until it catches up. Watch system.replicas.absolute_delay and queue_size.

Lost replica

It fell behind further than max_replicated_logs_to_keep, so Keeper marks it is_lost = 1 and trims the log anyway. On restart it clones a healthy replica: copies its part list into the queue, keeps local parts that match, and moves the rest to detached/.

Broken part

Checksum mismatch or missing files, found at startup or by the background part check. The part moves to detached/broken_* and a fetch of the same part is queued. If too many parts look broken at startup (max_suspicious_broken_parts), the table refuses to load rather than mass-fetch.

Keeper metadata lost

Local data is fine but Keeper has no table path: the table goes read-only. SYSTEM RESTORE REPLICA re-registers local parts in Keeper on each replica; parts missing on a replica are then fetched.

Look atTells youAct with
system.replicasis_readonly, is_session_expired, queue_size, absolute_delay, log_pointer vs log_max_indexSYSTEM RESTART REPLICA, SYSTEM SYNC REPLICA
system.replication_queueStuck entries: last_exception, num_tries, postpone_reasonFix the cause (disk, network, missing source part)
system.detached_partsreason: broken, ignored, unexpected, covered-by-broken, cloneALTER TABLE … ATTACH PART or DROP DETACHED PART
system.parts, system.part_logPart counts per partition, merge history and timingTune batching, partitions, pool size
system.merges, system.mutationsRunning merges, stuck mutations (latest_fail_reason)KILL MUTATION
Keeper path missingTable read-only, metadata goneSYSTEM RESTORE REPLICA; SYSTEM DROP REPLICA for a dead host
Say it in the room

"Detached parts are a quarantine, not garbage. Before deleting, check system.detached_parts: an ignored_ part from a clone may contain rows that never reached other replicas."

§10 · Local durability

Local durability: fsync and crash safety

By default ClickHouse does not fsync inserted parts. That is a deliberate trade: on one node an acknowledged insert can be lost on power failure, and replication (or object storage) provides durability instead. To defend that choice in an interview, you need to know where the bytes are at each moment and which crash loses them.

What an acknowledged insert means on one node

A plain MergeTree acknowledges after the part directory has been renamed into place. At that moment the part's files are almost certainly only in the kernel page cache as dirty pages, and the rename is only in the filesystem's in-memory journal.

  • Process crash (kill -9, OOM killer, segfault): the kernel still holds the pages and writes them back. The acknowledged part survives.
  • Power loss or kernel panic: dirty pages and an uncommitted journal are gone. The part may vanish, or its directory may survive with truncated files.

For ReplicatedMergeTree the ack also requires the Keeper commit, and with insert_quorum ≥ 2 a copy on another machine. Those change the answer more than any fsync setting does.

Page cache vs disk

write() copies bytes into the page cache and marks them dirty. The kernel writes them back later: after about 30 s by default (vm.dirty_expire_centisecs) or under memory pressure. ext4 commits its metadata journal every 5 s by default. The drive may hold data in its own volatile cache too. Data is durable only when fsync has returned, and only if the device honours cache flushes or has power-loss protection.

Page cachereached by write()survives process crashlost on power loss
FS journalnames, sizes, renames; committed every ~5 s or by fsyncsurvives process crashsurvives power loss once committed
Device cachereached by writeback or fsyncsurvives process crashlost on power loss unless flushed or protected
Stable mediaguaranteed once fsync returnssurvives process crashsurvives power loss
Bytes move left to right over time. fsync forces them all the way to the right for one file; nothing else gives a guarantee.

fsync, fdatasync, and the directory

  • fsync(fd) flushes the file's data and all its metadata (size, timestamps).
  • fdatasync(fd) flushes data and only the metadata needed to read it back, such as the file size. It skips timestamps, so it is cheaper. ClickHouse uses it where it can.
  • Neither persists the file's name. A name is an entry in the parent directory. Creating a file or renaming a directory changes the parent, so the parent must be fsynced too. Otherwise, after a power loss the files can be on disk with the rename missing, and the part comes back under its tmp_ name, which startup deletes.

Pillai et al. (OSDI 2014) showed that filesystems differ in which operations reach disk in order. For example, a rename can persist before the data it points at under some filesystems and mount options. Applications that rely on one filesystem's behaviour lose data on another.

// The crash-safe "publish a part" pattern
fd = open("tmp_insert_all_7_7_0/UserID.bin")
write(fd, …)
fdatasync(fd)            // fsync_after_insert: each file
…same for every .bin, .mrk, checksums.txt
rename("tmp_insert_all_7_7_0", "all_7_7_0")
dfd = open("hits/", O_DIRECTORY)
fsync(dfd)               // fsync_part_directory: persist the rename

// Without the first sync: the name can survive while the files are empty → broken part.
// Without the second: the files can survive while the name does not → part disappears.

How ClickHouse writes a part, and what a power loss can take

  1. Create tmp_insert_<part>/ (or tmp_merge_, tmp-fetch_) and write column files, marks and the primary index. Crash here: the tmp directory is deleted at startup. The client got no ack.
  2. Write checksums.txt with the size and hash of every file, plus count.txt and columns.txt. Startup uses these to check that a part is complete.
  3. Optionally fsync every file (fsync_after_insert, or the merge and fetch thresholds).
  4. Rename to the final name. The part is now visible. Power loss after this without directory fsync: the rename may or may not be on disk.
  5. Optionally fsync the part directory and the table directory (fsync_part_directory).
  6. ReplicatedMergeTree: commit the part, block hash and GET_PART to Keeper in one multi-op. Then acknowledge. Power loss in the next ~30 s without fsync: the rename may be on disk with the data missing. Startup sees sizes that do not match checksums.txt and detaches the part as broken_.
SettingWhat it syncsDefaultCost
fsync_after_insertEvery file of a newly inserted part, before the rename0One sync per file, about two per column for Wide parts. The docs warn it significantly slows inserts and advise against it with Wide parts.
fsync_part_directoryThe part directory (and its parent) after writes and renames0A couple of directory syncs per part operation. Few calls, but each forces a journal commit.
min_rows_to_fsync_after_merge
min_compressed_bytes_to_fsync_after_merge
Merged parts above the threshold0 (off)Paid once per large merge. Protects merges that are expensive to redo, at low relative cost.
min_compressed_bytes_to_fsync_after_fetchParts fetched from another replica0 (off)Same idea for replication traffic.
old_parts_lifetimeNot a sync: how long source parts stay on disk after a merge480 sDisk space. Keeps the sources until writeback has made the merged part durable, so a power loss can fall back to them.
non_replicated_deduplication_windowNot a sync: insert dedup for plain MergeTree0 (off)Needed if a client retries after an ambiguous failure on a single node; otherwise retries can duplicate rows.

Replication instead of fsync

Copies on other machines are independent of this machine's power. With insert_quorum = 2, an ack means a second replica already has the part. With quorum 1, a short window remains between the ack and the peers' fetch. A replica that restarts with a broken or missing part sees that Keeper lists the part, detaches what it has, and fetches it again. This only works if replicas fail independently, so put them in different racks or zones.

SharedMergeTree and S3

The commit point is a successful object-store PUT followed by a Keeper commit. S3 returns success only after storing the object redundantly, and Keeper fsyncs its Raft log. The server's local disk holds only cache, so local fsync settings do not affect durability. A crash before the Keeper commit leaves orphan objects that cleanup removes.

Restart after a crash

tmp_* directories are removed. Parts whose files are missing or whose sizes do not match checksums.txt go to detached/broken_*. In a replicated table, local parts Keeper does not know go to detached/unexpected_*, and parts Keeper lists but the replica lacks are fetched. If many parts look broken at once, max_suspicious_broken_parts stops the table from loading instead of quietly dropping data.

Simulator C · pull the plug

choose settings, then a crash point
Crash right after step… (click a step)

    Say it in the room

    "ClickHouse treats fsync as optional because the unit of durability is the replica set, not the disk. A part is durable when Keeper has it and enough replicas hold it, or, in SharedMergeTree, when the object store has acknowledged the PUT. I'd turn on directory fsync and fsync for large merges on a single-node deployment, and use insert_quorum=2 where acknowledged data must survive a power loss."

    §11 · Likely interviewer follow-ups

    Questions they push on, with short model answers

    §12

    Further reading

    PVLDB 17(12) · 2024
    ClickHouse – Lightning Fast Analytics for Everyone

    Schulze, Schreiber, Yatsishin, Dahimene, Milovidov. Read the storage-layer section (parts, merges, replication, consistency) and the query-processing section on how the sparse index and skipping indexes prune data.

    ClickHouse blog · Aug 2023
    ClickHouse Cloud boosts performance with SharedMergeTree and Lightweight Updates

    Tom Schreiber. The clearest comparison of ReplicatedMergeTree, zero-copy replication and SharedMergeTree, with insert and merge throughput benchmarks across node counts.

    SIGMOD 2016
    The Snowflake Elastic Data Warehouse

    Dageville et al. Immutable micro-partitions in S3, metadata in FoundationDB, stateless virtual warehouses with local caches. Compare it point by point with SharedMergeTree.

    O'Reilly · DDIA
    Designing Data-Intensive Applications, ch. 3 "Storage and Retrieval"

    Martin Kleppmann. LSM trees and SSTables, compaction strategies, and column-oriented storage. It gives you the vocabulary to map MergeTree onto. In the 2nd edition this material is chapter 4.

    OSDI 2014
    All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications

    Pillai, Chidambaram, Alagappan, Al-Kiswany, Arpaci-Dusseau, Arpaci-Dusseau. The ALICE tool found crash-consistency bugs in mature applications by replaying which file operations can reach disk in which order. Read it for the rename and directory-fsync patterns behind §10.

    ClickHouse docs
    MergeTree table settings (fsync)

    Reference for fsync_after_insert, fsync_part_directory, min_rows_to_fsync_after_merge, min_compressed_bytes_to_fsync_after_merge, min_compressed_bytes_to_fsync_after_fetch and old_parts_lifetime. Search the page for "fsync". On a live server: SELECT name, value, description FROM system.merge_tree_settings WHERE name LIKE '%fsync%'.

    ClickHouse docs
    Architecture overview (the VLDB paper as docs)

    Optional: the same content as the paper in web form, easier to skim the night before.