C++ memory model basics
Compilers and CPUs reorder memory operations. The memory model says which reorderings a correct program can observe, and it does that with one relation: happens-before.
- Sequenced-before
- Program order inside one thread.
- Synchronizes-with
- A release store (or release fence) to atomic
Mand an acquire load that reads that value, or a later value in its release sequence. - Happens-before
- The transitive closure of the two above (plus seq_cst ordering). If write W happens-before read R, R sees W or something newer.
- Data race
- Two conflicting accesses, at least one non-atomic, not ordered by happens-before. That is undefined behavior, not "a stale value".
- Modification order
- Every atomic has a single total order of writes that all threads agree on (coherence), even with relaxed.
std::atomic<bool> ready{false};
int payload = 0; // plain, non-atomic
// Thread 1
payload = 42;
ready.store(true, std::memory_order_release);
// Thread 2
while (!ready.load(std::memory_order_acquire)) {}
assert(payload == 42); // guaranteed: the release store
// synchronizes-with the acquire loadInterview framingAcquire/release is a one-way handshake on a single variable. It's what publishing a pointer, a SPSC queue or a "data is ready" flag needs. It does not give you a global order of independent events.
| Ordering | Guarantee | Typical use | Codegen (x86 / ARMv8) |
|---|---|---|---|
| relaxed | Atomicity and per-variable coherence only. No ordering with other variables. | Statistics counters: CurrentMetrics, ProfileEvents, refcount increments. | plain mov / ldr, str; RMW is lock xadd / ldadd |
| acquire | On loads: later reads and writes in this thread can't move above it. | Consumer side of a flag, lock acquisition. | mov / ldar (or ldapr) |
| release | On stores: earlier reads and writes can't move below it. | Producer side of a flag, unlock. | mov / stlr |
| acq_rel | Both, on a read-modify-write. | CAS that both takes and publishes; fetch_sub on a refcount that may hit zero. | lock cmpxchg / casal |
| seq_cst | Acquire/release plus one total order S over all seq_cst operations. The only ordering that forbids Store→Load reordering. | Dekker/Peterson-style handshakes, "set my flag, then check yours". | store becomes xchg (or mov+mfence) / stlr+ldar |
| consume | Dependency ordering. Compilers treat it as acquire, and it's being phased out of the standard. | Don't. | same as acquire |
Litmus-test explorer
Interactive · exhaustive over SC interleavingsPossible final register values
Locks vs lock-free
Lock-free is about progress guarantees, not speed. Under heavy contention a well-placed mutex often beats a CAS loop, because the expensive part is the cache line moving between cores, not the lock.
| Primitive | Uncontended | Contended | Use when | Watch out for |
|---|---|---|---|---|
| std::mutex | One CAS in user space (~20 ns) | Waiter sleeps in futex_wait; unlock pays a futex_wake syscall and wake-up latency (µs) | Default choice; critical sections of any length | Lock convoys, holding it across I/O, lock-order inversion (deadlock) |
| std::shared_mutex | Atomic RMW on a shared reader count | Readers still bounce one cache line; writers can starve or be starved depending on policy | Read-mostly data with long read sections | For short read sections it's often slower than a plain mutex. ClickHouse has its own futex-based SharedMutex and a fair FIFO RWLock for table locks. |
| Spinlock | One exchange | Waiters burn CPU; with TTAS they spin on a shared copy until release invalidates it | Very short sections, never-preempted contexts (kernel, pinned threads) | Lock-holder preemption in user space turns into whole lost time slices |
| Atomics | lock xadd: ~5–20 ns local | Every RMW needs the line in Modified state: ~50–100 ns per transfer, serialized | Counters, flags, sequence numbers | A "lock-free" counter hit by 64 cores is a serialization point too |
| CAS loop | Load + one CAS | Retries grow with contention; no fairness | Lock-free stacks/queues, max/min updates, state machines | ABA, memory reclamation, livelock-ish retry storms |
The CAS loop
// Lock-free "update max", the canonical CAS loop
void updateMax(std::atomic<uint64_t> & cur, uint64_t v)
{
uint64_t old = cur.load(std::memory_order_relaxed);
while (old < v &&
!cur.compare_exchange_weak(old, v,
std::memory_order_relaxed, // on success
std::memory_order_relaxed)) // on failure: old is reloaded
; // weak may fail spuriously on LL/SC machines, so loop
}struct SpinLock // test-and-test-and-set
{
std::atomic<bool> locked{false};
void lock()
{
for (;;)
{
if (!locked.exchange(true, std::memory_order_acquire))
return;
while (locked.load(std::memory_order_relaxed))
_mm_pause(); // spin on a Shared copy, no RMW traffic
}
}
void unlock() { locked.store(false, std::memory_order_release); }
};ABA, step by step
Treiber stackFalse sharing on a 64-byte cache line
MESI ping-pongCore 0 only writes a, core 1 only writes b. They never share data, yet in the packed layout each write needs exclusive ownership of the whole line, so it invalidates the other core's copy. Fix with alignas(std::hardware_destructive_interference_size) (64 on x86, 128 on some ARM and with adjacent-line prefetch). ClickHouse applies the same idea at a larger scale: MemoryTracker accumulates per-thread "untracked" bytes (max_untracked_memory, 4 MiB by default) before touching the shared atomic.
Contention and thread pools
Three pathologies to name on sight, then how ClickHouse decides how many threads a query actually gets.
Lock convoys
A hot lock makes threads queue. Each release must wake the next waiter (µs of futex wake + scheduling), and while that thread is being scheduled others pile up behind it. The lock's throughput is set by wake-up latency instead of the critical section, and it stays that way until load drops.
Context mutex and table RWLocks are the usual suspects at high QPS. Look at ContextLockWaitMicroseconds and RWLockReadersWaitMilliseconds in ProfileEvents, and futex_wait frames in Real-time trace samples.Priority inversion
A low-priority thread holds a lock that a high-priority thread needs, and medium-priority work keeps the low one off the CPU. The high-priority thread effectively runs at low priority. Mars Pathfinder is the textbook case; priority inheritance (PI futexes) is the textbook fix.
os_thread_priority (nice values) or the priority setting can deprioritize batch queries. If a deprioritized query holds a table lock or a cache entry a dashboard needs, the dashboard waits at batch speed.Oversubscription
More runnable threads than cores. You pay context switches, cold caches, longer run queues, and spinlocks degrade badly. Throughput flattens while latency keeps climbing.
max_threads defaults to the number of physical cores per query. 40 concurrent queries × 32 threads on 32 cores is 40× oversubscribed. Symptom: OSCPUWaitMicroseconds (runnable but not running) grows; runqlat confirms.How a query gets threads
max_threads workers, depending on how parallel the pipeline is.concurrent_threads_soft_limit_num or concurrent_threads_soft_limit_ratio_to_cores (default 2× cores). Every query keeps at least one thread. Slots are granted lazily as the pipeline needs them.max_thread_pool_size (10000), max_thread_pool_free_size, thread_pool_queue_size. When it can't schedule: CANNOT_SCHEDULE_TASK. Watch GlobalThread vs GlobalThreadActive.OSCPUWaitMicroseconds per query.CPU slot schedulers
concurrent_threads_scheduler decides who gets a freed slot when demand exceeds the soft limit.
| Value | Behavior | Failure mode |
|---|---|---|
| round_robin | Up to max_threads slots per query; the first is granted unconditionally. | Many max_threads = 1 queries each eat a slot and delay wide queries. |
| fair_round_robin | First thread is free (not counted); up to max_threads − 1 slots, round robin. | Fairer, but arrival order still matters. |
| max_min_fair default | Like fair_round_robin, but a freed slot goes to the query holding the fewest. | Best under heavy oversubscription: short queries aren't starved by long ones. |
Newer releasesWorkload scheduling (CREATE RESOURCE … (MASTER THREAD, WORKER THREAD), CREATE WORKLOAD) can treat CPU as a schedulable resource. With cpu_slot_preemption, slots are granted for a quantum (cpu_slot_quantum_ns, 10 ms) and re-requested, so fairness follows actual CPU time instead of thread count.
A growing count of delayed slots means queries are being throttled by the soft limit rather than by the hardware:
SELECT event, value FROM system.events WHERE event LIKE 'ConcurrencyControl%';
Contention simulator
Each thread loops: do W of private work, take the lock, spend S inside, release. Drag the critical section and watch where the lock becomes the ceiling, where the mutex convoys and where the spinlock collapses.
The model (shape, not benchmark numbers)
Throughput X(N) = smooth-min(parallel limit, lock limit), solved iteratively for lock utilization u.
Parallel limit = min(N, cores) / (W + S + u·ctx), with ctx = 1.5 µs of CPU per sleep/wake pair (mutex only).
Lock limit = shards / (S + handoff). Uncontended acquire costs 20 ns; under contention add a cache-line transfer h = 80 ns. Mutex handoff = 20 ns + h·u′ + 0.6·u′·3 µs (a sleeping waiter must be woken: the convoy term). Spinlock handoff = 20 ns + h·(u′ + 0.35·spinners) (invalidation storm), and past the core count S is stretched by (1 + (N − cores)/cores) for lock-holder preemption. u is the fraction of time the lock is actually held (S plus the acquire cost for the mutex, since the lock is free while a waiter wakes); u′ = u·(N−1)/N is the chance someone else holds it.
Oversubscription also costs every curve ~3% per extra multiple of cores for scheduling overhead.
Production debugging methodology
Interviewers listen for method more than tools. Scope the symptom, check resources systematically, then profile. Don't start with a guess and a restart.
USE: for every resource
Utilization (busy time), Saturation (queued work), Errors. Brendan Gregg's checklist works because it forces you to cover resources you weren't suspecting.
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU | OSUserTime, per-query UserTimeMicroseconds | OSCPUWaitMicroseconds, load vs cores, runqlat | cgroup throttling |
| Memory | MemoryTracking vs server limit, MemoryResident | page-cache churn, overcommit waits | code 241, OOM kills |
| Disk | iostat %util | await, queue depth, OSIOWaitMicroseconds | dmesg I/O errors, broken parts in detached/ |
| Network | NetworkSendBytes / NetworkReceiveBytes | NetworkReceiveElapsedMicroseconds | retransmits, socket timeouts |
| Thread pools | GlobalThreadActive, merge pool tasks vs size | pool queue length, delayed CPU slots | CANNOT_SCHEDULE_TASK |
| Locks | hard to see directly | ContextLockWaitMicroseconds, RWLock*Wait* | lock_acquire_timeout errors |
| Keeper | requests/s | outstanding requests, ZooKeeperWaitMicroseconds | session expired, ReadonlyReplica |
RED: for every service and query class
Rate, Errors, Duration. In ClickHouse the "service" view is system.query_log grouped by time and by normalized_query_hash.
SELECT
toStartOfMinute(event_time) AS minute,
count() AS queries,
countIf(type != 'QueryFinish') AS errors,
quantiles(0.5, 0.99)(query_duration_ms) AS p50_p99_ms
FROM system.query_log
WHERE event_date = today()
AND type != 'QueryStart'
GROUP BY minute
ORDER BY minute;The loop
- Pin down the symptomWhich queries, which users, which hosts, since when, what changed (deploys, config, data shape).
- RED on query_logIs it one query class, one tenant, one replica, or everything?
- USE on resourcesFind the saturated resource on the affected hosts.
- Diff ProfileEventsSame query, good vs bad run: more work, or same work waiting longer?
- Profiletrace_log CPU vs Real samples, flame graph, off-CPU stacks.
- Mitigate, then fix and preventKill or throttle first; then the durable change and an alert on the leading signal.
Latency vs throughput, and the tail
Utilization drives the queue
M/M/1, service 5 msThroughput rises linearly with ρ; latency goes as 1/(1−ρ). Going from 70% to 90% busy triples latency for 29% more throughput. That's why capacity targets sit well below 100%.
Fan-out amplifies the tail
Distributed query1 − (1 − p)n: a query over n shards is as slow as its slowest shard. Mitigations: use_hedged_requests (send to another replica if the first is slow to answer), max_replica_delay_for_distributed_queries, fewer shards per query (good sharding key), and avoiding hot replicas. Also: percentiles don't average across hosts, and load generators that wait for responses hide queueing (coordinated omission).
ClickHouse observability toolkit
The system tables are the debugger. Know which one answers which question, and have one query for each ready in muscle memory.
Incident drills
Pick an incident. At each step choose what you would do next; the signals panel fills with what that step reveals. Score counts first-try picks of the strongest move.
Likely interviewer follow-ups
Answer out loud first, then open the model answer. Each fits in about 45 seconds.
Further reading
Six sources that cover everything above. Read Preshing and the cppreference page before the memory-model questions, Gregg before the debugging ones.