Concurrency and Production Debugging - Systems - Jordi Villar
ClickHouse system design prep · interview module

Concurrency and Production Debugging

From the C++ memory model up to a pager going off at 3 a.m. Each section ends with something to poke at: litmus tests, a contention model, and incident drills that make you choose the next query to run.

Section 1

C++ memory model basics

Compilers and CPUs reorder memory operations. The memory model says which reorderings a correct program can observe, and it does that with one relation: happens-before.

Sequenced-before
Program order inside one thread.
Synchronizes-with
A release store (or release fence) to atomic M and an acquire load that reads that value, or a later value in its release sequence.
Happens-before
The transitive closure of the two above (plus seq_cst ordering). If write W happens-before read R, R sees W or something newer.
Data race
Two conflicting accesses, at least one non-atomic, not ordered by happens-before. That is undefined behavior, not "a stale value".
Modification order
Every atomic has a single total order of writes that all threads agree on (coherence), even with relaxed.
std::atomic<bool> ready{false};
int payload = 0;                  // plain, non-atomic

// Thread 1
payload = 42;
ready.store(true, std::memory_order_release);

// Thread 2
while (!ready.load(std::memory_order_acquire)) {}
assert(payload == 42);   // guaranteed: the release store
                         // synchronizes-with the acquire load

Interview framingAcquire/release is a one-way handshake on a single variable. It's what publishing a pointer, a SPSC queue or a "data is ready" flag needs. It does not give you a global order of independent events.

OrderingGuaranteeTypical useCodegen (x86 / ARMv8)
relaxedAtomicity and per-variable coherence only. No ordering with other variables.Statistics counters: CurrentMetrics, ProfileEvents, refcount increments.plain mov / ldr, str; RMW is lock xadd / ldadd
acquireOn loads: later reads and writes in this thread can't move above it.Consumer side of a flag, lock acquisition.mov / ldar (or ldapr)
releaseOn stores: earlier reads and writes can't move below it.Producer side of a flag, unlock.mov / stlr
acq_relBoth, on a read-modify-write.CAS that both takes and publishes; fetch_sub on a refcount that may hit zero.lock cmpxchg / casal
seq_cstAcquire/release plus one total order S over all seq_cst operations. The only ordering that forbids Store→Load reordering.Dekker/Peterson-style handshakes, "set my flag, then check yours".store becomes xchg (or mov+mfence) / stlr+ldar
consumeDependency ordering. Compilers treat it as acquire, and it's being phased out of the standard.Don't.same as acquire

Litmus-test explorer

Interactive · exhaustive over SC interleavings
Presets

Possible final register values

SC reachable by some interleaving weak allowed only through reordering forbidden by the model
Why, with these orderings
Example execution

Section 2

Locks vs lock-free

Lock-free is about progress guarantees, not speed. Under heavy contention a well-placed mutex often beats a CAS loop, because the expensive part is the cache line moving between cores, not the lock.

PrimitiveUncontendedContendedUse whenWatch out for
std::mutexOne CAS in user space (~20 ns)Waiter sleeps in futex_wait; unlock pays a futex_wake syscall and wake-up latency (µs)Default choice; critical sections of any lengthLock convoys, holding it across I/O, lock-order inversion (deadlock)
std::shared_mutexAtomic RMW on a shared reader countReaders still bounce one cache line; writers can starve or be starved depending on policyRead-mostly data with long read sectionsFor short read sections it's often slower than a plain mutex. ClickHouse has its own futex-based SharedMutex and a fair FIFO RWLock for table locks.
SpinlockOne exchangeWaiters burn CPU; with TTAS they spin on a shared copy until release invalidates itVery short sections, never-preempted contexts (kernel, pinned threads)Lock-holder preemption in user space turns into whole lost time slices
Atomicslock xadd: ~5–20 ns localEvery RMW needs the line in Modified state: ~50–100 ns per transfer, serializedCounters, flags, sequence numbersA "lock-free" counter hit by 64 cores is a serialization point too
CAS loopLoad + one CASRetries grow with contention; no fairnessLock-free stacks/queues, max/min updates, state machinesABA, memory reclamation, livelock-ish retry storms

The CAS loop

// Lock-free "update max", the canonical CAS loop
void updateMax(std::atomic<uint64_t> & cur, uint64_t v)
{
    uint64_t old = cur.load(std::memory_order_relaxed);
    while (old < v &&
           !cur.compare_exchange_weak(old, v,
               std::memory_order_relaxed,   // on success
               std::memory_order_relaxed))  // on failure: old is reloaded
        ;   // weak may fail spuriously on LL/SC machines, so loop
}
struct SpinLock   // test-and-test-and-set
{
    std::atomic<bool> locked{false};
    void lock()
    {
        for (;;)
        {
            if (!locked.exchange(true, std::memory_order_acquire))
                return;
            while (locked.load(std::memory_order_relaxed))
                _mm_pause();   // spin on a Shared copy, no RMW traffic
        }
    }
    void unlock() { locked.store(false, std::memory_order_release); }
};

ABA, step by step

Treiber stack

False sharing on a 64-byte cache line

MESI ping-pong
0increments
0cache-line transfers
0 nsest. time (~70 ns per transfer, ~1 ns local)

Core 0 only writes a, core 1 only writes b. They never share data, yet in the packed layout each write needs exclusive ownership of the whole line, so it invalidates the other core's copy. Fix with alignas(std::hardware_destructive_interference_size) (64 on x86, 128 on some ARM and with adjacent-line prefetch). ClickHouse applies the same idea at a larger scale: MemoryTracker accumulates per-thread "untracked" bytes (max_untracked_memory, 4 MiB by default) before touching the shared atomic.

Section 3

Contention and thread pools

Three pathologies to name on sight, then how ClickHouse decides how many threads a query actually gets.

Lock convoys

A hot lock makes threads queue. Each release must wake the next waiter (µs of futex wake + scheduling), and while that thread is being scheduled others pile up behind it. The lock's throughput is set by wake-up latency instead of the critical section, and it stays that way until load drops.

In ClickHouseThe global Context mutex and table RWLocks are the usual suspects at high QPS. Look at ContextLockWaitMicroseconds and RWLockReadersWaitMilliseconds in ProfileEvents, and futex_wait frames in Real-time trace samples.

Priority inversion

A low-priority thread holds a lock that a high-priority thread needs, and medium-priority work keeps the low one off the CPU. The high-priority thread effectively runs at low priority. Mars Pathfinder is the textbook case; priority inheritance (PI futexes) is the textbook fix.

In ClickHouseos_thread_priority (nice values) or the priority setting can deprioritize batch queries. If a deprioritized query holds a table lock or a cache entry a dashboard needs, the dashboard waits at batch speed.

Oversubscription

More runnable threads than cores. You pay context switches, cold caches, longer run queues, and spinlocks degrade badly. Throughput flattens while latency keeps climbing.

In ClickHousemax_threads defaults to the number of physical cores per query. 40 concurrent queries × 32 threads on 32 cores is 40× oversubscribed. Symptom: OSCPUWaitMicroseconds (runnable but not running) grows; runqlat confirms.

How a query gets threads

Query pipelinePipelineExecutor
Wants up to max_threads workers, depending on how parallel the pipeline is.
↓ asks for CPU slots
ConcurrencyControlserver-wide CPU slots
Soft limit from concurrent_threads_soft_limit_num or concurrent_threads_soft_limit_ratio_to_cores (default 2× cores). Every query keeps at least one thread. Slots are granted lazily as the pipeline needs them.
↓ granted slots become threads
Per-query ThreadPoolThreadFromGlobalPool
Short-lived pools borrow OS threads from the global pool rather than creating them.
↓
GlobalThreadPoolprocess-wide
max_thread_pool_size (10000), max_thread_pool_free_size, thread_pool_queue_size. When it can't schedule: CANNOT_SCHEDULE_TASK. Watch GlobalThread vs GlobalThreadActive.
↓ plus background pools (merges, mutations, fetches, replication schedule) competing for the same cores
Kernel schedulerN cores
Run-queue wait shows up as OSCPUWaitMicroseconds per query.

CPU slot schedulers

concurrent_threads_scheduler decides who gets a freed slot when demand exceeds the soft limit.

ValueBehaviorFailure mode
round_robinUp to max_threads slots per query; the first is granted unconditionally.Many max_threads = 1 queries each eat a slot and delay wide queries.
fair_round_robinFirst thread is free (not counted); up to max_threads − 1 slots, round robin.Fairer, but arrival order still matters.
max_min_fair defaultLike fair_round_robin, but a freed slot goes to the query holding the fewest.Best under heavy oversubscription: short queries aren't starved by long ones.

Newer releasesWorkload scheduling (CREATE RESOURCE … (MASTER THREAD, WORKER THREAD), CREATE WORKLOAD) can treat CPU as a schedulable resource. With cpu_slot_preemption, slots are granted for a quantum (cpu_slot_quantum_ns, 10 ms) and re-requested, so fairness follows actual CPU time instead of thread count.

A growing count of delayed slots means queries are being throttled by the soft limit rather than by the hardware:

SELECT event, value
FROM system.events
WHERE event LIKE 'ConcurrencyControl%';
Section 4

Contention simulator

Each thread loops: do W of private work, take the lock, spend S inside, release. Drag the critical section and watch where the lock becomes the ceiling, where the mutex convoys and where the spinlock collapses.

Cores
Y axis

The model (shape, not benchmark numbers)

Throughput X(N) = smooth-min(parallel limit, lock limit), solved iteratively for lock utilization u.

Parallel limit = min(N, cores) / (W + S + u·ctx), with ctx = 1.5 µs of CPU per sleep/wake pair (mutex only).

Lock limit = shards / (S + handoff). Uncontended acquire costs 20 ns; under contention add a cache-line transfer h = 80 ns. Mutex handoff = 20 ns + h·u′ + 0.6·u′·3 µs (a sleeping waiter must be woken: the convoy term). Spinlock handoff = 20 ns + h·(u′ + 0.35·spinners) (invalidation storm), and past the core count S is stretched by (1 + (N − cores)/cores) for lock-holder preemption. u is the fraction of time the lock is actually held (S plus the acquire cost for the mutex, since the lock is free while a waiter wakes); u′ = u·(N−1)/N is the chance someone else holds it.

Oversubscription also costs every curve ~3% per extra multiple of cores for scheduling overhead.

Section 5

Production debugging methodology

Interviewers listen for method more than tools. Scope the symptom, check resources systematically, then profile. Don't start with a guess and a restart.

USE: for every resource

Utilization (busy time), Saturation (queued work), Errors. Brendan Gregg's checklist works because it forces you to cover resources you weren't suspecting.

ResourceUtilizationSaturationErrors
CPUOSUserTime, per-query UserTimeMicrosecondsOSCPUWaitMicroseconds, load vs cores, runqlatcgroup throttling
MemoryMemoryTracking vs server limit, MemoryResidentpage-cache churn, overcommit waitscode 241, OOM kills
Diskiostat %utilawait, queue depth, OSIOWaitMicrosecondsdmesg I/O errors, broken parts in detached/
NetworkNetworkSendBytes / NetworkReceiveBytesNetworkReceiveElapsedMicrosecondsretransmits, socket timeouts
Thread poolsGlobalThreadActive, merge pool tasks vs sizepool queue length, delayed CPU slotsCANNOT_SCHEDULE_TASK
Lockshard to see directlyContextLockWaitMicroseconds, RWLock*Wait*lock_acquire_timeout errors
Keeperrequests/soutstanding requests, ZooKeeperWaitMicrosecondssession expired, ReadonlyReplica

RED: for every service and query class

Rate, Errors, Duration. In ClickHouse the "service" view is system.query_log grouped by time and by normalized_query_hash.

SELECT
    toStartOfMinute(event_time)            AS minute,
    count()                                AS queries,
    countIf(type != 'QueryFinish')         AS errors,
    quantiles(0.5, 0.99)(query_duration_ms) AS p50_p99_ms
FROM system.query_log
WHERE event_date = today()
  AND type != 'QueryStart'
GROUP BY minute
ORDER BY minute;

The loop

  1. Pin down the symptomWhich queries, which users, which hosts, since when, what changed (deploys, config, data shape).
  2. RED on query_logIs it one query class, one tenant, one replica, or everything?
  3. USE on resourcesFind the saturated resource on the affected hosts.
  4. Diff ProfileEventsSame query, good vs bad run: more work, or same work waiting longer?
  5. Profiletrace_log CPU vs Real samples, flame graph, off-CPU stacks.
  6. Mitigate, then fix and preventKill or throttle first; then the durable change and an alert on the leading signal.

Latency vs throughput, and the tail

Utilization drives the queue

M/M/1, service 5 ms
mean response
p99 response
in system (Little: L = λW)

Throughput rises linearly with ρ; latency goes as 1/(1−ρ). Going from 70% to 90% busy triples latency for 29% more throughput. That's why capacity targets sit well below 100%.

Fan-out amplifies the tail

Distributed query
Per-shard chance of a slow response
of queries wait for a slow shard

1 − (1 − p)n: a query over n shards is as slow as its slowest shard. Mitigations: use_hedged_requests (send to another replica if the first is slow to answer), max_replica_delay_for_distributed_queries, fewer shards per query (good sharding key), and avoiding hot replicas. Also: percentiles don't average across hosts, and load generators that wait for responses hide queueing (coordinated omission).

Section 6

ClickHouse observability toolkit

The system tables are the debugger. Know which one answers which question, and have one query for each ready in muscle memory.

Section 7

Incident drills

Pick an incident. At each step choose what you would do next; the signals panel fills with what that step reveals. Score counts first-try picks of the strongest move.

Section 8

Likely interviewer follow-ups

Answer out loud first, then open the model answer. Each fits in about 45 seconds.