Workload Scheduling - Systems - Jordi Villar
ClickHouse system design · resource scheduling & multi-tenant isolation

Workload
scheduling, deep dive

How one ClickHouse server splits CPU, memory, IO and query slots between tenants: the workload tree, the scheduling nodes it compiles into, the fairness math underneath, and what happens when demand exceeds capacity. Three simulators let you poke at the mechanics.

SELECT * FROM system.scheduler WHERE resource = 'cpu' AND type = 'request_queue' FORMAT Vertical Row 1: ────── resource: cpu path: …/production type: request_queue weight: 3 priority: 0 is_active: 1 queue_length: 14 vruntime: 938454.2 busy_periods: 63 tokens: ᴺᵁᴸᴸ
01 · The short version

Summary

One tree, many resources

You declare RESOURCEs (cpu, memory, query, per-disk read/write) and one WORKLOAD tree. Every workload compiles into scheduling nodes for every resource, visible row by row in system.scheduler.

Two sharing modes

Time-shared resources (CPU, IO, query slots) schedule small requests out of leaf queues. Space-shared memory schedules long-lived allocations that are admitted, grown, shrunk and possibly evicted.

Weights guarantee, max_* limit

weight gives a max-min fair share from below and stays work-conserving. Caps come from constraint nodes: in-flight limits, token buckets, memory limits.

Priority ≠ precedence

priority orders time-shared siblings: losers wait. precedence orders memory admission and eviction: losers can be killed. They are set independently.

Inside a leaf

scheduler picks how queries of one workload share it: fifo (default), fair, las, priority. The first two cannot starve a query; the last two can.

CPU is slots plus preemption

Threads hold CPU slots. With cpu_slot_preemption they renew every quantum and downscale if blocked too long; parking frees slots during non-CPU waits. max_cpus is a token bucket over running time.

Memory is reservations

Experimental. reserve_memory creates a pending allocation, admitted by precedence then normalized usage, evicted in reverse with LCA rules. A soft limit asks queries to spill first.

Per host, then overload

All limits are enforced locally on each host. When queues are full, reject cheaply with SERVER_OVERLOADED. Quotas and profiles are per-user counters and per-query caps, not schedulers.

02 · Model

The workload hierarchy and its scheduling nodes

Scheduling is off until you create at least one resource and one workload. A resource names something ClickHouse can meter: CPU threads (master, worker, or both), memory reservations, query slots, or reads and writes on a disk. A workload is a node in a single tree that every resource shares. Siblings divide what their parent gets according to weight, priority or precedence, and any workload can carry max_* limits for its whole subtree.

Queries enter the tree through the workload query setting, which must name a leaf. Unset means 'default'. Merges and mutations enter through the merge_workload and mutation_workload server or MergeTree settings. Workloads with no IN clause are roots; all roots hang off an implicit anonymous root at path /, so several root trees still compete by weight and priority.

CREATE RESOURCE cpu      (MASTER THREAD, WORKER THREAD);
CREATE RESOURCE memory   (MEMORY RESERVATION);
CREATE RESOURCE query    (QUERY);
CREATE RESOURCE s3_read  (READ DISK s3);
CREATE RESOURCE s3_write (WRITE DISK s3);          -- or: (READ ANY DISK, WRITE ANY DISK)

CREATE WORKLOAD all SETTINGS
    max_concurrent_threads_ratio_to_cores = 2,
    max_memory_ratio = 0.8,
    max_bytes_per_second = '2Gi';                 -- applies to s3_read and s3_write separately
CREATE WORKLOAD user        IN all  SETTINGS weight = 9;
CREATE WORKLOAD system      IN all;
CREATE WORKLOAD development IN user;
CREATE WORKLOAD production  IN user SETTINGS weight = 3;

-- per-resource override of one setting
CREATE OR REPLACE WORKLOAD all SETTINGS max_bytes_per_second = '1Gi' FOR s3_read,
                                        max_bytes_per_second = '3Gi' FOR s3_write;

SELECT count() FROM hits WHERE CounterID = 42 SETTINGS workload = 'production';
Workload tree: four resources feed root workload all, which splits into user (at least 90 percent) and system (at least 10 percent); user splits into development (at least 25 percent) and production (at least 75 percent). cpuMASTER+WORKER THREAD memoryMEMORY RESERVATION s3_readREAD DISK s3 s3_writeWRITE DISK s3 2 × cores threads 0.8 × RAM 2 GiB/s 2 GiB/s alllimits live here ≥ 90% ≥ 10% userweight = 9 systemweight = 1 ≥ 25% ≥ 75% developmentweight = 1 productionweight = 3 workload = 'production' merge_workload = 'system'
The logical tree. Percentages are lower bounds under contention (max-min fair shares), not caps: when development is idle, production can use all of user, and user can use all of all. Shares compose multiplicatively, so production is guaranteed 0.9 × 0.75 = 67.5% of each resource.

What the tree compiles into

Each workload becomes a small stack of scheduling nodes per resource. Policy nodes pick which child to serve next; constraint nodes block their subtree while a limit is hit; queues hold the actual requests or allocations. ClickHouse builds this automatically from the SQL, and you only meet node names when reading system.scheduler.

Scheduling nodes for two resources. Left, network_write: resource root, workload all, inflight_limit with 100 requests, fair policy node, then production weight 3 and development weight 1 each ending in a request_queue. Right, memory: resource root, all, allocation_limit 2 GiB, fair_allocation, then production and development each ending in an allocation_queue. TIME-SHARED SPACE-SHARED network_write all inflight_limitmax_requests = 100 ≤ 100 in flight fair (p0) 75% 25% productionweight = 3 developmentweight = 1 request_queue request_queue memory all allocation_limitmax_memory = 2Gi ≤ 2 GiB reserved fair_allocation 75% 25% productionweight = 3 developmentweight = 1 allocation_queue allocation_queue
resource root workload (logical) constraint policy queue (leaf)
Same SQL, two resources. With CREATE WORKLOAD all SETTINGS max_io_requests = 100, max_memory = '2Gi', the IO side gets an in-flight constraint and the memory side gets an allocation limit. Under a parent, a priority node sits above one fair node per priority level (hence "p0"); with all siblings at the default priority there is only one level.
Workload settingResourcesNode / mechanismSemantics
weightallchild weight in fair / fair_allocationMax-min fair share among siblings with equal priority or precedence. Guarantee from below, work conserving.
priorityCPU, IO, querypriority nodeStrict order between siblings, lower value first. Losers are delayed.
precedencememoryprecedence_allocationOrder of admission and eviction, lower value first. Losers can be killed.
max_io_requests, max_bytes_inflightdisk IOinflight_limit (max_requests, max_cost)Concurrency cap on requests and on bytes in flight.
max_bytes_per_second, max_burst_bytesdisk IObandwidth_limit (max_speed, max_burst)Token bucket on bytes.
max_concurrent_threads, …_ratio_to_coresCPUslot count limitCaps threads holding CPU slots; effectively lowers max_threads under load.
max_cpus, max_cpu_share, max_burst_cpu_secondsCPUtoken bucket over CPU timeCaps CPU consumption rate. Only with cpu_slot_preemption.
max_concurrent_queriesqueryslot count limitConcurrent queries in the subtree.
max_queries_per_second, max_burst_queriesquerytoken bucketRate at which new queries may start.
max_memorymemoryallocation_limitHard cap on reserved memory in the subtree; triggers eviction.
max_memory_before_spill, max_memory_to_spill_ratiomemorysoft_limit of allocation_limitAbove this, queries with reclaimable memory are asked to spill. Smaller of the two wins.
max_waiting_queriesquery, memory (leaf only, not inherited)queue boundBeyond it new queries fail with SERVER_OVERLOADED.
schedulerCPU, IOdiscipline of the leaf queuefifo, fair, las, priority between queries of one workload.

Exact node types for the CPU and query limits are described here by behaviour. Check system.scheduler on your build for the names it materializes.

Reading system.scheduler

One row per node, local to the host. Columns are type-specific and NULL elsewhere: queues expose queue_length, queue_cost and budget; fair children expose vruntime and fair nodes system_vruntime; inflight_limit exposes inflight_requests, inflight_cost, max_requests, max_cost; bandwidth_limit exposes max_speed, max_burst, tokens, throttling_us. Memory nodes add allocated, allocations, pending, admits, killers, victims, spills, reclaimable, reclaiming, soft_limit, rejects. Everyone gets dequeued_requests, canceled_requests, rejected_requests, their costs, busy_periods and throughput.

-- Who is backlogged right now, and how far ahead is each fair child?
SELECT resource, path, weight, priority, queue_length, queue_cost, vruntime, busy_periods, throughput
FROM system.scheduler WHERE type = 'request_queue' ORDER BY queue_length DESC;

-- Is a token bucket the bottleneck?
SELECT resource, path, max_speed, max_burst, tokens, throttling_us
FROM system.scheduler WHERE type = 'bandwidth_limit';

-- Memory pressure: who kills, who dies, who spills
SELECT path, type, allocated, allocations, pending, admits, killers, victims, spills, soft_limit, rejects
FROM system.scheduler WHERE resource = 'memory';

-- The table is per host; for a fleet view
SELECT hostName() AS h, path, queue_length FROM clusterAllReplicas(default, system.scheduler)
WHERE type = 'request_queue' AND queue_length > 0;

Persistence & ops

  • Definitions live on disk at workload_path or in Keeper at workload_zookeeper_path (recommended for consistency across nodes), or are pushed with ON CLUSTER.
  • Change with CREATE OR REPLACE WORKLOAD. A workload or resource referenced by another cannot be dropped.
  • system.workloads and system.resources can be backed up and restored together; restore modes are controlled by create_workloads_and_resources.

Config-defined, for cloud

  • <resources_and_workloads> in server config holds the same SQL. Those entities win over SQL ones and cannot be altered by users.
  • Pattern: infra limits on root in config, throw_on_unknown_workload = true, plus CREATE WORKLOAD default IN all so unmarked queries are governed. Users add children.
  • Without default existing, strict mode can break startup queries.
03 · Taxonomy

Time-shared vs space-shared resources

The split decides what "losing" means. A time-shared resource is consumed in small units and returned at once; a scheduler can always say "not yet" and the cost of delay is latency. A space-shared resource is held for the life of a query; once granted, the only way to take it back is to wait for release, ask the holder to shrink (spill), or kill it.

Time-shared: CPU, IO, query slotsSpace-shared: memory
Unit scheduledRequest: a CPU quantum, a chunk of bytes, a query startAllocation: one per query or background task
LifetimeShort; enqueued, dequeued, consumed, finishedQuery lifetime; resizes up and down while running
CostCPU nanoseconds, bytes, 1 per queryBytes reserved
Ordering knobpriority + weightprecedence + weight
Reclaim mechanismPreemption by delay: stop granting, throttleSpill (soft limit), evict = kill (hard limit)
Fairness stateVirtual runtime per child (SFQ)Normalized usage: (usage + request) / weight
Leaf noderequest_queueallocation_queue (running + pending)
Failure when fullWait; or SERVER_OVERLOADED if waiting queue boundedPending; victim gets killed; or SERVER_OVERLOADED
04 · Theory

Max-min fairness and weighted fair queueing

Max-min fairness. An allocation is max-min fair if no one can get more without taking from someone who already has less (normalized by weight). Computing it is water-filling: give every backlogged sibling capacity × wi/Σw; anyone who wants less than that keeps only what they want; redistribute the leftover among the rest; repeat. Two consequences matter for multi-tenancy: a light tenant gets its whole demand, and a heavy tenant cannot push anyone below their weighted share. That is isolation without static partitioning, because idle share is never wasted.

From fluid to packets. Demers, Keshav and Shenker (1989) turned the fluid ideal into a queueing discipline: emulate bit-by-bit round robin by stamping each packet with a virtual finish time and serving the smallest. Their key fix over Nagle's per-packet round robin was to count bytes, not packets, so a flow sending large packets cannot win. ClickHouse requests carry a cost in the same spirit: bytes for IO, CPU nanoseconds for CPU.

What ClickHouse runs. The fair node uses start-time fair queueing (SFQ). Each child has a vruntime. To dequeue, pick the active child with the smallest vruntime, then add cost / weight to it. The node remembers system_vruntime, the vruntime of the last served request. When an idle child becomes active again, its vruntime is set to system_vruntime. That reset is what stops an idle tenant from banking credit and then monopolizing the resource when it wakes up: share is "use it or lose it".

dequeue(fair):                          on child activation:
  c = argmin(vruntime) over active        c.vruntime = system_vruntime
  r = c.dequeue()
  system_vruntime = c.vruntime          priority node:
  c.vruntime += r.cost / c.weight         serve lowest priority value that is active
  return r                                 (each level is its own fair node)

Simulator · weighted fair queueing across workloads

Workloadweightpriorityarrivals / tickserved (last 60) vs max-min ideal ▏queueoldest waitstate

Each tick the server can serve C unit-cost requests. Bars show what each workload got; the black tick in each share bar is the max-min fair ideal for the current demands. Priority uses ClickHouse semantics: lower value is served first, and only same-priority siblings share by weight.

05 · Two orderings

Priority vs precedence: preemption vs eviction

Both are static orderings between siblings with "lower value wins", and both fall back to weights on ties. They differ in what the loser pays. Priority applies to time-shared resources and is non-destructive: a lower-priority workload is delayed or throttled, but its work so far is kept. Precedence applies to memory and can be destructive: a higher-precedence pending or growing allocation may evict (kill) a lower-precedence running one.

Keeping them separate lets you say "admin queries get CPU first" without also saying "admin queries may kill production queries to get memory". A typical setup gives a tenant high CPU priority but equal memory precedence.

priorityprecedence
ResourcesCPU, IO, query slotsMemory reservations
Effect on loserWaits; requests stay queued; may be downscaled (fewer threads)Stays pending; or gets killed if already running
Reversible?Yes, no work lostNo, killed work is lost and must be retried
Starvation riskYes, if a higher level is always backloggedYes, lower precedence may never be admitted or keeps being evicted
Within a levelSFQ by weightMax-min by normalized usage
system.schedulerpriority columnprecedence column; killers, victims
06 · Inside one workload

FIFO, fair, LAS and priority schedulers

weight and priority act between workloads. The workload's scheduler setting decides how the queries inside one leaf share a time-shared resource. It can be set per resource with FOR, and changing it swaps the algorithm in place. For CPU these policies reflect real CPU time only with cpu_slot_preemption = 1 (the default); without preemption each slot grant has a fixed cost.

schedulerServes nextStarvationGood forWeak spot
fifo (default)Oldest requestfreePredictable, historical behaviourNo isolation: one fat query delays everyone behind it; ignores weights
fairWFQ by the query's weight setting (default 1)freeMixed tenants inside one workloadLong and short queries get equal rate, so short ones still wait on a busy server
lasLeast attained service; levels drop each time consumption doublespossibleLowest mean latency for short queries without knowing sizes (approximates SRPT)A steady stream of short queries can starve a long one
priorityLowest workload_priority (Int64), FIFO within levelpossibleExplicit tiers set by the clientAny strict priority starves the bottom under sustained load

Weight lowering: aging in reverse. Under fair, a query's weight can drop once it crosses a threshold: weight_lowering_age_seconds, weight_lowering_cpu_seconds or weight_lowering_io_bytes, multiplied by weight_lowering_factor (clamped to [0, 1]; the first threshold to trip applies, they do not stack). This gets much of LAS's latency benefit for short queries while keeping a non-zero share for long ones, so nothing starves.

CREATE OR REPLACE WORKLOAD adhoc SETTINGS scheduler = 'fair' FOR cpu, scheduler = 'fifo' FOR s3_read;

SELECT ... SETTINGS workload = 'adhoc',
       weight_lowering_cpu_seconds = 10,   -- after 10 CPU-seconds…
       weight_lowering_factor = 0.1;       -- …this query competes at 1/10 weight
07 · Rate limits

Token buckets

A token bucket of rate r and depth b guarantees that over any window of length Δ, consumption ≤ r·Δ + b. Long-run average is capped at r; short bursts up to b pass untouched. ClickHouse uses the same shape three times:

ResourceRate rBurst bUnitsNotes
CPUmax_cpus or max_cpu_share × coresmax_burst_cpu_seconds (default 1 s)CPU-secondsNeeds cpu_slot_preemption. Counts only the Running state, not Ready or Blocked.
Disk IOmax_bytes_per_secondmax_burst_bytesbytesPer resource: reads and writes are separate buckets unless one resource covers both.
Query startsmax_queries_per_secondmax_burst_queriesqueriesBounds new query starts in any interval T by r·T + b.

The CPU case has a subtlety worth stating in an interview. With max_cpus = 0.001 and max_burst_cpu_seconds = 60, the workload can run 1 thread for 60 s, or 60 threads for 1 s, before throttling bites. If the burst is too small relative to the number of concurrent threads, many threads drain the bucket in the same instant and then all stall together, so the workload under-uses its max_cpus. The simulator below shows the related failure for discrete requests: a request larger than the bucket can never pass a strict bucket.

Simulator · token bucket

arrivals admitted tokens in bucket backlog r and b
08 · CPU

CPU slots, master vs worker threads, preemption and parking

CPU is scheduled as slots: permission for one thread to run. A query's first thread is its master (same for a merge or mutation); additional pipeline threads are workers, up to max_threads. When every slot is taken the server is in overload, and each released slot goes to whichever workload the tree selects. A query that cannot get workers still runs on its master and upscales later when slots free up.

Declaring master and worker threads as separate resources protects responsiveness: workers can saturate their pool (say 100) without blocking new queries from starting, because masters draw from their own, larger pool (say 1000). The price is that total threads are no longer tied to core count. Note that limiting master slots does not limit concurrent queries, since a slot can be released mid-query and picked up by another; for that use the QUERY resource.

CREATE RESOURCE worker_cpu (WORKER THREAD);
CREATE RESOURCE master_cpu (MASTER THREAD);
CREATE WORKLOAD all SETTINGS max_concurrent_threads = 100  FOR worker_cpu,
                             max_concurrent_threads = 1000 FOR master_cpu;
CPU slot lifecycle. A thread requests a slot, runs a quantum, renews the slot at the end of every quantum. If renewal is granted it keeps running; if it stays blocked longer than cpu_slot_preemption_timeout_ms the query downscales and the thread exits. During non-CPU waits a thread may park, releasing its slot, and re-acquires one when it resumes. Request slotmaster: query start Runningone quantum Renew slotmay block if overloaded Downscalethread exits Parkedslot released granted quantum renewed (cpu_slot_quantum_ns later) blocked >preemption_timeout_ms non-CPU wait (cpu_slot_parking) resumes CPU work Accounting for max_cpus: Running → counted Ready → not counted (waiting for a core) Blocked → not counted (IO, mutex, syscall)
Without cpu_slot_preemption a thread takes a slot at start and keeps it until its work is done, so fairness is measured in slot grants, not CPU seconds. Preemption makes the slot a lease that expires every quantum.

Preemption

With cpu_slot_preemption on, each thread renews its slot every cpu_slot_quantum_ns. Under overload the renewal may block; if it blocks longer than cpu_slot_preemption_timeout_ms the query scales down by a thread. CPU-time fairness then holds between workloads; between queries of one workload it can be violated in corner cases.

Parking

cpu_slot_parking lets a thread give its slot back at waits where it isn't using CPU: idle worker with no task, waiting on a remote or async source, a cached read fetching from storage, read-bandwidth throttling. A plain local file read does not park. It requires preemption and a single resource covering both master and worker threads.

Threads vs CPUs

max_concurrent_threads limits how many threads may hold slots; it shapes parallelism. max_cpus / max_cpu_share throttles consumption by token bucket; it caps rate even when the box is idle. Weights give guarantees; max_cpu_share gives ceilings.

Practical defaults

  • Don't set max_cpus on the root: it can never create pressure. Prefer max_concurrent_threads ≈ 2 × cores so half the threads can be in IO and the rest still oversubscribe the CPU, which is what makes the policy bite.
  • Declaring a CPU resource disables concurrent_threads_soft_limit_num/_ratio_to_cores.
  • use_concurrency_control = 0 bypasses CPU scheduling; pin it with a settings constraint.
  • Not yet applied to merges and mutations.
CREATE WORKLOAD all         SETTINGS max_concurrent_threads_ratio_to_cores = 2;
CREATE WORKLOAD admin       IN all SETTINGS max_concurrent_threads = 2, priority = -1;  -- first in line, tiny
CREATE WORKLOAD production  IN all SETTINGS weight = 4;                                -- ≥ 80% when contended
CREATE WORKLOAD analytics   IN production SETTINGS weight = 3, max_cpu_share = 0.7;   -- ≥ 60%, ≤ 70%
CREATE WORKLOAD ingestion   IN production;                                             -- ≥ 20%, no cap
CREATE WORKLOAD development IN all SETTINGS max_cpu_share = 0.3;                       -- ≥ 20%, ≤ 30% always
09 · Memory (experimental)

Reservations, admission, eviction and spilling

Memory is the hard one because you cannot pause an allocation. ClickHouse tracks every query's memory and sums it up the tree. With a MEMORY RESERVATION resource and a max_memory somewhere above, each query in a marked workload gets an allocation in its leaf.

  • reserve_memory > 0: the allocation starts pending and blocks until that much is available in every ancestor. reserve_memory = 0: starts running immediately, but still counts and can still be evicted. Queries without workload markup are neither queued nor evictable.
  • Running allocations grow and shrink with real usage. They never drop below reserve_memory. Set reserve_memory = max_memory_usage for a fixed, non-elastic reservation.
  • Admission: FIFO inside a leaf; across siblings, higher precedence first, then lowest normalized usage (usage + request) / weight. Eviction picks in reverse: lowest precedence, highest normalized usage.
  • Eviction is best effort: the victim dies at its next memory sync point, not instantly.
  • max_waiting_queries on a leaf bounds pending allocations; beyond it, SERVER_OVERLOADED.
Allocation lifecycle: a new allocation starts pending if reserve_memory is above zero, otherwise running. Pending is admitted to running. Running has independent increase and decrease requests, and ends either released on finish or killed on eviction. start Pendingreserved, waiting Runningincrease ↑ needs approvaldecrease ↓ needs approval Released Killed reserve > 0 reserve_memory = 0 admit finish evict

Who may kill whom

When a pending or growing allocation would breach a max_memory, the limit node looks for a victim in its subtree. The decision is made at the least common ancestor (LCA) of killer and victim, comparing the two children of the LCA on the paths to each:

  1. A pending allocation never evicts running allocations of its own workload (killer and victim coincide).
  2. Lower precedence never kills higher precedence.
  3. A pending allocation never kills one of equal precedence. Running allocations of equal precedence may evict each other when growing, based on normalized usage.

If no eligible victim exists or killing would not free enough, the request waits. That converts "too many queries at once" from MEMORY_LIMIT_EXCEEDED errors into a queue.

Spilling. max_memory_before_spill or max_memory_to_spill_ratio set a soft limit below max_memory. Above it, the scheduler asks queries holding reclaimable memory to spill: external sort runs, aggregation states, spillable hash joins, external DISTINCT, IN sets. Crossing the soft limit doesn't block or kill anything. Leave a gap between soft and hard limits so spilling has room to work before eviction starts. Estimates below min_bytes_to_spill are ignored.

CREATE RESOURCE memory (MEMORY RESERVATION);
CREATE WORKLOAD all        SETTINGS max_memory = '10Gi', max_memory_to_spill_ratio = 0.7;
CREATE WORKLOAD system     IN all  SETTINGS weight = 1;                    -- ≥ 1 GiB
CREATE WORKLOAD user       IN all  SETTINGS weight = 9;                    -- ≥ 9 GiB
CREATE WORKLOAD production IN user SETTINGS precedence = 1, weight = 3;
CREATE WORKLOAD staging    IN user SETTINGS precedence = 1, weight = 1;
CREATE WORKLOAD testing    IN user SETTINGS precedence = 2, max_waiting_queries = 20;

SELECT ... SETTINGS workload = 'production', reserve_memory = '2Gi', max_memory_usage = '6Gi';

Here testing only gets memory production and staging don't use, and is evicted first. Production and staging queue rather than kill each other. System has precedence 0, but its LCA with the user workloads is all, whose two children have equal precedence, so neither side can evict the other.

Simulator · memory admission, eviction and spilling

Workloadprecedenceweightarrival p/tickmax_waitingusednormrunpendadmitsvictimsspillsrejects

Now: running allocations (hatched = spilled)

Pending queue heads

Scheduler log

Flat tree: all three workloads are children of all. Each query reserves 0.4 to 1.8 GiB, grows to up to 1.9× its reservation, and runs 10 to 24 ticks; about 60% have reclaimable memory. Admission is strict: if the best-ranked pending head cannot fit or evict, nothing behind it is admitted that tick. Victims are chosen by lowest precedence, then highest normalized usage, then youngest query (least lost work); that last tie-break is this simulator's choice. A query whose growth is blocked runs at half speed until memory frees up.

10 · IO

IO scheduling for S3 disks

IO resources attach to disks: READ DISK s3, WRITE DISK s3, or ANY DISK. Each read or write is chopped into requests whose cost is the byte count, and each request waits in its workload's queue until the tree lets it through. For object storage this matters for three reasons: the NIC is shared by every query and by merges; S3 throttles per prefix and returns 503 SlowDown when pushed; and the number of in-flight requests drives both connection count and tail latency.

  • max_io_requests and max_bytes_inflight become an inflight_limit. Bounding concurrency bounds queueing inside the S3 client and keeps the scheduler, not the network stack, in charge of ordering.
  • max_bytes_per_second and max_burst_bytes become a bandwidth_limit token bucket.
  • Weights split bandwidth between tenants and between foreground queries and background merges (merge_workload, mutation_workload, which can also be set per table).
  • Limits are per resource. A workload with max_bytes_per_second = '10Mi' gets 10 MiB/s for reads and another 10 MiB/s for writes; one resource declared for both READ and WRITE shares one budget.
  • Request cost may be estimated before the IO completes. The budget column on queues tracks the mismatch between estimated and real cost (for example after a failed read), so fairness self-corrects over time.
<resources_and_workloads>
  CREATE RESOURCE s3_read  (READ DISK s3);
  CREATE RESOURCE s3_write (WRITE DISK s3);
  CREATE WORKLOAD all SETTINGS
      max_io_requests = 500 FOR s3_read,  max_bytes_per_second = '1280Mi' FOR s3_read,
      max_io_requests = 1000 FOR s3_write, max_bytes_per_second = '3200Mi' FOR s3_write;
  CREATE WORKLOAD default IN all;
  CREATE WORKLOAD merges  IN all SETTINGS weight = 1;
</resources_and_workloads>
11 · Scope

Per-host vs cluster-wide limits

Every scheduling decision is made on one host, and every max_* is a per-host number. Definitions are shared (Keeper storage or ON CLUSTER), so each node runs the same tree, but no node knows what the others are doing. A distributed query carries its workload setting to the shards, so each shard schedules its piece locally under the same name.

This keeps the hot path free of coordination: scheduling decisions happen per CPU quantum and per IO chunk, far too often for a network round trip. The cost is that a tenant's cluster-wide consumption is N × the per-host limit, and skew (one hot shard) is invisible to the others. Config-defined workloads also allow different limits on heterogeneous nodes.

Approach to a cluster-wide limitHowProsCons
Static splitPer-host limit = global / N (or by node capacity)Zero coordination; works todayWastes capacity under skew; must be recomputed on resize
Initiator admissionQUERY resource on the nodes that accept client trafficOne decision per query, cheapDoesn't bound fan-out work; many initiators still need a split
Leased tokensCentral allocator hands out time-bounded shares per host by demand (Google's Doorman pattern)Tracks skew; global max-min possibleAllocator is a dependency; must decide fail-open or fail-closed; lease length trades freshness for load
Gossip / usage reportsHosts publish usage periodically and rebalance local limitsNo single point of failureConverges slowly; overshoot during bursts

Compare quotas. Quota counters live on the server that received the query. In distributed processing usage is accumulated on the requestor, so a user moving to another server starts fresh there, and a restart resets counters. Same per-host trade-off, different mechanism.

12 · Adjacent controls

Quotas and settings profiles

Workloads shape how fast and in what order work runs. Quotas and settings profiles answer different questions: how much may this principal do per interval, and what is any single query allowed to do. A complete multi-tenant design uses all three.

WorkloadsQuotasSettings profiles & constraints
Keyed byWorkload (leaf named by the workload setting)user_name, ip_address, forwarded_ip_address, client_key, combinations, normalized_query_hashUser or role
MeasuresLive CPU time, bytes, slots, reserved memoryqueries, query_selects, query_inserts, errors, result_rows/bytes, read_rows/bytes, written_bytes, execution_time, queries_per_normalized_hash, …Per-query caps: max_memory_usage, max_threads, max_execution_time, max_bytes_to_read, max_concurrent_queries_for_user
WindowContinuousFOR [RANDOMIZED] INTERVAL n second…year, several per quotaPer query
On excessDelay, throttle, queue, evict, or SERVER_OVERLOADEDReject until the interval resets (or TRACKING ONLY)Query fails (or overflow_mode = break)
Role in isolationPerformance isolation under contentionBudgets, abuse limits, billing signalsBlast-radius limits; pinning the routing
CREATE SETTINGS PROFILE tenant_acme SETTINGS
    workload = 'acme' CONST,                 -- the tenant cannot pick another leaf
    use_concurrency_control = 1 CONST,       -- cannot bypass CPU scheduling
    max_threads = 16, max_execution_time = 120,
    reserve_memory = '512Mi', max_memory_usage = '8Gi'
TO acme_role;

CREATE QUOTA acme_q KEYED BY user_name
    FOR INTERVAL 1 hour MAX queries = 20000, read_bytes = 5000000000000, errors = 500,
    FOR RANDOMIZED INTERVAL 1 day MAX execution_time = 50000
TO acme_role;

Settings profiles also carry the routing: the docs suggest putting workload in a profile and making it constant, which is how a tenant's queries are guaranteed to land in their leaf. Profiles inherit, with the last definition winning.

13 · When demand exceeds capacity

Overload and SERVER_OVERLOADED

Scheduling decides who goes first; it cannot create capacity. Under sustained overload queues grow without bound, latency rises for everyone, clients time out and retry, and the retries add load. The SRE playbook is to keep queues short, reject early and cheaply, and make clients back off. ClickHouse has several rejection points:

TriggerErrorBehaviour
Leaf max_waiting_queries reached (QUERY slots or MEMORY pending)SERVER_OVERLOADEDBounded queue in front of the scheduler. Not inherited by children.
CPU overload ratio = OSCPUWaitMicroseconds / OSCPUVirtualTimeMicroseconds ≥ min_os_cpu_wait_time_ratio_to_throwSERVER_OVERLOADEDProbabilistic: probability rises linearly to 1 at max_os_cpu_wait_time_ratio_to_throw. Ignored while busy time is under os_cpu_busy_time_threshold.
Same ratio vs min/max_os_cpu_wait_time_ratio_to_drop_connectionconnection refusedServer-level, changeable without restart. Cheapest possible rejection.
Server max_concurrent_queries, user max_concurrent_queries_for_userTOO_MANY_SIMULTANEOUS_QUERIESOlder hard caps outside the workload tree.
Thresholds in <resource_overload_warnings>row in system.warningsCPU and memory overload warnings with hysteresis (warn ratio, clear ratio, duration).

Calculator · probabilistic rejection

Handling Overload, applied

  • Model cost, not QPS: one query can scan 1 KB or 1 TB. Request cost in bytes and CPU-ns is the right currency.
  • Per-customer limits so one tenant's spike doesn't become everyone's outage.
  • Client-side adaptive throttling: clients drop locally with probability max(0, (requests − K·accepts) / (requests + 1)), so rejections stop costing the server anything.
  • Criticality: shed the lowest tier first. Maps to priority/precedence plus per-tier max_waiting_queries.

Addressing Cascading Failures, applied

  • Bound queues; long queues just convert overload into latency and timeouts.
  • Retry budgets and jittered exponential backoff; never retry at every layer.
  • Propagate deadlines and drop work whose caller has given up. A blocked query in ClickHouse waits indefinitely and isn't in SHOW PROCESSLIST, so client timeouts plus max_waiting_queries are the guard.
  • Eviction kills work; killed work is retried; retries add load. Prefer spill and queue before kill.
14 · Practice

Likely interviewer follow-ups

Why are weights a guarantee and not a limit? How do I get a hard cap?fundamentals

Max-min fairness is work conserving: if a sibling is idle its share is redistributed, so weights only bind under contention. That's what you want for utilization. A hard cap is a separate constraint node: max_concurrent_threads, max_cpu_share, max_bytes_per_second, max_memory, max_concurrent_queries. Caps waste capacity when the box is idle, so use them for contractual tiers or to protect a shared dependency, and use weights for everything else.

Why SFQ with vruntime, and why reset to system_vruntime on activation?fairness

SFQ is O(log n) per dequeue with a heap (or O(n) for small n), handles variable cost, and needs no global clock: virtual time is just the start tag of the last served request. Resetting an activating child to system_vruntime prevents credit hoarding: an idle tenant that comes back can't claim the time it didn't use and starve the others. The cost is that bursty tenants get no reward for being idle; if you wanted that, you'd add a bounded credit (a token bucket in front) explicitly.

How do shares compose down the tree?fairness

Multiplicatively along the path, and only among active siblings at each level. With all→production (weight 4 of 5) →analytics (3 of 4), analytics is guaranteed 0.8 × 0.75 = 60%. Hierarchical WFQ also gives isolation of subtrees: a flood inside development can't take more than development's share from production, no matter how many queries it has. Flat per-query fairness would let a tenant win by sending more queries.

Without cpu_slot_preemption, what does fairness actually mean?cpu

Fairness in the number of slot grants, not CPU seconds. A thread holds its slot until it finishes its work, so one long scan can hold a slot indefinitely while short queries queue. Preemption turns the slot into a lease renewed every quantum, so the scheduler gets a decision point regularly and can downscale queries that keep losing. The trade-off is context switches and renewal overhead; the quantum size trades responsiveness against overhead.

Why two settings, priority and precedence, instead of one?design

Because delaying and killing have very different costs. You might want admin queries to jump the CPU queue but never kill a 40-minute production aggregation to get memory. Coupling them forces a choice between latency for admins and safety for production. Keeping them independent lets operators express both; the same workload can be priority −1 and precedence 0 equal to everyone.

Strict priority starves. How would you prevent it without losing the semantics?starvation

Options, roughly in order of how much they weaken priority: cap the high tier (max_concurrent_threads, max_cpu_share) so it can't saturate; aging, where a waiting request's effective priority rises with wait time; a minimum share for lower tiers (a small fair slice under the priority node, like Linux SCHED_DEADLINE's admission or CFS bandwidth reserving); or replacing priority with large weight ratios (e.g. 100:1) which keeps "almost strict" behaviour but is starvation-free. In ClickHouse the idiomatic answer is the first: give the high-priority workload a small cap, as in the admin example.

LAS vs fair with weight lowering: when would you pick each?schedulers

LAS minimizes mean response time for heavy-tailed workloads without knowing query sizes, so it's great for interactive dashboards mixed with occasional large scans. It can starve a long query under a continuous stream of short ones. fair with weight_lowering_cpu_seconds and a factor like 0.1 gets most of the benefit (short queries run at 10× the weight of long ones) while guaranteeing the long query a non-zero rate. If an SLA covers long queries, choose fair.

How would you choose the eviction victim, and how would you make eviction cheaper?memory

Order of preference: lowest precedence, then highest usage relative to weight (the tenant most over its fair share), then the query whose death wastes least work (youngest, or least CPU attained), and among those the one that frees enough memory with the fewest kills. Make it cheaper by spilling first (soft limit), by restartable or checkpointed operators, by automatic retry for idempotent SELECTs, and by returning a distinct error so clients can back off instead of retrying immediately.

Strict FIFO admission has head-of-line blocking. Would you add backfilling?memory

A large pending reservation blocks smaller ones behind it even when they'd fit. Backfilling (as in HPC batch schedulers) admits a later, smaller request if it fits now, but can starve the big one forever. Safe variants: EASY backfill (only admit if it doesn't delay the head's earliest start, which needs duration estimates), or a reservation that accumulates freed memory for the head while letting others use only what's left. In a database, durations are poorly known, so I'd bound starvation with a max wait after which backfilling stops for that head.

Each resource is scheduled independently. Is that a problem?extension

It can be. A tenant can be first in line for CPU but stuck behind IO throttling, holding CPU slots (or memory) while waiting. Parking addresses the CPU half. More generally, multi-resource fairness is Dominant Resource Fairness: equalize each tenant's dominant share (its largest fraction of any resource). DRF is strategy-proof and Pareto efficient, but needs per-query resource vectors up front, which analytical queries don't have. A pragmatic extension is to feed IO wait into CPU slot release (parking) and memory admission into CPU admission, so a query doesn't hold one resource while blocked on another.

How would you build a cluster-wide limit for a tenant?distributed

Keep per-host scheduling for the hot path and add a slow control loop above it. A coordinator (or Keeper-backed leader) collects per-host demand every few seconds and assigns per-host max_* values whose sum is the global limit, max-min fair across hosts by demand. Hosts treat the assignment as a lease; if it expires they fall back to global / N (fail-static). Overshoot during a burst is bounded by lease length × burst. This is the Doorman design, and it fits ClickHouse because workloads are already hot-reloadable with CREATE OR REPLACE WORKLOAD.

How do you size max_burst_cpu_seconds and other buckets?tokens

The burst must exceed the largest single request cost, or strict buckets deadlock (see the simulator). For CPU, burst should cover a quantum for every thread the workload might run concurrently, roughly max_concurrent_threads × quantum, plus room for the latency target: a burst of B lets an idle workload run B CPU-seconds at full parallelism, which is what makes short dashboard queries fast. Too large a burst defeats the cap over short horizons and lets a tenant spike a shared box.

How would you extend CPU scheduling to merges and mutations?extension

Merges already route IO via merge_workload. For CPU they need preemption points: a merge processes blocks, so renew the slot per block and allow pausing between blocks. Two risks: starving merges leads to too many parts and then insert failures (TOO_MANY_PARTS), which is a correctness-adjacent outage; and merges hold memory for the merge's lifetime. So give the merge workload a guaranteed weight rather than low priority, and make its memory precedence high enough that it isn't evicted, with backpressure on inserts as the release valve.

Scheduling vs admission control vs throttling: distinguish them.fundamentals

Scheduling orders work that is already accepted (fair, priority nodes). Throttling caps rate regardless of contention (bandwidth_limit, token buckets). Admission control decides whether work may start at all (memory reservations, query slots, max_waiting_queries, overload rejection). Good systems admit only what they can finish, schedule fairly what they admit, and throttle only where a contract or a fragile dependency demands it.

How would you debug "production is slow even though it has weight 9"?operations

Start at system.scheduler. Is production's queue long and its vruntime low (it's waiting for capacity, so look at the constraint above it: is_satisfied = 0, inflight_requests = max_requests, or throttling_us climbing)? Is a higher-priority sibling active? Are queries even landing in production (check the workload setting and throw_on_unknown_workload)? Is anyone bypassing with use_concurrency_control = 0? For memory, look at pending, victims. Then check the resource isn't the problem at all: the slow part may be an unscheduled resource such as local disk.

Priority inversion: can a low-priority query block a high-priority one?design

Yes, via shared state the scheduler doesn't see: locks, a shared hash table being built for a cache, or memory held by a low-precedence query while the high-priority one waits on reservation. Classical fix is priority inheritance: the holder temporarily inherits the waiter's priority. In ClickHouse terms you'd boost the holder's workload for the duration, or simply make sure contended structures aren't held across scheduling points. Parking and preemption timeouts reduce the window.

15 · Sources

Further reading

  • ClickHouse docs
    Workload scheduling

    Resources, the workload tree, CPU slots and preemption, query-aware schedulers, memory reservations and spilling, query slots, node types. The primary source for this page.

  • ClickHouse docs
    system.scheduler

    Every column per node type: queue stats, vruntime, inflight and token-bucket state, memory admits, killers, victims and spills.

  • Paper, 1989
    Demers, Keshav, Shenker. Analysis and Simulation of a Fair Queueing Algorithm

    SIGCOMM ’89. Max-min fairness, bit-by-bit round robin emulation with finish tags, why cost must be in bytes. PDF.

  • Google SRE book
    Chapter 21: Handling Overload

    Cost-based limits, per-customer limits, client-side adaptive throttling, criticality, retry handling.

  • Google SRE book
    Chapter 22: Addressing Cascading Failures

    Resource exhaustion, queue management, load shedding, deadlines, retry amplification.

  • ClickHouse docs
    Quotas · CREATE QUOTA

    Interval-based limits, quota keys, accounting on the requestor server.

  • ClickHouse docs
    Settings profiles

    Grouped settings with inheritance; together with constraints, the way to pin a tenant's workload.

  • Also useful
    Server overload

    CPU wait/busy ratio, probabilistic query rejection, connection dropping, overload warnings.