Capacity Planning: Finding the Knee Before Production Does

Capacity planning has a reputation for spreadsheets, and the useful version is much smaller: find the load at which latency starts bending upward, then run comfortably below it. Everything else is arithmetic.

The curve#

Every queueing system has this shape. Below the knee, an arriving request finds a free worker immediately. Above it, requests queue, and queue time is added to every subsequent request — so latency does not rise linearly with load, it rises hyperbolically.

The collapse region is the one people underestimate: past saturation, throughput actively falls, because the system is doing work for clients that have already given up, and because the queue itself consumes memory and CPU.

Find the knee by ramping load and watching p99, not average:

export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-arrival-rate',      // arrival rate, NOT virtual users
      startRate: 50, timeUnit: '1s',
      preAllocatedVUs: 100, maxVUs: 2000,
      stages: [
        { target: 100, duration: '2m' },
        { target: 200, duration: '2m' },
        { target: 400, duration: '2m' },
        { target: 800, duration: '2m' },
      ],
    },
  },
  thresholds: { http_req_duration: ['p(99)<1000'] },
};

ramping-arrival-rate keeps to a schedule regardless of how the system responds. A closed model — fixed virtual users each waiting for a response — slows its own request rate when the system slows, which hides exactly the behaviour you are trying to measure. That is coordinated omission, explained in Latency spikes.

Watch for k6's dropped_iterations metric. If it is non-zero, the generator could not keep the schedule and the results understate the tail again.

Little's Law#

L      = average number of requests in the system
lambda = arrival rate (requests per second)
W      = average time in the system (seconds)

Three ways it earns its keep:

Sizing a pool. 500 req/s with 40ms average service time needs 500 × 0.040 = 20 concurrent workers just to keep up, with zero margin. Size for 30–40 and you have headroom.

Predicting a queue. If concurrency is capped at 10 and λ × W says you need 20, the other 10 requests per unit time must queue — and the queue grows without bound until something rejects.

Deriving the maximum. With 20 workers and a 40ms service time, maximum throughput is 20 / 0.040 = 500 req/s. Anything above that queues by definition.

Target:              1,000 req/s
Per request:         5ms CPU + 45ms database wait = 50ms
Concurrency needed:  1000 x 0.050 = 50 in flight
 
Request threads:     200      OK
DB connections:      15       <- the constraint
 
Max via the pool:    15 / 0.045 = 333 req/s
 
At 1,000 req/s arriving, 667 req/s worth of requests queue for a connection,
and the queue grows forever.

That arithmetic locates the bottleneck before any load test runs, and it is exactly the reasoning behind connection-pool exhaustion. The answer here is not more threads — it is a larger connection pool (up to what the database can take), a faster query, or fewer database calls per request.

Headroom#

Measured knee:            450 req/s per instance
Peak traffic:             3,000 req/s
Instances needed at knee: 7
 
Zone failure: lose 1 of 3 zones, remaining instances take +50%
  -> size for 3000 x 1.5 = 4,500 req/s -> 10 instances
Growth to the next review (20%)          -> 12 instances
Scaling lag: 90s pod startup vs a 10s spike -> keep 2 warm spare -> 14

The reasoning is more useful than the number. Ask: what is the largest single failure this must survive, how fast can traffic actually arrive, and how long does adding capacity take? A service that takes ninety seconds to start cannot autoscale in response to a spike that arrives in ten — it needs standing headroom instead.

Target utilisation follows from the failure domain: two zones means running near 50%, three near 66%, five near 80%. Running at 90% with any redundancy requirement is arithmetic that does not work.

What to measure#

The last three answer the question a load test should answer: which resource is the bottleneck? Knowing the knee is 450 req/s is less useful than knowing that at 450 req/s the connection pool saturates, because the second tells you what to change.

Test degradation, not just capacity#

export const options = {
  scenarios: {
    beyond_capacity: {
      executor: 'constant-arrival-rate',
      rate: 900, timeUnit: '1s',       // deliberately 2x the knee
      duration: '10m',
      preAllocatedVUs: 500, maxVUs: 3000,
    },
  },
};

The questions this answers are the ones that matter in an incident:

• Does it return fast 503s, or does latency grow without limit?
• Does memory stay flat, or does an unbounded queue fill the heap?
• Does it recover when load returns to normal, or stay degraded?
• Do circuit breakers open, and do they close again?

A service that fails cleanly at 900 req/s is far better than one that reaches 950 and then falls over and stays down. That property — graceful degradation — is worth testing explicitly, and it is the thing that distinguishes the defences in Cascading failure from theory.

Making the test honest#

Realistic data volumes. A table with a thousand rows uses different query plans from one with ten million. Load-test against production-shaped data or the results are fiction.

Realistic key distribution. Uniformly random keys give a 100% cache miss rate; a Zipfian distribution — a few very popular items — matches real traffic and exercises hot-key behaviour.

Warm up first. The JIT needs thousands of invocations. Discard the first two minutes.

Include the network. Testing from the same host removes real latency and TLS handshake cost.

Test the whole system. A single service in isolation says nothing about what happens when three services contend for the same database.

What gets asked#

"How would you size this service?" or "how do you know it can handle Black Friday?" The strong answer is a method: find the knee with an open-model ramp, use Little's Law to identify which resource saturates first, then add headroom for the largest single failure and for scaling lag.

The detail that separates candidates is testing beyond the knee. Most people size for the target load; far fewer verify what the service does when the target is exceeded — which is the situation the plan exists for.