Thread-Pool Starvation and Queue Collapse

Starvation is the quieter cousin of deadlock: no cycle, no JVM warning, and every thread technically doing something. The service simply stops making progress while every dashboard says it is alive.

What it looks like#

"order-worker-1" #31 waiting on condition [0x00007f2a1c0]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at jdk.internal.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
        at com.zaxxer.hikari.util.ConcurrentBag.borrow(ConcurrentBag.java:151)
        at com.acme.orders.OrderService.enrich(OrderService.java:112)
 
"order-worker-2" ... identical
"order-worker-3" ... identical
      ... all sixteen identical ...

Sixteen workers, all parked waiting for a database connection. Nothing is deadlocked — each thread will proceed the moment a connection frees — but none can, so the pool does no work at all. Every task submitted behind them waits.

The JVM reports no deadlock, because there is no cycle in the wait-for graph. You have to notice the pattern yourself: N identical stacks where N is the pool size.

Why the queue makes it worse#

// LinkedBlockingQueue with capacity Integer.MAX_VALUE.
// Never rejects, so maximumPoolSize is never reached and tasks accumulate.
ExecutorService pool = Executors.newFixedThreadPool(16);

At a thousand requests per second with a five-minute downstream outage, that queue holds three hundred thousand tasks. Each holds a request object, headers, a security context and whatever the task captured — commonly 2–10KB. That is 600MB–3GB of queued work, and the heap fails long before the dependency recovers.

There is a second, subtler harm: even after the dependency recovers, the service spends minutes working through a backlog of requests whose clients timed out long ago. The work is wasted and the recovery is delayed.

ThreadPoolExecutor pool = new ThreadPoolExecutor(
        16, 32, 60L, TimeUnit.SECONDS,
        new ArrayBlockingQueue<>(200),                       // bounded
        namedThreadFactory("order-worker-"),
        new ThreadPoolExecutor.CallerRunsPolicy());          // backpressure

Sizing the queue is a latency decision, not a memory one: queueSize / throughput is the worst-case wait. At 80 requests per second, a queue of 200 means a task at the back waits 2.5 seconds. If your client times out at 3 seconds, a longer queue is pure waste.

The nested-pool deadlock#

ExecutorService pool = Executors.newFixedThreadPool(8);
 
public Report build(String id) throws Exception {
    return pool.submit(() -> {
        // Running on a pool thread, and blocking on work that must ALSO
        // run on a pool thread.
        Future<Section> section = pool.submit(() -> loadSection(id));
        return new Report(section.get());
    }).get();
}

Once eight outer tasks are in flight, all eight threads are blocked in section.get() and no thread remains to execute any inner task. This is a permanent deadlock that the JVM cannot detect.

The same shape appears with parallelStream() inside a task already running on the common ForkJoinPool, and with a @Async method calling another @Async method on the same executor.

Three fixes: use separate pools for the two levels; restructure so no task waits on the same pool (thenCompose rather than get()); or use virtual threads, where blocking does not consume a limited resource.

Bulkheads#

@Configuration
public class ExecutorConfig {
 
    // Sized independently. A slow payment gateway cannot starve inventory.
    @Bean("paymentExecutor")
    ThreadPoolTaskExecutor paymentExecutor() {
        return executor("payment-", 8, 8, 50);
    }
 
    @Bean("inventoryExecutor")
    ThreadPoolTaskExecutor inventoryExecutor() {
        return executor("inventory-", 16, 16, 100);
    }
 
    // Optional enrichment: small, and allowed to fail fast.
    @Bean("recommendationExecutor")
    ThreadPoolTaskExecutor recommendationExecutor() {
        return executor("recs-", 4, 4, 10);
    }
 
    private ThreadPoolTaskExecutor executor(String prefix, int core, int max, int queue) {
        var e = new ThreadPoolTaskExecutor();
        e.setCorePoolSize(core);
        e.setMaxPoolSize(max);
        e.setQueueCapacity(queue);
        e.setThreadNamePrefix(prefix);
        e.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy());
        e.setWaitForTasksToCompleteOnShutdown(true);
        e.setAwaitTerminationSeconds(30);
        return e;
    }
}

Now a recommendations outage costs four threads and ten queued tasks. Checkout is untouched. The named thread prefixes also mean the next thread dump identifies the culprit in one glance.

The lighter-weight alternative is a Semaphore per dependency on a shared pool — Resilience4j's Bulkhead — which limits concurrency without the memory cost of separate pools. Both are covered in CountDownLatch, Semaphore and friends.

Monitoring#

@Bean
MeterBinder poolMetrics(ThreadPoolTaskExecutor paymentExecutor) {
    return registry -> {
        ThreadPoolExecutor tp = paymentExecutor.getThreadPoolExecutor();
        Gauge.builder("executor.queue.depth", tp, e -> e.getQueue().size())
             .tag("pool", "payment").register(registry);
        Gauge.builder("executor.active", tp, ThreadPoolExecutor::getActiveCount)
             .tag("pool", "payment").register(registry);
    };
}

Queue depth is the metric to page on. Rejections are a lagging indicator — by the time they appear, users are seeing errors.

The incident, end to end#

1. Symptom. p99 latency climbs from 180ms to 30 seconds over ten minutes. CPU at 12%. No errors yet.
2. Thread dump. All 16 order-worker threads parked in ConcurrentBag.borrow — waiting for a database connection.
3. Look one layer down. The connection pool is exhausted because a query lost its index after a schema migration and now takes eight seconds. See The N+1 query and the slow endpoint.
4. Why total outage rather than slow? The executor queue was unbounded, so every arriving request queued behind the slow ones. Heap climbed towards the limit.
5. Mitigate. Recreate the index. Latency recovers in ninety seconds.
6. Fix. Bound the queue at 200, switch to CallerRunsPolicy, split payment and order executors.
7. Guardrail. Alert on executor.queue.depth above 50 for two minutes, and a load test in CI that asserts the service returns 503s rather than filling the heap when a dependency is stubbed slow.

Step 4 is the insight worth carrying into an interview: the root cause was a missing index, but the reason it became an outage rather than a slowdown was the unbounded queue. Distinguishing the trigger from the amplifier is what senior incident analysis looks like.

What gets asked#

"What happens when a downstream service gets slow?" is the setup. The answer walks the chain: workers block, throughput collapses, the queue grows, and — depending on whether it is bounded — you either shed load or exhaust the heap. Then name the two defences: bound the queue with a rejection policy, and bulkhead per dependency.