ExecutorService and Thread-Pool Sizing

Thread pools are where concurrency theory meets an actual production incident. The default factory methods are convenient and two of the three are dangerous under load, for reasons that are entirely mechanical once you know how ThreadPoolExecutor decides what to do with a submitted task.

The decision, step by step#

That middle branch is the whole source of confusion. People expect the pool to grow under load; it grows only when the queue refuses a task. With a LinkedBlockingQueue of default capacity — which is Integer.MAX_VALUE — the queue never refuses anything.

The three convenience factories#

// corePoolSize = maximumPoolSize = n, UNBOUNDED queue.
// Never rejects. Never grows. Accumulates tasks until OutOfMemoryError.
Executors.newFixedThreadPool(n);
 
// corePoolSize = 0, maximumPoolSize = Integer.MAX_VALUE, SynchronousQueue.
// The queue holds nothing, so every task either finds an idle thread or
// creates a new one. Unbounded thread creation under a traffic spike.
Executors.newCachedThreadPool();
 
// A fixed pool of one, with all the same problems plus a single point of stall.
Executors.newSingleThreadExecutor();

Both failure modes are real. A fixed pool behind a slow downstream dependency queues every arriving request; at a thousand requests per second with a thirty-second outage that is thirty thousand task objects, each holding a request context, and the heap fills. A cached pool under the same spike creates thousands of platform threads and the JVM dies with unable to create native thread.

ThreadPoolExecutor pool = new ThreadPoolExecutor(
        16,                                  // corePoolSize
        32,                                  // maximumPoolSize
        60L, TimeUnit.SECONDS,               // idle timeout for non-core threads
        new ArrayBlockingQueue<>(500),       // BOUNDED — this is the important line
        namedThreadFactory("order-worker-"),
        new ThreadPoolExecutor.CallerRunsPolicy());
 
pool.allowCoreThreadTimeOut(true);           // let core threads retire when idle too

Rejection policies#

CallerRunsPolicy is the one to know. When the pool and queue are full, the thread that submitted the task executes it inline. That thread is usually the request-accepting thread, so while it is busy running the task it is not accepting more work — the slowdown propagates naturally back to the source. It converts an unbounded memory problem into a bounded latency problem, which is almost always the better failure.

A custom handler is often worth writing, if only to record a metric:

new RejectedExecutionHandler() {
    @Override public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
        rejections.increment();               // Micrometer counter — alert on this
        throw new RejectedExecutionException("order pool saturated, queue=" +
                executor.getQueue().size());
    }
}

Sizing#

CPU-bound work — parsing, hashing, in-memory computation. The right number is close to Runtime.getRuntime().availableProcessors(). More threads than cores cannot execute more work; they only add context switches and cache pressure. One extra thread is sometimes used to cover an occasional page fault.

I/O-bound work — database calls, HTTP requests. Threads spend most of their time waiting, so more of them than cores is correct. The classic formula from Java Concurrency in Practice:

threads = cores × targetUtilisation × (1 + waitTime / serviceTime)
 
8 cores, target 0.8, task waits 90ms and computes 10ms:
threads = 8 × 0.8 × (1 + 90/10) = 8 × 0.8 × 10 = 64

The formula is a starting point, not an answer, because the real constraint is usually downstream. Sixty-four threads all calling a database with a twenty-connection pool means forty-four of them are queued for a connection — see Connection-pool exhaustion. Size the pool to what the dependency can absorb, and let the queue and rejection policy handle the rest.

Submitting and collecting results#

Future<Order> f = pool.submit(() -> loadOrder(id));
Order order = f.get(2, TimeUnit.SECONDS);          // always use the timeout overload
 
// Run several and wait for all — the executor handles the fan-out
List<Callable<Order>> tasks = ids.stream()
        .map(id -> (Callable<Order>) () -> loadOrder(id))
        .toList();
List<Future<Order>> results = pool.invokeAll(tasks, 5, TimeUnit.SECONDS);
 
// Take results in completion order rather than submission order
CompletionService<Order> cs = new ExecutorCompletionService<>(pool);
ids.forEach(id -> cs.submit(() -> loadOrder(id)));
for (int i = 0; i < ids.size(); i++) {
    Order o = cs.take().get();                     // blocks until the NEXT one finishes
}

The defence is to wrap every task body in a try/catch that logs, or to always call get().

Shutting down#

pool.shutdown();                                     // stop accepting; finish what is queued
try {
    if (!pool.awaitTermination(30, TimeUnit.SECONDS)) {
        List<Runnable> abandoned = pool.shutdownNow();   // interrupt running tasks
        log.warn("forced shutdown, {} tasks never started", abandoned.size());
        if (!pool.awaitTermination(10, TimeUnit.SECONDS)) {
            log.error("pool did not terminate");
        }
    }
} catch (InterruptedException e) {
    pool.shutdownNow();
    Thread.currentThread().interrupt();
}

Neither shutdown() nor shutdownNow() blocks — both return immediately, which surprises people. awaitTermination is what waits. And shutdownNow interrupts the worker threads, which only stops tasks that honour interruption; a task ignoring InterruptedException runs to completion regardless.

In Spring, a ThreadPoolTaskExecutor bean with setWaitForTasksToCompleteOnShutdown(true) and setAwaitTerminationSeconds(30) does this for you as part of context shutdown. Since Java 19, ExecutorService implements AutoCloseable, so try (var pool = ...) performs shutdown and await on exit.

Monitoring#

pool.getActiveCount();          // threads currently executing
pool.getQueue().size();         // depth — the leading indicator of trouble
pool.getPoolSize();             // threads alive
pool.getCompletedTaskCount();   // throughput over time

Queue depth is the metric to alert on. A pool that is healthy has a queue near zero; a rising queue means arrival rate exceeds service rate, and it will keep rising until something breaks. By the time rejections appear, the latency damage is already done.

What gets asked#

"How would you size a thread pool?" is the standard opener, and the strong answer distinguishes CPU-bound from I/O-bound and mentions the downstream constraint. Then: what is wrong with newFixedThreadPool (unbounded queue); what happens when the pool is saturated (the four rejection policies, and why CallerRunsPolicy gives backpressure); and how you shut one down cleanly.