HikariCP Connection-Pool Exhaustion

The service stops responding. The logs fill with one exception, repeated thousands of times per minute. Everything looks healthy — CPU low, memory fine, the database responsive — and no request completes.

The symptom#

java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available,
request timed out after 30000ms (total=10, active=10, idle=0, waiting=47)
    at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
    at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
    at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(...)
    at com.acme.orders.OrderService.findRecent(OrderService.java:64)

Read the four numbers in the parentheses first:

That third row is worth knowing: if total is below the configured maximum, the problem is upstream of the pool. The database is refusing connections, or max_connections is reached across all clients.

Cause 1: a leak#

public List<Order> findRecent() throws SQLException {
    Connection conn = dataSource.getConnection();     // borrowed
    PreparedStatement ps = conn.prepareStatement(SQL);
    ResultSet rs = ps.executeQuery();
    return map(rs);
    // Never closed. On an exception path, never closed either.
    // Ten requests and the pool is empty, permanently.
}

public List<Order> findRecent() throws SQLException {
    try (Connection conn = dataSource.getConnection();
         PreparedStatement ps = conn.prepareStatement(SQL);
         ResultSet rs = ps.executeQuery()) {
        return map(rs);
    }
}

A leak has a distinctive shape: the pool degrades monotonically. Available connections drop by one per leaked call and never recover, so the service works after a restart and fails a predictable time later.

spring.datasource.hikari.leak-detection-threshold=20000

HikariCP then logs the full stack trace of the code that borrowed any connection held longer than 20 seconds:

WARN c.z.h.pool.ProxyLeakTask - Connection leak detection triggered for
conn0: url=jdbc:postgresql://... , stack trace follows
java.lang.Exception: Apparent connection leak detected
    at com.acme.orders.OrderService.findRecent(OrderService.java:64)

That single line names the method. Leak detection costs almost nothing, and it is worth running permanently in production rather than only during an incident.

Cause 2: held too long#

The connection is returned correctly — eventually. This is far more common than an outright leak and harder to spot.

@Transactional
public void completeOrder(String orderId) {
    Order order = repository.findById(orderId).orElseThrow();
    order.setStatus(COMPLETE);
 
    // The transaction is open. The connection is checked out and IDLE
    // for the entire duration of this call.
    paymentGateway.capture(order.paymentRef());     // 2-8 seconds over the network
    emailClient.sendConfirmation(order.email());    // another second
 
    repository.save(order);
}

Twenty pool slots, a three-second external call, and the pool sustains fewer than seven requests per second before queuing. When the payment gateway slows down, the pool empties immediately.

public void completeOrder(String orderId) {
    Order order = loadAndMarkPending(orderId);      // short transaction, connection released
 
    CaptureResult result = paymentGateway.capture(order.paymentRef());   // no transaction
 
    finalise(orderId, result);                       // short transaction
    events.publish(new OrderCompleted(orderId));     // email sent asynchronously
}
 
@Transactional
protected Order loadAndMarkPending(String orderId) { ... }
 
@Transactional
protected void finalise(String orderId, CaptureResult result) { ... }

Two related offenders: @Transactional on a controller method, which extends the transaction across view rendering and response serialisation; and @Transactional(readOnly = false) on read paths, which prevents routing to a replica.

Cause 3: genuinely undersized#

Only after ruling out the first two. The sizing formula from PostgreSQL's own guidance:

connections = ((core_count × 2) + effective_spindle_count)
 
For a modern 8-core database on SSD:  (8 × 2) + 1 = 17

That number is per application instance × instance count, and the total must stay under the database's max_connections with headroom for migrations, admin sessions and replicas.

spring.datasource.hikari.maximum-pool-size=15
spring.datasource.hikari.minimum-idle=15          # = max: avoids latency spikes from creating one
spring.datasource.hikari.connection-timeout=3000  # fail fast — NOT the 30s default
spring.datasource.hikari.max-lifetime=1800000     # under any database or proxy idle timeout
spring.datasource.hikari.idle-timeout=600000
spring.datasource.hikari.leak-detection-threshold=20000
spring.datasource.hikari.validation-timeout=2000

Two of those deserve explanation. connection-timeout=3000 is deliberate: waiting thirty seconds for a connection means holding a request thread for thirty seconds, so the thread pool fills too and the failure spreads. Failing in three seconds and returning a 503 keeps the failure contained.

max-lifetime must be shorter than any idle timeout imposed by the database, a proxy such as PgBouncer, or a cloud load balancer. Otherwise the pool hands out connections the far end has already closed, producing intermittent "connection reset" errors that look random.

The relationship with the thread pool#

Tomcat's default of 200 threads in front of a 15-connection pool means 185 threads can be parked waiting. They hold their stacks, their request objects and their thread-locals, and they do nothing.

Two better shapes: size the request thread pool closer to what the connection pool can feed, or use a Semaphore bulkhead so requests are rejected quickly rather than queued — see CountDownLatch, Semaphore and friends.

Monitoring#

management.metrics.enable.hikaricp=true

pending is the leading indicator. It rises before any request fails, which gives you the window to act. active pinned at maximum with pending at zero is a pool running at exactly capacity — healthy but with no headroom.

The incident, end to end#

1. Symptom. All endpoints returning 500. Logs full of Connection is not available.
2. Read the numbers. active=15 idle=0 waiting=112 — fully checked out with a large queue.
3. Leak or slow? Restart one instance: it recovers and degrades again within four minutes. Steady degradation under load points at holding, not leaking.
4. Find it. leakDetectionThreshold was already on; the log names completeOrder holding connections for 8+ seconds.
5. Correlate. The payment gateway's p99 went from 200ms to 8s twenty minutes earlier.
6. Mitigate. Add a 2-second timeout to the payment client, deploy. The pool recovers.
7. Fix. Move the gateway call out of the transaction.
8. Guardrail. An ArchUnit test failing the build if any @Transactional method reaches a client package, plus an alert on hikaricp.connections.pending.

Step 8 is the part that distinguishes a strong answer. The fix stops this outage; the guardrail stops the next one.

What gets asked#

"Your service is timing out but CPU and memory are fine — what do you check?" leads here often. Answer with the pool statistics in the exception, the three causes, and the rule about network calls inside transactions. Adding that a bigger pool is usually the wrong fix, with the reason, is what turns a correct answer into a memorable one.