Debugging a 100% CPU Spike in Production

The pager goes off: CPU at 100%, latency climbing, the service still passing health checks. This is the most common production incident in Java, and there is a mechanical sequence that gets from the symptom to a line number in about three minutes.

The sequence#

top -H -p 12345
# Press Shift+P to sort by CPU
 
#    PID USER      PR  NI    VIRT    RES  %CPU  %MEM  COMMAND
#  12402 app       20   0   8.4g   3.1g  99.7   19.4  java
#  12403 app       20   0   8.4g   3.1g  98.9   19.4  java
#  12401 app       20   0   8.4g   3.1g   0.3   19.4  java

Two threads pinned. On a container, top -H inside the pod works; from outside, use ps -mo pid,tid,pcpu,comm -p <pid> or kubectl exec.

printf '%x\n' 12402
# 3072

The JVM records each thread's native id in the dump as nid, in hexadecimal. On Linux that value is the same OS thread id top printed, in a different base.

jcmd 12345 Thread.print > /tmp/dump1.txt
grep -A 30 'nid=0x3072' /tmp/dump1.txt

"order-worker-7" #47 prio=5 os_prio=0 cpu=184320.11ms tid=0x00007f2a nid=0x3072 runnable
   java.lang.Thread.State: RUNNABLE
        at java.util.regex.Pattern$Loop.match(Pattern.java:4785)
        at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
        at java.util.regex.Pattern$Curly.match0(Pattern.java:4279)
        at java.util.regex.Matcher.match(Matcher.java:1728)
        at com.acme.validation.EmailValidator.isValid(EmailValidator.java:23)
        at com.acme.api.OrderController.create(OrderController.java:88)

There it is: catastrophic regex backtracking in an email validator, reached from a controller. Note the cpu=184320.11ms field in the header — that is cumulative CPU for the thread, and comparing it across dumps is an independent confirmation.

for i in 1 2 3; do jcmd 12345 Thread.print > /tmp/dump$i.txt; sleep 5; done
grep -A 5 'nid=0x3072' /tmp/dump*.txt

Identical stacks across all three means the thread is genuinely stuck there. A moving stack means it is doing real work quickly, which is a different problem.

Cause 1: catastrophic regex backtracking#

// Looks reasonable. Is exponential on non-matching input.
private static final Pattern EMAIL = Pattern.compile("^([a-zA-Z0-9]+)+@example\\.com$");
 
EMAIL.matcher("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!").matches();
// The nested quantifier (x+)+ makes the engine try every way of splitting
// the input. 30 characters is roughly 2^30 attempts — minutes of CPU.

The signature is nested quantifiers: (a+)+, (a*)*, (a|aa)+. Java's regex engine is a backtracking engine, so on input that fails to match it explores every possible decomposition.

This is also a denial-of-service vulnerability — ReDoS — when the input comes from a user. Fixes: rewrite the pattern without nesting; use a possessive quantifier ((a+)++) or an atomic group ((?>a+)) to forbid backtracking; validate length before matching; or use a non-backtracking engine such as RE2/J for user-supplied patterns.

Cause 2: an infinite or near-infinite loop#

// 1. A condition that a concurrent modification can skip past.
while (index != target) { index += step; }
 
// 2. Java 7 HashMap corrupted by concurrent put — the chain became circular.
//    Stack shows: at java.util.HashMap.get(HashMap.java:303)
//    Fixed in Java 8, but the general lesson holds: a shared HashMap can
//    corrupt in ways that produce a hot loop with no exception.

The tell is a stack that is short, identical across dumps, and sits in a method you would not expect to be slow. HashMap.get at 100% CPU is the classic, and it means a map is being shared between threads without synchronisation — see ConcurrentHashMap internals.

Cause 3: GC thrashing#

top -H -p 12345
#  12500 app  99.9  "GC task thread#0 (ParallelGC)"
#  12501 app  99.8  "GC task thread#1 (ParallelGC)"

jstat -gcutil 12345 1000 10
#   S0     S1     E      O      M     YGC   YGCT    FGC    FGCT     GCT
#   0.00  0.00  98.21  99.87  95.12   842  120.4    118   890.2   1010.6

Old generation at 99.87% and 118 full collections is not a CPU problem at all — the collector is running constantly and reclaiming nothing. Switch to the memory investigation in Every OutOfMemoryError and The seven classic memory leaks.

This branch is worth calling out in an interview, because mistaking GC thrashing for a hot loop sends you looking for a bug that does not exist.

Cause 4: unbounded retries#

while (true) {
    try {
        return client.call();
    } catch (Exception e) {
        // No backoff, no attempt limit. When the dependency is down,
        // every request thread spins as fast as the network allows.
    }
}

CPU climbs the moment a downstream dependency fails, and the retry traffic often prevents that dependency from recovering. Covered in Cascading failure.

The related shape is a livelock: threads running, retrying, backing off in lockstep and never progressing. Identical to a hot loop in every metric, and distinguished only by reading the stacks.

Faster, with a profiler#

./asprof -d 30 -e cpu -f /tmp/cpu.html 12345

If async-profiler is available, this is quicker and more precise than thread dumps: a flame graph shows the hot path immediately, including inlined frames that dumps miss. And if JFR is already running continuously — which it should be — the recording already contains the incident:

jcmd 12345 JFR.dump name=continuous filename=/tmp/incident.jfr

Thread dumps remain the universal fallback: they need no agent, no flags and no installation, and they work on any JVM on any machine. Know both.

Mitigating while you investigate#

Capture evidence before restarting, because the restart destroys it:

jcmd $PID Thread.print > /tmp/threads.txt
jcmd $PID GC.heap_info  > /tmp/heap.txt
jstat -gcutil $PID 1000 10 > /tmp/gc.txt
top -H -p $PID -b -n 1 > /tmp/top.txt

Then take the instance out of the load balancer rather than killing it, if you can afford to leave it running — a stuck instance you can still inspect is far more valuable than a healthy one you cannot reproduce the failure on.

Telling this as a story#

This topic is the strongest single answer to "tell me about a production issue you debugged", because it has a clean narrative arc: a specific symptom, a mechanical diagnostic sequence, a root cause with a real explanation, and a fix plus a guardrail. Structure it that way — see The incident playbook.

The guardrail matters as much as the fix. For the regex case: a length check before matching, a static-analysis rule banning nested quantifiers, and an alert on p99 latency for that endpoint. A candidate who describes the guardrail is describing someone who prevents the next incident, not just someone who survived this one.