Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

The Incident Playbook: Answering "Tell Me About an Outage"

The order of operations during an incident, the USE and RED methods for narrowing a cause fast, writing a blameless postmortem, and how to turn a real outage into an interview answer that scores.

Intermediate8 min readUpdated
On this page

Every senior interview asks some version of "tell me about a production issue you debugged". It is scored on process, not heroics — and the same process is what actually works at 3am.

Key Takeaways

  • The order is detect → mitigate → diagnose → fix → prevent. Mitigation comes before understanding.
  • RED (rate, errors, duration) narrows down which service. USE (utilisation, saturation, errors) narrows down which resource.
  • Capture evidence before restarting. A restart destroys the only copy of the state.
  • A postmortem is blameless because systems that punish reporting stop hearing about problems.
  • Structure the interview answer as symptom, investigation, cause, fix, prevention — with numbers.

The order of operations

Mitigation is step three, before diagnosis. Understanding why can wait; the users cannot.

Mitigate before you understand. The instinct to find the root cause first is the most expensive one in incident response. Available mitigations, roughly in order of how quickly they work:

MitigationWhen
Roll back the last deployAnything that started shortly after a release
Turn off a feature flagA specific feature is implicated
Scale outLoad-related, and the bottleneck is not shared
Restart the affected instanceA leak, a stuck thread, a corrupted pool
Shed loadEverything is overloaded and you must protect the core
Fail overOne zone, one replica, one dependency is bad
the thirty-second evidence kit
PID=$(pgrep -f myapp.jar)
jcmd $PID Thread.print          > /tmp/inc/threads.txt
jcmd $PID GC.heap_info          > /tmp/inc/heap.txt
jcmd $PID VM.native_memory summary > /tmp/inc/native.txt
jstat -gcutil $PID 1000 10      > /tmp/inc/gc.txt
top -H -p $PID -b -n 2          > /tmp/inc/top.txt
jcmd $PID JFR.dump name=continuous filename=/tmp/inc/rec.jfr   # if JFR is running
# Only if you can afford the pause, and the node is drained:
jcmd $PID GC.heap_dump /tmp/inc/heap.hprof

Better still: take the instance out of the load balancer and leave it running. A broken instance you can still inspect is worth more than a healthy one you cannot reproduce the failure on.

RED, then USE

Under pressure, checklists beat intuition.

RED — per service. Rate (requests per second), Errors (failures per second), Duration (latency distribution). Scan these across your services and the unhealthy one usually stands out immediately. Follow the dependency graph downward until you find the service where the trouble starts rather than where it shows.

USE — per resource. For every resource, check Utilisation (percentage busy), Saturation (queue depth) and Errors.

ResourceUtilisationSaturationErrors
CPU% busyRun queue length, throttled seconds—
MemoryHeap usedGC frequency, allocation stallOutOfMemoryError
Thread poolActive / maxQueue depthRejections
Connection poolActive / maxPendingAcquire timeouts
DiskIO utilIO queue depthIO errors
NetworkBandwidth usedRetransmitsDrops, resets

The saturation column is the one people skip, and it is where the answer usually is. CPU at 40% with a run queue of 30 is a saturated machine; a connection pool at 100% utilisation with zero pending is fine. Utilisation without saturation is not a diagnosis.

Narrowing down fast

Four questions that eliminate most of the search space in the first two minutes:

What changed? Deploys, config changes, feature flags, schema migrations, certificate rotations, dependency releases. The overwhelming majority of incidents follow a change, and correlating the start time against the deploy timeline is the single highest-yield step.

Is it all instances or one? All instances points at a shared dependency, a deploy or traffic. One instance points at that host, its cache state, or a leak that happened to hit there first.

Is it all endpoints or one? One endpoint narrows it to a query, a dependency or a code path. All endpoints points at a shared resource — pool, GC, CPU, database.

Did it start suddenly or gradually? Sudden means a change or a dependency failing. Gradual means a leak, unbounded growth, or load creeping past a threshold.

a decision tree in four lines
All instances + all endpoints + sudden     -> deploy, config, or shared dependency
All instances + one endpoint  + sudden     -> that endpoint's query or dependency
One instance  + all endpoints + gradual    -> leak or pool exhaustion on that host
All instances + all endpoints + gradual    -> leak everywhere, or load exceeded capacity

The postmortem

Blameless is not a courtesy; it is a design choice. A process that assigns fault teaches people to hide problems, and you stop hearing about the near-misses that would have prevented the next outage. Write about what the system allowed, not what a person did.

the sections that matter
## Summary
One paragraph: what broke, who was affected, for how long.
 
## Impact
Numbers. 14,200 failed checkouts over 47 minutes. Estimated revenue impact.
 
## Timeline (UTC)
09:12  Deploy of v2.14.0 begins
09:31  First alert: p99 latency above 2s
09:34  On-call acknowledges
09:41  Thread dumps captured, root cause suspected
09:44  Rolled back to v2.13.4
09:52  Latency recovers
09:59  Incident closed
 
## Root cause
The technical mechanism, in enough detail that a new engineer could
reproduce it.
 
## What went well
Detection took 19 minutes. Rollback took 3.
 
## What went badly
No alert on connection-pool pending count; we found it by reading a dump.
 
## Action items
| Action | Owner | Due |
| Alert on hikaricp.connections.pending > 0 for 60s | ... | ... |
| ArchUnit rule: no HTTP client inside @Transactional | ... | ... |

Two things make a postmortem useful rather than ceremonial. Time to detect and time to mitigate, tracked as numbers over time — they measure whether your operational practice is improving. And action items with owners and dates, because a postmortem with no assigned work is a diary entry.

Turning it into an interview answer

Two to three minutes, five parts, a number in each:

Symptom. "Checkout p99 went from 200ms to 30 seconds over about ten minutes. CPU was at 12%, so it was not a compute problem."

Investigation. "I took three thread dumps five seconds apart. All sixteen worker threads had the same stack — parked in HikariCP's ConcurrentBag.borrow, waiting for a connection. So I looked at the pool, and it was fully checked out with 112 requests pending."

Root cause. "A change the previous week moved a payment-gateway call inside a @Transactional method. The gateway's p99 had degraded to eight seconds, and each request was holding a database connection for that whole time."

Fix. "I mitigated by adding a two-second timeout to the gateway client and deploying — the pool recovered in about a minute. The real fix was splitting the transaction so the external call happens between two short transactions."

Prevention. "I added an alert on pool pending count, and an ArchUnit test that fails the build if any @Transactional method reaches the client package."

What that answer demonstrates, in order: you read evidence rather than guessing; you know the tools; you distinguish mitigation from fix; you found a cause with a mechanism; and you left the system harder to break.

Common failures in this answer. Blaming a person or a team. Describing a problem you did not personally work on. Skipping mitigation and going straight to the elegant fix. Having no numbers. Having no prevention step — which reads as "I put out fires" rather than "I stop them".

If you have no war story yet

Say so honestly, then substitute something real: a bug you diagnosed in staging, a performance problem you found in a load test, an issue you followed in someone else's postmortem. Then describe how you would approach it — the process above is the answer, and demonstrating the process credibly beats inventing an incident, which interviewers detect easily by asking one specific follow-up.

The other route is to build the story. The LoadLab project reproduces each failure in this phase on purpose, so you can debug a real memory leak, a real pool exhaustion and a real CPU spike with the real tools — and then talk about them from experience.

What gets asked

Directly: "tell me about a production issue you debugged." Indirectly: "how would you investigate a slow service?" or "walk me through what you do when you get paged." All three are the same question, and the answer is the sequence at the top of this page.

Frequently Asked Questions

What is the difference between mitigation and a fix?
Mitigation stops the bleeding — a rollback, a feature flag, a restart, scaling up, shedding load. A fix removes the cause. During an incident, mitigate first and always; understanding can wait, users cannot. Teams that insist on finding the root cause before acting turn ten-minute incidents into two-hour ones, and interviewers listen specifically for whether a candidate knows the difference.
What are the USE and RED methods?
Two complementary checklists. USE is for resources — for every resource, check Utilisation, Saturation and Errors. RED is for services — for every service, check Rate, Errors and Duration. Under pressure they stop you skipping the obvious: run RED across your services to find which one is unhealthy, then USE on that service resources to find which one is exhausted.
How much detail should an interview answer contain?
Two to three minutes, structured as symptom, investigation, root cause, fix, prevention, with a concrete number in each part. Name the tools you actually ran. Then stop and let them ask. The most common failure is not too little detail but too much narrative before reaching the point — the interviewer is scoring your diagnostic process, not the drama.

Related tutorials