Heap Dump Analysis with MAT

A heap dump is a complete snapshot of every object in the JVM. Eclipse MAT turns it from an unreadable binary into an answer, provided you know which three views to use.

Capturing one#

jcmd <pid> GC.heap_dump /var/dumps/heap.hprof        # preferred
jmap -dump:live,format=b,file=/var/dumps/heap.hprof <pid>   # older, same effect

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/dumps/

The live option runs a full GC first, so the dump contains only reachable objects. That makes the file smaller and the analysis cleaner, and it is what you want for leak hunting. Omit it when you need to see objects that are unreachable but not yet collected — rare.

Three practical cautions:

The pause is real. Roughly one to ten seconds per gigabyte. On a live node behind a load balancer, drain it first. On Kubernetes, remove the pod from the service by changing its labels rather than deleting it.

Disk space. The file is about the size of the live heap. A 16GB heap needs 16GB free, and needs to be written somewhere that survives the container.

It is sensitive data. A dump contains every String in memory: passwords in flight, tokens, customer records. Treat the file with the same controls as a database export, and delete it when the investigation ends.

Shallow versus retained#

This distinction is the whole point of the tool.

Shallow size is the object's own footprint: header plus fields, where a reference field counts as 4 or 8 bytes regardless of what it points to.

Retained size is the object plus everything reachable only through it. If an object is also reachable by another path, it is not counted — which is exactly right, because collecting this object would not free it.

Sorting by retained size immediately identifies which few objects hold the heap. Sorting by shallow size tells you almost nothing.

The three views#

1. Leak Suspects report. MAT runs this automatically on open. It looks for objects with disproportionate retained size and produces a summary like "one instance of OrderCache occupies 1,932,842,104 bytes (78.4%)". It is right often enough to try first, and it names the accumulation point — the object under which the memory sits.

2. Dominator tree. The one to spend time in. Object A dominates B if every path from a GC root to B passes through A. Sorted by retained size, the top of this tree is where your heap has gone. Expand downwards until the retained size splits across many similar children — that fan-out point is the collection doing the retaining.

3. Histogram. Instance count and shallow size per class. Useful for the different question of "why are there 4 million String objects?" Group by class loader to spot a classloader leak immediately: the same class name appearing under several loaders is the signature.

Path to GC roots#

Once you know what is retained, this tells you who is holding it.

com.acme.OrderCache @ 0x7f2a1c0
 └─ CACHE (java.util.HashMap) @ 0x7f2a200
     └─ table (java.util.HashMap$Node[]) @ 0x7f2a240
         └─ [1847] (java.util.HashMap$Node) @ 0x7f2b100
             └─ value (com.acme.Order) @ 0x7f2b180
 
GC root: system class → static field OrderCache.CACHE

The last line is the diagnosis: a static field. Always select "exclude weak/soft references" when running this query — otherwise MAT reports paths through caches and WeakHashMap entries that would not actually prevent collection, and you chase the wrong reference.

Two dumps, compared#

The single most effective technique for a slow leak:

jcmd <pid> GC.heap_dump /var/dumps/t1.hprof
sleep 3600
jcmd <pid> GC.heap_dump /var/dumps/t2.hprof

In MAT, open both and use Histogram → Compare to another Heap Dump. The delta column shows which classes gained instances. A leak shows as one or two classes growing steadily while everything else stays flat, and it removes all the guesswork about which large object is legitimate.

OQL#

MAT has a SQL-like query language for questions the standard views cannot answer.

SELECT * FROM java.util.HashMap m WHERE m.size > 100000

SELECT * FROM byte[] b WHERE b.@length > 10000000

SELECT * FROM INSTANCEOF java.sql.Connection c WHERE c.closed = false

SELECT s.toString(), COUNT(*) FROM java.lang.String s GROUP BY s.toString()

That last one regularly finds tens of megabytes of identical strings — repeated JSON field names, enum labels parsed from input, log message templates. String.intern() or a small canonicalising map at the parse boundary fixes it.

An alternative: JFR without the pause#

jcmd <pid> JFR.start name=leak settings=profile duration=10m filename=leak.jfr

The OldObjectSample event in a JDK Flight Recorder profile samples objects that survive collection and records their allocation stack trace. That is something a heap dump cannot give you: not just what is retained, but the line of code that created it.

It is far cheaper than a heap dump — a few percent overhead, no multi-second pause — so it is the right first step on a system you cannot take out of rotation. Open the recording in JDK Mission Control and look at the Live Objects view. See Profiling with JFR.

A worked example#

A service restarts with OutOfMemoryError every 36 hours. The sequence:

1. jstat -gcutil confirms occupancy after Full GC climbing from 40% to 95% over a day. It is a leak, not an undersized heap.
2. Take a dump on a drained canary. Leak Suspects reports "1.9GB in one instance of ConcurrentHashMap".
3. Dominator tree: the map holds 900,000 SessionContext objects at roughly 2KB each.
4. Path to GC roots: reached from a static final Map in SessionRegistry.
5. Read the code: sessions are added on login and removed on explicit logout — but not on timeout, and most users never log out.
6. Fix: replace the map with a Caffeine cache with expireAfterAccess(30, MINUTES), and add a gauge metric for its size so the next version of this is visible on a dashboard rather than in a crash.

That five-step shape — confirm, capture, dominate, root, read — is the answer to give when asked how you would approach a leak.

What gets asked#

"Walk me through diagnosing a memory leak" is the question, and the expected structure is exactly the worked example above. Two details that mark experience: knowing that dumping pauses the JVM and that you should drain the node first, and knowing the difference between shallow and retained size well enough to explain why a 48-byte HashMap is the top entry in the report.