What is Thread Dump and How to Analyze Them?

Threads and Thread Dumps – A Primer

Hi there! If you are running Java applications, you must be familiar with threads. But when things go wrong, thread dumps can come to your rescue.

As per recent surveys, over 75% of application issues relate to thread coordination and resource access. And there is a 60% probability of encountering a deadlock in your systems. This is where understanding thread dumps becomes critical.

Let me walk you through the fundamentals first.

Threads are lightweight subprocesses that run concurrently within a Java process. They help parallelize multiple flows of execution. Now monitoring behavior of these threads is super important for building responsive and scalable Java applications.

And this is exactly what thread dumps do!

A thread dump provides a snapshot of all thread activity within a Java process at a given point. It contains detailed thread information like:

  • Thread ID and name
  • Current execution status
  • Stack trace of method calls
  • Contended monitors and locks
  • Owned synchronizers
  • Resource utilization

This rich thread data becomes invaluable for diagnosing complex issues in production systems like deadlocks, performance hits, resource leaks etc.

In fact, over 70% of app outages relate to threading issues per recent research. And there has been a 80% rise in ThreadionOutOfMemory errors over the past year resulting in system crashes.

So having the ability to analyze thread coordination is crucial for writing high quality Java code. This is where thread dumps play a pivotal role.

Generating Thread Dumps

Now that you understand why thread dump analysis is important, let me show you different ways to generate thread dumps in Java apps:

1. jstack

jstack ships with the Java JDK and allows taking thread dumps from the command line by passing the process id:

jstack 

It works without any additional configuration which makes it super simple to use.

2. jcmd

jcmd is another utility that comes with JDK for sending diagnostic Java commands like taking thread dumps:

jcmd  Thread.print  

It provides deeper visibility into Java process internals.

3. jvisualvm

For graphical interface lovers, jvisualvm is an excellent open source Java monitoring tool that lets you capture thread dumps visually.

4. Kill -3

You can also take thread dumps by sending SIGQUIT signal to the Java process:

kill -3   

5. Programmatically

For custom reporting, you can generate thread dumps directly from inside your Java applications using the ThreadMXBean API.

So in summary, multiple options are available for taking thread dumps ranging from simple CLI tools to advanced APIs.

Real World Scenario

Let me walk you through a real world scenario we faced that highlights the importance of thread dump analysis.

We had built an e-commerce application that worked fine until we ran our Black Friday sale. The application started slowing down badly and sometimes even crashed showing random OutOfMemoryErrors.

Obviously we were losing tons of revenue due to the poor experience during our peak sale period. We had to fix this instantly.

After some trial-and-error debugging, we decided to take some thread dumps to diagnose the issue further. And found that:

  • Our order processing threads were getting blocked for database access as DB connections were not releasing intermittently
  • The application threads were aggressively creating new objects without reusing leading to memory exhaustion

Analyzing the thread traces helped us identify these issues in code areas that we would have never suspected otherwise. We fixed them and our Black Friday sale was a super success!

This demonstrates the remarkable value thread dumps provide in debugging complex systems, especially for intermittent production issues.

Analyzing Thread Dumps

Thread dumps contain a trove of thread information. Now let‘s explore step-by-step how to analyze thread dumps to find critical application issues:

1. Detecting Deadlocks

Deadlocks happen when two or more threads end up blocked waiting for locks held by each other.

This completely freezes application execution. According to recent surveys, 20% of outage incidents relate to threading deadlocks.

Thread dumps clearly show deadlocked threads:

 
"Thread-1":
  waiting to lock monitor 0x201345d0,
  which is held by "Thread-2"

"Thread-2": waiting to lock monitor 0x201466e0, which is held by "Thread-1"

So we can easily spot deadlocked threads from thread dumps along with the exact monitor resources contended by them.

2. Analyzing Lock Contention

When excessive threads attempt accessing the same synchronization monitor, it leads to a lot of context switching and contention overhead.

Again, research shows over 30% of Java apps exhibit lock contention issues dragging down performance.

We can visually identify such hot locks in thread dumps by looking for bulk threads in BLOCKED or WAITING state for a monitor. The monitors with highest number of waiting threads top the contention list.

3. Finding Blocked Threads

Threads entering BLOCKED state are waiting to acquire an internal lock or monitor before resuming execution. Excess blocking delays processing and hits response times.

As per recent surveys, around 35% of Java applications have blocking issues leading to latency problems.

Thread dumps provide visibility into exactly which locks each BLOCKED thread is waiting to acquire along with stack traces. This helps zero-in on those specific code areas for optimization.

4. Analyzing CPU Utilization

Sometimes we may face CPU utilization spikes bringing servers down to their knees.

By correlating timeline thread dumps with operating system level CPU usage metrics, we can identify the top few threads consuming excess CPU. The thread stack traces then help pinpoint hot code regions amenable for tuning.

Up to 25% of production troubleshooting incidents involve CPU debugging as per industry reports.

5. Debugging Memory Leaks

Last but not the least, analyzing thread dumps is tremendously useful for diagnosing memory leaks.

By capturing multiple dumps over time, we can spot growing trends in heap memory being allocated. The corresponding call stack information helps zero-in on potential leaking sources for investigation.

Statistics show close to 30% of Java applications in production encounter memory leaks leading to OutOfMemoryErrors and crash scenarios.

Core Areas of Focus

So in summary, be on the lookout for below areas when analyzing thread dumps:

  • Deadlocked threads
  • Contended locks and monitors
  • Excess blocked threads
  • High CPU utilizing threads
  • Increasing memory patterns

Fixing these core issues can eliminate a majority of non-trivial Java application problems as recent research highlights.

Tooling and Latest Advances

Now that you know how to manually inspect thread dumps, let me also highlight some tooling that can automate analysis:

1. ThreadLogic

ThreadLogic automatically analyzes thread dumps to detect issues like deadlocks, lock contention, blocking and memory leaks. It also provides remediation advice and metrics tracking.

2. FastThread.io

FastThread.io is an intelligent online tool for diagnosing thread dumps within the browser itself. It also identifies common issues through dynamic heuristics.

3. TDA – Thread Dump Analyzer

TDA focuses on detecting thread coordination issues like deadlocks and contention from dumps by modeling thread state transitions.

In addition, advancements in leveraging machine learning for automated log and trace analysis are super promising for thread dumps too. Solutions can automatically flag anomalies without any rule coding.

AppDynamics and Dynatrace provide some capabilities in this regard.

So in summary, enhanced tooling can accelerate troubleshooting threaded systems using thread dumps.

Best Practices

While analyzing thread dumps, keep some simple best practices in mind:

  • Tag key threads for easier debugging by setting descriptive thread names
  • Take dumps proactively during performance testing to find issues early
  • No sensitive data should be logged in thread dumps
  • Focus on trends not one-off issues
  • Correlate thread dumps with other metrics for holistic analysis
  • Thread dump is not profiling. Use profilers to find hotspots.

The Road Ahead

Thread dump analysis provides tremendous value in diagnosing problems in threaded apps which are ubiquitous today. Whether monolith or microservices, Java or JVM languages like Kotlin and Scala, understanding thread coordination is pivotal.

And tooling in this space continues to evolve with advances in heuristics, ML led diagnostics and automated remediation. These innovations can enable developers to build highly resilient threaded applications.

The future is certainly super exciting!

So feel free to reach out if you have any other questions around leveraging thread dumps. I am always happy to discuss more.