Skip to content

Last updated：2026-10-12

C++ Deadlock Troubleshooting

How do you troubleshoot deadlocks in concurrent C++ programming?

Recently I was writing a parallelism-enabled operator in Apache Arrow: a nested loop join. Since parallelism is involved, you inevitably run into the well-known deadlock problem. Suppose you hit a deadlock — how do you debug and locate it?

That is the goal of this article: to help you quickly master deadlock detection and deadlock debugging in concurrent programming. It is pure hands-on content, so let's get into it.

1. Introduction

To explain deadlocks better, let's introduce one with a program.

std::mutex gMutex;

int t2() {
  std::lock_guard<std::mutex> m(gMutex);
  return 0;
}

int t1() {
  std::lock_guard<std::mutex> l(gMutex);
  return t2();
}

Looking at this program, everyone can tell something is wrong — it deadlocks! The problem is that both the t1() function and the t2() function acquire the global mutex gMutex, but after acquiring the lock, t1() calls t2(), and inside t2() the code tries to acquire gMutex once again.

t1 has already taken the lock without releasing it, while t2 tries to take it again; both sides wait for the other to release first and fall into an endless wait. In real projects the code is never as simple as this — it is far more complicated. For example, this is what the code I wrote in Apache Arrow looks like:

Status OnBuildSideFinished(size_t thread_index) {
  std::lock_guard<std::mutex> guard(probe_side_mutex_);
  // do something
  accumulate_build_ready_ = true;
  return scheduler_->StartTaskGroup(thread_index,task_group_probe_,queued_batches_to_probe_.batch_count()); 
}

As you can see, this code is much more complicated than the scenario above: it has at least 5 nested stack levels, and the rest of the code is not even shown. But these two cases are essentially the same deadlock model.

2. Debugging

Now that the deadlock model has been explained, how do you locate this kind of problem?

There are two approaches you can take. The first is to run the program directly and then attach gdb to it.

For example:

./a.out

Then, after finding the process ID:

gdb -p xxx

At this point we can get the threads that are currently waiting.

(gdb) info threads 
  Id   Target Id                                 Frame 
* 1    Thread 0x7ffff7fe2740 (LWP 32301) "a.out" 0x00007ffff7bc8017 in pthread_join () from /lib64/libpthread.so.0
  2    Thread 0x7ffff6fd0700 (LWP 32305) "a.out" 0x00007ffff7bcd54d in __lll_lock_wait () from /lib64/libpthread.so.0

Then go look at the stack of __lll_lock_wait; for example, here I looked at thread 2, then inspected the stack to get the line numbers of t1 and t2, which pinpoints exactly where the problem is — very intuitive!

Besides this approach, you can also start the program directly under gdb; it will hang, and after killing it with ctrl + c you get the same information as above.

For example:

(gdb) r
Starting program: /home/light/a.out 
Missing separate debuginfos, use: debuginfo-install glibc-2.17-326.el7_9.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Hello World!
[New Thread 0x7ffff6fd0700 (LWP 32305)]
^C
Thread 1 "a.out" received signal SIGINT, Interrupt.
0x00007ffff7bc8017 in pthread_join () from /lib64/libpthread.so.0
Missing separate debuginfos, use: debuginfo-install libgcc-4.8.5-44.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64
(gdb) info threads 
  Id   Target Id                                 Frame 
* 1    Thread 0x7ffff7fe2740 (LWP 32301) "a.out" 0x00007ffff7bc8017 in pthread_join () from /lib64/libpthread.so.0
  2    Thread 0x7ffff6fd0700 (LWP 32305) "a.out" 0x00007ffff7bcd54d in __lll_lock_wait () from /lib64/libpthread.so.0

Alright, that's all for this section. If you found it interesting, feel free to share this hardcore article! Meanwhile, let me also recommend my thread pool project — I have recorded video walkthroughs for this project, together with one-click docker deployment (I plan to record that episode for everyone later on); if you are interested, just get in touch with me.

Comments