Skip to content

Last updated：2026-10-12

GDB Multi-Threaded All-Stop Debugging

When we debug a multi-threaded program with gdb, threads can execute at any time. When we try to debug a specific thread (for example with next), it can cause a switch to another thread, which does not match our expectation of advancing to the next line in the current thread.

Thread 2 "arrow-compute-n" hit Breakpoint 1, arrow::compute::NestedLoopJoinNode::InputReceived (this=0x5bbf90, input=0x5ba840, batch=...)
    at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:399
399         size_t thread_index = thread_indexer_();
(gdb) n
[Switching to Thread 0x7ffff2c08700 (LWP 29121)]

Thread 3 "arrow-compute-n" hit Breakpoint 1, arrow::compute::NestedLoopJoinNode::InputReceived (this=0x5bbf90, input=0x5bb060, batch=...)
    at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:399
399         size_t thread_index = thread_indexer_();

So what should we do at this point?

That is where scheduler-locking, today's topic, comes in. GDB cannot single-step all threads in lockstep. Because thread scheduling depends on the operating system of your debug target (and is not controlled by GDB), other threads may execute several statements while the current thread completes a single step. Moreover, in general, when the program stops, other threads stop in the middle of a statement rather than at a clean statement boundary. You may even find, after continuing or single-stepping, that your program has stopped in another thread. This happens whenever another thread hits a breakpoint, signal, or exception before the first thread finishes the operation you requested.

The scheduler-locking feature provided by gdb can help you lock the scheduler to allow only a single thread to run, changing GDB's default behavior.

By default the scheduler-locking value equals replay

(gdb) show scheduler-locking
Mode for locking scheduler during execution is "replay".

In replay mode the behavior is similar to on; in recording mode, or during normal execution, it is similar to off. This is the default mode.

  • on

For example, executing t 2 switches to thread 2; setting on at this point keeps debugging confined to the current thread.

  • off

The opposite of on: no locking, and any thread may run at any time.

  • step

While single-stepping, the behavior is similar to on; otherwise it is similar to off. When you step, threads other than the current one never get a chance to run, whereas when you use commands like continue, until, or finish, they are completely free to run.

For example, commands like next and step will debug inside the process you want while the other threads hang; commands like continue let all the threads run.

Example

As you can see, when step is set, continue switches between threads.

(gdb) info threads
  Id   Target Id                                           Frame 
  1    Thread 0x7ffff7e88780 (LWP 31464) "arrow-compute-n" 0x00007ffff37e3de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2    Thread 0x7ffff3409700 (LWP 31494) "arrow-compute-n" arrow::compute::NestedLoopJoinNode::InputReceived (this=0x5bbf90, input=0x5bb060, batch=...)
    at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:399
* 3    Thread 0x7ffff2c08700 (LWP 31495) "arrow-compute-n" arrow::compute::NestedLoopJoinNode::InputReceived (this=0x5bbf90, input=0x5bb060, batch=...)
    at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:399
(gdb) c
Continuing.
[Switching to Thread 0x7ffff3409700 (LWP 31494)]

Thread 2 "arrow-compute-n" hit Breakpoint 1, arrow::compute::NestedLoopJoinNode::InputReceived (this=0x5bbf90, input=0x5bb060, batch=...)
    at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:399
399         size_t thread_index = thread_indexer_();
(gdb) info threads
  Id   Target Id                                           Frame 
  1    Thread 0x7ffff7e88780 (LWP 31464) "arrow-compute-n" 0x00007ffff37e3de2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 2    Thread 0x7ffff3409700 (LWP 31494) "arrow-compute-n" arrow::compute::NestedLoopJoinNode::InputReceived (this=0x5bbf90, input=0x5bb060, batch=...)
    at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:399
  3    Thread 0x7ffff2c08700 (LWP 31495) "arrow-compute-n" 0x00007ffff605a372 in arrow::compute::NestedLoopJoinNode::InputReceived (this=0x5bbf90, input=0x5bb060, batch=...)
    at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:399

Continuing again reaches the next breakpoint:

(gdb) info threads
  Id   Target Id                                           Frame 
  1    Thread 0x7ffff7e88780 (LWP 31464) "arrow-compute-n" 0x00007ffff37e3a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2    Thread 0x7ffff3409700 (LWP 31494) "arrow-compute-n" 0x00007ffff5781b50 in arrow::compute::NestedLoopJoinNode::OnBuildSideBatch(unsigned long, arrow::compute::ExecBatch)@plt ()
   from /code/arrow/cpp/build/debug/libarrow.so.200
* 3    Thread 0x7ffff2c08700 (LWP 31495) "arrow-compute-n" 0x00007ffff5781b50 in arrow::compute::NestedLoopJoinNode::OnBuildSideBatch(unsigned long, arrow::compute::ExecBatch)@plt ()
   from /code/arrow/cpp/build/debug/libarrow.so.200
(gdb) c
Continuing.
[Switching to Thread 0x7ffff3409700 (LWP 31494)]

Thread 2 "arrow-compute-n" hit Breakpoint 3, 0x00007ffff5781b50 in arrow::compute::NestedLoopJoinNode::OnBuildSideBatch(unsigned long, arrow::compute::ExecBatch)@plt ()
   from /code/arrow/cpp/build/debug/libarrow.so.200
(gdb) info threads
  Id   Target Id                                           Frame 
  1    Thread 0x7ffff7e88780 (LWP 31464) "arrow-compute-n" 0x00007ffff37e3a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 2    Thread 0x7ffff3409700 (LWP 31494) "arrow-compute-n" 0x00007ffff5781b50 in arrow::compute::NestedLoopJoinNode::OnBuildSideBatch(unsigned long, arrow::compute::ExecBatch)@plt ()
   from /code/arrow/cpp/build/debug/libarrow.so.200
  3    Thread 0x7ffff2c08700 (LWP 31495) "arrow-compute-n" 0x00007ffff5781b56 in arrow::compute::NestedLoopJoinNode::OnBuildSideBatch(unsigned long, arrow::compute::ExecBatch)@plt ()
   from /code/arrow/cpp/build/debug/libarrow.so.200

You can see the switching between threads. If you want to stay within one thread, you can simply next, for example:

(gdb) t 2
[Switching to thread 2 (Thread 0x7ffff3409700 (LWP 32396))]
#0  0x00007ffff605ae0d in arrow::compute::NestedLoopJoinNode::OnBuildSideBatch (this=0x5bbf90, thread_index=1, batch=...) at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:487
487         if (!batch.length) {
(gdb) n
490         std::lock_guard<std::mutex> guard(build_side_mutex_);
(gdb) n
491         build_accumulator_.InsertBatch(std::move(batch));

If you execute a print operation that triggers a function call, the program stops and cannot output the result we expect.

(gdb) p batch.ToString()
[Switching to Thread 0x7ffff2c08700 (LWP 32397)]

Thread 3 "arrow-compute-n" hit Breakpoint 1, arrow::compute::NestedLoopJoinNode::InputReceived (this=0x5bbf90, input=0x5ba840, batch=...)
    at /code/arrow/cpp/src/arrow/compute/exec/nested_loop_node.cc:399
399         size_t thread_index = thread_indexer_();
The program stopped in another thread while making a function call from GDB.
Evaluation of the expression containing the function
(arrow::compute::ExecBatch::ToString() const) will be abandoned.
When the function is done executing, GDB will silently stop.

We can now set on, and printing works!

(gdb) set scheduler-locking on
(gdb) p batch.ToString()
$2 = "ExecBatch\n    # Rows: 2\n    0: Array[null,4]\n    1: Array[true,false]\n"

https://sourceware.org/gdb/current/onlinedocs/gdb.html/All_002dStop-Mode.html

Comments