Skip to content

Last updated：2026-10-12

Hands-On Memory Leak Troubleshooting in C++

While investigating a memory problem in a database kernel (Greenplum/PostgreSQL family) recently, I got a string of very classic WARNINGs: problem in alloc set ExecutorState: bad single-chunk. What this log means: the memory pool (alloc set) found, during validation, that a chunk's header had been corrupted — nine times out of ten, some code wrote out of bounds and trampled the neighboring chunk's metadata.

The hard part of such problems is: where the error surfaces is not where the bug lives. The WARNING is only printed when the memory is freed and validated, while the out-of-bounds write could have happened in any piece of code long before. Below is the complete investigation process from that session — let us see how to recover the "crime scene" with gdb.

Read the scene information first:

regression=# SELECT t1.a, t1.c, t2.b, t2.c FROM prt1_e t1, prt2_e t2 WHERE (t1.a + t1.b)/2 = (t2.b + t2.a)/2 AND t1.c = 0 ORDER BY t1.a, t2.b;
WARNING:  problem in alloc set ExecutorState: bad single-chunk 0x7cfdd48 in block 0x7cfc580  (seg1 slice1 127.0.1.1:7003 pid=4295)
WARNING:  problem in alloc set ExecutorState: found inconsistent memory block 0x7cfc580  (seg1 slice1 127.0.1.1:7003 pid=4295)
WARNING:  problem in alloc set ExecutorState: bad single-chunk 0x7cfdd48 in block 0x7cfc580  (seg1 slice1 127.0.1.1:7003 pid=4295)
WARNING:  problem in alloc set ExecutorState: found inconsistent memory block 0x7cfc580  (seg1 slice1 127.0.1.1:7003 pid=4295)
WARNING:  problem in alloc set ExecutorState: bad single-chunk 0x87f85c8 in block 0x87f6e00  (seg0 slice1 127.0.1.1:7002 pid=4294)
WARNING:  problem in alloc set ExecutorState: found inconsistent memory block 0x87f6e00  (seg0 slice1 127.0.1.1:7002 pid=4294)
WARNING:  problem in alloc set ExecutorState: bad single-chunk 0x7d3c5c8 in block 0x7d3ae00  (seg2 slice1 127.0.1.1:7004 pid=4296)
WARNING:  problem in alloc set ExecutorState: found inconsistent memory block 0x7d3ae00  (seg2 slice1 127.0.1.1:7004 pid=4296)
WARNING:  problem in alloc set ExecutorState: bad single-chunk 0x87f85c8 in block 0x87f6e00  (seg0 slice1 127.0.1.1:7002 pid=4294)
WARNING:  problem in alloc set ExecutorState: bad single-chunk 0x7d3c5c8 in block 0x7d3ae00  (seg2 slice1 127.0.1.1:7004 pid=4296)
WARNING:  problem in alloc set ExecutorState: found inconsistent memory block 0x87f6e00  (seg0 slice1 127.0.1.1:7002 pid=4294)
WARNING:  problem in alloc set ExecutorState: found inconsistent memory block 0x7d3ae00  (seg2 slice1 127.0.1.1:7004 pid=4296)

From the WARNINGs we can extract several key pieces of information:

(1) the bad chunk's address is known: 0x7cfdd48 (bad single-chunk points at it);

(2) the block it lives in is 0x7cfc580, and the same block's "found inconsistent memory block" means the whole block's chain validation also failed;

(3) multiple segments (seg0/seg1/seg2) report it at the same time, so this is not environment flakiness but a deterministic logic bug.

With an exact address in hand, the plan is very direct: set a conditional breakpoint on the memory allocator (aset.c's AllocSetAlloc) to intercept the one call that "returns this address on allocation", then put a hardware watchpoint on that address — whoever writes it is the culprit.

The gdb session:

Debugging:

(gdb) b aset.c:1109 if chunk=0x7cfdd48
Breakpoint 1 at 0x105d420: file aset.c, line 1109.
(gdb) c
Continuing.

Thread 1 "postgres" hit Breakpoint 1, AllocSetAlloc (context=0x787ac50, size=4096) at aset.c:1109
1109        block->freeptr += (chunk_size + ALLOC_CHUNKHDRSZ);
(gdb) p chunk
$1 = (AllocChunk) 0x7cfdd48
(gdb) p chunk->size
$2 = 9187201950435737471
(gdb) p &chunk->size
$3 = (Size *) 0x7cfdd48
(gdb) watch (*((Size *) 0x7cfdd48))
Hardware watchpoint 2: (*((Size *) 0x7cfdd48))
(gdb) i b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x000000000105d420 in AllocSetAlloc at aset.c:1109
    stop only if chunk=0x7cfdd48
    breakpoint already hit 1 time
2       hw watchpoint  keep y                      (*((Size *) 0x7cfdd48))
(gdb) d 1
(gdb) c

Retrospective

The playbook of this case transfers directly to memory-corruption investigations in any C/C++ project:

(1) first extract the exact address from the error log — it is the only reliable anchor;

(2) the conditional breakpoint b aset.c:1109 if chunk=0x7cfdd48 catches the allocation site and confirms who got this address;

(3) watch (*(Size*)0x7cfdd48), a hardware watchpoint, waits for the culprit to show up — hardware breakpoints do not rewrite memory and do not affect program behavior, which makes them the most reliable tool for memory-corruption problems;

(4) once running, the call stack at the moment the watchpoint hits is the complete scene of the out-of-bounds write.

Note that in step one, p chunk->size printed 9187201950435737471 (0x7F7F7F7F7F7F7F7F) — a pile of 0x7F, the classic signature of scribbled memory: the trampling code was writing a bunch of flags/fill values. Globally searching the codebase for this value often finds the culprit directly.

One question to leave you: if the out-of-bounds write happens in a forked child process while you are debugging the parent, how should the flow above be adjusted? Feel free to discuss in the comments.

Comments