On 03/19/2014 06:50 PM, Allen Zhang wrote:
1, I noticed that the main thread was 4097 and 4098 died. what is the child process created straight after main? My guess is this 4098 child process manages TCP connections. Is this correct?
2, Why does debugging level has an impact on this? Is it because higher debugging level introduces some delay?
That's hard to say. However, changing any aspect of the execution behaviour changes the state of the program, and can certainly have an impact on when it crashes, and whether it crashes at all.
The nature of memory bugs is that memory boundaries are often overstepped, but this does not necessarily result in a crash. The crash arises from the consequences of accessing that out-of-bounds memory, such as when the program ingests garbage from that memory area because it has been written to by something else. And, all of this behaviour varies with the order of operations, the particular libc you are using, its version, and the memory footprint of various other executed components.
The way to troubleshoot an issue like this is to analyse the core dump that is generated by the process that died due to the segmentation fault (signal 11). You should be able to find that core dump somewhere on your system. When you do, you can read it with 'gdb':
gdb /path/to/kamailio/binary /path/to/core.4098
Note that by default, many values will be optimised out. To get a fuller picture, you may need to compile Kamailio without -Ox compiler optimisations, and with additional debug information, e.g. -g.
-- Alex