On Mittwoch, 20. Mai 2009, Christian Koch wrote:
[..] After solving all the performance issues with syslog we now made a stress test with memdebug enabled. Now kamailio terminates itself (perhaps because memory is corrupted?).
Hi Christian,
yes, this looks like the memory is corrupt, as it aparently crashes in a core function (added content from the log file), which normally should be pretty stable:
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 15), called from proxy.c: hostent_cpy(148) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 16), returns address 0x8265b20 frag. 0x8265b08 (size=32) on 1 -th hit May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 4), called from proxy.c: hostent_cpy(159) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 4), returns address 0x81bcbe8 frag. 0x81bcbd0 (size=4) on 1 -th hit May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params (0x81677e0, 8), called from proxy.c: hostent_cpy(182) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params(0x81677e0, 0x8265b20), called from proxy.c: hostent_cpy(185) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: freeing frag. 0x8265b08 alloc'ed from proxy.c: hostent_cpy(148) May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: params(0x81677e0, (nil)), called from proxy.c: hostent_cpy(187)
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27684]: CRITICAL:core:qm_free: bad pointer (nil) (out of memory block!) - aborting May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27697]: CRITICAL:core:receive_fd: EOF on 12 May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: child process 27684 exited by a signal 6 May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: core was not generated
Can you configure the kamailio server that it generates a core file? Then take a look to the backtrace where the invalid memory access was done, to verify if its really crashed in the core function, or perhaps some other parts has a problem here. Further informations: http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27679]: INFO:core:handle_sigs: terminating due to SIGCHLD May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: INFO:core:sig_usr: signal 15 received May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: Memory status (pkg): May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: qm_status (0x81677e0): May 20 15:31:55 AmbriaSip1 /usr/local/sbin/kamailio[27681]: heap size= 1048576
The complete output of the memory status is available here: https://rcpt.yousendit.com/690295962/7b39d332264f086b1bf0f134c026fad3
From the logs it seems that indeed a log of memory was allocated from the pv core. One of the main callers is pv_parse_ht_name, which is from the htable module. Not sure if this is a valid condition that it allocates that much pkg_mem, Daniel, can you perhaps take a look?
Cheers, Henning