[SR-Users] Kamailio 3.1.3 uncontrolled core dump

Fri May 13 10:19:26 CEST 2011

Hi everybody.

As posted in
http://lists.sip-router.org/pipermail/sr-users/2011-May/068514.html we
are having an uncontrolled core dump. It seems that the crash occurs
when some users registers to Kamailio. After upgrading to Kamilio
release 3.1.3 we are able to generate the following GDB:

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5_6.2)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/kamailio-3.1.3/sbin/kamailio...done.
Reading symbols from /lib/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/db_mysql.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules/db_mysql.so
Reading symbols from /usr/lib/mysql/libmysqlclient.so.15...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/mysql/libmysqlclient.so.15
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libm.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libssl.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib/libssl.so.6
Reading symbols from /lib/libcrypto.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib/libcrypto.so.6
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/libsrdb2.so.1...done.
Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/libsrdb2.so.1
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/libsrdb1.so.1...done.
Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/libsrdb1.so.1
Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libselinux.so.1
Reading symbols from /lib/libsepol.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib/libsepol.so.1
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/mi_fifo.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/mi_fifo.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/libkmi.so.1...done.
Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/libkmi.so.1
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/kex.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/kex.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/libkcore.so.1...done.
Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/libkcore.so.1
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/tm.so...done.
Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/tm.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/tmx.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/tmx.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/sl.so...done.
Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/sl.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/rr.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/rr.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/pv.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/pv.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/maxfwd.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/maxfwd.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/usrloc.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/usrloc.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/registrar.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/registrar.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/textops.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/textops.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/siputils.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/siputils.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/xlog.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/xlog.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/sanity.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules/sanity.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/ctl.so...done.
Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/ctl.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/mi_rpc.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules/mi_rpc.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/acc.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/acc.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/auth.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules/auth.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/auth_db.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/auth_db.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/permissions.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/permissions.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/alias_db.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/alias_db.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/domain.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/domain.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/nathelper.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/nathelper.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/rtpproxy.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/rtpproxy.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/tls.so...done.
Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/tls.so
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/sqlops.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/sqlops.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/group.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules_k/group.so
Reading symbols from
/usr/local/kamailio-3.1.3/lib/kamailio/modules/dialplan.so...done.
Loaded symbols for
/usr/local/kamailio-3.1.3/lib/kamailio/modules/dialplan.so
Reading symbols from /lib/libpcre.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Core was generated by `/usr/local/kamailio-3.1.3/sbin/kamailio -P
/var/run/kamailio.pid -m 64 -u kamai'.
Program terminated with signal 11, Segmentation fault.
#0  pv_set_ruri_user (msg=0x83622c8, param=0x8320078, op=254,
val=0xbfbaa83c) at pv_core.c:1756
1756		val->rs.s[val->rs.len] = '\0';
(gdb) (gdb) (gdb) exit
(gdb) quit

Last debug messages before the crash are:

May 13 09:32:06 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]:
ERROR: <core> [resolve.c:1540]: ERROR: sip_hostport2su: could not
resolve hostname: "@192.168.64.36" 
May 13 09:32:06 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]:
ERROR: tm [ut.h:318]: failed to resolve "@192.168.64.36" 
May 13 09:32:06 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]:
ERROR: tm [t_fwd.c:1528]: ERROR: t_forward_nonack: failure to add
branches 
May 13 09:32:06 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]:
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: Unresolvable
destination (478/SL) 
May 13 09:32:14 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3043]:
ERROR: <core> [resolve.c:1540]: ERROR: sip_hostport2su: could not
resolve hostname: "@192.168.64.36" 
May 13 09:32:14 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3043]:
ERROR: tm [ut.h:318]: failed to resolve "@192.168.64.36" 
May 13 09:32:14 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3043]:
ERROR: tm [t_fwd.c:1528]: ERROR: t_forward_nonack: failure to add
branches 
May 13 09:32:14 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3043]:
ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: Unresolvable
destination (478/SL) 
May 13 09:32:16 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: :
<core> [pass_fd.c:293]: ERROR: receive_fd: EOF on 21 
May 13 09:32:16 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]:
ERROR: <core> [tcp_main.c:4315]: connect 192.168.67.21:49320 failed
(timeout)  
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: :
<core> [pass_fd.c:293]: ERROR: receive_fd: EOF on 21 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]:
ERROR: <core> [io_wait.h:628]: ERROR: io_watch_del: trying to delete
already erased entry 21 in the hash(-1, 0, 0xb40a02e4) flags 0) 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]:
ERROR: <core> [io_wait.h:628]: ERROR: io_watch_del: trying to delete
already erased entry 23 in the hash(-1, 0, 0x83657ec) flags 0) 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: :
<core> [pass_fd.c:209]: ERROR: send_fd: sendmsg failed sending 36 on 23:
Broken pipe (32) 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]:
ERROR: <core> [tcp_main.c:3917]: ERROR: send2child: send_fd failed for
0xb421c7e4 (flags 0x4018), fd 36 
May 13 09:32:16 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3027]:
ALERT: <core> [main.c:741]: child process 3043 exited by a signal 11 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3027]:
ALERT: <core> [main.c:744]: core was generated 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3027]:
INFO: <core> [main.c:756]: INFO: terminating due to SIGCHLD 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3046]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3045]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3039]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3035]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3034]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3042]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3041]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3040]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3038]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3037]:
INFO: <core> [main.c:807]: INFO: signal 15 received 
May 13 09:32:24 tip1 hb_standby[3440]: Going standby [foreign].
May 13 09:32:25 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3027]:
ERROR: ctl [ctl.c:379]: ERROR: ctl: could not delete unix socket
/tmp/ser_ctl: Operation not permitted (1) 
May 13 09:32:25 tip1 heartbeat: [2048]: info: tip1.umh.es wants to go
standby [foreign]
May 13 09:32:36 tip1 heartbeat: [2048]: WARN: No reply to standby
request.  Standby request cancelled.

Any help would be appreciated.

Cheers,

Ricardo Dominguez