Hi everybody.

As posted in http://lists.sip-router.org/pipermail/sr-users/2011-May/068514.html we are having an uncontrolled core dump. It seems that the crash occurs when some users registers to Kamailio. After upgrading to Kamilio release 3.1.3 we are able to generate the following GDB:

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5_6.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/local/kamailio-3.1.3/sbin/kamailio...done. Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/db_mysql.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/db_mysql.so Reading symbols from /usr/lib/mysql/libmysqlclient.so.15...(no debugging symbols found)...done. Loaded symbols for /usr/lib/mysql/libmysqlclient.so.15 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libssl.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libssl.so.6 Reading symbols from /lib/libcrypto.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.6 Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/libsrdb2.so.1...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/libsrdb2.so.1 Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/libsrdb1.so.1...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/libsrdb1.so.1 Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.2 Reading symbols from /usr/lib/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.3 Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcom_err.so.2 Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libk5crypto.so.3 Reading symbols from /usr/lib/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /lib/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libselinux.so.1 Reading symbols from /lib/libsepol.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libsepol.so.1 Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/mi_fifo.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/mi_fifo.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/libkmi.so.1...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/libkmi.so.1 Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/kex.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/kex.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/libkcore.so.1...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/libkcore.so.1 Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/tm.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/tm.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/tmx.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/tmx.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/sl.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/sl.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/rr.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/rr.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/pv.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/pv.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/maxfwd.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/maxfwd.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/usrloc.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/usrloc.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/registrar.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/registrar.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/textops.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/textops.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/siputils.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/siputils.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/xlog.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/xlog.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/sanity.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/sanity.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/ctl.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/ctl.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/mi_rpc.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/mi_rpc.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/acc.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/acc.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/auth.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/auth.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/auth_db.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/auth_db.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/permissions.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/permissions.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/alias_db.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/alias_db.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/domain.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/domain.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/nathelper.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/nathelper.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/rtpproxy.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/rtpproxy.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/tls.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/tls.so Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/sqlops.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/sqlops.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/group.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules_k/group.so Reading symbols from /usr/local/kamailio-3.1.3/lib/kamailio/modules/dialplan.so...done. Loaded symbols for /usr/local/kamailio-3.1.3/lib/kamailio/modules/dialplan.so Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.0 Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_dns.so.2 Core was generated by `/usr/local/kamailio-3.1.3/sbin/kamailio -P /var/run/kamailio.pid -m 64 -u kamai'. Program terminated with signal 11, Segmentation fault. #0 pv_set_ruri_user (msg=0x83622c8, param=0x8320078, op=254, val=0xbfbaa83c) at pv_core.c:1756 1756 val->rs.s[val->rs.len] = '\0'; (gdb) (gdb) (gdb) exit (gdb) quit

Last debug messages before the crash are:

May 13 09:32:06 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]: ERROR: <core> [resolve.c:1540]: ERROR: sip_hostport2su: could not resolve hostname: "@192.168.64.36" May 13 09:32:06 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]: ERROR: tm [ut.h:318]: failed to resolve "@192.168.64.36" May 13 09:32:06 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]: ERROR: tm [t_fwd.c:1528]: ERROR: t_forward_nonack: failure to add branches May 13 09:32:06 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]: ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: Unresolvable destination (478/SL) May 13 09:32:14 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3043]: ERROR: <core> [resolve.c:1540]: ERROR: sip_hostport2su: could not resolve hostname: "@192.168.64.36" May 13 09:32:14 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3043]: ERROR: tm [ut.h:318]: failed to resolve "@192.168.64.36" May 13 09:32:14 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3043]: ERROR: tm [t_fwd.c:1528]: ERROR: t_forward_nonack: failure to add branches May 13 09:32:14 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3043]: ERROR: sl [sl_funcs.c:282]: ERROR: sl_reply_error used: Unresolvable destination (478/SL) May 13 09:32:16 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: : <core> [pass_fd.c:293]: ERROR: receive_fd: EOF on 21 May 13 09:32:16 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: ERROR: <core> [tcp_main.c:4315]: connect 192.168.67.21:49320 failed (timeout) May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: : <core> [pass_fd.c:293]: ERROR: receive_fd: EOF on 21 May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: ERROR: <core> [io_wait.h:628]: ERROR: io_watch_del: trying to delete already erased entry 21 in the hash(-1, 0, 0xb40a02e4) flags 0) May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: ERROR: <core> [io_wait.h:628]: ERROR: io_watch_del: trying to delete already erased entry 23 in the hash(-1, 0, 0x83657ec) flags 0) May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: : <core> [pass_fd.c:209]: ERROR: send_fd: sendmsg failed sending 36 on 23: Broken pipe (32) May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: ERROR: <core> [tcp_main.c:3917]: ERROR: send2child: send_fd failed for 0xb421c7e4 (flags 0x4018), fd 36 May 13 09:32:16 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3027]: ALERT: <core> [main.c:741]: child process 3043 exited by a signal 11 May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3027]: ALERT: <core> [main.c:744]: core was generated May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3027]: INFO: <core> [main.c:756]: INFO: terminating due to SIGCHLD May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3047]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3046]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3045]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3044]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3039]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3035]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3034]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3042]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3041]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3040]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3038]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:17 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3037]: INFO: <core> [main.c:807]: INFO: signal 15 received May 13 09:32:24 tip1 hb_standby[3440]: Going standby [foreign]. May 13 09:32:25 tip1 /usr/local/kamailio-3.1.3/sbin/kamailio[3027]: ERROR: ctl [ctl.c:379]: ERROR: ctl: could not delete unix socket /tmp/ser_ctl: Operation not permitted (1) May 13 09:32:25 tip1 heartbeat: [2048]: info: tip1.umh.es wants to go standby [foreign] May 13 09:32:36 tip1 heartbeat: [2048]: WARN: No reply to standby request. Standby request cancelled.

Any help would be appreciated.

Cheers,

Ricardo Dominguez