[Serusers] Serious problems with SER : crash. Need Help

[Ricardo Martinez]

Hello list
	I'm having serious problems with my "production" SER machine.  Since
i installed "SER" program in my linux machine i have been experimenting
"freezing" and "crashing" events without any particular reason.  The
freezing events were solved eliminating the call to the exec_dset() function
from the ser.cfg file.  Now i'm using avp_radius to acomplish the same
function.  But now instead of the "freezing" events i'm having "crashing"
events every 4 or 5 days.  I'm really don't know what could possibly be
happening, so i discover that SER leaves a core.dumped files in my "root"
directory from my linux machine.  I'm not an expert in C or Linux, but i
discover a basic way to check this core files using "gdb" tool.
I'm not even now if i'm using the tool ok but this is the output : 

[root@]# gdb /usr/local/sbin/ser ../core.18662 
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
Core was generated by `/usr/local/sbin/ser -P /var/run/ser.pid'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/local//lib/ser/modules/mysql.so...done.
Loaded symbols for /usr/local//lib/ser/modules/mysql.so
Reading symbols from /usr/lib/mysql/libmysqlclient.so.10...done.
Loaded symbols for /usr/lib/mysql/libmysqlclient.so.10
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /usr/local//lib/ser/modules/sl.so...done.
Loaded symbols for /usr/local//lib/ser/modules/sl.so
Reading symbols from /usr/local//lib/ser/modules/tm.so...done.
Loaded symbols for /usr/local//lib/ser/modules/tm.so
Reading symbols from /usr/local//lib/ser/modules/rr.so...done.
Loaded symbols for /usr/local//lib/ser/modules/rr.so
Reading symbols from /usr/local//lib/ser/modules/maxfwd.so...done.
Loaded symbols for /usr/local//lib/ser/modules/maxfwd.so
Reading symbols from /usr/local//lib/ser/modules/usrloc.so...done.
Loaded symbols for /usr/local//lib/ser/modules/usrloc.so
Reading symbols from /usr/local//lib/ser/modules/registrar.so...done.
Loaded symbols for /usr/local//lib/ser/modules/registrar.so
Reading symbols from /usr/local//lib/ser/modules/textops.so...done.
Loaded symbols for /usr/local//lib/ser/modules/textops.so
Reading symbols from /usr/local//lib/ser/modules/exec.so...done.
Loaded symbols for /usr/local//lib/ser/modules/exec.so
Reading symbols from /usr/local//lib/ser/modules/mediaproxy.so...done.
Loaded symbols for /usr/local//lib/ser/modules/mediaproxy.so
Reading symbols from /usr/local//lib/ser/modules/domain.so...done.
Loaded symbols for /usr/local//lib/ser/modules/domain.so
Reading symbols from /usr/local//lib/ser/modules/dbtext.so...done.
Loaded symbols for /usr/local//lib/ser/modules/dbtext.so
Reading symbols from /usr/local//lib/ser/modules/nathelper.so...done.
Loaded symbols for /usr/local//lib/ser/modules/nathelper.so
Reading symbols from /usr/local/lib/ser/modules/auth.so...done.
Loaded symbols for /usr/local/lib/ser/modules/auth.so
Reading symbols from /usr/local/lib/ser/modules/auth_radius.so...done.
Loaded symbols for /usr/local/lib/ser/modules/auth_radius.so
Reading symbols from /usr/local/lib/libradiusclient-ng.so.2...done.
Loaded symbols for /usr/local/lib/libradiusclient-ng.so.2
Reading symbols from /usr/local/lib/ser/modules/group_radius.so...done.
Loaded symbols for /usr/local/lib/ser/modules/group_radius.so
Reading symbols from /usr/local/lib/ser/modules/acc.so...done.
Loaded symbols for /usr/local/lib/ser/modules/acc.so
Reading symbols from /usr/local/lib/ser/modules/avp_radius.so...done.
Loaded symbols for /usr/local/lib/ser/modules/avp_radius.so
Reading symbols from /usr/local/lib/ser/modules/avpops.so...done.
Loaded symbols for /usr/local/lib/ser/modules/avpops.so
Reading symbols from /usr/local/lib/ser/modules/xlog.so...done.
Loaded symbols for /usr/local/lib/ser/modules/xlog.so
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
#0  0x42070fa4 in _IO_str_overflow_internal () from /lib/tls/libc.so.6

(gdb) bt
#0  0x42070fa4 in _IO_str_overflow_internal () from /lib/tls/libc.so.6
#1  0x4206fb28 in _IO_default_xsputn_internal () from /lib/tls/libc.so.6
#2  0x4204528a in vfprintf () from /lib/tls/libc.so.6
#3  0x4206432c in vsprintf () from /lib/tls/libc.so.6
#4  0x4204f18d in sprintf () from /lib/tls/libc.so.6
#5  0x4016f889 in UseMediaProxy (msg=0x8106a28, str1=0x0, str2=0x0) at
mediaproxy.c:1454
#6  0x0804efe2 in do_action (a=0x8105238, msg=0x8106a28) at action.c:610
#7  0x0805079c in run_actions (a=0x8105238, msg=0x8105238) at action.c:718
#8  0x0804ef01 in do_action (a=0x8105268, msg=0x8106a28) at action.c:600
#9  0x0805079c in run_actions (a=0x8105238, msg=0x8105268) at action.c:718
#10 0x0804ef01 in do_action (a=0x8105298, msg=0x8106a28) at action.c:600
#11 0x0805079c in run_actions (a=0x8105238, msg=0x8105298) at action.c:718
#12 0x0804ed9f in do_action (a=0x8104188, msg=0x8106a28) at action.c:375
#13 0x0805079c in run_actions (a=0x8105238, msg=0x8106a28) at action.c:718
#14 0x0804ef01 in do_action (a=0x81041e8, msg=0x8106a28) at action.c:600
#15 0x0805079c in run_actions (a=0x8105238, msg=0x81041e8) at action.c:718
#16 0x0804ef01 in do_action (a=0x8104b48, msg=0x8106a28) at action.c:600
#17 0x0805079c in run_actions (a=0x8105238, msg=0x8104b48) at action.c:718
#18 0x08069449 in receive_msg (
    buf=0x80c48a0 "INVITE sip:0056085603736 at sipproxy.mydomain.com
SIP/2.0\r\nVia: SIP/2.0/UDP
192.168.1.2:5061;branch=z9hG4bK-a10b77eb;rport\r\nFrom: 5555847148
<sip:5555847148 at sipproxy.mydomain.com>;tag=802df038f5b36902o1\r"...,
len=1119, rcv_info=0xbfffe350)
    at receive.c:165
#19 0x08079e8f in udp_rcv_loop () at udp_server.c:458
#20 0x0805b480 in main_loop () at main.c:1032
#21 0x0805c7ac in main (argc=0, argv=0x4213ad38) at main.c:1568
#22 0x42015574 in __libc_start_main () from /lib/tls/libc.so.6

>From another crash (five days ago).


[root at sipvoiss root]# gdb /usr/local/sbin/ser ../core.3694 
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
Core was generated by `/usr/local/sbin/ser -P /var/run/ser.pid'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/local//lib/ser/modules/mysql.so...done.
Loaded symbols for /usr/local//lib/ser/modules/mysql.so
Reading symbols from /usr/lib/mysql/libmysqlclient.so.10...done.
Loaded symbols for /usr/lib/mysql/libmysqlclient.so.10
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /usr/local//lib/ser/modules/permissions.so...done.
Loaded symbols for /usr/local//lib/ser/modules/permissions.so
Reading symbols from /usr/local//lib/ser/modules/sl.so...done.
Loaded symbols for /usr/local//lib/ser/modules/sl.so
Reading symbols from /usr/local//lib/ser/modules/tm.so...done.
Loaded symbols for /usr/local//lib/ser/modules/tm.so
Reading symbols from /usr/local//lib/ser/modules/rr.so...done.
Loaded symbols for /usr/local//lib/ser/modules/rr.so
Reading symbols from /usr/local//lib/ser/modules/maxfwd.so...done.
Loaded symbols for /usr/local//lib/ser/modules/maxfwd.so
Reading symbols from /usr/local//lib/ser/modules/usrloc.so...done.
Loaded symbols for /usr/local//lib/ser/modules/usrloc.so
Reading symbols from /usr/local//lib/ser/modules/registrar.so...done.
Loaded symbols for /usr/local//lib/ser/modules/registrar.so
Reading symbols from /usr/local//lib/ser/modules/textops.so...done.
Loaded symbols for /usr/local//lib/ser/modules/textops.so
Reading symbols from /usr/local//lib/ser/modules/exec.so...done.
Loaded symbols for /usr/local//lib/ser/modules/exec.so
Reading symbols from /usr/local//lib/ser/modules/mediaproxy.so...done.
Loaded symbols for /usr/local//lib/ser/modules/mediaproxy.so
Reading symbols from /usr/local//lib/ser/modules/domain.so...done.
Loaded symbols for /usr/local//lib/ser/modules/domain.so
Reading symbols from /usr/local//lib/ser/modules/dbtext.so...done.
Loaded symbols for /usr/local//lib/ser/modules/dbtext.so
Reading symbols from /usr/local//lib/ser/modules/nathelper.so...done.
Loaded symbols for /usr/local//lib/ser/modules/nathelper.so
Reading symbols from /usr/local/lib/ser/modules/auth.so...done.
Loaded symbols for /usr/local/lib/ser/modules/auth.so
Reading symbols from /usr/local/lib/ser/modules/auth_radius.so...done.
Loaded symbols for /usr/local/lib/ser/modules/auth_radius.so
Reading symbols from /usr/local/lib/libradiusclient-ng.so.2...done.
Loaded symbols for /usr/local/lib/libradiusclient-ng.so.2
Reading symbols from /usr/local/lib/ser/modules/group_radius.so...done.
Loaded symbols for /usr/local/lib/ser/modules/group_radius.so
Reading symbols from /usr/local/lib/ser/modules/acc.so...done.
Loaded symbols for /usr/local/lib/ser/modules/acc.so
Reading symbols from /usr/local/lib/ser/modules/avp_radius.so...done.
Loaded symbols for /usr/local/lib/ser/modules/avp_radius.so
Reading symbols from /usr/local/lib/ser/modules/avpops.so...done.
Loaded symbols for /usr/local/lib/ser/modules/avpops.so
Reading symbols from /usr/local/lib/ser/modules/xlog.so...done.
Loaded symbols for /usr/local/lib/ser/modules/xlog.so
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
#0  0x42070fa4 in _IO_str_overflow_internal () from /lib/tls/libc.so.6
(gdb) 
(gdb) bt
#0  0x42070fa4 in _IO_str_overflow_internal () from /lib/tls/libc.so.6
#1  0x4206fb28 in _IO_default_xsputn_internal () from /lib/tls/libc.so.6
#2  0x4204528a in vfprintf () from /lib/tls/libc.so.6
#3  0x4206432c in vsprintf () from /lib/tls/libc.so.6
#4  0x4204f18d in sprintf () from /lib/tls/libc.so.6
#5  0x4017b889 in UseMediaProxy (msg=0x8107178, str1=0x0, str2=0x0) at
mediaproxy.c:1454
#6  0x0804efe2 in do_action (a=0x8105568, msg=0x8107178) at action.c:610
#7  0x0805079c in run_actions (a=0x8105568, msg=0x8105568) at action.c:718
#8  0x0804ef01 in do_action (a=0x8105598, msg=0x8107178) at action.c:600
#9  0x0805079c in run_actions (a=0x8105568, msg=0x8105598) at action.c:718
#10 0x0804ef01 in do_action (a=0x81055c8, msg=0x8107178) at action.c:600
#11 0x0805079c in run_actions (a=0x8105568, msg=0x81055c8) at action.c:718
#12 0x0804ed9f in do_action (a=0x81044b8, msg=0x8107178) at action.c:375
#13 0x0805079c in run_actions (a=0x8105568, msg=0x8107178) at action.c:718
#14 0x0804ef01 in do_action (a=0x8104518, msg=0x8107178) at action.c:600
#15 0x0805079c in run_actions (a=0x8105568, msg=0x8104518) at action.c:718
#16 0x0804ef01 in do_action (a=0x8104e78, msg=0x8107178) at action.c:600
#17 0x0805079c in run_actions (a=0x8105568, msg=0x8104e78) at action.c:718
#18 0x08069449 in receive_msg (
    buf=0x80c48a0 "INVITE sip:5503100 at sipproxy.mydomain.com SIP/2.0\r\nVia:
SIP/2.0/UDP 192.168.0.2:5060;branch=z9hG4bK-4cd3c2cf;rport\r\nFrom:
5555848177
<sip:5555848177 at sipproxy.mydomain.com>;tag=e9d67edee86d16dbo0\r\nTo: <"...,
len=1139, rcv_info=0xbfffe650)
    at receive.c:165
#19 0x08079e8f in udp_rcv_loop () at udp_server.c:458
#20 0x0805b480 in main_loop () at main.c:1032
#21 0x0805c7ac in main (argc=0, argv=0x4213ad38) at main.c:1568
#22 0x42015574 in __libc_start_main () from /lib/tls/libc.so.6

Does this make sense to somone?.  Can i obtain more information from the
core.dumped file?

For the record i'm using : 

[root at sipvoiss root]# uname -a
Linux sipvoiss 2.4.20-8 #1 Thu Mar 13 17:54:28 EST 2003 i686 i686 i386
GNU/Linux

[root at sipvoiss root]# more /etc/issue
Red Hat Linux release 9 (Shrike)
Kernel \r on an \m

[root at sipvoiss root]# ser -V
version: ser 0.9.3 (i386/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MEM, SHM_MMAP, PKG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
@(#) $Id: main.c,v 1.197 2004/12/03 19:09:31 andrei Exp $
main.c compiled on 09:52:20 Jul  7 2005 with gcc 3.2

[root at sipvoiss root]#  /usr/local/mediaproxy/mediaproxy.py --version
mediaproxy.py 1.3.1

I hope that someone could help me on this one.
Thanks in advance

Regards,
Ricardo Martinez.-