Hi Dmitri! Is this bug reproduce-able? If yes, please activate core dumping in /etc/defaults/kamailio and configure location of core file in /etc/init.d/kamailio (search for "core") regards klaus Am 08.02.2010 12:16, schrieb Dmitri: > Hello list, > > I have fresh installation of Debian Squeeze on X86_64. > I downloaded source tarball > http://www.kamailio.org/pub/kamailio/latest/src/kamailio-3.0.0_src.tar.gz > > and did 'make deb' there. > So, I got following debian packages: > > kamailio_3.0.0_amd64.deb, kamailio-berkeley-modules_3.0.0_amd64.deb, > kamailio-carrierroute-modules_3.0.0_amd64.deb > kamailio-cpl-modules_3.0.0_amd64.deb, > kamailio-ldap-modules_3.0.0_amd64.deb, > kamailio-memcached-modules_3.0.0_amd64.deb > kamailio-mysql-modules_3.0.0_amd64.deb, kamailio-nth_3.0.0_amd64.deb, > kamailio-perl-modules_3.0.0_amd64.deb > kamailio-postgres-modules_3.0.0_amd64.deb, > kamailio-presence-modules_3.0.0_amd64.deb, > kamailio-purple-modules_3.0.0_amd64.deb > kamailio-radius-modules_3.0.0_amd64.deb, > kamailio-snmpstats-modules_3.0.0_amd64.deb, > kamailio-tls-modules_3.0.0_amd64.deb > kamailio-unixodbc-modules_3.0.0_amd64.deb, > kamailio-utils-modules_3.0.0_amd64.deb, > kamailio-xmlrpc-modules_3.0.0_amd64.deb > kamailio-xmpp-modules_3.0.0_amd64.deb > > I installed packages: > kamailio_3.0.0_amd64.deb, kamailio-mysql-modules_3.0.0_amd64.deb, > kamailio-presence-modules_3.0.0_amd64.deb, > kamailio-xmlrpc-modules_3.0.0_amd64.deb > > Created database, changed passwords for openser and openserro users in > MySQL as well as in kamailio config files, enabled MySQL. > DUMP_CORE=yes in /etc/default/kamailio, also added '-w > /var/run/kamailio' to options. > Created test user with kamctl. > Started kamailio with init script: /etc/init.d/kamailio start: > Feb 7 22:05:10 kamailio kamailio: INFO: <core> [tcp_main.c:4150]: > init_tcp: using epoll_lt as the io watch method (auto detected) > Feb 7 22:05:10 kamailio /usr/sbin/kamailio[2678]: INFO: usrloc > [hslot.c:53]: locks array size 512 > Feb 7 22:05:10 kamailio /usr/sbin/kamailio[2678]: INFO: <core> > [udp_server.c:166]: INFO: udp_init: SO_RCVBUF is initially 124928 > Feb 7 22:05:10 kamailio /usr/sbin/kamailio[2678]: INFO: <core> > [udp_server.c:217]: INFO: udp_init: SO_RCVBUF is finally 262142 > Feb 7 22:05:10 kamailio /usr/sbin/kamailio[2678]: INFO: <core> > [udp_server.c:166]: INFO: udp_init: SO_RCVBUF is initially 124928 > Feb 7 22:05:10 kamailio /usr/sbin/kamailio[2678]: INFO: <core> > [udp_server.c:217]: INFO: udp_init: SO_RCVBUF is finally 262142 > Feb 7 22:05:10 kamailio /usr/sbin/kamailio[2678]: INFO: <core> > [udp_server.c:166]: INFO: udp_init: SO_RCVBUF is initially 124928 > Feb 7 22:05:10 kamailio /usr/sbin/kamailio[2678]: INFO: <core> > [udp_server.c:217]: INFO: udp_init: SO_RCVBUF is finally 262142 > Feb 7 22:05:10 kamailio /usr/sbin/kamailio[2695]: INFO: ctl > [io_listener.c:224]: io_listen_loop: using epoll_lt io watch method > (config) > > Then I tried to connect to kamailio server with softphone Zoiper, and > kamailio crashes immediately: > > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2700]: : <core> > [pass_fd.c:283]: ERROR: receive_fd: EOF on 21 > Feb 7 22:05:20 kamailio kernel: [ 1051.407643] kamailio[2689]: segfault > at c2 ip 00007fa8b5947b19 sp 00007fff48ff9700 error 4 in > tm.so[7fa8b5917000+72000] > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2678]: ALERT: <core> > [main.c:722]: child process 2689 exited by a signal 11 > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2678]: ALERT: <core> > [main.c:725]: core was not generated > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2678]: INFO: <core> > [main.c:737]: INFO: terminating due to SIGCHLD > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2691]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2699]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2698]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2687]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2697]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2690]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2696]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2688]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2695]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2680]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2694]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2693]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2692]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2686]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2685]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2681]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2684]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2682]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2700]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2683]: INFO: <core> > [main.c:788]: INFO: signal 15 received > Feb 7 22:05:20 kamailio /usr/sbin/kamailio[2678]: ERROR: ctl > [ctl.c:379]: ERROR: ctl: could not delete unix socket /tmp/ser_ctl: > Operation not permitted (1) > Feb 7 22:05:20 kamailio kernel: [ 1051.432646] kamailio[2678]: segfault > at 12 ip 00007fa8b592ae43 sp 00007fff48ff9f40 error 4 in > tm.so[7fa8b5917000+72000] > > For unknown reason core was not generated. > Any ideas ? > > Many thanks in advance, > Dmitri > > > _______________________________________________ > Kamailio (OpenSER) - Users mailing list > Users(a)lists.kamailio.org > http://lists.kamailio.org/cgi-bin/mailman/listinfo/users > http://lists.openser-project.org/cgi-bin/mailman/listinfo/users

Hi Klaus, Yes, it is permanent error. And I enabled core dumping in /etc/default/kamailio and specified location in init script, but core was not generated.... if test "$DUMP_CORE" = "yes" ; then # set proper ulimit ulimit -c unlimited directory for the core dump files COREDIR=/tmp [ -d $COREDIR ] || mkdir $COREDIR chmod 777 $COREDIR echo "$COREDIR/core.%e.sig%s.%p" > /proc/sys/kernel/core_pattern fi 08.02.2010 14:47, Klaus Darilion пишет: _______________________________________________ Kamailio (OpenSER) - Users mailing list Users(a)lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users

I don't know it this applies to you but also check the following: 1. You have enough space in the partition where $COREDIR is created 2. Check that the user that runs kamailio (usually openser) has read write access on that directory Also if you grep for core in the /var/log/ directory I believe that there might be some lines either in dmesg or in messages that tell you if a core was created or not. Hope what I said is not redundant and it helps. Marius

Hello, 1. Yes, there is at least 2 times more space than core dump needs. 2. I could find nothing, unfortunately...

I can only see that problem is related to tm.so library: kamailio kernel: [ 1051.432646] kamailio[2678]: segfault at 12 ip 00007fa8b592ae43 sp 00007fff48ff9f40 error 4 in tm.so[7fa8b5917000+72000] BR, Dmitri _______________________________________________ Kamailio (OpenSER) - Users mailing list Users(a)lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users

Hello, No, I do not using those modules. I created debian modules from latest git sources as well and have same crash. Modules I am using are: grep -e '^loadmodule.*' ./kamailio.cfg loadmodule "db_mysql.so" loadmodule "mi_fifo.so" loadmodule "kex.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "maxfwd.so" loadmodule "usrloc.so" loadmodule "registrar.so" loadmodule "textops.so" loadmodule "uri_db.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "ctl.so" loadmodule "mi_rpc.so" loadmodule "acc.so" loadmodule "auth.so" loadmodule "auth_db.so" loadmodule "presence.so" loadmodule "presence_xml.so" loadmodule "nathelper.so" 08.02.2010 21:07, Daniel-Constantin Mierla пишет:

Am 08.02.2010 20:00, schrieb Dmitri: # Try specifying things manually, as user root: echo "/tmp/core.%e.sig%s.%p" > /proc/sys/kernel/core_pattern # verify configuration changes cat /proc/sys/kernel/core_pattern # set ulimit ulimit -c unlimited # start kamailio manually kamailio now kamailio should be running as user root and should be able to core dump into /tmp Daniel, I can't remember. Is it necessary to turn off some compiler optimizations in Makefile.defs to get a useful backtrace? (or was that in Asterisk?)

Hello Klaus, Thank you for suggestion. Now I got core, and output of gdb you may see in attached file. Core was about 40M of size. I have to say that i never used kamailio before and this crash is my first experience. BR, Dmitri 08.02.2010 21:37, Klaus Darilion пишет: _______________________________________________ Kamailio (OpenSER) - Users mailing list Users(a)lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users

Hello, Updated gdb output in attached file. BR, Dmitri 08.02.2010 22:51, Daniel-Constantin Mierla пишет:

Hello, No, it is not local, it is reacheable over internet, real dns domain.

Waiting for your instructions.

BR, Dmitri. 08.02.2010 23:27, Daniel-Constantin Mierla пишет:

Hello, The PC has private address on eth0 interface and public on ppp0 default route looks like: default dev ppp0 scope link Tomorrow I'll try to play with ngrep.

BR, Dmitri 08.02.2010 23:59, Daniel-Constantin Mierla пишет:

Then, open Makefile.defs file and search for MEMDBG, update the intial value to 1: MEMDBG ?= 1 The you re-compile/re-generate debs. To doublecheck, once everything was re-installed, run kamailio -V, you should see DBG_QM_MALLOC in flags string. Set debug lever to some big value, say 5, test again and send over the syslog messages along with the backtrace from the new core. Thanks for assistance, Daniel

Hi Dnitri, On 2/9/10 9:19 AM, Dmitri wrote: it is very strange. I just did the same steps and I get it. Can you doublecheck there was no typo when you checkout and having the wrong branch? Cheers, Daniel -- Daniel-Constantin Mierla eLearning class for Kamailio 3.0.0 Starting Feb 8, 2010 *http://www.asipto.com/

debug=4

debug=2

"mysql://openser:openserrw@localhost/openser")

"mysql://openser:openserrw@localhost/openser")

"mysql://openser:openserrw@localhost/openser")

# "mysql://openser:openserrw@localhost/openser")

# "mysql://openser:openserrw@localhost/openser")

"mysql://openser:openserrw@localhost/openser")

there is no crash, but errors in config. Seems you have some is_method() without parameters: Feb 9 13:19:02 kamailio kamailio[3425]: ERROR: textops [textops.c:1636]: empty method name Feb 9 13:19:02 kamailio kamailio[3425]:<core> [mem/q_malloc.c:428]: qm_free(0x874ba0, 0x8e4a78), called from textops: textops.c: fixup_method(1637) Feb 9 13:19:02 kamailio kamailio[3425]:<core> [mem/q_malloc.c:450]: qm_free: freeing frag. 0x8e4a48 alloc'ed from textops: textops.c: fixup_method(1626) Feb 9 13:19:02 kamailio kamailio[3425]: ERROR:<core> [route.c:1026]: fixing failed (code=-1) at cfg:/etc/kamailio/kamailio.cfg:304 Feb 9 13:19:02 kamailio kamailio[3425]: ERROR:<core> [route.c:1026]: fixing failed (code=-1) at cfg:/etc/kamailio/kamailio.cfg:308 Feb 9 13:19:02 kamailio kamailio[3425]: ERROR:<core> [route.c:1026]: fixing failed (code=-1) at cfg:/etc/kamailio/kamailio.cfg:327 Feb 9 13:19:02 kamailio kamailio[3425]: ERROR:<core> [route.c:1026]: fixing failed (code=-1) at cfg:/etc/kamailio/kamailio.cfg:333 Check the line reported in above logs. Cheers, Daniel

Hi Dmitri, your installation is very weird ... I just took everything, compiled and run with default config file, no problem. Do you build the packages and run them on the same system? Or you have a build system and then copy&install on another one? Please send here the output of: uname -a kamailio -V The changes to the database url should have no effect, as you haven't enabled mysql (not visibile in the diff) they are within #!defines that are not enabled. Cheers, Daniel

Hello Daniel, I am very sorry about my installation.. I proceeded by documents, but got a lot of errors. It is true, i forgot to put definitions to global parameter. Yes, I build on one system, and then move debs to other one. All depends on second machine are satisfied. Both systems are identical hw and sw (debian squeezy). I enabled at the top of config file following thinks, but without success: #!define WITH_MYSQL #!define WITH_AUTH #!define WITH_USRLOCDB uname -a Linux kamailio 2.6.32-trunk-amd64 #1 SMP Sun Jan 10 22:40:40 UTC 2010 x86_64 GNU/Linux kamailio -V version: kamailio 3.0.0 (x86_64/linux) dc20fc flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. @(#) $Id$ main.c compiled on 12:34:49 Feb 9 2010 with gcc 4.4.3 You may see current kamalio.cfg and last syslog in attached file BR, Dmitri 09.02.2010 21:49, Daniel-Constantin Mierla пишет:

Hi Dmitri, I got zoiper communicator (mac os x, v1.15.6377) and tested with it kamailio 3.0.0 from git (on ubuntu 9.10). Registrations and calls work ok, nothing wrong spotted in signaling or syslog. Looks very likely there is an issue between build system and deployed system, some library or binary corrupted or mismatched. Or sove very particular situation in your case, but since you said you run default config, is pretty much improbable. Let me know if you have new details and whether you could run it on build system. Cheers, Daniel On 2/9/10 10:27 PM, Dmitri wrote:

Hello Daniel, I fetched source code from git using # git clone --depth 1 git://git.sip-router.org/sip-router kamailio # cd kamailio # git checkout -b kamailio_3.0 origin/kamailio_3.0 But there is no MEMDBG option in Makefile.defs devel:/usr/src/kamailio-3.0.0/kamailio# grep MEMDBG ./Makefile.defs devel:/usr/src/kamailio-3.0.0/kamailio# Should I add MEMDBG there ?

BR, Dmitri 09.02.2010 0:26, Daniel-Constantin Mierla пишет: