[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  ts_append (msg=0x7f883bfd5490, ruri=0x7fff65fbfe30, table=0x7f883bf7e390 "location") at ts_append.c:60

60 ts_append.c: No such file or directory.

(gdb) bt

#0  ts_append (msg=0x7f883bfd5490, ruri=0x7fff65fbfe30, table=0x7f883bf7e390 "location") at ts_append.c:60

#1  0x00007f8837f638fc in w_ts_append (_msg=0x7f883bfd5490, _table=0x7f883bf7e390 "location", _ruri=0x7f88339aa634 "\002") at tsilo.c:225

#2  0x0000000000534db0 in do_action (h=0x7fff65fc1a40, a=0x7f883bf7d278, msg=0x7f883bfd5490) at action.c:1059

#3  0x00000000005339e8 in run_actions (h=0x7f8837f65540, a=0x5, msg=0x7f88339aa634) at action.c:1548

#4  0x00000000005352e6 in do_action (h=0x7fff65fc1a40, a=0x7f883bf7dc50, msg=0x7f883bfd5490) at action.c:1048

#5  0x00000000005339e8 in run_actions (h=0x7f8837f65540, a=0x5, msg=0x7f88339aa634) at action.c:1548

#6  0x00000000005352e6 in do_action (h=0x7fff65fc1a40, a=0x7f883bf7df48, msg=0x7f883bfd5490) at action.c:1048

#7  0x00000000005339e8 in run_actions (h=0x7f8837f65540, a=0x5, msg=0x7f88339aa634) at action.c:1548

#8  0x00000000005354a3 in do_action (h=0x7fff65fc1a40, a=0x7f883bfa8ab8, msg=0x7f883bfd5490) at action.c:677

#9  0x00000000005339e8 in run_actions (h=0x7f8837f65540, a=0x5, msg=0x7f88339aa634) at action.c:1548

#10 0x00000000005354a3 in do_action (h=0x7fff65fc1a40, a=0x7f883bfa8ab8, msg=0x7f883bfd5490) at action.c:677

#11 0x00000000005339e8 in run_actions (h=0x7f8837f65540, a=0x5, msg=0x7f88339aa634) at action.c:1548

#12 0x00000000005352e6 in do_action (h=0x7fff65fc1a40, a=0x7f883bf56068, msg=0x7f883bfd5490) at action.c:1048

#13 0x00000000005339e8 in run_actions (h=0x7f8837f65540, h@entry=0x7fff65fc1a40, a=0x5, a@entry=0x7f883bf4f958, msg=0x7f88339aa634, msg@entry=0x7f883bfd5490) at action.c:1548

#14 0x000000000053f885 in run_top_route (a=0x7f883bf4f958, msg=0x7f883bfd5490, c=<optimized out>) at action.c:1634

#15 0x00000000005407e6 in receive_msg (buf=0x0, len=5, rcv_info=0x7f88339b8638) at receive.c:196

#16 0x00000000005b806b in tcp_read_req (con=0x7f88339b8620, bytes_read=0x7fff65fc1d50, read_flags=0x7fff65fc1d54) at tcp_read.c:1382

#17 0x00000000005bb4e1 in handle_io (fm=0xb, events=5, idx=865773108) at tcp_read.c:1568

#18 0x00000000005c191a in io_wait_loop_epoll (h=<optimized out>, t=<optimized out>, repeat=<optimized out>) at io_wait.h:1061

#19 tcp_receive_loop (unix_sock=938890560) at tcp_read.c:1733

#20 0x00000000004d3447 in tcp_init_children () at tcp_main.c:4787

#21 0x0000000000506863 in main_loop () at main.c:1658

#22 0x000000000041b944 in main (argc=0, argv=0x0) at main.c:2533


It seems to be on reception of REGISTER from client.



2015-09-10 12:47 GMT+02:00 Daniel-Constantin Mierla <miconda@gmail.com>:
Hello,

can you install the kamailio-dbg package to get access to debugging sysmbols? Once that package is installed, get the backtrace again and send it here.

Cheers,
Daniel


On 10/09/15 10:55, Thibault Gueslin wrote:
I have done again the trace and obtained 2 core dumps:

Here is the second:

Core was generated by `kamailio -f /etc/kamailio/kamailio.cfg -w /tmp'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  0x0000000000000264 in ?? ()

(gdb) bt full

#0  0x0000000000000264 in ?? ()

No symbol table info available.

#1  0x00007fac88d39ebb in free_faked_req () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#2  0x00007fac88d3a1e7 in run_failure_handlers () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#3  0x00007fac88d3bf59 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#4  0x00007fac88d43105 in relay_reply () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#5  0x00007fac88d4764b in reply_received () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#6  0x00000000004fbfee in ?? ()

No symbol table info available.

#7  0x000000000054074c in receive_msg ()

No symbol table info available.

#8  0x00000000005b806b in tcp_read_req ()

No symbol table info available.

#9  0x00000000005bb704 in ?? ()

No symbol table info available.

#10 0x00000000005c191a in tcp_receive_loop ()

No symbol table info available.

#11 0x00000000004d3447 in tcp_init_children ()

No symbol table info available.

#12 0x0000000000506863 in main_loop ()

No symbol table info available.

#13 0x000000000041b944 in main ()

No symbol table info available.

The scenario is to call a remote which is started. After starting the regisration, the call is presented, then kill the app,  then started again, call is presented, wait while ringing....

 After a few times, kamailio crashs.


(sent to mailing list now)

2015-09-09 17:20 GMT+02:00 Daniel-Constantin Mierla <miconda@gmail.com>:
This is during shut down clean up. Have you stopped kamailio manually? If not, aave you got more than one core file? If not, enable one core file per process.

Cheers,
Daniel


On 09/09/15 17:03, Thibault Gueslin wrote:

It seems related to tsilo module and tm module:


#0  0x00007f6a7c8978f5 in lock_entry () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tsilo.so


#0  0x00007f6a7c8978f5 in lock_entry () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tsilo.so

(gdb) bt full

#0  0x00007f6a7c8978f5 in lock_entry () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tsilo.so

No symbol table info available.

#1  0x00007f6a7c89d185 in ts_onreply () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tsilo.so

No symbol table info available.

#2  0x00007f6a7f688f67 in run_trans_callbacks_internal () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#3  0x00007f6a7f68921e in run_trans_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#4  0x00007f6a7f6043e0 in free_cell () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#5  0x00007f6a7f605c58 in free_hash_table () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#6  0x00007f6a7f67f9a1 in tm_shutdown () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so

No symbol table info available.

#7  0x0000000000509142 in destroy_modules ()

No symbol table info available.

#8  0x00000000004ff559 in cleanup ()

No symbol table info available.

#9  0x00000000005004e8 in ?? ()

No symbol table info available.

#10 0x00000000005023cb in handle_sigs ()

No symbol table info available.

#11 0x0000000000506a3e in main_loop ()

No symbol table info available.

#12 0x000000000041b944 in main ()

No symbol table info available.


version: kamailio 4.3.1 (x86_64/linux) 

flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB

poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.

id: unknown 

compiled with gcc 4.9.2


I have seen some commits related to tm module after release 4.3.1


Should it work better in 4.3.2 ?


Regards


thibault



2015-09-09 14:26 GMT+02:00 Daniel-Constantin Mierla <miconda@gmail.com>:
Hello,

do you get other error messages in syslog before the one "CRITICAL: ..."?

It is hard to say it is from something already known or new. Right now there is an open issue for saving dialog variables in database that was not yet sorted out. Another one related to tm and memory manager, reported after 4.3.1, got a safety fix. Apart of these two, I don't remember any active crash report for the moment.

If you get it very often, then it should be easy to catch and troubleshoot properly. Can you run it so it generates a core file?

The easiest way is to run it as root, with 'ulimit -c unlimited' executed before starting kamailio.

Can you share what operating system are you using and exact version of kamailio (output of kamailio -v)?

Cheers,
Daniel


On 09/09/15 12:15, Thibault Gueslin wrote:
After upgrading to kamailio 4.3.1 and modifying the config, I get this crash very often:
...

CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 21

ALERT: <core> [main.c:728]: handle_sigs(): child process 9341 exited by a signal 11

ALERT: <core> [main.c:731]: handle_sigs(): core was not generated

INFO: <core> [main.c:743]: handle_sigs(): terminating due to SIGCHLD

INFO: <core> [main.c:794]: sig_usr(): signal 15 received

INFO: <core> [main.c:794]: sig_usr(): signal 15 received

INFO: <core> [main.c:794]: sig_usr(): signal 15 received

...


Is it known ? or may be related to my config ?


Thank you


Thibault 



_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat


-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat


-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat