[SR-Users] tm.so --> segfault at 3135352e36 ip 00007f761bb57ed1 sp 00007fff9db8b1c0 error 4 in tm.so

Abdoul Osséni abdoul.osseni at gmail.com
Tue Feb 26 11:13:40 CET 2019


Hello,

The value of debug level I had during the crash is 2.
---
debug=2
---

I checked from my monitoring tools and system logs if the server has
encounter any issue (freeze, network lost, database issues, ...) but I
found nothing.

[image: image.png]



[image: image.png]

I use Debian (8.6 default version).

[image: image.png]
Regards

Abdoul


Le lun. 25 févr. 2019 à 18:34, Daniel-Constantin Mierla <miconda at gmail.com>
a écrit :

> Hello,
>
> that's strange, but a while ago someone else reported an issue with same
> backtrace.
>
> So the crash happens at the last line in the next snippet from
> reply_received() function in the tm module:
>
>     uac=&t->uac[branch];
>     LM_DBG("org. status uas=%d, uac[%d]=%d local=%d is_invite=%d)\n",
>         t->uas.status, branch, uac->last_received,
>         is_local(t), is_invite(t));
>     last_uac_status=uac->last_received;
>
> The backtrace and info locals say that uac is null (0x0). According to my
> knowledge, the address of a field in a structure cannot be null and uac is
> set to &t->uac[branch]. Moreover, uac->last_received is printed in the
> LM_DBG() above the line of crash, if uac was 0x0, the crash should have
> happened there.
>
> Then uac is a local variable, so it is on the stack of the process, in its
> private memory. There is no other assign or copy operation between the line
> of code where the uac is set and the crash. So overall, should be no race
> condition there. Either the kernel was doing something wrong, or maybe the
> coredump was somehow corrupted.
>
> What was the value of debug level you had during the crash (debug
> parameter in kamailio.cfg)?
>
> Could there have been any freeze of the operating system for long time and
> then a resume?
>
> Can you give the output of command:
>
> uname -a
>
> What kind of linux distro and version you are running?
>
> Cheers,
> Daniel
> On 25.02.19 15:27, Abdoul Osséni wrote:
>
> Hello,
>
> Please see attached the output of the gdb commands.
>
> Can you check with all core files and see if the backtrace is the same?
>
> --> Yes the backtrace is the same.
>
> Sorry, I use kamailio v5.2
>
> root at sbc:/var/cores# kamailio -V
> version: kamailio 5.2.1 (x86_64/linux) cd2583
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
> USE_DST_BLACKLIST, HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024,
> BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: cd2583
> compiled on 07:33:25 Jan 31 2019 with gcc 4.9.2
>
>
> Kamailio is running on a bare metal server.
>
>
> Thanks
>
> Abdoul
>
>
> Le lun. 25 févr. 2019 à 14:40, Daniel-Constantin Mierla <miconda at gmail.com>
> a écrit :
>
>> Hello,
>>
>> can you give the output for next gdb commands:
>>
>> bt full
>>
>> info locals
>>
>> list
>>
>> Can you check with all core files and see if the backtrace is the same?
>>
>> What is the version of Kamailio? Is it running on a bare metal server or
>> a virtual machine/container?
>>
>> Cheers,
>> Daniel
>> On 25.02.19 14:21, Abdoul Osséni wrote:
>>
>> Hello,
>>
>> Hello dear list,
>>
>> Today, I have had mutiples crashes. It seems it linked to tm.so module.
>>
>> -rw------- 1 root kamailio 4299702272 Feb 25 13:08
>> core.kamailio.sig11.29204
>> -rw------- 1 root kamailio 1453023232 Feb 25 13:12
>> core.kamailio.sig11.29203
>> -rw------- 1 root kamailio 1416065024 Feb 25 13:12
>> core.kamailio.sig11.29207
>> -rw------- 1 root kamailio 4299681792 Feb 25 13:16
>> core.kamailio.sig11.19047
>> -rw------- 1 root kamailio 2108506112 Feb 25 13:20
>> core.kamailio.sig11.19043
>> -rw------- 1 root kamailio 4299689984 Feb 25 13:34
>> core.kamailio.sig11.19247
>> -rw------- 1 root kamailio 4299681792 Feb 25 13:34
>> core.kamailio.sig11.19246
>> -rw------- 1 root kamailio 4299698176 Feb 25 13:35
>> core.kamailio.sig11.19248
>> -rw------- 1 root kamailio 4299689984 Feb 25 13:35
>> core.kamailio.sig11.19243
>> -rw------- 1 root kamailio 4299685888 Feb 25 13:35
>> core.kamailio.sig11.19244
>> -rw------- 1 root kamailio 4299689984 Feb 25 13:36
>> core.kamailio.sig11.19242
>>
>> root at sbc:/var/cores# gdb /usr/local/sbin/kamailio
>> core.kamailio.sig11.29204
>> GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
>> Copyright (C) 2014 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/>.
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from /usr/local/sbin/kamailio...done.
>> [New LWP 29204]
>>
>> warning: .dynamic section for "/lib/x86_64-linux-gnu/libpthread.so.0" is
>> not at the expected address (wrong library or version mismatch?)
>> Warning: couldn't activate thread debugging using libthread_db: Cannot
>> find new threads: generic error
>>
>> warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading
>> has been declined by your `auto-load safe-path' set to
>> "$debugdir:$datadir/auto-load".
>> To enable execution of this file add
>>         add-auto-load-safe-path /lib/x86_64-linux-gnu/libthread_db-1.0.so
>> line to your configuration file "/root/.gdbinit".
>> To completely disable this security protection add
>>         set auto-load safe-path /
>> line to your configuration file "/root/.gdbinit".
>> For more information about this security protection see the
>> "Auto-loading safe path" section in the GDB manual.  E.g., run from the
>> shell:
>>         info "(gdb)Auto-loading safe path"
>>
>> warning: Unable to find libthread_db matching inferior's thread library,
>> thread debugging will not be available.
>> Warning: couldn't activate thread debugging using libthread_db: Cannot
>> find new threads: generic error
>>
>> warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading
>> has been declined by your `auto-load safe-path' set to
>> "$debugdir:$datadir/auto-load".
>>
>> warning: Unable to find libthread_db matching inferior's thread library,
>> thread debugging will not be available.
>> Core was generated by `/usr/local/sbin/kamailio -P
>> /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  0x00007f68b9bb4515 in reply_received (p_msg=0x7f693ccd5cc8) at
>> t_reply.c:2240
>> 2240            last_uac_status=uac->last_received;
>> (gdb)
>>
>>
>> Any ideas?
>>
>> Regards
>>
>> Abdoul
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
>> Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in Washington, DC, USA -- www.asipto.com
>>
>> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
> Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in Washington, DC, USA -- www.asipto.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190226/18113ccf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 3351 bytes
Desc: not available
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190226/18113ccf/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 2005 bytes
Desc: not available
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190226/18113ccf/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 56725 bytes
Desc: not available
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190226/18113ccf/attachment-0002.png>


More information about the sr-users mailing list