[SR-Users] core dump if silo table > ~5000 records

Daniel-Constantin Mierla miconda at gmail.com
Wed Sep 18 09:04:11 CEST 2019


Hello,

the crash exposed by the backtrace sent here is from shut down
procedure. If there was a crash, then it happened earlier and you have
to enable core dumps per process/pid in order to get more core files,
one of them should have the crash at runtime. The one at shutdown might
be a side effect of the one at runtime, or can be independent of it. But
first needs to be sorted out the one at runtime.

See more details about core files per process/pid at:

  - https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile

Reproduce and sent the backtraces from all dumped corefiles.

Cheers,
Daniel

On 14.09.19 16:36, Henning Westerholt wrote:
>
> Hello David,
>
> strange. Do you have a timer in the msilo module configured that is
> executed every 20s?
>
> About the core dump - this looks like the kamailio is already shutting
> down from some earlier error. The crash is not in the msilo code but
> in the tsilo code.
>
> Can you print the value of urecord in frame 0?
>
> Cheers,
>
> Henning
>
> Am 14.09.19 um 15:45 schrieb David Villasmil:
>> bt full:
>>
>>
>> root at sip:/home/carlos# gdb /usr/local/kamailio5/sbin/kamailio
>> /usr/local/kamailio5/sbin/core
>> GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
>> Copyright (C) 2014 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>> copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/>.
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from /usr/local/kamailio5/sbin/kamailio...done.
>> [New LWP 22299]
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library
>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> Core was generated by `/usr/local/kamailio5/sbin/kamailio -f
>> /usr/local/kamailio5/etc/kamailio/kamaili'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  0x00007f32cd989140 in free_ts_urecord (urecord=0x7f328e347270) at
>> ts_hash.c:54
>> 54 urecord->transactions = urecord->transactions->next;
>> (gdb) bt
>> #0  0x00007f32cd989140 in free_ts_urecord (urecord=0x7f328e347270) at
>> ts_hash.c:54
>> #1  0x00007f32cd989e9d in destroy_ts_table () at ts_hash.c:145
>> #2  0x00007f32cd9920d4 in destroy () at tsilo.c:193
>> #3  0x000000000051af99 in destroy_modules () at core/sr_module.c:811
>> #4  0x0000000000418199 in cleanup (show_status=1) at main.c:529
>> #5  0x0000000000419817 in shutdown_children (sig=15, show_status=1)
>> at main.c:671
>> #6  0x000000000041c1de in handle_sigs () at main.c:763
>> #7  0x0000000000423ddd in main_loop () at main.c:1751
>> #8  0x0000000000429a61 in main (argc=13, argv=0x7fff1d3773f8) at
>> main.c:2639
>>
>>
>>
>> (gdb) bt full
>> #0  0x00007f32cd989140 in free_ts_urecord (urecord=0x7f328e347270) at
>> ts_hash.c:54
>>         __FUNCTION__ = "free_ts_urecord"
>>         ptr = 0x72743b3536323831
>> #1  0x00007f32cd989e9d in destroy_ts_table () at ts_hash.c:145
>>         ts_u = 0x7f3200000023
>>         l_ts_u = 0x7f328e347270
>>         i = 7
>>         __FUNCTION__ = "destroy_ts_table"
>> #2  0x00007f32cd9920d4 in destroy () at tsilo.c:193
>> No locals.
>> #3  0x000000000051af99 in destroy_modules () at core/sr_module.c:811
>>         t = 0x7f32d68eb248
>>         foo = 0x7f32d68eaa18
>>         __FUNCTION__ = "destroy_modules"
>> #4  0x0000000000418199 in cleanup (show_status=1) at main.c:529
>>         memlog = 16711680
>>         __FUNCTION__ = "cleanup"
>> #5  0x0000000000419817 in shutdown_children (sig=15, show_status=1)
>> at main.c:671
>>         __FUNCTION__ = "shutdown_children"
>> #6  0x000000000041c1de in handle_sigs () at main.c:763
>>         chld = 0
>>         chld_status = 139
>>         memlog = 4291896
>>         __FUNCTION__ = "handle_sigs"
>> #7  0x0000000000423ddd in main_loop () at main.c:1751
>>         i = 8
>>         pid = 22340
>>         si = 0x0
>>         si_desc = "udp receiver child=7 sock=169.239.137.3:5060
>> <http://169.239.137.3:5060>\000\b\000\000P\335ό\001\000\000\000\300\244\244\214\062\177\000\000\220p7\035\377\177\000\000\372\071d\000\000\000\000\000PxA\000\000\000\000\000\350R\237\326\062\177",
>> '\000' <repeats 14 times>,
>> "\001\000\000\000\340p7\035\377\177\000\000I\273d\000\000\000\000"
>>         nrprocs = 8
>>         woneinit = 1
>>         __FUNCTION__ = "main_loop"
>> #8  0x0000000000429a61 in main (argc=13, argv=0x7fff1d3773f8) at
>> main.c:2639
>>         cfg_stream = 0x202a010
>>         c = -1
>>         r = 0
>>         tmp = 0x7fff1d3788bf ""
>>         tmp_len = 0
>>         port = 1
>>         proto = 32562
>>         options = 0x72b100
>> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
>>         ret = -1
>>         seed = 1981610615
>>         rfd = 4
>>         debug_save = 0
>>         debug_flag = 0
>>         dont_fork_cnt = 0
>>         n_lst = 0x7fff1d3772b0
>>         p = 0x7fff1d377468 "\361\210\067\035\377\177"
>>         st = {st_dev = 15, st_ino = 10098, st_nlink = 2, st_mode =
>> 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 80,
>> st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1568455518,
>> tv_nsec = 802350062}, st_mtim = {
>>             tv_sec = 1568457084, tv_nsec = 898346157}, st_ctim =
>> {tv_sec = 1568457084, tv_nsec = 898346157}, __glibc_reserved = {0, 0, 0}}
>>         __FUNCTION__ = "main"
>> (gdb) 
>>
>>
>> Regards,
>>
>> David Villasmil
>> email: david.villasmil.work at gmail.com
>> <mailto:david.villasmil.work at gmail.com>
>> phone: +34669448337
>>
>>
>> On Sat, Sep 14, 2019 at 2:35 PM David Villasmil
>> <david.villasmil.work at gmail.com
>> <mailto:david.villasmil.work at gmail.com>> wrote:
>>
>>     Hello guys,
>>
>>     i've got a kamailo:
>>
>>     version: kamailio 5.0.0 (x86_64/linux)
>>     flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
>>     DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
>>     PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
>>     USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>>     USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>     ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
>>     16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>     poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>     id: unknown
>>     compiled on 19:50:58 Mar 28 2017 with gcc 4.9.2
>>
>>     which was core dumping every 20 seconds and we had no idea why.
>>     We emptied out the silo table, which had:
>>
>>     mysql> select count(*) from silobackup;
>>     +----------+
>>     | count(*) |
>>     +----------+
>>     |     5240 |
>>     +----------+
>>
>>     records.
>>     Once we truncated it, kamailio started working again.
>>
>>     Is there a limit to how many records msilo can handle?
>>
>>     I will look into the data in the table, but in case there's a
>>     limit, i'd like to know.
>>
>>     core available on request.
>>
>>     Thanks
>>
>>     Regards,
>>
>>     David Villasmil
>>     email: david.villasmil.work at gmail.com
>>     <mailto:david.villasmil.work at gmail.com>
>>     phone: +34669448337
>>
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> -- 
> Henning Westerholt - https://skalatan.de/blog/
> Kamailio services - https://skalatan.de/services
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190918/6b610081/attachment.html>


More information about the sr-users mailing list