[SR-Users] Coredump while exchanging diameter capabilities
Carsten Bock
carsten at ng-voice.com
Thu Jan 18 06:17:51 CET 2018
Hi,
The log clearly states, what went wrong and where. There was insufficient
memory allocated at some point.
I'm traveling right now, but I will check either tonight from the Hotel or
tomorrow.
Thanks for reporting,
Carsten
Am 17.01.2018 09:51 schrieb "Tsvetan Filev" <tsvetan.filev at inno-networks.com
>:
> Hi Rick.
>
> Yes I have coredumps and backtraces but I installed the rpms from here
> https://www.kamailio.org/wiki/packages/rpms (http://download.opensuse.org/
> repositories/home:/kamailio:/v5.1.x-rpms/openSUSE_Leap_42.3/ ) so I'm
> missing the debug symbols.
> Can I find the kamailio debug rpm (kamailio-debuginfo-5.1.0-21.7.x86_64)
> somewhere ?
>
> Once I have the symbols I will create an issue on GitHub.
>
> Here is what I get for now:
>
> # gdb kamailio core.kamailio.479.e6ada9851c9848a98237890cfddfaf
> e3.6560.1516177422000000
>
> GNU gdb (GDB; openSUSE Leap 42.3) 8.0.1
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.
> html> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-suse-linux".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://bugs.opensuse.org/> <http://bugs.opensuse.org/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from kamailio...(no debugging symbols found)...done.
> [New LWP 6560]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/sbin/kamailio -DD -P
> /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamai'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007f637dfb38c7 in __GI_raise (sig=sig at entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:55
> 55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> Missing separate debuginfos, use: zypper install
> kamailio-debuginfo-5.1.0-21.7.x86_64
> (gdb) bt full
> #0 0x00007f637dfb38c7 in __GI_raise (sig=sig at entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:55
> resultvar = 0
> pid = 6560
> selftid = 6560
> #1 0x00007f637dfb4c9a in __GI_abort () at abort.c:78
> save_stage = 1
> act = {__sigaction_handler = {sa_handler = 0x28af01000000,
> sa_sigaction = 0x28af01000000}, sa_mask = {__val = {140724448160064, 0,
> 140065300084096, 6721723, 140065147406864, 140065144188928,
> 140065137572536, 140065144419712, 21474836483, 140065144188928,
> 9019276028016, 4, 0, 234976, 4, 8177296}}, sa_flags =
> 2123556608, sa_restorer = 0x0}
> sigs = {__val = {32, 0 <repeats 15 times>}}
> #2 0x000000000066906e in qm_debug_frag ()
> No symbol table info available.
> #3 0x000000000066beaf in qm_free ()
> No symbol table info available.
> #4 0x00000000006755f5 in qm_shm_free ()
> No symbol table info available.
> #5 0x00007f6374c38a1d in AAAFreeAVP () from /usr/lib64/kamailio/modules/
> cdp.so
> No symbol table info available.
> #6 0x00007f6374c144ac in AAAFreeAVPList () from
> /usr/lib64/kamailio/modules/cdp.so
> No symbol table info available.
> #7 0x00007f6374beb0f9 in save_peer_applications () from
> /usr/lib64/kamailio/modules/cdp.so
> No symbol table info available.
> #8 0x00007f6374beb16d in Process_CEA () from /usr/lib64/kamailio/modules/
> cdp.so
> No symbol table info available.
> #9 0x00007f6374be4a22 in sm_process () from /usr/lib64/kamailio/modules/
> cdp.so
> No symbol table info available.
> #10 0x00007f6374bd15d6 in receive_message () from
> /usr/lib64/kamailio/modules/cdp.so
> No symbol table info available.
> #11 0x00007f6374bc7032 in do_receive () from /usr/lib64/kamailio/modules/
> cdp.so
> No symbol table info available.
> #12 0x00007f6374bcae19 in receive_loop () from /usr/lib64/kamailio/modules/
> cdp.so
> No symbol table info available.
> #13 0x00007f6374bc4563 in receiver_process () from
> /usr/lib64/kamailio/modules/cdp.so
> No symbol table info available.
> #14 0x00007f6374be0c8c in diameter_peer_start () from
> /usr/lib64/kamailio/modules/cdp.so
> No symbol table info available.
> #15 0x00007f6374bd2e5f in cdp_child_init () from
> /usr/lib64/kamailio/modules/cdp.so
> No symbol table info available.
> #16 0x000000000054b705 in init_mod_child ()
> No symbol table info available.
> #17 0x000000000054b422 in init_mod_child ()
> No symbol table info available.
> #18 0x000000000054b422 in init_mod_child ()
> No symbol table info available.
> #19 0x000000000054b422 in init_mod_child ()
> No symbol table info available.
> #20 0x000000000054b422 in init_mod_child ()
> No symbol table info available.
> #21 0x000000000054ba3d in init_child ()
> No symbol table info available.
> #22 0x0000000000426925 in main_loop ()
> No symbol table info available.
> #23 0x000000000042d21e in main ()
> No symbol table info available.
> (gdb) info locals
> save_stage = 1
> \act = {__sigaction_handler = {sa_handler = 0x28af01000000, sa_sigaction =
> 0x28af01000000}, sa_mask = {__val = {140724448160064, 0, 140065300084096,
> 6721723, 140065147406864, 140065144188928, 140065137572536,
> 140065144419712, 21474836483, 140065144188928, 9019276028016,
> 4, 0, 234976, 4, 8177296}}, sa_flags = 2123556608, sa_restorer = 0x0}
> sigs = {__val = {32, 0 <repeats 15 times>}}
> (gdb) list
> 50 in ../sysdeps/unix/sysv/linux/raise.c
>
> Regards.
>
> On 16.01.2018 19:51, Rick Barenthin wrote:
>
> Hey,
>
> looks like an error with memory allocation. If you haven't done yet, it
> would be nice if you could create an issue on GitHub. Do you still have the
> coredump? If so please create an trace with gdb and attache it to the issue.
>
> Greetings Rick
>
> Am 16.01.2018 09:50 schrieb "Tsvetan Filev" <tsvetan.filev at inno-networks.
> com>:
>
> Hi.
>
> I get a coredump which is caused by bad memory handling during the
> diameter capability exchange process.
> Here is part of the log file:
> ===================================
> 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]: 87(13388)
> INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started...
> 2018-01-16T09:16:40.296393+02:00 linux-o12d systemd-coredump[13196]:
> Process 13193 (kamailio) of user 479 dumped core.
> 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]: 94(13395)
> DEBUG: cdp [peermanager.c:263]: peer_timer(): peer_timer(): taking care of
> peers...
> 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]: 94(13395)
> DEBUG: cdp [peermanager.c:280]: peer_timer(): peer_timer(): Peer
> hss.epc.mnc019.mcc425.3gppnetwork.org State 0
> 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]: 94(13395)
> DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer
> hss.epc.mnc019.mcc425.3gppnetwork.org State Closed Event Start
> 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]: 94(13395)
> INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req():
> Peer hss.epc.mnc019.mcc425.3gppnetwork.org
> 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]: 94(13395)
> INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to
> connect to 10.82.10.85 port 3868
> 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]: 94(13395)
> INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer
> hss.epc.mnc019.mcc425.3gppnetwork.org:3868 connected
> 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [receiver.c:697]: receive_loop(): select_recv(): There is
> something on the fd exchange pipe
> 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [receiver.c:706]: receive_loop(): select_recv(): fd exchange
> pipe says fd [22] for peer 0x7f0626b91c98:[hss.epc.mnc019
> .mcc425.3gppnetwork.org]
> 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer
> hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_Conn_Ack Event
> I_Rcv_Conn_Ack
> 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [diameter_msg.c:184]: AAANewMessage(): AAANewMessage: param
> session received null and it's a request!!
> 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer(): AAABuildMsgBuffer():
> len=204
> 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [receiver.c:994]: peer_send_msg(): peer_send_msg(): Pipe push
> [0x7f0626c02548]
> 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [receiver.c:751]: receive_loop(): select_recv(): There is
> something on the send pipe
> 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [receiver.c:764]: receive_loop(): select_recv(): Send pipe says
> [0x7f0626c02548] 8
> 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage(): AAAFreeMessage: Freeing
> message (0x7f0626c02548) 257
> 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [receiver.c:574]: do_receive(): receive_loop(): [
> hss.epc.mnc019.mcc425.3gppnetwork.org] Recv Version 1 Length 360
> 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [receiver.c:1088]: receive_message(): receive_message(): [
> hss.epc.mnc019.mcc425.3gppnetwork.org] Recv msg 257
> 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:90]: sm_process(): sm_process(): Peer
> hss.epc.mnc019.mcc425.3gppnetwork.org State Wait_I_CEA Event I_Rcv_CEA
> 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:692]: count_Supported_Vendor_Id_AVPS():
> Found 4 Supported_Vendor AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application 0: 5535
> 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 10415
> 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 13019
> 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 58637
> 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 5535
> 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 10415
> 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 13019
> 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 58637
> 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 5535
> 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 10415
> 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 13019
> 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:743]: save_peer_applications(): Found
> Supported Vendor for Application 0: 58637
> 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found
> Supported Vendor for Application 1: 5535
> 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found
> Supported Vendor for Application 1: 10415
> 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found
> Supported Vendor for Application 1: 13019
> 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]: 92(13393)
> DEBUG: cdp [peerstatemachine.c:756]: save_peer_applications(): Found
> Supported Vendor for Application 1: 58637
> 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]: 92(13393)
> CRITICAL: <core> [core/mem/q_malloc.c:145]: qm_debug_frag(): BUG: qm: prev.
> fragm. tail overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]!
> Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked
> by cdp: diameter_avp.c:142.
> 2018-01-16T09:16:41.054292 <54292>+02:00 linux-o12d kamailio[13298]:
> 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process 13393
> exited by a signal 6
> 2018-01-16T09:16:41.054403 <54403>+02:00 linux-o12d kamailio[13298]:
> 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated
> 2018-01-16T09:16:41.054422 <54422>+02:00 linux-o12d kamailio[13298]:
> 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due to
> SIGCHLD
> 2018-01-16T09:16:41.054438 <54438>+02:00 linux-o12d kamailio[13298]:
> 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to
> SIGCHLD
> 2018-01-16T09:16:41.054464 <54464>+02:00 linux-o12d kamailio[13298]:
> 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received
> ...
> ===================================
>
> Attached is wireshark trace.
> Here is my DiameterPeer.xml:
>
> ===================================
> <?xml version="1.0" encoding="UTF-8"?>
> <DiameterPeer
> FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org"
> Realm="epc.mnc019.mcc425.3gppnetwork.org"
> Vendor_Id="10415"
> Product_Name="CDiameterPeer"
> AcceptUnknownPeers="1"
> DropUnknownOnDisconnect="1"
> Tc="30"
> Workers="4"
> QueueLength="8"
> TransactionTimeout="5"
> SessionsHashSize="128"
> DefaultAuthSessionTimeout="3600"
> MaxAuthSessionTimeout="3600">
>
> <Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org" Realm="
> epc.mnc019.mcc425.3gppnetwork.org" port="3868" />
> <Acceptor port="3869" bind="10.82.10.56" />
> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX -->
> <DefaultRoute FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org"
> metric="10" />
> </DiameterPeer>
> ===================================
>
> It looks like buffer overflow to me but I'm not sure.
>
> kamailio version is:
> ===================================
> kamailio -v
> version: kamailio 5.1.0 (x86_64/linux)
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE,
> USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC,
> TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
> HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: unknown
> compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5
> ===================================
>
> OS is openSUSE Leap 42.3.
> Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4 08:11:03 UTC
> 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux
>
> Do you have any idea what might be wrong ?
> It could be bad config but still it should say something in the log
> without a crash.
>
> Regards.
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20180118/ceaae5c0/attachment.html>
More information about the sr-users
mailing list