[SR-Users] Coredump while exchanging diameter capabilities
Tsvetan Filev
tsvetan.filev at inno-networks.com
Wed Jan 17 09:49:14 CET 2018
Hi Rick.
Yes I have coredumps and backtraces but I installed the rpms from here
https://www.kamailio.org/wiki/packages/rpms
(http://download.opensuse.org/repositories/home:/kamailio:/v5.1.x-rpms/openSUSE_Leap_42.3/
) so I'm missing the debug symbols.
Can I find the kamailio debug rpm (kamailio-debuginfo-5.1.0-21.7.x86_64)
somewhere ?
Once I have the symbols I will create an issue on GitHub.
Here is what I get for now:
# gdb kamailio
core.kamailio.479.e6ada9851c9848a98237890cfddfafe3.6560.1516177422000000
GNU gdb (GDB; openSUSE Leap 42.3) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kamailio...(no debugging symbols found)...done.
[New LWP 6560]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamai'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f637dfb38c7 in __GI_raise (sig=sig at entry=6) at
../sysdeps/unix/sysv/linux/raise.c:55
55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
Missing separate debuginfos, use: zypper install
kamailio-debuginfo-5.1.0-21.7.x86_64
(gdb) bt full
#0 0x00007f637dfb38c7 in __GI_raise (sig=sig at entry=6) at
../sysdeps/unix/sysv/linux/raise.c:55
resultvar = 0
pid = 6560
selftid = 6560
#1 0x00007f637dfb4c9a in __GI_abort () at abort.c:78
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x28af01000000,
sa_sigaction = 0x28af01000000}, sa_mask = {__val = {140724448160064, 0,
140065300084096, 6721723, 140065147406864, 140065144188928,
140065137572536, 140065144419712, 21474836483, 140065144188928,
9019276028016, 4, 0, 234976, 4, 8177296}}, sa_flags =
2123556608, sa_restorer = 0x0}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x000000000066906e in qm_debug_frag ()
No symbol table info available.
#3 0x000000000066beaf in qm_free ()
No symbol table info available.
#4 0x00000000006755f5 in qm_shm_free ()
No symbol table info available.
#5 0x00007f6374c38a1d in AAAFreeAVP () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#6 0x00007f6374c144ac in AAAFreeAVPList () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#7 0x00007f6374beb0f9 in save_peer_applications () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#8 0x00007f6374beb16d in Process_CEA () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#9 0x00007f6374be4a22 in sm_process () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#10 0x00007f6374bd15d6 in receive_message () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#11 0x00007f6374bc7032 in do_receive () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#12 0x00007f6374bcae19 in receive_loop () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#13 0x00007f6374bc4563 in receiver_process () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#14 0x00007f6374be0c8c in diameter_peer_start () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#15 0x00007f6374bd2e5f in cdp_child_init () from
/usr/lib64/kamailio/modules/cdp.so
No symbol table info available.
#16 0x000000000054b705 in init_mod_child ()
No symbol table info available.
#17 0x000000000054b422 in init_mod_child ()
No symbol table info available.
#18 0x000000000054b422 in init_mod_child ()
No symbol table info available.
#19 0x000000000054b422 in init_mod_child ()
No symbol table info available.
#20 0x000000000054b422 in init_mod_child ()
No symbol table info available.
#21 0x000000000054ba3d in init_child ()
No symbol table info available.
#22 0x0000000000426925 in main_loop ()
No symbol table info available.
#23 0x000000000042d21e in main ()
No symbol table info available.
(gdb) info locals
save_stage = 1
\act = {__sigaction_handler = {sa_handler = 0x28af01000000, sa_sigaction
= 0x28af01000000}, sa_mask = {__val = {140724448160064, 0,
140065300084096, 6721723, 140065147406864, 140065144188928,
140065137572536, 140065144419712, 21474836483, 140065144188928,
9019276028016,
4, 0, 234976, 4, 8177296}}, sa_flags = 2123556608, sa_restorer = 0x0}
sigs = {__val = {32, 0 <repeats 15 times>}}
(gdb) list
50 in ../sysdeps/unix/sysv/linux/raise.c
Regards.
On 16.01.2018 19:51, Rick Barenthin wrote:
> Hey,
>
> looks like an error with memory allocation. If you haven't done yet,
> it would be nice if you could create an issue on GitHub. Do you still
> have the coredump? If so please create an trace with gdb and attache
> it to the issue.
>
> Greetings Rick
>
> Am 16.01.2018 09:50 schrieb "Tsvetan Filev"
> <tsvetan.filev at inno-networks.com
> <mailto:tsvetan.filev at inno-networks.com>>:
>
> Hi.
>
> I get a coredump which is caused by bad memory handling during the
> diameter capability exchange process.
> Here is part of the log file:
> ===================================
> 2018-01-16T09:16:39.890992+02:00 linux-o12d kamailio[13298]:
> 87(13388) INFO: cdp [worker.c:332]: worker_process(): [0] Worker
> process started...
> 2018-01-16T09:16:40.296393+02:00 linux-o12d
> systemd-coredump[13196]: Process 13193 (kamailio) of user 479
> dumped core.
> 2018-01-16T09:16:40.871483+02:00 linux-o12d kamailio[13298]:
> 94(13395) DEBUG: cdp [peermanager.c:263]: peer_timer():
> peer_timer(): taking care of peers...
> 2018-01-16T09:16:40.871545+02:00 linux-o12d kamailio[13298]:
> 94(13395) DEBUG: cdp [peermanager.c:280]: peer_timer():
> peer_timer(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org> State 0
> 2018-01-16T09:16:40.871596+02:00 linux-o12d kamailio[13298]:
> 94(13395) DEBUG: cdp [peerstatemachine.c:90]: sm_process():
> sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org> State Closed Event
> Start
> 2018-01-16T09:16:40.871612+02:00 linux-o12d kamailio[13298]:
> 94(13395) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req():
> I_Snd_Conn_Req(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org>
> 2018-01-16T09:16:40.871636+02:00 linux-o12d kamailio[13298]:
> 94(13395) INFO: cdp [receiver.c:869]: peer_connect():
> peer_connect(): Trying to connect to 10.82.10.85 port 3868
> 2018-01-16T09:16:40.871782+02:00 linux-o12d kamailio[13298]:
> 94(13395) INFO: cdp [receiver.c:937]: peer_connect():
> peer_connect(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org:3868
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org:3868> connected
> 2018-01-16T09:16:40.871813+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [receiver.c:697]: receive_loop():
> select_recv(): There is something on the fd exchange pipe
> 2018-01-16T09:16:40.871828+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [receiver.c:706]: receive_loop():
> select_recv(): fd exchange pipe says fd [22] for peer
> 0x7f0626b91c98:[hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org>]
> 2018-01-16T09:16:40.871910+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process():
> sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org> State Wait_Conn_Ack
> Event I_Rcv_Conn_Ack
> 2018-01-16T09:16:40.871933+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [diameter_msg.c:184]: AAANewMessage():
> AAANewMessage: param session received null and it's a request!!
> 2018-01-16T09:16:40.872011+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [diameter_msg.c:81]: AAABuildMsgBuffer():
> AAABuildMsgBuffer(): len=204
> 2018-01-16T09:16:40.872041+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [receiver.c:994]: peer_send_msg():
> peer_send_msg(): Pipe push [0x7f0626c02548]
> 2018-01-16T09:16:40.872064+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [receiver.c:751]: receive_loop():
> select_recv(): There is something on the send pipe
> 2018-01-16T09:16:40.872084+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [receiver.c:764]: receive_loop():
> select_recv(): Send pipe says [0x7f0626c02548] 8
> 2018-01-16T09:16:40.872104+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [diameter_msg.c:410]: AAAFreeMessage():
> AAAFreeMessage: Freeing message (0x7f0626c02548) 257
> 2018-01-16T09:16:40.872277+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [receiver.c:574]: do_receive():
> receive_loop(): [hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org>] Recv Version 1
> Length 360
> 2018-01-16T09:16:40.872322+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [receiver.c:1088]: receive_message():
> receive_message(): [hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org>] Recv msg 257
> 2018-01-16T09:16:40.872345+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:90]: sm_process():
> sm_process(): Peer hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org> State Wait_I_CEA
> Event I_Rcv_CEA
> 2018-01-16T09:16:40.872372+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:692]:
> count_Supported_Vendor_Id_AVPS(): Found 4 Supported_Vendor
> AVPS92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 5535
> 2018-01-16T09:16:40.872389+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 10415
> 2018-01-16T09:16:40.872405+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 13019
> 2018-01-16T09:16:40.872420+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 58637
> 2018-01-16T09:16:40.872438+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 5535
> 2018-01-16T09:16:40.872453+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 10415
> 2018-01-16T09:16:40.872468+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 13019
> 2018-01-16T09:16:40.872486+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 58637
> 2018-01-16T09:16:40.872504+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 5535
> 2018-01-16T09:16:40.872523+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 10415
> 2018-01-16T09:16:40.872539+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 13019
> 2018-01-16T09:16:40.872554+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:743]:
> save_peer_applications(): Found Supported Vendor for Application
> 0: 58637
> 2018-01-16T09:16:40.872570+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:756]:
> save_peer_applications(): Found Supported Vendor for Application
> 1: 5535
> 2018-01-16T09:16:40.872586+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:756]:
> save_peer_applications(): Found Supported Vendor for Application
> 1: 10415
> 2018-01-16T09:16:40.872601+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:756]:
> save_peer_applications(): Found Supported Vendor for Application
> 1: 13019
> 2018-01-16T09:16:40.872616+02:00 linux-o12d kamailio[13298]:
> 92(13393) DEBUG: cdp [peerstatemachine.c:756]:
> save_peer_applications(): Found Supported Vendor for Application
> 1: 58637
> 2018-01-16T09:16:40.872634+02:00 linux-o12d kamailio[13298]:
> 92(13393) CRITICAL: <core> [core/mem/q_malloc.c:145]:
> qm_debug_frag(): BUG: qm: prev. fragm. tail
> overwritten(28af01000000, 0)[0x7f0626c038d0:0x7f0626c03908]!
> Memory allocator was called from cdp: diameter_avp.c:365. Fragment
> marked by cdp: diameter_avp.c:142.
> 2018-01-16T09:16:41.054292+02:00 linux-o12d kamailio[13298]:
> 0(13298) ALERT: <core> [main.c:746]: handle_sigs(): child process
> 13393 exited by a signal 6
> 2018-01-16T09:16:41.054403+02:00 linux-o12d kamailio[13298]:
> 0(13298) ALERT: <core> [main.c:749]: handle_sigs(): core was generated
> 2018-01-16T09:16:41.054422+02:00 linux-o12d kamailio[13298]:
> 0(13298) INFO: <core> [main.c:771]: handle_sigs(): terminating due
> to SIGCHLD
> 2018-01-16T09:16:41.054438+02:00 linux-o12d kamailio[13298]:
> 0(13298) DEBUG: <core> [main.c:773]: handle_sigs(): terminating
> due to SIGCHLD
> 2018-01-16T09:16:41.054464+02:00 linux-o12d kamailio[13298]:
> 2(13303) INFO: <core> [main.c:826]: sig_usr(): signal 15 received
> ...
> ===================================
>
> Attached is wireshark trace.
> Here is my DiameterPeer.xml:
>
> ===================================
> <?xml version="1.0" encoding="UTF-8"?>
> <DiameterPeer
> FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org
> <http://ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org>"
> Realm="epc.mnc019.mcc425.3gppnetwork.org
> <http://epc.mnc019.mcc425.3gppnetwork.org>"
> Vendor_Id="10415"
> Product_Name="CDiameterPeer"
> AcceptUnknownPeers="1"
> DropUnknownOnDisconnect="1"
> Tc="30"
> Workers="4"
> QueueLength="8"
> TransactionTimeout="5"
> SessionsHashSize="128"
> DefaultAuthSessionTimeout="3600"
> MaxAuthSessionTimeout="3600">
>
> <Peer FQDN="hss.epc.mnc019.mcc425.3gppnetwork.org
> <http://hss.epc.mnc019.mcc425.3gppnetwork.org>"
> Realm="epc.mnc019.mcc425.3gppnetwork.org
> <http://epc.mnc019.mcc425.3gppnetwork.org>" port="3868" />
> <Acceptor port="3869" bind="10.82.10.56" />
> <Auth id="16777216" vendor="10415" /> <!--3GPP CxDX -->
> <DefaultRoute
> FQDN="ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org
> <http://ims110-scscf.epc.mnc019.mcc425.3gppnetwork.org>"
> metric="10" />
> </DiameterPeer>
> ===================================
>
> It looks like buffer overflow to me but I'm not sure.
>
> kamailio version is:
> ===================================
> kamailio -v
> version: kamailio 5.1.0 (x86_64/linux)
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
> USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
> 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: unknown
> compiled on 05:30:36 Jan 15 2018 with gcc 4.8.5
> ===================================
>
> OS is openSUSE Leap 42.3.
> Kernel: Linux linux-o12d 4.4.104-39-default #1 SMP Thu Jan 4
> 08:11:03 UTC 2018 (7db1912) x86_64 x86_64 x86_64 GNU/Linux
>
> Do you have any idea what might be wrong ?
> It could be bad config but still it should say something in the
> log without a crash.
>
> Regards.
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
>
>
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20180117/54dc44bd/attachment.html>
More information about the sr-users
mailing list