[SR-Users] kamailio-5.2.0 Crash - udp_send failed

Mon Dec 7 13:53:37 CET 2020

Hello,

get also the output of "bt full" from gdb.

I see you use ims moudules, they are known to have some limitations for
various settings. Is this something that you can reproduce, or is
something that happens from time to time without any pattern?

Cheers,
Daniel

On 06.12.20 20:13, Hamid Hashmi wrote:
> Hello 
>
> I have updated Kamailio from 5.2.0 to 5.4.2 but the service is still
> crashing with sig 11 and sig 6. Please find the attached 'bt full' traces.
>
> Sig 11:
> |Core was generated by `/usr/local/sbin/kamailio -f
> /usr/local/etc/kamailio_pcscf/kamailio.cfg -P /var/'. Program
> terminated with signal SIGSEGV, Segmentation fault. #0
>  0x00000000006a16fc in timer_list_expire (t=1339518316,
> h=0x7facee484fd0, slow_l=0x7facee488208, slow_mark=7936) at
> core/timer.c:846 846                     _timer_rm_list(tl); /* detach
> */ (gdb) bt #0  0x00000000006a16fc in timer_list_expire (t=1339518316,
> h=0x7facee484fd0, slow_l=0x7facee488208, slow_mark=7936) at
> core/timer.c:846 #1  0x00000000006a1c19 in timer_handler () at
> core/timer.c:922 #2  0x00000000006a20ba in timer_main () at
> core/timer.c:961 #3  0x000000000042ac0a in main_loop () at main.c:1753
> #4  0x000000000043400c in main (argc=13, argv=0x7ffc256fe358) at
> main.c:2856|
> Sig 6:
> |Using host libthread_db library
> "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by
> `/usr/local/sbin/kamailio -f
> /usr/local/etc/kamailio_pcscf/kamailio.cfg -P /var/'. Program
> terminated with signal SIGABRT, Aborted. #0  0x00007fad95f56c37 in
> raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0
>  0x00007fad95f56c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007fad95f5a028 in abort () from
> /lib/x86_64-linux-gnu/libc.so.6 #2  0x000000000041e39d in
> sig_alarm_abort (signo=14) at main.c:687 #3  <signal handler called>
> #4  0x00007fad960183c7 in syscall () from
> /lib/x86_64-linux-gnu/libc.so.6 #5  0x00007fad6f5c024b in futex_get
> (lock=0x7facee70baa0) at ../../core/mem/../futexlock.h:121 #6
>  0x00007fad6f5c2167 in get_peer_by_fqdn (fqdn=0x7faceeb377b0) at
> peermanager.c:240 #7  0x00007fad6f5e6c31 in get_first_connected_route
> (cdp_session=0x7faceeb37768, r=0x7facee70a220, app_id=16777236,
> vendor_id=10415) at routing.c:110 #8  0x00007fad6f5ea8b2 in
> get_routing_peer (cdp_session=0x7faceeb37768, m=0x7facef617fb0) at
> routing.c:272 #9  0x00007fad6f5ac72d in AAASendMessage
> (message=0x7facef617fb0, callback_f=0x0, callback_param=0x0) at
> diameter_comm.c:137 #10 0x00007fad6ef05dbb in rx_send_str
> (rx_session_id=0x7facf0849d50) at rx_str.c:157 #11 0x00007fad6eeb25c4
> in callback_dialog (dlg=0x7facf0036d70, type=8192,
> params=0x7fad72b2a1c0 <params>) at ims_qos_mod.c:453 #12
> 0x00007fad728973af in run_dlg_callbacks (type=8192,
> dlg=0x7facf0036d70, req=0x0, rpl=0x0, dir=0, dlg_data=0x0) at
> dlg_cb.c:274 #13 0x00007fad728d5315 in destroy_dlg
> (dlg=0x7facf0036d70) at dlg_hash.c:219 #14 0x00007fad728d5b1b in
> destroy_dlg_table () at dlg_hash.c:291 #15 0x00007fad7290b71e in
> mod_destroy () at ims_dialog.c:627 #16 0x0000000000616ae1 in
> destroy_modules () at core/sr_module.c:746 #17 0x000000000041ce17 in
> cleanup (show_status=1) at main.c:563 #18 0x000000000041e6ef in
> shutdown_children (sig=15, show_status=1) at main.c:706 #19
> 0x0000000000421755 in handle_sigs () at main.c:806 #20
> 0x000000000042bc85 in main_loop () at main.c:1817 #21
> 0x000000000043400c in main (argc=13, argv=0x7ffc256fe358) at main.c:2856|
>
> PS: I am using ubuntu 14.04.  
>
> Regards
>
> *Hamid R. Hashmi*
>
> ------------------------------------------------------------------------
> *From:* Henning Westerholt <hw at skalatan.de>
> *Sent:* Saturday, September 26, 2020 12:04 PM
> *To:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
> *Cc:* Hamid Hashmi <hamid2kviii at hotmail.com>
> *Subject:* Re: kamailio-5.2.0 Crash - udp_send failed
>  
> Hello,
>
> you are using a old and non released dev version of kamailio. These
> are meant only for development purposes.
>
> Please update to a proper version (like the latest one based on 5.3 or
> 5.4 branch.
>
> Then try to reproduce the bug again and report on the list or by
> creating an issue.
>
> Cheers,
>
> Henning
>
>
> -- 
> Henning Westerholt - https://skalatan.de/blog/ <https://skalatan.de/blog/>
> Kamailio services - https://skalatan.de/services
> <https://skalatan.de/services>
>
> ------------------------------------------------------------------------
> *From:* sr-users <sr-users-bounces at lists.kamailio.org> on behalf of
> Hamid Hashmi <hamid2kviii at hotmail.com>
> *Sent:* Friday, September 25, 2020 9:28:07 PM
> *To:* sr-users at lists.kamailio.org <sr-users at lists.kamailio.org>
> *Subject:* [SR-Users] kamailio-5.2.0 Crash - udp_send failed
>  
> I am using "kamailio 5.2.0-dev6 (x86_64/linux) 48ed62-dirty" what
> could be the reason of the crash
>
> its usually crash with the following logs
>
> |Sep 25 16:58:41 pcscf [Proxy-CSCF][20771]: ERROR: <core>
> [core/udp_server.c:596]: udp_send():
> sendto(sock,0x7f36eee8c560,418,0,10.100.227.202:5060,16): Invalid
> argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20771]: CRITICAL:
> <core> [core/udp_server.c:601]: udp_send(): invalid
> sendtoparameters#012one possible reason is the server is bound to
> localhost and#012attempts to send to the net Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20771]: ERROR: tm [../../core/forward.h:219]:
> msg_send_buffer(): udp_send failed Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20771]: WARNING: tm [t_reply.c:1581]:
> t_retransmit_reply(): send pr buffer failed Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20756]: ERROR: <core> [core/udp_server.c:596]:
> udp_send(): sendto(sock,0x7f36ef8eb948,358,0,10.100.177.91:5060,16):
> Invalid argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20756]:
> CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid
> sendtoparameters#012one possible reason is the server is bound to
> localhost and#012attempts to send to the net Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20756]: ERROR: sl [../../core/forward.h:219]:
> msg_send_buffer(): udp_send failed Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20744]: ERROR: <core> [core/udp_server.c:596]:
> udp_send(): sendto(sock,0x7f36ef8b4a80,358,0,10.100.72.230:5060,16):
> Invalid argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20744]:
> CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid
> sendtoparameters#012one possible reason is the server is bound to
> localhost and#012attempts to send to the net Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20744]: ERROR: sl [../../core/forward.h:219]:
> msg_send_buffer(): udp_send failed Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20783]: ERROR: <core> [core/udp_server.c:596]:
> udp_send(): sendto(sock,0x7f36ef8ac740,416,0,10.100.72.230:5060,16):
> Invalid argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20783]:
> CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid
> sendtoparameters#012one possible reason is the server is bound to
> localhost and#012attempts to send to the net Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20783]: ERROR: tm [../../core/forward.h:219]:
> msg_send_buffer(): udp_send failed Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20752]: ERROR: <core> [core/udp_server.c:596]:
> udp_send(): sendto(sock,0x7f36ef8b5098,356,0,10.100.56.38:5060,16):
> Invalid argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20805]: ERROR:
> <core> [core/udp_server.c:596]: udp_send():
> sendto(sock,0x7f36ef8e8f18,416,0,10.100.177.91:5060,16): Invalid
> argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20805]: CRITICAL:
> <core> [core/udp_server.c:601]: udp_send(): invalid
> sendtoparameters#012one possible reason is the server is bound to
> localhost and#012attempts to send to the net Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20805]: ERROR: tm [../../core/forward.h:219]:
> msg_send_buffer(): udp_send failed Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20752]: CRITICAL: <core> [core/udp_server.c:601]:
> udp_send(): invalid sendtoparameters#012one possible reason is the
> server is bound to localhost and#012attempts to send to the net Sep 25
> 16:58:41 pcscf [Proxy-CSCF][20752]: ERROR: sl
> [../../core/forward.h:219]: msg_send_buffer(): udp_send failed Sep 25
> 16:58:41 pcscf [Proxy-CSCF][20777]: ERROR: <script>: Re-Transmit:
> REGISTER (sip:410060269353579 at ims.mnc006.mcc410.3gppnetwork.org
> (10.100.236.57:5060) to
> sip:410060269353579 at ims.mnc006.mcc410.3gppnetwork.org,
> 1SDq5cuY3iDYFKLATTp) Sep 25 16:58:41 pcscf [Proxy-CSCF][20761]: ERROR:
> <core> [core/udp_server.c:596]: udp_send():
> sendto(sock,0x7f36ef8e8a08,354,0,10.100.2.76:5060,16): Invalid
> argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20761]: CRITICAL:
> <core> [core/udp_server.c:601]: udp_send(): invalid
> sendtoparameters#012one possible reason is the server is bound to
> localhost and#012attempts to send to the net Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20761]: ERROR: sl [../../core/forward.h:219]:
> msg_send_buffer(): udp_send failed Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20767]: ERROR: <core> [core/udp_server.c:596]:
> udp_send(): sendto(sock,0x7f36ef8ad100,360,0,10.100.144.137:5060,16):
> Invalid argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20767]:
> CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid
> sendtoparameters#012one possible reason is the server is bound to
> localhost and#012attempts to send to the net Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20767]: ERROR: sl [../../core/forward.h:219]:
> msg_send_buffer(): udp_send failed Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20777]: ERROR: <core> [core/udp_server.c:596]:
> udp_send(): sendto(sock,0x7f36ef8eb860,360,0,10.100.140.226:5060,16):
> Invalid argument(22) Sep 25 16:58:41 pcscf [Proxy-CSCF][20777]:
> CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid
> sendtoparameters#012one possible reason is the server is bound to
> localhost and#012attempts to send to the net Sep 25 16:58:41 pcscf
> [Proxy-CSCF][20777]: ERROR: sl [../../core/forward.h:219]:
> msg_send_buffer(): udp_send failed Sep 25 16:58:43 pcscf
> [Proxy-CSCF][20861]: CRITICAL: <core> [core/pass_fd.c:277]:
> receive_fd(): EOF on 77 Sep 25 16:58:44 pcscf [Proxy-CSCF][20853]:
> ERROR: <core> [core/tcp_read.c:301]: tcp_read_data(): error reading:
> Connection timed out (110) ([10.100.64.49]:32888 -> Sep 25 16:58:44
> pcscf [Proxy-CSCF][20853]: ERROR: <core> [core/tcp_read.c:302]:
> tcp_read_data(): -> [10.161.224.228]:5060) Sep 25 16:58:44 pcscf
> [Proxy-CSCF][20853]: ERROR: <core> [core/tcp_read.c:1485]:
> tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7f36a739fe78
> r: 0x7f36a739fef8 Sep 25 16:58:44 pcscf [Proxy-CSCF][20742]: ALERT:
> <core> [main.c:748]: handle_sigs(): child process 20815 exited by a
> signal 11 Sep 25 16:58:44 pcscf [Proxy-CSCF][20742]: ALERT: <core>
> [main.c:751]: handle_sigs(): core was generated Sep 25 16:58:44 pcscf
> kernel: [52607827.581845] net_ratelimit: 16 callbacks suppressed Sep
> 25 16:58:44 pcscf [Proxy-CSCF][20742]: CRITICAL: cdp
> [diameter_peer.c:423]: diameter_peer_destroy():
> destroy_diameter_peer(): Bye Bye from C Diameter Peer test Sep 25
> 16:59:01 pcscf CRON[24805]: (root) CMD (/bin/bash
> /root/volumes/monitoring/src/services_status.sh -p
> http://10.161.224.25:9091 -s kamailio_pcscf,sems -n pcscf) Sep 25
> 16:59:01 pcscf CRON[24804]: (CRON) info (No MTA installed, discarding
> output) Sep 25 16:59:44 pcscf [Proxy-CSCF][20742]: CRITICAL: <core>
> [main.c:654]: sig_alarm_abort(): shutdown timeout triggered, dying...|
> but last time its also crashed with the following logs as well
>
> |Sep 25 18:39:59 pcscf [Proxy-CSCF][28015]: ERROR: <core>
> [core/tcp_read.c:301]: tcp_read_data(): error reading: Connection
> timed out (110) ([10.100.183.34]:39022 -> Sep 25 18:39:59 pcscf
> [Proxy-CSCF][28015]: ERROR: <core> [core/tcp_read.c:302]:
> tcp_read_data(): -> [10.161.224.228]:5060) Sep 25 18:39:59 pcscf
> [Proxy-CSCF][28015]: ERROR: <core> [core/tcp_read.c:1485]:
> tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7f92ca8d0150
> r: 0x7f92ca8d01d0 Sep 25 18:39:59 pcscf [Proxy-CSCF][28010]: ERROR:
> cdp [authstatemachine.c:404]: auth_client_statefull_sm_process():
> auth_client_statefull_sm_process(): Received invalid event 6 while in
> state Open! Sep 25 18:39:59 pcscf [Proxy-CSCF][28010]: ERROR: cdp
> [authstatemachine.c:404]: auth_client_statefull_sm_process():
> auth_client_statefull_sm_process(): Received invalid event 6 while in
> state Open! Sep 25 18:40:00 pcscf [Proxy-CSCF][28034]: CRITICAL:
> <core> [core/pass_fd.c:277]: receive_fd(): EOF on 77 Sep 25 18:40:01
> pcscf [Proxy-CSCF][28026]: ERROR: <core> [core/tcp_read.c:301]:
> tcp_read_data(): error reading: Connection timed out (110)
> ([10.100.100.218]:1337 -> Sep 25 18:40:01 pcscf [Proxy-CSCF][28026]:
> ERROR: <core> [core/tcp_read.c:302]: tcp_read_data(): ->
> [10.161.224.228]:5060) Sep 25 18:40:01 pcscf [Proxy-CSCF][28026]:
> ERROR: <core> [core/tcp_read.c:1485]: tcp_read_req(): ERROR:
> tcp_read_req: error reading - c: 0x7f92ca87f070 r: 0x7f92ca87f0f0 Sep
> 25 18:40:01 pcscf [Proxy-CSCF][27919]: ALERT: <core> [main.c:748]:
> handle_sigs(): child process 27989 exited by a signal 11 Sep 25
> 18:40:01 pcscf [Proxy-CSCF][27919]: ALERT: <core> [main.c:751]:
> handle_sigs(): core was generated Sep 25 18:40:01 pcscf kernel:
> [52613904.372973] net_ratelimit: 29 callbacks suppressed Sep 25
> 18:40:01 pcscf [Proxy-CSCF][27919]: CRITICAL: cdp
> [diameter_peer.c:423]: diameter_peer_destroy():
> destroy_diameter_peer(): Bye Bye from C Diameter Peer test Sep 25
> 18:40:01 pcscf CRON[30581]: (root) CMD (/bin/bash
> /root/volumes/monitoring/src/services_status.sh -p
> http://10.161.224.25:9091 -s kamailio_pcscf,sems -n pcscf) Sep 25
> 18:40:01 pcscf CRON[30580]: (CRON) info (No MTA installed, discarding
> output) Sep 25 18:40:02 pcscf sems[2858]: [#7f2080044700]
> [update_uac_request, trans_layer.cpp:2080] WARNING:  While sending 200
> ACK: no matching transaction Sep 25 18:41:01 pcscf
> [Proxy-CSCF][27919]: CRITICAL: <core> [main.c:654]: sig_alarm_abort():
> shutdown timeout triggered, dying...|
>
> PS: Kamailio is not listening on localhost. 
>
> Regards
>
> *Hamid R. Hashmi*
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20201207/c0982841/attachment.htm>