sr-dev July 2020

sr-dev@lists.kamailio.org

26 participants
389 discussions

[kamailio/kamailio] Deadlocks after ungracefully terminated TLS connections (#2395)

by Laszlo

This bug report concerns a Kamailio instance that's functioning as an edge proxy, accepting requests from clients over TLS and forwarding them to internal components over UDP. I've observed occasional "deadlocks" that all take place after there were connectivity issues between Kamailio and one or more of the clients (logging messages such as "tcp_send failed", "handle_tcpconn_ev(): connect x failed", "tcp_read_data(): error reading: Connection reset by peer"). Environment: * Ubuntu 16.04 * Kamailio 5.2.7 (5.2.7+ubuntu16.04) Relevant config: ``` # for TLS clients socket_workers=10 listen=tls:wan.ip:TLS_PORT # to be able to forward incoming TCP requests to UDP destinations socket_workers=10 listen=udp:wan.ip:TLS_PORT # for RPC commands socket_workers=2 listen=tcp:127.0.0.1:HTTP_PORT ``` ## Reproduction After a lot of trial and error, I've found a way to somewhat reproducibly trigger the problem using sipp. It involves setting up a large amount of simultaneous calls dialing into the edge proxy using TLS, to a destination UA that will answer the call and keep it open for a few seconds, then using `ctrl-c` to suddenly quit sipp which will ungracefully terminate the TCP/TLS connections. It takes me a couple of iterations to trigger the problem, e.g repeat until Kamailio starts failing: * start sipp * terminate sipp with ctrl-c * (quickly) restart sipp * terminate sipp * etc The amount of simultaneous calls has to be sufficiently large. In my environment I'm unable to reproduce the problem at all with 10-20 calls, while 50 calls always triggers the problem after a couple of iterations. Example sipp commandline: `sipp -nd -sf caller.xml -d 2000 -r 50 -l 500 -m 5000 -max_socket 1000 -t ln -tls_cert cert.crt -tls_key key.key kamailio.edge.proxy:5061` ## Failure mode 1 In one of the failure modes, it appears that all available TCP/TLS workers are permanently bricked; sipp's calls will simply timeout and not trigger any dialplan handling. Interestingly, even though I have dedicated workers for TLS (SIP client) and TCP (RPC commands), and my reproduction scenario doesn't connect to the HTTP_PORT at all, this failure scenario bricks the TCP (RPC) listeners as well; any curl requests to it will simply timeout as well. Kamailio will hang for a while on (systemd) restart. ## Failure mode 2 In another failure mode, it looks like the TCP workers are unaffected but the UDP perform forwarding to UDP destinations are broken. Sipp's calls will enter Kamailio, but forwarding (`relay()`) to the next hop will timeout even though the destination is functioning just fine. Since this failure mode doesn't break RPC I was able to enable debug logging: ``` 2020-07-07T17:26:58.770411+02:00 kamailio[24056]: INFO: From Customer 2020-07-07T17:26:58.770697+02:00 kamailio[24056]: DEBUG: pv [pv_trans.c:590]: tr_eval_string(): i=0 j=2 2020-07-07T17:26:58.770971+02:00 kamailio[24056]: DEBUG: dispatcher [dispatch.c:2055]: ds_manage_routes(): set [4] 2020-07-07T17:26:58.771248+02:00 kamailio[24056]: DEBUG: dispatcher [dispatch.c:2161]: ds_manage_routes(): using alg [0] hash [250849449] 2020-07-07T17:26:58.771535+02:00 kamailio[24056]: DEBUG: dispatcher [dispatch.c:2205]: ds_manage_routes(): selected [0-4-0/1] <sip:wan.ip:8065> 2020-07-07T17:26:58.771817+02:00 kamailio[24056]: DEBUG: dispatcher [dispatch.c:2226]: ds_manage_routes(): using first entry [4/1] 2020-07-07T17:26:58.772090+02:00 kamailio[24056]: DEBUG: dispatcher [dispatch.c:2269]: ds_manage_routes(): using entry [4/0] 2020-07-07T17:26:58.772362+02:00 kamailio[24056]: DEBUG: dispatcher [dispatch.c:2016]: ds_select_dst_limit(): selected target destinations: 2 2020-07-07T17:26:58.773001+02:00 kamailio[24056]: DEBUG: pv [pv_core.c:2611]: pv_set_force_sock(): trying to set send-socket to [udp:wan.ip:5061] 2020-07-07T17:26:58.773353+02:00 kamailio[24056]: DEBUG: <core> [core/socket_info.c:559]: grep_sock_info(): checking if host==us: 13==13 && [wan.ip] == [wan.ip] 2020-07-07T17:26:58.773829+02:00 kamailio[24056]: DEBUG: <core> [core/socket_info.c:566]: grep_sock_info(): checking if port 5061 (advertise 0) matches port 5061 2020-07-07T17:26:58.774122+02:00 kamailio[24056]: NOTICE: Relaying (cid: 1-9151(a)sipp.ip). 2020-07-07T17:26:58.774417+02:00 kamailio[24056]: DEBUG: siputils [checks.c:120]: has_totag(): no totag 2020-07-07T17:26:58.774708+02:00 kamailio[24056]: DEBUG: tm [t_lookup.c:1344]: t_newtran(): msg (0x7f69d1cc4980) id=1265/24056 global id=1265/24056 T start=(nil) 2020-07-07T17:26:58.774989+02:00 kamailio[24056]: DEBUG: tm [t_lookup.c:499]: t_lookup_request(): start searching: hash=43232, isACK=0 2020-07-07T17:26:58.775315+02:00 kamailio[24056]: DEBUG: tm [t_lookup.c:457]: matching_3261(): RFC3261 transaction matching failed - via branch [z9hG4bK-9151-1-1] 2020-07-07T17:26:58.775612+02:00 kamailio[24056]: DEBUG: tm [t_lookup.c:682]: t_lookup_request(): no transaction found 2020-07-07T17:26:58.775908+02:00 kamailio[24056]: DEBUG: <core> [core/md5utils.c:67]: MD5StringArray(): MD5 calculated: 0a5d4a8d13f9606523da9c4d193b2261 2020-07-07T17:26:58.776270+02:00 kamailio[24056]: DEBUG: <core> [core/dns_cache.c:3197]: dns_srv_sip_resolve(): (wan.ip, 0, 0), ip, ret=0 2020-07-07T17:26:58.776701+02:00 kamailio[24056]: DEBUG: <core> [core/msg_translator.c:1765]: check_boundaries(): no multi-part body 2020-07-07T17:26:58.777012+02:00 kamailio[24056]: DEBUG: <core> [core/msg_translator.c:2935]: create_via_hf(): id added: <;i=a701>, rcv proto=3 2020-07-07T17:26:58.777596+02:00 kamailio[24056]: DEBUG: tm [t_funcs.c:375]: t_relay_to(): new transaction forwarded 2020-07-07T17:26:58.777948+02:00 kamailio[24056]: DEBUG: <core> [core/receive.c:354]: receive_msg(): request-route executed in: 1927 usec 2020-07-07T17:26:58.778243+02:00 kamailio[24056]: DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 2020-07-07T17:26:58.779265+02:00 kamailio[24056]: message repeated 5 times: [ DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil)] 2020-07-07T17:26:58.779431+02:00 kamailio[24056]: DEBUG: <core> [core/xavp.c:507]: xavp_destroy_list(): destroying xavp list (nil) 2020-07-07T17:26:58.779753+02:00 kamailio[24056]: DEBUG: <core> [core/receive.c:458]: receive_msg(): cleaning up 2020-07-07T17:27:00.705303+02:00 kamailio[24053]: DEBUG: tm [t_reply.c:1266]: t_should_relay_response(): ->>>>>>>>> T_code=0, new_code=408 2020-07-07T17:27:00.705917+02:00 kamailio[24053]: DEBUG: tm [t_lookup.c:1041]: t_check_msg(): msg (0x7f69b1017e80) id=1264/24056 global id=1264/24056 T start=0x7f69b10167c0 2020-07-07T17:27:00.706187+02:00 kamailio[24053]: DEBUG: tm [t_lookup.c:1116]: t_check_msg(): T (0x7f69b10167c0) already found for msg (0x7f69b1017e80)! 2020-07-07T17:27:00.706533+02:00 kamailio[24053]: DEBUG: tmx [t_var.c:539]: pv_get_tm_reply_code(): reply code is <408> 2020-07-07T17:27:00.706879+02:00 kamailio[24053]: DEBUG: tm [t_lookup.c:1041]: t_check_msg(): msg (0x7f69b1017e80) id=1264/24056 global id=1264/24056 T start=0x7f69b10167c0 2020-07-07T17:27:00.707198+02:00 kamailio[24053]: DEBUG: tm [t_lookup.c:1116]: t_check_msg(): T (0x7f69b10167c0) already found for msg (0x7f69b1017e80)! 2020-07-07T17:27:00.707522+02:00 kamailio[24053]: DEBUG: tmx [t_var.c:594]: pv_get_tm_reply_reason(): reply reason is [Request Timeout] { "DSC": "tcp receiver (tls:wan.ip:5061) child=0", "IDX": 14, "PID": 24056 }, { "DSC": "slow timer", "IDX": 11, "PID": 24053 }, ``` Kamailio will restart (systemd) normally and everything will function normally again afterwards. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2395

3 years, 4 months

[kamailio/kamailio] core: main - Protect async-signal-unsafe code in sig_usr() with SIG_DEBUG (#2127)

by Guillem Jover

#### Pre-Submission Checklist - [ ] Commit message has the format required by CONTRIBUTING guide - [ ] Commits are split per component (core, individual modules, libs, utils, ...) - [ ] Each component has a single commit (if not, squash them into one commit) - [ ] No commits to README files for modules (changes must be done to docbook files in `doc/` subfolder, the README file is autogenerated) #### Type Of Change - [ ] Small bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds new functionality) - [ ] Breaking change (fix or feature that would change existing functionality) #### Checklist: - [ ] PR should be backported to stable branches - [ ] Tested changes locally - [ ] Related to issue #XXXX (replace XXXX with an open issue number) #### Description The sig_usr() signal handler contains debug code to print memory usage and statistics that is not async-signal-safe, which can cause crashes or misbehavior. This code which only affects explicit SIGUSR1 signals on children processes, and is not guarded with the SIG_DEBUG macro. You can view, comment on, or merge this pull request online at: https://github.com/kamailio/kamailio/pull/2127 -- Commit Summary -- * core: main - Protect async-signal-unsafe code in sig_usr() with SIG_DEBUG -- File Changes -- M src/main.c (2) -- Patch Links -- https://github.com/kamailio/kamailio/pull/2127.patch https://github.com/kamailio/kamailio/pull/2127.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/pull/2127

3 years, 4 months

[kamailio/kamailio] Expand the list of required default 3rd party libraries for Kamailio (#969)

by Olle E. Johansson

(from devmeeting #Fosdem) - adding new dependencies as core dependencies - Curl - libssl - uuid - libunistring - libxml2 Any other candidates? They have to be libraries that are commonly installed on most Linux/Unix system. This would mean that a lot of more modules will be built by default, like TLS, HTTP_CLIENT, WEBSOCKET. We may also look over the names of the packages/groups for the build system and linux distros -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/969

3 years, 4 months

[kamailio/kamailio] topos module seems to break re-SUBSCRIBE routing to downstream destinations (#1724)

by George Diamantopoulos

### Description Kamailio seems to fail to properly route in-dialog SUBSCRIBE with the topos module enabled. In this configuration, SUBSCRIBEs are expected to be delivered to a downstream UAS, an asterisk instance in the internal network. record_route() is executed for all initial requests, including SUBSCRIBEs. Routing is performed correctly with the topos module disabled, no other modifications were needed.  ### Troubleshooting I haven't been able to work around this issue in any other way other than disabling the topos module. #### Reproduction The configuration used is a heavily modified version of the example configuration file, but the handling of SUBSCRIBEs is generally simple. The initial SUBSCRIBE is subject to some tests, the R-URI is edited and dispatched to a farm of asterisk servers based on some criteria. Record Routing is enforced, so subsequent in-dialog re-SUBSCRIBEs are expected to be routed with loose_route(), which doesn't seem to handle things properly.  #### Debugging Data I'm attaching kamailio logs with debug=4, cfg_trace  #### Log Messages See attachments  #### SIP Traffic Please see: https://www.cloudshark.org/captures/d2643aae7f2a  ### Possible Solutions Using topoh instead for topos, but unfortunately this will break communication for other reasons (messages too large/UDP fragmentation for some peers). ### Additional Information * **Kamailio Version** Tried with kamailio 5.1.6 and kamailio 5.3.0-dev0 deb.kamailio.org repositories were used to acquire binaries for both versions * **Operating System**: Debian Stretch  [resubscribe.log](https://github.com/kamailio/kamailio/files/2586440/resubsc… -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1724

3 years, 5 months

[kamailio/kamailio] uri_restore_rcv_alias() tried twice for local request (#2014)

by Alex Hermann

### Description While debugging an issue with locally generated in-dialog requests I stumbled upon some weirdness with the alias functionality from the nathelper module. The dialog modules tries to handle an alias parameter when sending an in-dialog request by calling ```uri_restore_rcv_alias()``` in the function ```build_dlg_t()```. A few instructions later, the same is attempted by the tm module in the function ```req_within()```. It seems to me that the attempt by the dialog module is redundant and can be removed. And while at it, IMHO it should be removed from the tm module too. This is functionality introduced by the nathelper module. Nowhere in the documentation it is mentioned there is some automatic handling of the alias parameter. If handling is needed for local requests, the script writer can call ```handle_uri_alias()``` from within local_route. If the functionality has to stay, may I suggest that the code in the nathelper module makes use of the ```uri_restore_rcv_alias()``` function instead of having its own duplicate implementation. Less maintenance, easier debugging. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2014

3 years, 5 months

[kamailio/kamailio] rtpengine: stats for rtpengine command and reply from rtpengine (#1699)

by Surendra Tiwari

### Description rightnow, there is no stats about command sent to rtpengine and getting reply from rtpengine. There should be a stats that will tell that number of offer request sent: number of answer request sent: number of delete request sent: number of failed request for offer [after retry]: number of failed request for answer [after retry]: number of failed request for delete [after retry]: This will help to identify actual number of failed request.  ### Troubleshooting #### Reproduction  #### Debugging Data  ``` (paste your debugging data here) ``` #### Log Messages  ``` (paste your log messages here) ``` #### SIP Traffic  ``` (paste your sip traffic here) ``` ### Possible Solutions  ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` (paste your output here) ``` * **Operating System**:  ``` (paste your output here) ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1699

3 years, 5 months

[kamailio/kamailio] 100% CPU usage after NTP date update (#2046)

by Benjamin Marty

#### Reproduction Install newest Kamailio 5.2. Change date e.g. 2 Years into the future. (Make sure you have no daemon running which prohibits those operations): ``` sudo date -s "2014-12-25 $(date +%H:%M:%S)" ``` ``` sudo date -s "2016-12-25 $(date +%H:%M:%S)" ``` Kamailio now uses the whole CPU to do stuff: ``` top - 08:10:35 up 14 days, 1:29, 2 users, load average: 3.77, 1.83, 1.05 Tasks: 212 total, 11 running, 201 sleeping, 0 stopped, 0 zombie %Cpu(s): 95.9 us, 1.5 sy, 2.2 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st MiB Mem : 926.1 total, 116.3 free, 382.5 used, 427.3 buff/cache MiB Swap: 100.0 total, 40.4 free, 59.6 used. 423.4 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 13264 root 20 0 89720 3944 2676 R 44.9 0.4 0:05.15 kamailio 13266 root 20 0 89720 4000 2736 R 37.3 0.4 0:04.63 kamailio 13267 root 20 0 89720 4912 3488 R 36.0 0.5 0:04.37 kamailio 13270 root 20 0 89720 4540 3240 R 36.0 0.5 0:05.85 kamailio 13269 root 20 0 89720 4000 2736 R 34.7 0.4 0:04.81 kamailio 13265 root 20 0 89720 3944 2680 R 34.4 0.4 0:04.61 kamailio 13261 root 20 0 89720 3952 2676 R 32.2 0.4 0:04.44 kamailio 13262 root 20 0 89720 3944 2676 R 31.8 0.4 0:04.72 kamailio 13256 root 20 0 89720 4752 3540 R 30.3 0.5 0:07.04 kamailio 13268 root 20 0 89720 4000 2736 R 30.3 0.4 0:04.49 kamailio 18021 root 20 0 40116 11884 5728 D 13.7 1.3 0:01.11 apt-get ``` There are multiple mailing list entries regarding this issue: https://lists.kamailio.org/pipermail/sr-users/2015-October/090465.html https://lists.kamailio.org/pipermail/sr-users/2015-November/090487.html https://lists.kamailio.org/pipermail/sr-users/2019-June/105967.html https://lists.kamailio.org/pipermail/sr-users/2019-August/106428.html I was able to get rid of the issue with uncommenting the mentioned part of the code. But I'm quite sure that's not a fix :) ![image](https://user-images.githubusercontent.com/11159719/63680341-b6d67880-c7f3-11e9-825b-529e7ec26b2e.png) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2046

3 years, 5 months

[kamailio/kamailio] commit 0436af5abd8b73e breaks startup for jsonrpcs module (#2391)

by Henning Westerholt

The commit 0436af5abd8b73e breaks startup for jsonrpcs module. Logs: ``` Jul 5 10:04:41 kam01 kamailio[4951]: ERROR: jsonrpcs [jsonrpcs_sock.c:358]: jsonrpc_dgram_init_server(): failed to change the owner/group for /var/run/kamailio/kamailio_rpc.sock to 33.0;Operation not permitted[1] Jul 5 10:04:41 kam01 kamailio[4951]: CRITICAL: jsonrpcs [jsonrpcs_sock.c:423]: jsonrpc_dgram_init_socks(): initializing datagram server function returned error Jul 5 10:04:41 kam01 kamailio[4951]: ERROR: jsonrpcs [jsonrpcs_sock.c:246]: jsonrpc_dgram_mod_init(): init datagram sockets function failed Jul 5 10:04:41 kam01 kamailio[4951]: ERROR: jsonrpcs [jsonrpcs_mod.c:1150]: mod_init(): cannot initialize datagram transport Jul 5 10:04:41 kam01 kamailio[4951]: ERROR: <core> [core/sr_module.c:849]: init_mod(): Error while initializing module jsonrpcs (/usr/lib/x86_64-linux-gnu/kamailio/modules/jsonrpcs.so) ``` Relevant cfg: ``` loadmodule "jsonrpcs.so" modparam("jsonrpcs", "pretty_format", 1) modparam("jsonrpcs", "dgram_user", "www-data") modparam("jsonrpcs", "dgram_group", "root") modparam("jsonrpcs", "transport", 7) ``` Previously Kamailio was started as root, and then configured with -u $USER and -g $GROUP to drop privileges to the lower user/group. During startup Kamailio could create the socket with the correct permissions. The mentioned commit changes this to: start Kamailio directly as user Kamailio and group Kamailio, which don't have privileges create the sockets with the correct permissions. Not sure about the motivation about this change, so I don't want to simple revert it. Maybe @sergey-safarov or other developers can comment. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2391

3 years, 5 months

[kamailio/kamailio] add rtpproxy in kamailio/kamailio-ci:master-alpine (#2109)

by Gaurav Bmotra

hi can any one help me to add rtpproxy in kamailio/kamailio-ci:master-alpine thank you -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2109

3 years, 5 months

[kamailio/kamailio] Can't install kamailio-sipcapture-daemon-config package due to missing dependency (#1861)

by pwakano

kamailio-sipcapture-daemon-config can't be installed because it depends on the kamailio-sipcapture package, which can't be found...  Use the http://download.opensuse.org/repositories/home:/kamailio:/v5.2.x-rpms/CentO… repo and try to install the kamailio-sipcapture-daemon-config. The dependency can't be resolved and so the package does not install. It is not clear if the dependency is missing or if it is not valid. ``` # yum install kamailio-sipcapture-daemon-config Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.optus.net * epel: mirror.optus.net * extras: mirror.intergrid.com.au * updates: mirror.intergrid.com.au Resolving Dependencies --> Running transaction check ---> Package kamailio-sipcapture-daemon-config.x86_64 0:5.2.1-1.el7.centos will be installed --> Processing Dependency: kamailio-sipcapture = 5.2.1 for package: kamailio-sipcapture-daemon-config-5.2.1-1.el7.centos.x86_64 --> Finished Dependency Resolution Error: Package: kamailio-sipcapture-daemon-config-5.2.1-1.el7.centos.x86_64 (home_kamailio_v5.2.x-rpms) Requires: kamailio-sipcapture = 5.2.1 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest # yum deplist kamailio-sipcapture-daemon-config Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * base: mirror.optus.net * epel: mirror.optus.net * extras: mirror.intergrid.com.au * updates: mirror.intergrid.com.au package: kamailio-sipcapture-daemon-config.x86_64 5.2.1-1.el7.centos dependency: /bin/sh provider: bash.x86_64 4.2.46-31.el7 dependency: kamailio-sipcapture = 5.2.1 Unsatisfied dependency ```      * **Operating System**:  ``` # uname -a Linux voice-conn-kamailio-1 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/redhat-release CentOS Linux release 7.6.1810 (Core) ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1861

3 years, 6 months

← Newer
1
...
5
6
7
8
9
10
11
...
39
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

sr-dev July 2020