<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description
Kamailio is unable to create a SCTP association to IP address that is reachable normally when dispatcher.list is configured as following: ``` 760320003 sip:10.59.144.2:5063;transport=sctp 9 0 dstid=1 760320003 sip:10.59.144.130:5063;transport=sctp 9 0 dstid=1 ``` 10.59.144.2 and 10.59.144.130 are IPs of same endpoint treated as primary and secondary, Kamailio can't send probe (OPTIONS requests) as SCTP association can't be created.
Issue doesn't exist on RHEL/Centos 6.
### Troubleshooting
#### Reproduction
Use previously mentioned dispatcher list with changed IP addresses to match local configuration.
SCTP part of kamailio.cfg
``` # if not OS default is used #modparam("sctp", "sctp_socket_rcvbuf", 14096) #modparam("sctp", "sctp_socket_sndbuf", 14096) # Number of milliseconds before an unsent message/chunk is dropped. modparam("sctp", "sctp_send_ttl", 10000) modparam("sctp", "sctp_autoclose", 300) # How many times to attempt re-sending a message on a re-opened association, if the sctp stack did give up sending it (it's not related to sctp protocol level retransmission). Useful to improve reliability with peers that reboot/restart or fail over to another machine. modparam("sctp", "sctp_send_retries", 1) modparam("sctp", "sctp_srto_initial", 200) modparam("sctp", "sctp_srto_max", 250) modparam("sctp", "sctp_srto_min", 150) modparam("sctp", "sctp_asocmaxrxt", 4) modparam("sctp", "sctp_init_max_attempts", 3) modparam("sctp", "sctp_init_max_timeo", 1000) modparam("sctp", "sctp_hbinterval", 1000) modparam("sctp", "sctp_pathmaxrxt", 2) modparam("sctp", "sctp_sack_delay", 10) modparam("sctp", "sctp_sack_freq", 1) modparam("sctp", "sctp_max_burst", 4) ```
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` (paste your debugging data here) ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: dispatcher [dispatch.c:3234]: ds_ping_result_helper(): probing set, but not mode DS_PROBE_INACTIVE Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: dispatcher [dispatch.c:3268]: ds_ping_set(): probing set #760320003, URI sip:10.59.144.2:5063;transport=sctp Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: dispatcher [dispatch.c:3293]: ds_ping_set(): Default ping_from: sip:siplb@sdp.t-mobile.at Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: tm [uac.c:435]: t_uac_prepare(): next_hop=sip:10.59.144.2:5063;transport=sctp Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: <core> [core/resolve.c:1244]: srv_sip_resolvehost(): 10.59.144.2:5063 proto=4 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2281]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.130:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.130:5063: SCTP_ADDR_ADDED: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: <core> [core/resolve.c:1356]: srv_sip_resolvehost(): returning 0xa7c280 (10.59.144.2:5063 proto=4) Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: tm [uac.c:158]: dlg2hash(): hashid 27287 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2281]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.130:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.130:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2280]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.130:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.130:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: tm [uac.c:652]: send_prepared_request_impl(): uac: 0x7fbf4eb4e5b8 branch: 0 to 10.59.144.2:5063 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2294]: DEBUG: tm [../../core/onsend.h:69]: run_onsend(): required parameters are not available - ignoring Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2282]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.2:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.2:5063: SCTP_ADDR_ADDED: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2278]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.2:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.2:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2282]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.2:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.2:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2285]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.2:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.2:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2283]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.130:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.130:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2279]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.130:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.130:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:13 sdp-lbref02 /usr/sbin/kamailio[2279]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.2:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.2:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:14 sdp-lbref02 /usr/sbin/kamailio[2280]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.2:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.2:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:14 sdp-lbref02 /usr/sbin/kamailio[2278]: DEBUG: sctp [sctp_server.c:2317]: sctp_handle_notification(): sctp notification from 10.59.144.130:5063 on sdp-lbref02-sigtran01:5060: SCTP_PEER_ADDR_CHANGE: 10.59.144.130:5063: SCTP_ADDR_MADE_PRIM: assoc_id 0 Aug 6 18:19:15 sdp-lbref02 /usr/sbin/kamailio[2285]: DEBUG: sctp [sctp_server.c:2340]: sctp_handle_notification(): sctp notification from 10.59.144.130:5063 on sdp-lbref02-sigtran01:5060: SCTP_ASSOC_CHANGE: SCTP_CANT_STR_ASSOC: assoc_id 0, ostreams 0, istreams 0 Aug 6 18:19:15 sdp-lbref02 /usr/sbin/kamailio[2282]: DEBUG: sctp [sctp_server.c:2340]: sctp_handle_notification(): sctp notification from 10.59.144.2:5063 on sdp-lbref02-sigtran01:5060: SCTP_ASSOC_CHANGE: SCTP_CANT_STR_ASSOC: assoc_id 0, ostreams 0, istreams 0 Aug 6 18:19:15 sdp-lbref02 /usr/sbin/kamailio[2285]: DEBUG: sctp [sctp_server.c:2299]: sctp_handle_notification(): sctp notification from 10.59.144.2:5063 on sdp-lbref02-sigtran01:5060: SCTP_SEND_FAILED: error 0, assoc_id 0, flags 0 Aug 6 18:19:15 sdp-lbref02 /usr/sbin/kamailio[2285]: DEBUG: sctp [sctp_server.c:2098]: sctp_handle_send_failed(): sctp: RETRY-ing (1) ```
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` (paste your sip traffic here) ```
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
Issue is gone when one of IP addresses is removed from dispatcher.list.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` kamailio -v version: kamailio 5.3.5 (x86_64/linux) 9e70e8 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 9e70e8 compiled on 12:22:26 Jun 22 2020 with gcc 8.3.1
```
Also tested latest 5.2 and 5.4
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) -->
``` [root@sdp-lbref02 ~]# uname -a Linux sdp-lbref02 4.18.0-193.14.3.el8_2.x86_64 #1 SMP Mon Jul 20 15:02:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [root@sdp-lbref02 ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.2 (Ootpa) ```
I don't have any sctp testbed, but since this popped up with an OS upgrade, likely there are some changes in the sctp lib api/behaviour that require the module to be enhanced to follow them.
For whoever is going to work on this enhancement, here are some useful comments made by @fredposner on matrix chat when discussing this:
``` centos 8 made a lot of changes... I know there's a different method in which the OS itself handles SCTP.
> Blacklisting SCTP module by default in RHEL 8
To increase security, a set of kernel modules have been moved to the kernel-modules-extra package. These are not installed by default. As a consequence, non-root users cannot load these components as they are blacklisted by default. To use one of these kernel modules, the system administrator must install kernel-modules-extra and explicitly remove the module blacklist. As a result, non-root users will be able to load the software component automatically. ```
FYI we had issue with other application using SCTP on RHEL 8.2 the result it that there will be a fix in Linux kernel shipped with RHEL.
https://github.com/netty/netty/issues/10455
@pgrabowski do I understand it correctly that it is an issue related to a change in RHEL? So this issue could probably be closed on our side then?
Looks like a kernel downgrade resolve issue. Also, RHEL testing kernel has a fix.
Closed #2429.
Closing based on the last comments. Enhancements can be contributed with pull requests.