sr-dev March 2025

sr-dev@lists.kamailio.org

10 participants
171 discussions

[kamailio/kamailio] Memory usage increases everytime tls.reload is executed (Issue #3823)
by Sebastian Denz 27 Mar '25

27 Mar '25

### Description We are using Kamailio 5.7.4 on Debian 12 (from http://deb.kamailio.org/kamailio57) with rtpengine as an Edgeproxy for our clients. The instance terminates SIP/TLS (with Cliencertificates) and forwards the SIP Traffic to internal systems. After some days we are getting errors like this `tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: shm=7318616 threshold1=8912896` First we thought Kamailio just doesnt have enough memory, so we doubled it.. But after some days the Logmessage (and Userissues) occured again. So we monitored the shmmem statistics and found that used and max_used are constantly growing til it reaches the limit. As i mentioned we are using client-certificates and so we are also using the CRL feature. We do have a systemd-timer which fetches the CRL every hour and runs 'kamcmd tls.reload' when finished. Our tls.cfg looks like this: ``` [server:default] method = TLSv1.2+ private_key = /etc/letsencrypt/live/hostname.de/privkey.pem certificate = /etc/letsencrypt/live/hostname.de/fullchain.pem ca_list = /etc/kamailio/ca_list.pem ca_path = /etc/kamailio/ca_list.pem crl = /etc/kamailio/combined.crl.pem verify_certificate = yes require_certificate = yes [client:default] verify_certificate = yes require_certificate = yes ``` After testing a bit we found that every time tls.reload is executed Kamailio consumes a bit more memory which eventually leads to all the memory being consumed which leads to issues for our users. See following example: ``` [0][root@edgar-dev:~]# while true ; do /usr/sbin/kamcmd tls.reload ; /usr/sbin/kamcmd core.shmmem ; sleep 1 ; done Ok. TLS configuration reloaded. { total: 268435456 free: 223001520 used: 41352552 real_used: 45433936 max_used: 45445968 fragments: 73 } Ok. TLS configuration reloaded. { total: 268435456 free: 222377960 used: 41975592 real_used: 46057496 max_used: 46069232 fragments: 78 } Ok. TLS configuration reloaded. { total: 268435456 free: 221748664 used: 42604992 real_used: 46686792 max_used: 46698080 fragments: 77 } Ok. TLS configuration reloaded. { total: 268435456 free: 221110832 used: 43242408 real_used: 47324624 max_used: 47335608 fragments: 81 } ^C [130][root@edgar-dev:~]# ``` ### Troubleshooting #### Reproduction Everytime tls.reload is called the memory consumptions grows.. #### Debugging Data  ``` If you let me know what would be interesting for tracking this down, i am happy to provide logs/debugging data! ``` #### Log Messages  ``` If you let me know what would be interesting for tracking this down, i am happy to provide logs/debugging data! ``` #### SIP Traffic SIP doesnt seem to be relevant here ### Possible Solutions Calling tls.reload less often or restart kamailio before memory is consumed ;) ### Additional Information ``` version: kamailio 5.7.4 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 12.2.0 ``` * **Operating System**: ``` * Debian GNU/Linux 12 (bookworm) * Linux edgar-dev 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3823 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3823(a)github.com>

8 30

Build failed in Jenkins: kamailiodev-nightly-binaries » amd64,bullseye #2949
by kamailio＠sipwise.com 27 Mar '25

27 Mar '25

See <https://kamailio.sipwise.com/job/kamailiodev-nightly-binaries/architecture=…> Changes: ------------------------------------------ [...truncated 66.82 KiB...] I: new cache content 'dwz_0.13+20210201-1_amd64.deb' added I: new cache content 'libmaxminddb0_1.5.2-1_amd64.deb' added I: new cache content 'ruby_1%3a2.7+2_amd64.deb' added I: new cache content 'sgml-data_2.0.11+nmu1_all.deb' added I: new cache content 'autopoint_0.21-4_all.deb' added I: new cache content 'libgnutls-openssl27_3.7.1-5+deb11u7_amd64.deb' added I: new cache content 'libncurses6_6.2+20201114-2+deb11u2_amd64.deb' added I: new cache content 'libjson-c-dev_0.15-2+deb11u1_amd64.deb' added I: new cache content 'ruby-net-telnet_0.1.1-2_all.deb' added I: new cache content 'python3-dev_3.9.2-3_amd64.deb' added I: new cache content 'libprotobuf-dev_3.12.4-1+deb11u1_amd64.deb' added I: new cache content 'libpsl5_0.21.0-1.2_amd64.deb' added I: new cache content 'libpython3.9-minimal_3.9.2-1+deb11u3_amd64.deb' added I: new cache content 'libphonenumber8_8.12.16-4_amd64.deb' added I: new cache content 'python3.9-minimal_3.9.2-1+deb11u3_amd64.deb' added I: new cache content 'libjs-jquery_3.5.1+dfsg+~3.5.5-7_all.deb' added I: new cache content 'libsecsipid1_1.1.1~git20210601.d388d64-1+b2_amd64.deb' added I: new cache content 'mariadb-common_1%3a10.5.28-0+deb11u1_all.deb' added I: new cache content 'icu-devtools_67.1-7_amd64.deb' added I: new cache content 'libboost-system1.74-dev_1.74.0-9_amd64.deb' added I: new cache content 'libelf1_0.183-1_amd64.deb' added I: new cache content 'libevent-core-2.1-7_2.1.12-stable-1_amd64.deb' added I: new cache content 'libmnl0_1.0.4-3_amd64.deb' added I: new cache content 'libjs-underscore_1.9.1~dfsg-3_all.deb' added I: new cache content 'libreadline-dev_8.1-1_amd64.deb' added I: new cache content 'ruby-dev_1%3a2.7+2_amd64.deb' added I: new cache content 'libradcli4_1.2.11-1+b2_amd64.deb' added I: new cache content 'pkg-config_0.29.2-1_amd64.deb' added I: new cache content 'libpython2.7_2.7.18-8+deb11u1_amd64.deb' added I: new cache content 'libsigsegv2_2.13-1_amd64.deb' added I: new cache content 'm4_1.4.18-5_amd64.deb' added I: new cache content 'libmongoc-dev_1.17.6-1_amd64.deb' added I: new cache content 'librabbitmq-dev_0.10.0-1_amd64.deb' added I: new cache content 'libsqlite3-dev_3.34.1-3+deb11u1_amd64.deb' added I: new cache content 'python3-lib2to3_3.9.2-1_all.deb' added I: new cache content 'libreadline8_8.1-1_amd64.deb' added I: new cache content 'python2_2.7.18-3_amd64.deb' added I: new cache content 'default-libmysqlclient-dev_1.0.7_amd64.deb' added I: new cache content 'libmariadb-dev-compat_1%3a10.5.28-0+deb11u1_amd64.deb' added I: new cache content 'librdkafka-dev_1.6.0-1_amd64.deb' added I: new cache content 'libuv1-dev_1.40.0-2+deb11u1_amd64.deb' added I: new cache content 'libprotobuf23_3.12.4-1+deb11u1_amd64.deb' added I: new cache content 'libjs-sphinxdoc_3.4.3-2_all.deb' added I: new cache content 'libboost-atomic1.74.0_1.74.0-9_amd64.deb' added I: new cache content 'libpython3-dev_3.9.2-3_amd64.deb' added I: new cache content 'docbook-xml_4.5-9_all.deb' added I: new cache content 'ruby-power-assert_1.1.7-2_all.deb' added I: new cache content 'libjansson4_2.13.1-1.1_amd64.deb' added I: new cache content 'libudev-dev_247.3-7+deb11u6_amd64.deb' added I: new cache content 'libedit2_3.1-20191231-2+b1_amd64.deb' added I: new cache content 'libpci3_1%3a3.7.0-5_amd64.deb' added I: new cache content 'automake_1%3a1.16.3-2_all.deb' added I: new cache content 'flex_2.6.4-8_amd64.deb' added I: new cache content 'libdebhelper-perl_13.3.4_all.deb' added I: new cache content 'libldap2-dev_2.4.57+dfsg-3+deb11u1_amd64.deb' added I: new cache content 'libpq-dev_13.20-0+deb11u1_amd64.deb' added I: new cache content 'libsub-override-perl_0.09-2_all.deb' added I: new cache content 'librabbitmq4_0.10.0-1_amd64.deb' added I: new cache content 'libmemcached11_1.0.18-4.2_amd64.deb' added I: new cache content 'libruby2.7_2.7.4-1+deb11u5_amd64.deb' added I: new cache content 'libsnappy-dev_1.1.8-1_amd64.deb' added I: new cache content 'libpcre2-posix2_10.36-2+deb11u1_amd64.deb' added I: new cache content 'python-is-python2_2.7.18-9_all.deb' added I: new cache content 'odbcinst1debian2_2.3.6-0.1+b1_amd64.deb' added I: new cache content 'libbson-dev_1.17.6-1_amd64.deb' added I: new cache content 'libjansson-dev_2.13.1-1.1_amd64.deb' added I: new cache content 'libgmpxx4ldbl_2%3a6.2.1+dfsg-1+deb11u1_amd64.deb' added I: new cache content 'libsnmp-dev_5.9+dfsg-4+deb11u1_amd64.deb' added I: new cache content 'libmongocrypt0_1.1.0-1_amd64.deb' added I: new cache content 'libboost-serialization1.74-dev_1.74.0-9_amd64.deb' added I: new cache content 'libzstd-dev_1.4.8+dfsg-2.1_amd64.deb' added I: new cache content 'libsnmp40_5.9+dfsg-4+deb11u1_amd64.deb' added I: new cache content 'libsctp-dev_1.0.18+dfsg-1_amd64.deb' added I: new cache content 'libhashkit-dev_1.0.18-4.2_amd64.deb' added I: new cache content 'libcap-dev_1%3a2.44-1_amd64.deb' added I: new cache content 'xml-core_0.18+nmu1_all.deb' added I: new cache content 'libxml2-dev_2.9.10+dfsg-6.7+deb11u6_amd64.deb' added I: new cache content 'python3.9-dev_3.9.2-1+deb11u3_amd64.deb' added I: new cache content 'libmpdec3_2.5.1-1_amd64.deb' added I: new cache content 'libmosquitto-dev_2.0.11-1+deb11u2_amd64.deb' added I: new cache content 'libsasl2-2_2.1.27+dfsg-2.1+deb11u1_amd64.deb' added I: new cache content 'libboost-system1.74.0_1.74.0-9_amd64.deb' added I: new cache content 'libperl-dev_5.32.1-4+deb11u4_amd64.deb' added I: new cache content 'ruby-rubygems_3.2.5-2_all.deb' added I: new cache content 'libxml2_2.9.10+dfsg-6.7+deb11u6_amd64.deb' added I: new cache content 'libevent-extra-2.1-7_2.1.12-stable-1_amd64.deb' added I: new cache content 'ruby-minitest_5.13.0-1_all.deb' added I: new cache content 'libwrap0-dev_7.6.q-31_amd64.deb' added I: new cache content 'libexpat1_2.2.10-2+deb11u6_amd64.deb' added I: new cache content 'python2-dev_2.7.18-3_amd64.deb' added I: new cache content 'libboost-date-time1.74-dev_1.74.0-9_amd64.deb' added I: new cache content 'librdkafka++1_1.6.0-1_amd64.deb' added I: new cache content 'bsdextrautils_2.36.1-8+deb11u2_amd64.deb' added I: new cache content 'libmongocrypt-dev_1.1.0-1_amd64.deb' added I: new cache content 'python3-distutils_3.9.2-1_all.deb' added I: new cache content 'libcurl4_7.74.0-1.3+deb11u14_amd64.deb' added I: new cache content 'libyaml-0-2_0.2.2-1_amd64.deb' added I: new cache content 'libpython2.7-dev_2.7.18-8+deb11u1_amd64.deb' added I: new cache content 'rake_13.0.3-1_all.deb' added I: new cache content 'libunbound8_1.13.1-1+deb11u4_amd64.deb' added I: new cache content 'libboost-thread-dev_1.74.0.3_amd64.deb' added I: new cache content 'erlang-base_1%3a23.2.6+dfsg-1+deb11u1_amd64.deb' added I: new cache content 'libgnutls-dane0_3.7.1-5+deb11u7_amd64.deb' added I: new cache content 'libtasn1-6-dev_4.16.0-2+deb11u2_amd64.deb' added I: new cache content 'libprocps8_2%3a3.3.17-5_amd64.deb' added I: new cache content 'libtool_2.4.6-15_all.deb' added I: new cache content 'libpython3.9-stdlib_3.9.2-1+deb11u3_amd64.deb' added I: new cache content 'python2.7_2.7.18-8+deb11u1_amd64.deb' added I: new cache content 'libgnutlsxx28_3.7.1-5+deb11u7_amd64.deb' added I: new cache content 'debhelper_13.3.4_all.deb' added I: new cache content 'libarchive-zip-perl_1.68-1_all.deb' added I: new cache content 'libnetsnmptrapd40_5.9+dfsg-4+deb11u1_amd64.deb' added I: new cache content 'libcap2_1%3a2.44-1_amd64.deb' added I: new cache content 'libwebsockets-dev_4.0.20-2_amd64.deb' added I: new cache content 'libunistring-dev_0.9.10-4_amd64.deb' added I: new cache content 'bison_2%3a3.7.5+dfsg-1_amd64.deb' added I: new cache content 'libpython3.9_3.9.2-1+deb11u3_amd64.deb' added I: new cache content 'libevent-dev_2.1.12-stable-1_amd64.deb' added I: new cache content 'libpcre2-32-0_10.36-2+deb11u1_amd64.deb' added I: new cache content 'libmagic1_1%3a5.39-3+deb11u1_amd64.deb' added I: new cache content 'libwebsockets16_4.0.20-2_amd64.deb' added I: new cache content 'libphonenumber-dev_8.12.16-4_amd64.deb' added I: new cache content 'libdb5.3-dev_5.3.28+dfsg1-0.8_amd64.deb' added I: new cache content 'libpython3.9-dev_3.9.2-1+deb11u3_amd64.deb' added I: new cache content 'libprotobuf-lite23_3.12.4-1+deb11u1_amd64.deb' added I: new cache content 'libevent-openssl-2.1-7_2.1.12-stable-1_amd64.deb' added I: new cache content 'libsasl2-dev_2.1.27+dfsg-2.1+deb11u1_amd64.deb' added I: new cache content 'libdb-dev_5.3.1+nmu1_amd64.deb' added I: new cache content 'liblua5.1-0_5.1.5-8.1+b3_amd64.deb' added I: new cache content 'libhiredis-dev_0.14.1-1_amd64.deb' added I: new cache content 'libsctp1_1.0.18+dfsg-1_amd64.deb' added I: new cache content 'libmaxminddb-dev_1.5.2-1_amd64.deb' added I: new cache content 'python2-minimal_2.7.18-3_amd64.deb' added I: new cache content 'libbrotli1_1.0.9-2+b2_amd64.deb' added I: new cache content 'libnghttp2-14_1.43.0-1+deb11u2_amd64.deb' added I: new cache content 'libssl-dev_1.1.1w-0+deb11u2_amd64.deb' added I: new cache content 'libhashkit2_1.0.18-4.2_amd64.deb' added I: new cache content 'libsensors-dev_1%3a3.6.0-7_amd64.deb' added I: new cache content 'libmariadb3_1%3a10.5.28-0+deb11u1_amd64.deb' added I: new cache content 'libidn2-dev_2.3.0-5_amd64.deb' added I: new cache content 'man-db_2.9.4-2_amd64.deb' added I: new cache content 'media-types_4.0.0_all.deb' added I: new cache content 'libboost-chrono1.74-dev_1.74.0-9_amd64.deb' added I: new cache content 'po-debconf_1.0.21+nmu1_all.deb' added I: new cache content 'mailcap_3.69_all.deb' added I: new cache content 'libpython2.7-minimal_2.7.18-8+deb11u1_amd64.deb' added I: new cache content 'libboost-thread1.74.0_1.74.0-9_amd64.deb' added I: new cache content 'dh-autoreconf_20_all.deb' added I: new cache content 'openssl_1.1.1w-0+deb11u2_amd64.deb' added I: new cache content 'libboost1.74-dev_1.74.0-9_amd64.deb' added I: new cache content 'libboost-serialization1.74.0_1.74.0-9_amd64.deb' added I: new cache content 'groff-base_1.22.4-6_amd64.deb' added I: new cache content 'libsecsipid-dev_1.1.1~git20210601.d388d64-1+b2_amd64.deb' added I: new cache content 'libexpat1-dev_2.2.10-2+deb11u6_amd64.deb' added I: new cache content 'python3.9_3.9.2-1+deb11u3_amd64.deb' added I: new cache content 'uuid-dev_2.36.1-8+deb11u2_amd64.deb' added I: new cache content 'unixodbc-dev_2.3.6-0.1+b1_amd64.deb' added I: new cache content 'libuchardet0_0.0.7-1_amd64.deb' added I: new cache content 'libmosquitto1_2.0.11-1+deb11u2_amd64.deb' added I: new cache content 'libsasl2-modules-db_2.1.27+dfsg-2.1+deb11u1_amd64.deb' added I: new cache content 'ca-certificates_20210119_all.deb' added I: new cache content 'libuv1_1.40.0-2+deb11u1_amd64.deb' added I: new cache content 'python3-minimal_3.9.2-3_amd64.deb' added I: new cache content 'pci.ids_0.0~2021.02.08-1_all.deb' added I: new cache content 'libpcre2-dev_10.36-2+deb11u1_amd64.deb' added I: new cache content 'libev-dev_1%3a4.33-1_amd64.deb' added I: new cache content 'libjson-c5_0.15-2+deb11u1_amd64.deb' added I: new cache content 'libsensors-config_1%3a3.6.0-7_all.deb' added I: new cache content 'gettext-base_0.21-4_amd64.deb' added I: new cache content 'libncurses-dev_6.2+20201114-2+deb11u2_amd64.deb' added I: new cache content 'ruby-xmlrpc_0.3.0-2_all.deb' added I: new cache content 'autoconf_2.69-14_all.deb' added I: new cache content 'libpq5_13.20-0+deb11u1_amd64.deb' added I: new cache content 'libboost-chrono1.74.0_1.74.0-9_amd64.deb' added I: new cache content 'libbson-1.0-0_1.17.6-1_amd64.deb' added I: new cache content 'mysql-common_5.8+1.0.7_all.deb' added I: new cache content 'libgeocoding8_8.12.16-4_amd64.deb' added I: new cache content 'libevent-2.1-7_2.1.12-stable-1_amd64.deb' added I: new cache content 'python2.7-dev_2.7.18-8+deb11u1_amd64.deb' added I: new cache content 'mime-support_3.66_all.deb' added I: new cache content 'libboost-thread1.74-dev_1.74.0-9_amd64.deb' added I: new cache content 'libmariadb-dev_1%3a10.5.28-0+deb11u1_amd64.deb' added I: new cache content 'libpython2.7-stdlib_2.7.18-8+deb11u1_amd64.deb' added I: new cache content 'python3_3.9.2-3_amd64.deb' added I: new cache content 'ruby2.7_2.7.4-1+deb11u5_amd64.deb' added I: new cache content 'libpython2-dev_2.7.18-3_amd64.deb' added I: new cache content 'libev4_1%3a4.33-1_amd64.deb' added I: new cache content 'odbcinst_2.3.6-0.1+b1_amd64.deb' added I: new cache content 'libmicrohttpd12_0.9.72-2+deb11u1_amd64.deb' added I: new cache content 'libmnl-dev_1.0.4-3_amd64.deb' added I: new cache content 'libglib2.0-0_2.66.8-1+deb11u5_amd64.deb' added I: new cache content 'file_1%3a5.39-3+deb11u1_amd64.deb' added I: new cache content 'libcurl4-openssl-dev_7.74.0-1.3+deb11u14_amd64.deb' added I: new cache content 'libmd0_1.0.3-3_amd64.deb' added I: new cache content 'erlang-dev_1%3a23.2.6+dfsg-1+deb11u1_amd64.deb' added I: new cache content 'libmemcached-dev_1.0.18-4.2_amd64.deb' added I: new cache content 'dh-strip-nondeterminism_1.12.0-1_all.deb' added I: new cache content 'libsnmp-base_5.9+dfsg-4+deb11u1_all.deb' added I: new cache content 'libmicrohttpd-dev_0.9.72-2+deb11u1_amd64.deb' added I: new cache content 'libfile-stripnondeterminism-perl_1.12.0-1_all.deb' added I: new cache content 'rubygems-integration_1.18_all.deb' added I: new cache content 'libhiredis0.14_0.14.1-1_amd64.deb' added I: new cache content 'libsnappy1v5_1.1.8-1_amd64.deb' added I: new cache content 'libpipeline1_1.5.3-1_amd64.deb' added I: new cache content 'libradcli-dev_1.2.11-1+b2_amd64.deb' added I: new cache content 'libicu-dev_67.1-7_amd64.deb' added I: new cache content 'libpci-dev_1%3a3.7.0-5_amd64.deb' added I: new cache content 'python2.7-minimal_2.7.18-8+deb11u1_amd64.deb' added I: new cache content 'librtmp1_2.4+20151223.gitfa8646d.1-2+b2_amd64.deb' added I: unmounting /tmp/apt-jdg_6HCIxX filesystem I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: Cleaning COW directory I: forking: rm -rf /var/cache/pbuilder/build/cow.14953 + '[' 1 -eq 0 ']' + bailout 1 'Error: Failed to build with cowbuilder.' + '[' -n 1 ']' + EXIT=1 + '[' -n 'Error: Failed to build with cowbuilder.' ']' + echo 'Error: Failed to build with cowbuilder.' Error: Failed to build with cowbuilder. + rm -f /run/lock/bullseye-amd64.building.14599 + '[' -r /run/lock/bullseye-amd64.update.14599 ']' + '[' '' = true ']' + echo '*** Getting rid of files in <https://kamailio.sipwise.com/job/kamailiodev-nightly-binaries/architecture=…> to avoid problems in next run. ***' *** Getting rid of files in <https://kamailio.sipwise.com/job/kamailiodev-nightly-binaries/architecture=…> to avoid problems in next run. *** + rm -f '<https://kamailio.sipwise.com/job/kamailiodev-nightly-binaries/architecture=…'> + '[' -n /tmp/tmp.L2ki1Oqlxa ']' + rm -rf /tmp/tmp.L2ki1Oqlxa + '[' -n /tmp/apt-jdg_6HCIxX ']' + sudo rm -rf /tmp/apt-jdg_6HCIxX + '[' -n 416 ']' ++ sed -e 's/^$[0-9]*$.*/\1/' + SECONDS=42 ++ date + echo '*** Finished execution of /usr/bin/build-and-provide-package at Wed Mar 26 01:05:43 UTC 2025 [running 42 seconds] ***' *** Finished execution of /usr/bin/build-and-provide-package at Wed Mar 26 01:05:43 UTC 2025 [running 42 seconds] *** + exit 1 Build step 'Execute shell' marked build as failure Archiving artifacts Recording fingerprints TAP Reports Processing: START Looking for TAP results report in workspace using pattern: report/*.tap Did not find any matching files. Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?

1 1

[kamailio/kamailio] Segfault when using kamailio 5.8.4 as a presence proxy (Issue #4102)
by Joe Mordica 27 Mar '25

27 Mar '25

### Description We are now using kamailio on the edge as a presence proxy to scale presence. We have 3 internal kamailio instances handling the actual subscriptions and publishing. This works well, however, once per 24-36hours we experience a crash on the border proxy instance. We did not experience this when using the border proxy as a presence server (there was no proxying SUBSCRIBE). ### Troubleshooting #### Reproduction I believe this is happening when a customers network randomly experiences issues where it doesn't receive our replies to subscribes (and continues to send a subscribe). Logs are below. #### Debugging Data  Here's the output while replacing any sensitive info in the payload. ``` GNU gdb (Debian 13.1-3) 13.1 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio... Reading symbols from /usr/lib/debug/.build-id/ce/c19d0ed1a928e4e25a84ae20d77e9c904a4892.debug... warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing [New LWP 26] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `kamailio -DD -E -m 8000 -M 512 -f /etc/kamailio/kamailio.cfg -P /var/run/kamail'. Program terminated with signal SIGSEGV, Segmentation fault. #0 via_builder (len=len@entry=0x7ffed7e3ecb8, msg=msg@entry=0x7ff9529548b0, send_info=send_info@entry=0x7ff764d20bb0, branch=branch@entry=0x7ffed7e3ecc0, extra_params=0x0, hp=hp@entry=0x7ffed7e3ebb0) at core/msg_translator.c:2910 2910 core/msg_translator.c: No such file or directory. (gdb) bt full #0 via_builder (len=len@entry=0x7ffed7e3ecb8, msg=msg@entry=0x7ff9529548b0, send_info=send_info@entry=0x7ff764d20bb0, branch=branch@entry=0x7ffed7e3ecc0, extra_params=0x0, hp=hp@entry=0x7ffed7e3ebb0) at core/msg_translator.c:2910 via_len = <optimized out> extra_len = <optimized out> line_buf = <optimized out> max_len = <optimized out> via_prefix_len = <optimized out> address_str = <optimized out> port_str = 0x0 send_sock = 0x0 comp_len = <optimized out> comp_name_len = <optimized out> port = <optimized out> proto = <optimized out> ip = {af = 536870912, len = 0, u = {addrl = {534820896, 1586192}, addr32 = {534820896, 0, 1586192, 0}, addr16 = {47136, 8160, 0, 0, 13328, 24, 0, 0}, addr = " \270\340\037\000\000\000\000\0204\030\000\000\000\000"}} from = 0x0 local_addr = {s = {sa_family = 16, sa_data = "#\000\000\000\000\000\020\000\000\000\000\000\000"}, sin = {sin_family = 16, sin_port = 35, sin_addr = {s_addr = 0}, sin_zero = "\020\000\000\000\000\000\000"}, sin6 = {sin6_family = 16, sin6_port = 35, sin6_flowinfo = 0, sin6_addr = {__in6_u = { __u6_addr8 = "\020\000\000\000\000\000\000\000\003\000\000\000\000\000\000", __u6_addr16 = {16, 0, 0, 0, 3, 0, 0, 0}, __u6_addr32 = {16, 0, 3, 0}}}, sin6_scope_id = 3000082688}, sas = {ss_family = 16, __ss_padding = "#\000\000\000\000\000\020\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000\000\241\321\262cg\357`@\000\000\000\000\000\000\000X\f-\177!V\000\000\020\016\217R\371\177\000\000\001\000\000\000!V\000\000*\000\000\000\000\000\000\000\001\000\000\000\376\177\000\000\004\360;\177!V\000\000\250,A\177!V\000\000\000\355\343\327\376\177\000\000\000\241\321\262cg\357`h\256\006R\371\177\000", __ss_align = 94701866020241}} con = 0x0 rxavp = 0x0 xname = {s = 0x0, len = -2141140176} __func__ = "via_builder" #1 0x000056217f155030 in create_via_hf (len=len@entry=0x7ffed7e3ecb8, msg=msg@entry=0x7ff9529548b0, send_info=send_info@entry=0x7ff764d20bb0, branch=branch@entry=0x7ffed7e3ecc0) at core/msg_translator.c:3232 via = <optimized out> extra_params = {s = 0x0, len = 0} hp = {host = 0x56217f4fb380 <default_global_address>, port = 0x56217f4fb370 <default_global_port>} sbuf = "\331\023\000\000\001", '\000' <repeats 18 times> --Type <RET> for more, q to quit, c to continue without paging--info locals slen = <optimized out> xparams = <optimized out> id_buf = <optimized out> id_len = 0 __func__ = "create_via_hf" __llevel = <optimized out> __kld = <optimized out> #2 0x000056217f1592f7 in build_req_buf_from_sip_req (msg=msg@entry=0x7ff9529548b0, returned_len=returned_len@entry=0x7ffed7e3eeac, send_info=send_info@entry=0x7ff764d20bb0, mode=mode@entry=128) at core/msg_translator.c:2096 len = 581 new_len = <optimized out> received_len = 0 rport_len = 0 uri_len = 0 via_len = 0 body_delta = 0 line_buf = 0x0 received_buf = 0x0 rport_buf = 0x0 new_buf = 0x0 buf = 0x56217f5866c0 <buf> "SUBSCRIBE sip:74@sip.domain.co SIP/2.0\r\nVia: SIP/2.0/UDP 12.171.207.82:25744;branch=z9hG4bK1805670803;rport\r\nFrom: \"Heather Myers\" <sip:1086448@sip.domain.co>;tag=1805518213\r\nTo: <sip:74@sip.domain.co>\r\nCa"... path_buf = {s = 0x0, len = 0} offset = 0 s_offset = 0 size = <optimized out> via_anchor = 0x7ff952951300 via_lump = <optimized out> via_rm = <optimized out> via_insert_param = 0x0 path_anchor = <optimized out> path_lump = <optimized out> branch = {s = 0x7ff952954ed8 "z9hG4bKc7ed.5819b4295fe83f981a2fc19b7f739d7e.0", len = 46} flags = 262273 --Type <RET> for more, q to quit, c to continue without paging--list udp_mtu = <optimized out> di = {send_sock = 0x0, to = {s = {sa_family = 18608, sa_data = "\225R\371\177\000\000\002\000\023\331\n4\001\r"}, sin = {sin_family = 18608, sin_port = 21141, sin_addr = { s_addr = 32761}, sin_zero = "\002\000\023\331\n4\001\r"}, sin6 = {sin6_family = 18608, sin6_port = 21141, sin6_flowinfo = 32761, sin6_addr = {__in6_u = { __u6_addr8 = "\002\000\023\331\n4\001\r\000\000\000\000\000\000\000", __u6_addr16 = {2, 55571, 13322, 3329, 0, 0, 0, 0}, __u6_addr32 = {3641901058, 218182666, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 18608, __ss_padding = "\225R\371\177\000\000\002\000\023\331\n4\001\r", '\000' <repeats 103 times>, __ss_align = 0}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} ret = <optimized out> __func__ = "build_req_buf_from_sip_req" error00 = <optimized out> #3 0x00007ff951fbe90d in prepare_new_uac (t=t@entry=0x7ff764d20880, i_req=i_req@entry=0x7ff9529548b0, branch=branch@entry=0, uri=<optimized out>, uri@entry=0x7ff9529548e8, path=<optimized out>, path@entry=0x7ff952954f60, next_hop=<optimized out>, fsocket=0x7ff9528f0e10, snd_flags=..., fproto=<optimized out>, flags=<optimized out>, instance=<optimized out>, ruid=<optimized out>, location_ua=<optimized out>) at ./src/modules/tm/t_fwd.c:482 shbuf = 0x0 add_rm_backup = <optimized out> body_lumps_backup = <optimized out> parsed_uri_bak = {user = { s = 0x56217f5866ce <buf+14> "74(a)sip.domain.co SIP/2.0\r\nVia: SIP/2.0/UDP 12.171.207.82:25744;branch=z9hG4bK1805670803;rport\r\nFrom: \"Heather Myers\" <sip:1086448@sip.domain.co>;tag=1805518213\r\nTo: <sip:74@sip.domain.co>\r\nCall-ID: 0_18056"..., len = 2}, passwd = {s = 0x0, len = 0}, host = { s = 0x56217f5866d1 <buf+17> "sip.domain.co SIP/2.0\r\nVia: SIP/2.0/UDP 12.171.207.82:25744;branch=z9hG4bK1805670803;rport\r\nFrom: \"Heather Myers\" <sip:1086448@sip.domain.co>;tag=1805518213\r\nTo: <sip:74@sip.domain.co>\r\nCall-ID: 0_18056078"..., len = 11}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = { s = 0x0, len = 0}, port_no = 0, proto = 0, type = SIP_URI_T, flags = 0, transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = { s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = { s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} ret = -1 len = 32761 parsed_uri_ok_bak = <optimized out> free_new_uri = 1 msg_uri_bak = {s = 0x0, len = 0} dst_uri_bak = {s = 0x7ff9529525f0 "sip:10.52.1.13:5081", len = 19} dst_uri_backed_up = 1 path_bak = {s = 0x0, len = 0} free_path = 1 instance_bak = {s = 0x0, len = 0} ``` #### Log Messages  ``` ERROR 2025-01-10T06:00:31.847348563Z [resource.labels.containerName: kamailio] 8(28) CRITICAL: {1 1 SUBSCRIBE 0_1907143137(a)192.168.100.28} tm [timer.h:188]: _set_fr_retr(): already added: 0x7ff763e62250 , tl=0x7ff763e62270!!! ERROR 2025-01-10T06:00:31.847388483Z [resource.labels.containerName: kamailio] 8(28) CRITICAL: {1 1 SUBSCRIBE 0_1907143137(a)192.168.100.28} tm [t_fwd.c:1613]: t_send_branch(): BUG: retransmission already started for: 0x7ff763e62250 ERROR 2025-01-10T06:00:31.847394809Z [resource.labels.containerName: kamailio] 8(28) ERROR: {1 1 SUBSCRIBE 0_1907143137(a)192.168.100.28} sl [sl_funcs.c:428]: sl_reply_error(): stateless error reply used: No error (0/SL) ERROR 2025-01-10T06:00:31.847400738Z [resource.labels.containerName: kamailio] 8(28) BUG: {1 1 SUBSCRIBE 0_1907143137(a)192.168.100.28} tm [t_lookup.c:2052]: t_unref(): REQ_ERR DELAYED should have been caught much earlier for 0x7ff763e61f70: 24 (hex 18) ERROR 2025-01-10T06:00:47.346744773Z [resource.labels.containerName: kamailio] 45(65) ERROR: <core> [core/tcp_main.c:4818]: tcpconn_main_timeout(): connect 64.63.142.90:12190 failed (timeout) ERROR 2025-01-10T06:00:58.274915089Z [resource.labels.containerName: kamailio] 45(65) CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 20 ERROR 2025-01-10T06:01:02.715789076Z [resource.labels.containerName: kamailio] 0(1) ALERT: <core> [main.c:805]: handle_sigs(): child process 26 exited by a signal 11 ERROR 2025-01-10T06:01:02.715827333Z [resource.labels.containerName: kamailio] 0(1) ALERT: <core> [main.c:809]: handle_sigs(): core was generated ERROR 2025-01-10T06:01:02.718815287Z [resource.labels.containerName: kamailio] 45(65) CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 25 ERROR 2025-01-10T06:01:02.720985493Z [resource.labels.containerName: kamailio] 45(65) CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 23 ERROR 2025-01-10T06:01:02.725450552Z [resource.labels.containerName: kamailio] 45(65) CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 30 ``` #### SIP Traffic  ``` (paste your sip traffic here) ``` ### Possible Solutions I can atleast say that when this segfault happens that these logs are always present right before the crash. However, I also see instances of these logs without a crash immediately following. On the proxy we are doing this is withindlg route to send subscribes to the presence route: ``` if KSR.is_SUBSCRIBE() and KSR.is_myself_ruri() then -- in-dialog subscribe requests ksr_route_presence(); KSR.x.exit(); end ``` Any then this is our ksr_route_presence(): ``` function ksr_route_presence() if not KSR.is_SUBSCRIBE() then return 1; end local fuser = KSR.kx.get_fuser(); if fuser=='' then KSR.xlog.xinfo('Subscription attempt without username from IP ' .. KSR.kx.get_srcip() .. ' - Rejecting\n'); KSR.sl.sl_send_reply(404, 'Not Found'); KSR.x.exit(); end KSR.pv.sets('$avp(s:tenant)', string.match(fuser, '-(.*)')); local setId = '6' .. KSR.kx.get_def('DEFAULT_DISPATCH_SET'); local modeId = '7'; if KSR.dispatcher.ds_select_dst(setId, modeId)<0 then KSR.sl.send_reply(404, 'No destination'); KSR.x.exit(); end KSR.corex.set_send_socket(KSR.kx.get_def('PRIVATE_LISTEN_IP') .. ':' .. KSR.kx.get_def('INTERNAL_PORT')); KSR.tm.t_on_failure('ksr_route_rtf_dispatch'); ksr_route_relay(); end ``` ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.8.4 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 12.2.0 ``` * **Operating System**:  ``` PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ``` ``` Linux gke-us-south1-external-sip-800ca69e-qyb3 5.15.0-1048-gke #53-Ubuntu SMP Tue Nov 28 00:39:01 UTC 2023 x86_64 GNU/Linux ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4102 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4102(a)github.com>

4 29

[kamailio/kamailio] If fix_nated_contact is used, then remove_hf("Contact"), the contact header will not be deleted cleanly. (Issue #4190)
by QuincyGao 26 Mar '25

26 Mar '25

QuincyGao created an issue (kamailio/kamailio#4190)  ### Description  I want to use opensips as SBC, When the agent registers to kamailio, the contact content is modified to the ip and port of kamailio, and then Register is forwarded to the agent. remove_hf("Contact") can't remove the whole Contact( field, vlue). here is my config: ``` route[REGISTRAR] { if (!is_method("REGISTER")) return; if(isflagset(FLT_NATS)) { setbflag(FLB_NATB); #!ifdef WITH_NATSIPPING # do SIP NAT pinging setbflag(FLB_NATSIPPING); #!endif } fix_nated_contact(); if (!save("location","0x02")) { #if (!save("location")) { sl_reply_error(); } $fs="udp:172.16.4.111:5461"; $du="sip:172.16.4.114:5060"; remove_hf("Contact"); $var(contact) = "Contact: <sip:" +$fU+"@"+"172.16.4.111:5461>;expires=120\r\n" ; append_hf("$var(contact)"); route(RELAY); exit; } ``` here is the sip message: ``` 2025/03/26 16:36:07.706093 172.16.4.111:5461 -> 172.16.4.114:5060 REGISTER sip:172.16.4.111:5460 SIP/2.0 Via: SIP/2.0/UDP 172.16.4.111:5461;branch=z9hG4bKfc7e.d2df36f3b5681afeb71e442e645de38d.0 Via: SIP/2.0/UDP 172.16.80.3:49309;received=172.16.80.3;rport=49309;branch=z9hG4bKPj8ab8bd1d3c3a4c1bb0cd639540fa3183 Max-Forwards: 69 From: <sip:1008@172.16.4.111>;tag=95f7f5601f8b40458a1efe6923ab0a29 To: <sip:1008@172.16.4.111> Call-ID: f84b643f5d284d18a0d8175e1ef4d747 CSeq: 33264 REGISTER User-Agent: MicroSIP/3.21.6 sip:1008@172.16.80.3:49309;obExpires: 120 Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS Content-Length: 0 Contact: <sip:1008@172.16.4.111:5461>;expires=120 ``` you can see the problem is that: `sip:1008@172.16.80.3:49309;obExpires: 120` , `sip:1008@172.16.80.3:49309` is the value of the Contact field after fix_nated_contact(). ### Troubleshooting #### Reproduction  #### Debugging Data  ``` (paste your debugging data here) ``` #### Log Messages  ``` (paste your log messages here) ``` #### SIP Traffic  ``` (paste your sip traffic here) ``` ### Possible Solutions  ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.8.5 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: compiled on 11:27:01 Mar 21 2025 with gcc 4.8.5 ``` * **Operating System**:  ``` Centos7.9 ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4190 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4190(a)github.com>

1 0

[kamailio/kamailio] dmq send dialog message failed (Issue #4189)
by QuincyGao 26 Mar '25

26 Mar '25

QuincyGao created an issue (kamailio/kamailio#4189)  ### Description  I used dmq on two machines, registered two softphones on the 111 machine, and then used one phone to call another. After the call was connected, dmq sent a dialog message with an error message 400 bad request. Here is the 111 config, 113 almost same to 111: ``` loadmodule "dmq.so" modparam("dmq", "server_address", "sip:172.16.4.111:5464") modparam("dmq", "server_socket", "udp:172.16.4.111:5464") modparam("dmq", "notification_address", "sip:172.16.4.113:5464") modparam("dmq", "notification_channel", "peers") modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 30) loadmodule "dmq_usrloc.so" modparam("dmq_usrloc", "enable", 1) modparam("dmq_usrloc", "sync", 1) modparam("dmq_usrloc", "batch_msg_contacts", 50) # 50 contacts / message modparam("dmq_usrloc", "batch_size", 10000) # 10000 contacts / batch modparam("dmq_usrloc", "batch_usleep", 500000) loadmodule "dialog.so" modparam("dialog", "enable_stats", 1) modparam("dialog", "enable_dmq", 1) ... if (is_method("INVITE")) { setflag(FLT_ACC); # do accounting dlg_manage(); } ``` Here is the sip: <img width="836" alt="Image" src="https://github.com/user-attachments/assets/df609e21-77bb-43e9-a8aa-112069a3…" /> the 113 log error: ``` dialog [dlg_dmq.c:127]: dlg_dmq_handle_msg(): no content length header found ``` ### Troubleshooting #### Reproduction  The inevitable bug #### Debugging Data  ``` (paste your debugging data here) ``` #### Log Messages  ``` 113 log: 22(4389) DEBUG: <core> [core/udp_server.c:708]: udp_rcv_loop(): received on udp socket: (106/100/767) [[KDMQ sip:dialog@172.16.4.113:5464 SIP/2.0 0D 0A Via: SIP/2.0/UDP 172.16.4.111:5464;branch=z9hG4bK28e.2ec7]] 22(4389) DEBUG: <core> [core/receive.c:190]: ksr_evrt_received(): event route core:msg-received not defined 22(4389) DEBUG: <core> [core/parser/parse_fline.c:249]: parse_first_line(): first line type 1 (request) flags 1 22(4389) DEBUG: <core> [core/parser/msg_parser.c:722]: parse_msg(): SIP Request: 22(4389) DEBUG: <core> [core/parser/msg_parser.c:724]: parse_msg(): method: <KDMQ> 22(4389) DEBUG: <core> [core/parser/msg_parser.c:726]: parse_msg(): uri: <sip:dialog@172.16.4.113:5464> 22(4389) DEBUG: <core> [core/parser/msg_parser.c:728]: parse_msg(): version: <SIP/2.0> 22(4389) DEBUG: <core> [core/parser/parse_hname2.c:316]: parse_sip_header_name(): parsed header name [Via] type 1 22(4389) DEBUG: <core> [core/parser/parse_via.c:1312]: parse_via_param(): Found param type 232, <branch> = <z9hG4bK28e.2ec78467000000000000000000000000.0>; state=16 22(4389) DEBUG: <core> [core/parser/parse_via.c:2663]: parse_via(): end of header reached, state=5 22(4389) DEBUG: <core> [core/parser/msg_parser.c:592]: parse_headers(): Via found, flags=2 22(4389) DEBUG: <core> [core/parser/msg_parser.c:594]: parse_headers(): this is the first via 22(4389) DEBUG: <core> [core/parser/parse_hname2.c:316]: parse_sip_header_name(): parsed header name [To] type 3 22(4389) DEBUG: <core> [core/parser/parse_addr_spec.c:904]: parse_addr_spec(): end of header reached, state=10 22(4389) DEBUG: <core> [core/parser/msg_parser.c:173]: get_hdr_field(): <To> [32]; uri=[sip:dialog@172.16.4.113:5464] 22(4389) DEBUG: <core> [core/parser/msg_parser.c:176]: get_hdr_field(): to body (32)[<sip:dialog@172.16.4.113:5464> ], to tag (0)[] 22(4389) DEBUG: <core> [core/parser/parse_hname2.c:316]: parse_sip_header_name(): parsed header name [From] type 4 22(4389) DEBUG: <core> [core/parser/parse_hname2.c:316]: parse_sip_header_name(): parsed header name [CSeq] type 5 22(4389) DEBUG: <core> [core/parser/msg_parser.c:154]: get_hdr_field(): cseq <CSeq>: <10> <KDMQ> 22(4389) DEBUG: <core> [core/parser/parse_hname2.c:316]: parse_sip_header_name(): parsed header name [Call-ID] type 6 22(4389) DEBUG: <core> [core/receive.c:397]: receive_msg(): --- received sip message - request - call-id: [6ecc0d71033645ea-1972(a)172.16.4.111] - cseq: [10 KDMQ] 22(4389) DEBUG: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| <core> [core/receive.c:263]: ksr_evrt_pre_routing(): event route core:pre-routing not defined 22(4389) DEBUG: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| <core> [core/receive.c:474]: receive_msg(): preparing to run routing scripts... 22(4389) exec: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| *** cfgtrace:dbg_cfg_trace(): request_route=[DEFAULT_ROUTE] c=[/usr/local/bin/../etc/kamailio/conf/kamailio.cfg] l=584 a=26 n=xlog 22(4389) INFO: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| <script>: 172.16.4.111:5464|172.16.4.113:5464|udp|KDMQ 22(4389) exec: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| *** cfgtrace:dbg_cfg_trace(): request_route=[DEFAULT_ROUTE] c=[/usr/local/bin/../etc/kamailio/conf/kamailio.cfg] l=592 a=16 n=if 22(4389) exec: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| *** cfgtrace:dbg_cfg_trace(): request_route=[DEFAULT_ROUTE] c=[/usr/local/bin/../etc/kamailio/conf/kamailio.cfg] l=589 a=25 n=is_method 22(4389) exec: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| *** cfgtrace:dbg_cfg_trace(): request_route=[DEFAULT_ROUTE] c=[/usr/local/bin/../etc/kamailio/conf/kamailio.cfg] l=590 a=24 n=dmq_process_message 22(4389) DEBUG: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| dmq [message.c:154]: ki_dmq_process_message_rc(): dmq_handle_message [KDMQ sip:dialog@172.16.4.113:5464] 22(4389) DEBUG: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| dmq [message.c:168]: ki_dmq_process_message_rc(): dmq_handle_message peer found: dialog 22(4389) DEBUG: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| <core> [core/parser/parse_addr_spec.c:185]: parse_to_param(): add param: tag=6c399121368d030ed41b62758342ea91-0405b528 22(4389) DEBUG: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| <core> [core/parser/parse_addr_spec.c:904]: parse_addr_spec(): end of header reached, state=29 22(4389) DEBUG: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| dialog [dlg_dmq.c:124]: dlg_dmq_handle_msg(): dmq message received 22(4389) ERROR: |25-03-26 15:25:01|6ecc0d71033645ea-1972@172.16.4.111| dialog [dlg_dmq.c:127]: dlg_dmq_handle_msg(): no content length header found ``` #### SIP Traffic  ``` (paste your sip traffic here) ``` ### Possible Solutions  ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.8.5 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: compiled on 11:27:01 Mar 21 2025 with gcc 4.8.5 ``` * **Operating System**:  ``` centos7.9 ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4189 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4189(a)github.com>

1 1

git:master:87cbab90: modules: readme files regenerated - tm ... [skip ci]
by Kamailio Dev 26 Mar '25

26 Mar '25

Module: kamailio Branch: master Commit: 87cbab9017452821f1c5a04c97e327def9083260 URL: https://github.com/kamailio/kamailio/commit/87cbab9017452821f1c5a04c97e327d… Author: Kamailio Dev <kamailio.dev(a)kamailio.org> Committer: Kamailio Dev <kamailio.dev(a)kamailio.org> Date: 2025-03-26T07:31:10+01:00 modules: readme files regenerated - tm ... [skip ci] --- Modified: src/modules/tm/README --- Diff: https://github.com/kamailio/kamailio/commit/87cbab9017452821f1c5a04c97e327d… Patch: https://github.com/kamailio/kamailio/commit/87cbab9017452821f1c5a04c97e327d… --- diff --git a/src/modules/tm/README b/src/modules/tm/README index 43a859df662..cbe98e02e6f 100644 --- a/src/modules/tm/README +++ b/src/modules/tm/README @@ -156,12 +156,15 @@ Daniel-Constantin Mierla 5.2. tm.t_uac_start 5.3. tm.t_uac_wait 5.4. tm.t_uac_wait_block - 5.5. tm.cancel - 5.6. tm.hash_stats - 5.7. tm.reply - 5.8. tm.reply_callid - 5.9. tm.clean - 5.10. tm.stats + 5.5. tm.t_uac_start_noack + 5.6. tm.t_uac_wait_block_noack + 5.7. tm.t_uac_wait_noack + 5.8. tm.cancel + 5.9. tm.hash_stats + 5.10. tm.reply + 5.11. tm.reply_callid + 5.12. tm.clean + 5.13. tm.stats 6. Event Routes @@ -439,12 +442,15 @@ Chapter 1. Admin Guide 5.2. tm.t_uac_start 5.3. tm.t_uac_wait 5.4. tm.t_uac_wait_block - 5.5. tm.cancel - 5.6. tm.hash_stats - 5.7. tm.reply - 5.8. tm.reply_callid - 5.9. tm.clean - 5.10. tm.stats + 5.5. tm.t_uac_start_noack + 5.6. tm.t_uac_wait_block_noack + 5.7. tm.t_uac_wait_noack + 5.8. tm.cancel + 5.9. tm.hash_stats + 5.10. tm.reply + 5.11. tm.reply_callid + 5.12. tm.clean + 5.13. tm.stats 6. Event Routes @@ -3181,12 +3187,15 @@ reply_route { 5.2. tm.t_uac_start 5.3. tm.t_uac_wait 5.4. tm.t_uac_wait_block - 5.5. tm.cancel - 5.6. tm.hash_stats - 5.7. tm.reply - 5.8. tm.reply_callid - 5.9. tm.clean - 5.10. tm.stats + 5.5. tm.t_uac_start_noack + 5.6. tm.t_uac_wait_block_noack + 5.7. tm.t_uac_wait_noack + 5.8. tm.cancel + 5.9. tm.hash_stats + 5.10. tm.reply + 5.11. tm.reply_callid + 5.12. tm.clean + 5.13. tm.stats 5.1. tm.list @@ -3222,7 +3231,22 @@ reply_route { up to 80 seconds, if no reply is received, it returns a RPC fault 500 code. While waiting, the RPC process cannot handle other RPC commands. -5.5. tm.cancel +5.5. tm.t_uac_start_noack + + Similar to tm.t_uac_start, but do not automatically send ACK for local + INVITE requests. + +5.6. tm.t_uac_wait_block_noack + + Similar to tm.t_uac_wait_block, but do not automatically send ACK for + local INVITE requests. + +5.7. tm.t_uac_wait_noack + + Similar to tm.t_uac_wait, but do not automatically send ACK for local + INVITE requests. + +5.8. tm.cancel Generates and sends a CANCEL for an existing local SIP request. @@ -3230,14 +3254,14 @@ reply_route { * callid - callid of the INVITE request to be cancelled. * cseq - cseq of the INVITE request to be cancelled. -5.6. tm.hash_stats +5.9. tm.hash_stats Gets information about the load of TM internal hash table. Parameters: * none -5.7. tm.reply +5.10. tm.reply Generates and sends a reply for an existing inbound SIP transaction. @@ -3251,7 +3275,7 @@ reply_route { * body - (optional, may not be present) reply body (if present, requires the “Content-Type” and “Content-length” headers) -5.8. tm.reply_callid +5.11. tm.reply_callid Generates and sends a reply for an existing inbound SIP transaction. @@ -3266,14 +3290,14 @@ reply_route { * body - (optional, may not be present) reply body (if present, requires the “Content-Type” and “Content-length” headers) -5.9. tm.clean +5.12. tm.clean Trigger a hard clean of expired transactions. Parameters: * none -5.10. tm.stats +5.13. tm.stats Gets information about current and past TM transaction handling.

1 0

git:master:30fc24ec: tm: docs for rpc commands with no-ack
by Daniel-Constantin Mierla 26 Mar '25

26 Mar '25

Module: kamailio Branch: master Commit: 30fc24ecc80952054a228d93f3aed0c45b0a4af4 URL: https://github.com/kamailio/kamailio/commit/30fc24ecc80952054a228d93f3aed0c… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2025-03-26T07:28:24+01:00 tm: docs for rpc commands with no-ack --- Modified: src/modules/tm/doc/rpc.xml --- Diff: https://github.com/kamailio/kamailio/commit/30fc24ecc80952054a228d93f3aed0c… Patch: https://github.com/kamailio/kamailio/commit/30fc24ecc80952054a228d93f3aed0c… --- diff --git a/src/modules/tm/doc/rpc.xml b/src/modules/tm/doc/rpc.xml index 8f31ad34364..808ecbfcba0 100644 --- a/src/modules/tm/doc/rpc.xml +++ b/src/modules/tm/doc/rpc.xml @@ -82,6 +82,36 @@ </para> </section> + <section id="tm.rpc.t_uac_start_noack"> + <title> + <function moreinfo="none">tm.t_uac_start_noack</function> + </title> + <para> + Similar to tm.t_uac_start, but do not automatically send ACK for local + INVITE requests. + </para> + </section> + + <section id="tm.rpc.t_uac_wait_block_noack"> + <title> + <function moreinfo="none">tm.t_uac_wait_block_noack</function> + </title> + <para> + Similar to tm.t_uac_wait_block, but do not automatically send ACK for + local INVITE requests. + </para> + </section> + + <section id="tm.rpc.t_uac_wait_noack"> + <title> + <function moreinfo="none">tm.t_uac_wait_noack</function> + </title> + <para> + Similar to tm.t_uac_wait, but do not automatically send ACK for local + INVITE requests. + </para> + </section> + <section id="tm.rpc.cancel"> <title> <function moreinfo="none">tm.cancel</function>

1 0

git:master:466d3ad3: tm: rpc commands for sending requests without automatic ack
by Daniel-Constantin Mierla 26 Mar '25

26 Mar '25

Module: kamailio Branch: master Commit: 466d3ad386d861b7a5eb39ba123beffd8059ec45 URL: https://github.com/kamailio/kamailio/commit/466d3ad386d861b7a5eb39ba123beff… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2025-03-26T07:15:29+01:00 tm: rpc commands for sending requests without automatic ack --- Modified: src/modules/tm/rpc_uac.c Modified: src/modules/tm/rpc_uac.h Modified: src/modules/tm/tm.c --- Diff: https://github.com/kamailio/kamailio/commit/466d3ad386d861b7a5eb39ba123beff… Patch: https://github.com/kamailio/kamailio/commit/466d3ad386d861b7a5eb39ba123beff…

1 0

git:master:d99f0404: tm: t flags field made int
by Daniel-Constantin Mierla 26 Mar '25

26 Mar '25

Module: kamailio Branch: master Commit: d99f040463805a8f7438409e41a9a270ec298092 URL: https://github.com/kamailio/kamailio/commit/d99f040463805a8f7438409e41a9a27… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2025-03-25T22:21:52+01:00 tm: t flags field made int - short size is full, flag 15 already used --- Modified: src/modules/tm/h_table.h Modified: src/modules/tm/uac.h --- Diff: https://github.com/kamailio/kamailio/commit/d99f040463805a8f7438409e41a9a27… Patch: https://github.com/kamailio/kamailio/commit/d99f040463805a8f7438409e41a9a27… --- diff --git a/src/modules/tm/h_table.h b/src/modules/tm/h_table.h index 59e229d068d..3e8136cbdac 100644 --- a/src/modules/tm/h_table.h +++ b/src/modules/tm/h_table.h @@ -346,9 +346,9 @@ typedef struct cell /* sequence number within hash collision slot */ unsigned int label; /* different information about the transaction */ - unsigned short flags; + unsigned int flags; /* number of forks */ - short nr_of_outgoings; + int nr_of_outgoings; /* free operations counter - debug */ int fcount; diff --git a/src/modules/tm/uac.h b/src/modules/tm/uac.h index a76d8ea63a7..00dc32d2a5b 100644 --- a/src/modules/tm/uac.h +++ b/src/modules/tm/uac.h @@ -32,6 +32,7 @@ #define DEFAULT_CSEQ 10 /* Default CSeq number */ + /* structure for UAC interface * * You can free the memory allocated

1 0

[kamailio/kamailio] [rtpengine] failure to ping rtpengine results in consistent core dump in kamailio 5.8.2 (Issue #4122)
by Brandon Armstead 26 Mar '25

26 Mar '25

### RTPEngine + RTIMER + SEGFAULT + unable to ping When unable to ping RTPEngine instances kamailio crashes. ### Troubleshooting Determined issue was related to the ping failures, resolved AWS SG issue, allowing successful pings. Issue subsequently went away. #### Reproduction Use rtpengine with RDS backend Load rtpengine with ~ 20 destinations you do not expect a response from on Use rtimer to call rtpengine.reload via jsonrpcs: ``` modparam("rtimer", "timer", "name=rtpengine;interval=60;mode=0;") modparam("rtimer", "exec", "timer=rtpengine;route=RELOAD_RTPENGINE") route[RELOAD_RTPENGINE] { xinfo("RTPEngine database reloading"); jsonrpc_exec('{"jsonrpc": "2.0", "method": "rtpengine.reload", "id": 1}'); } ``` #### Debugging Data Got a core dump -- although I deleted quickly (not thinking) as they were running instances out of disk quickly. bt full did not show much to step through all though clearly pointed to rtpengine. #### Log Messages ``` send_rtpp_command(): timeout waiting reply for command "ping" from RTPEngine <udp:10.0.X.X> ``` ### Possible Solutions TBD / do not have ping failures under heavy load. ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.8.2 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 10.2.1 ``` * **Operating System**: ``` uname -a Linux 5.10.0-27-cloud-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) x86_64 GNU/Linux cat /proc/version Linux version 5.10.0-27-cloud-amd64 (debian-kernel(a)lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.205-2 (2023-12-31) lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4122 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/4122(a)github.com>

3 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

sr-dev March 2025