<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description Hi, I am a junior with Kamailio, if I show you where I am wrong, I will. Often during a call between two sip clients (located on the same subnet with pcscf) Kamailio P-CSCF crashes with a core dump. But sometimes the call goes through normally, I can’t see why this is happening. <!--
-->
### Troubleshooting
#### Reproduction It is reproduced often, the error is most likely somewhere with me. <!-- -->
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: https://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio... (No debugging symbols found in /usr/sbin/kamailio)
warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing [New LWP 14196] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0xb6f1eab9 in ?? () from /usr/lib/i386-linux-gnu/kamailio/modules/ims_ipsec_pcscf.so (gdb) bt full #0 0xb6f1eab9 in ?? () from /usr/lib/i386-linux-gnu/kamailio/modules/ims_ipsec_pcscf.so No symbol table info available. #1 0xb6f25b1e in ipsec_forward () from /usr/lib/i386-linux-gnu/kamailio/modules/ims_ipsec_pcscf.so No symbol table info available. #2 0xb6f2901a in ?? () from /usr/lib/i386-linux-gnu/kamailio/modules/ims_ipsec_pcscf.so No symbol table info available. #3 0x004bc49d in do_action () No symbol table info available. #4 0x004bae10 in run_actions () No symbol table info available. #5 0x004ca3a0 in run_top_route () No symbol table info available. #6 0xb73c1695 in reply_received () from /usr/lib/i386-linux-gnu/kamailio/modules/tm.so No symbol table info available. #7 0x0052fa14 in ?? () No symbol table info available. #8 0x005b9170 in receive_msg () No symbol table info available. #9 0x006bb830 in udp_rcv_loop () No symbol table info available. #10 0x004b74e8 in main_loop () No symbol table info available. #11 0x004ab0f1 in main () No symbol table info available. ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` Dec 10 15:33:07 pcscf /usr/sbin/kamailio[14194]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set(): script error-invalid id_set to be selected Dec 10 15:33:07 pcscf /usr/sbin/kamailio[14194]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2 Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14204]: ERROR: <script>: INVITE (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14204]: ERROR: ims_ipsec_pcscf [cmd.c:806]: ipsec_forward(): Contact doesn't exist Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14204]: ERROR: ims_ipsec_pcscf [cmd.c:806]: ipsec_forward(): Contact doesn't exist Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14193]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: <script>: INVITE (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set(): script error-invalid id_set to be selected Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2 Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14205]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set(): script error-invalid id_set to be selected Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14205]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2 Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14210]: ERROR: ims_ipsec_pcscf [cmd.c:252]: fill_contact(): Reply No contact headers Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14210]: ERROR: ims_ipsec_pcscf [cmd.c:799]: ipsec_forward(): Error filling in contact data Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14201]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set(): script error-invalid id_set to be selected Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14201]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2 Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14274]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 22 Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14191]: ALERT: <core> [main.c:782]: handle_sigs(): child process 14196 exited by a signal 11 Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14191]: ALERT: <core> [main.c:785]: handle_sigs(): core was generated Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: WARNING: <core> [core/daemonize.c:596]: mem_lock_pages(): failed to lock the memory pages (disable swap): Cannot allocate memory [12] Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set(): script error-invalid id_set to be selected Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: WARNING: tm [tm.c:521]: fixup_routes(): t_on_failure("NATMANAGE"): empty/non existing route Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: ERROR: <script>: event_route[htable:mod-init] { Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: ERROR: ims_ipsec_pcscf [cmd.c:950]: ipsec_destroy(): Contact doesn't exist Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14400]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14406]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14403]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14405]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14401]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14404]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14402]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14399]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14396]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14334]: ERROR: <script>: PRACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14334]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14334]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14337]: ERROR: <script>: PRACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14337]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14337]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14329]: ERROR: <script>: PRACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14329]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14329]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14341]: ERROR: <script>: PRACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14341]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14341]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14328]: ERROR: <script>: ACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14328]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14328]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14326]: ERROR: <script>: ACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14326]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14326]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14327]: ERROR: <script>: BYE (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14327]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14327]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: <script>: BYE (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa) Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set(): script error-invalid id_set to be selected Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2 Dec 10 15:35:30 pcscf /usr/sbin/kamailio[14396]: ERROR: <script>: Preloading NAT-PING. Rows: 921 Dec 10 15:35:30 pcscf /usr/sbin/kamailio[14396]: ERROR: <script>: OPTIONS to sip:alice@192.168.56.107:57838;transport=udp via sip:192.168.56.107:57838... ```
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` (paste your sip traffic here)
``` [pcscf_dump.zip](https://github.com/kamailio/kamailio/files/7692913/pcscf_dump.zip)
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.4.4 (i386/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 10.2.1 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) -->
``` Linux pcscf.ims.mnc001.mcc001.3gppnetwork.org 5.10.0-9-686-pae #1 SMP Debian 5.10.70-1 (2021-09-30) i686 GNU/Linux ``` [kamailio_cfg.zip](https://github.com/kamailio/kamailio/files/7692905/kamailio_cfg.zip) [kamailio_log.zip](https://github.com/kamailio/kamailio/files/7692934/kamailio_log.zip)
Colleagues, there is an opportunity to give a hint why it crashes ims_ipsec_pcscf ?
Hi Eugeniy, You need to install debug symbols as Henning mentioned and provide the backtrace with symbols to investigate. If possible provide P-cscf logs and Wireshark trace.
BR Aleksandar Yosifov
On Tue, Dec 28, 2021 at 2:52 PM Eugeniy notifications@github.com wrote:
Colleagues, there is an opportunity to give a hint why it crashes ims_ipsec_pcscf ?
— Reply to this email directly, view it on GitHub https://github.com/kamailio/kamailio/issues/2970#issuecomment-1002089365, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABO7UZO5JE2IPFKDMH4ESLLUTGXFNANCNFSM5JZDNT7Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message ID: kamailio/kamailio/issues/2970/1002089365@github.com _______________________________________________ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Thanks for the report, can you install maybe kamailio debug symbols (e.g. from packages) and do the backtrace again? The initial one is missing any detailed information.
Thanks for the answer! Did I understand correctly that I need to install "kamailio-dbg"? I installed Kamailio via "apt install" from the repository.
kamailio-dbg should do the trick
kamailio-dbg должен помочь
I apologize, but the packages cannot install kamailio-dbg on Debian, the repositories do not have any kamailio-dbg, kamailio-debbug, kamailio-dbg. Maybe I need to include from the debbig module parameters https://www.kamailio.org/docs/modules/devel/modules/debugger.html?
At least in the kamailio debian repository they are there:
root@sbct1: # apt-cache policy kamailio-dbg kamailio-dbg: Installiert: (keine) Installationskandidat: 5.5.3+bpo11 Versionstabelle: 5.5.3+bpo11 500 500 http://deb.kamailio.org/kamailio55 bullseye/main amd64 Packages
If you're using the packages from official Debian repo try adding following to your sources list
deb http://deb.debian.org/debian-debug bullseye main
On Tue, Dec 28, 2021, 15:00 Henning Westerholt notifications@github.com wrote:
At least in the kamailio debian repository they are there:
root@sbct1: # apt-cache policy kamailio-dbg kamailio-dbg: Installiert: (keine) Installationskandidat: 5.5.3+bpo11 Versionstabelle: 5.5.3+bpo11 500 500 http://deb.kamailio.org/kamailio55 bullseye/main amd64 Packages
— Reply to this email directly, view it on GitHub https://github.com/kamailio/kamailio/issues/2970#issuecomment-1002119121, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABO7UZO7MWDJEOVSWZFXL5TUTG7HRANCNFSM5JZDNT7Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message ID: kamailio/kamailio/issues/2970/1002119121@github.com _______________________________________________ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Thanks for the tips. The old core is gone, but the new one.
``` GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: https://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /sbin/kamailio... Reading symbols from /usr/lib/debug/.build-id/f0/69ca129a40c5415b1162a2672ca9a1eb6443a0.debug...
warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing [New LWP 32251] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'. Program terminated with signal SIGABRT, Aborted. #0 0xb7fcb559 in __kernel_vsyscall () (gdb) bt full #0 0xb7fcb559 in __kernel_vsyscall () No symbol table info available. #1 0xb7cc2e02 in __libc_signal_restore_set (set=0xbfb295ec) at ../sysdeps/unix/sysv/linux/internal-signals.h:86 resultvar = <optimized out> #2 __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48 set = {__val = {8192, 0, 0, 2021, 0, 0, 3083523171, 400, 0, 3216152096, 3085381632, 3085383488, 79848, 1, 0, 1, 23021476, 23021468, 15, 23021468, 1844847104, 3216152452, 0, 3083925755, 3085383488, 23021448, 3086841612, 3083491989, 5012815, 3085381632, 0, 3216152352}} pid = <optimized out> tid = <optimized out> ret = 0 #3 0xb7cab306 in __GI_abort () at abort.c:79 save_stage = <optimized out> act = {__sigaction_handler = {sa_handler = 0xb7fc5c70, sa_sigaction = 0xb7fc5c70}, sa_mask = {__val = {0, 10800, 22903096, 0, 0, 0, 0, 0, 4979331, 1, 0, 9883664, 3086866326, 4979331, 3086977408, 3216152596, 3086977856, 3086770192, 1, 1, 0, 0, 9883648, 3086974976, 4927488, 3216152744, 3084441447, 130, 3083435384, 3216152632, 0, 3086866153}}, sa_flags = 9883648, sa_restorer = 0x897078} sigs = {__val = {32, 0 <repeats 31 times>}} #4 0x004d14cf in sig_alarm_abort (signo=14) at main.c:699 __func__ = "sig_alarm_abort" #5 <signal handler called> No symbol table info available. #6 0xb7fcb557 in __kernel_vsyscall () No symbol table info available. #7 0xb7d8da07 in syscall () at ../sysdeps/unix/sysv/linux/i386/syscall.S:29 No locals. #8 0xb6f3f936 in futex_get (lock=0xb08d3b6c) at ../../core/mem/../futexlock.h:121 v = 1 i = -1225347072 #9 0xb6f41668 in lock_ulslot (_d=0xb08e5b88, i=206) at udomain.c:295 No locals. #10 0xb6f40be1 in free_udomain (_d=0xb08e5b88) at udomain.c:156 i = 206 __func__ = "free_udomain" #11 0xb6f36508 in free_all_udomains () at dlist.c:296 ptr = 0xb08e5b04 __func__ = "free_all_udomains" #12 0xb6f39c69 in destroy () at ims_usrloc_pcscf_mod.c:253 No locals. #13 0x006d3660 in destroy_modules () at core/sr_module.c:839 t = 0xb75253a4 foo = 0xb7524d3c __func__ = "destroy_modules" #14 0x004d002b in cleanup (show_status=1) at main.c:575 memlog = 0 __func__ = "cleanup" #15 0x004d182b in shutdown_children (sig=15, show_status=1) at main.c:718 __func__ = "shutdown_children" #16 0x004d4612 in handle_sigs () at main.c:816 chld = 0 chld_status = 139 any_chld_stopped = 1 memlog = 5041487 __func__ = "handle_sigs" #17 0x004e0af6 in main_loop () at main.c:1903 --Type <RET> for more, q to quit, c to continue without paging-- i = 16 pid = 32331 si = 0x0 si_desc = "udp receiver child=15 sock=0.0.0.0:5064\000\060:5060\000\277\000Ж\000\371}\000\000\254\245\262\277裲\277\a\267e\000\210\234V\267\f\000\000\000\060\232Q\267ζe\000\000\000\000\000\226\301\215\000\005\000\000\000\000\000\000\000\334uTue De\030\244\262\277\365\267e\000\344(\232\000\332\vN\000\062\061\n" nrprocs = 16 woneinit = 1 __func__ = "main_loop" #18 0x004e9bd0 in main (argc=9, argv=0xbfb2a884) at main.c:3053 cfg_stream = 0x15101c0 c = -1 r = 0 tmp = 0xbfb2ae7a "" tmp_len = 1 port = 0 proto = -1209441992 ahost = 0x0 aport = 0 options = 0x899ef0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2727312077 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xbfb2a67f p = 0x1 <error: Cannot access memory at address 0x1> st = {st_dev = 22, __pad1 = 0, st_ino = 931, st_mode = 16888, st_nlink = 2, st_uid = 111, st_gid = 115, st_rdev = 0, __pad2 = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1640709398, tv_nsec = 701244783}, st_mtim = {tv_sec = 1640709398, tv_nsec = 701244783}, st_ctim = {tv_sec = 1640709398, tv_nsec = 701244783}, __glibc_reserved4 = 0, __glibc_reserved5 = 0} tbuf = "\000\000\000\000\000\000\000\000\004\000\000\020\000\360\377\377Linux", '\000' <repeats 60 times>, "pcscf.ims.mnc001.mcc001.3gppnetwork.org", '\000' <repeats 26 times>, "\065.10.0-9-686-pae", '\000' <repeats 22 times>, "\fw\375\267\225ZʷO}L", '\000' <repeats 16 times>... option_index = 0 long_options = {{name = 0x89c26a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x8974d6 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x89c26f "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x89c275 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x89c27b "substdef", has_arg = 1, flag = 0x0, val = 1026}, { name = 0x89c284 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x89c28e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x89c298 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x89c2a3 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x89c2ac "log-engine", has_arg = 1, flag = 0x0, val = 1031}, { name = 0x89c2b7 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x89c2bd "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x89c2c7 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main"
```
Updated Kamailio to: ``` version: kamailio 5.5.3 (i386/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 10.2.1 ```
It looks like the second core dump back trace was from another child that stopped because of an error in another child. Do you have only this core dump, or are there others? If more, try to find the one with the process ID from the first log file output related to the crash.
Sorry, wrong core ``` root@pcscf:/usr/local/src/kamailio# gdb /sbin/kamailio /tmp/core-kamailio-11-111-115-32593-1640711154 GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: https://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /sbin/kamailio... Reading symbols from /usr/lib/debug/.build-id/f0/69ca129a40c5415b1162a2672ca9a1eb6443a0.debug...
warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing [New LWP 32593] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0xb6f06afa in fill_contact (ci=0xbfd2249c, m=0xb7555b1c) at cmd.c:271
warning: Source file is more recent than executable. 271 ci->received_host.len = ip_addr2sbuf(&req->rcv.src_ip, srcip, 50); (gdb) bt full #0 0xb6f06afa in fill_contact (ci=0xbfd2249c, m=0xb7555b1c) at cmd.c:271 t = 0xb092ef0c cb = 0xb75a9d54 vb = 0xb75cc99c req = 0xb09308e8 srcip = 0x0 __func__ = "fill_contact" #1 0xb6f107e3 in ipsec_forward (m=0xb7555b1c, d=0xb08e6b88, _cflags=0) at cmd.c:799 ci = {searchflag = 0, extra_search_criteria = 0, aor = {s = 0x0, len = 0}, via_host = { s = 0x9bf4e6 <buf+454> "192.168.56.1:64334;rport=64334;received=192.168.56.1;branch=z9hG4bK527528804\r\nRecord-Route: sip:mt@192.168.56.120;lr=on;ftag=527524296;rm=7\r\nRecord-Route: <sip:mt@192.168.56.106:6060;lr=on;ftag=5275"..., len = 12}, via_port = 64334, via_prot = 1, received_host = {s = 0x0, len = 0}, received_port = 0, received_proto = 0, path = 0x0, expires = 0, callid = 0x0, public_ids = 0x0, num_public_ids = 0, service_routes = 0x0, num_service_routes = 0, rx_regsession_id = 0x0, reg_state = PCONTACT_ANY} pcontact = 0x0 ret = -1 dst_proto = 1 '\001' dst_port = 0 src_port = 0 via_host = {af = 3075613068, len = 3075665560, u = {addrl = {6627768, 3076040100, 3084758229, 9719808}, addr32 = {6627768, 3076040100, 3084758229, 9719808}, addr16 = {8632, 101, 42404, 46936, 44245, 47069, 20480, 148}, addr = "\270!e\000\244\245X\267լݷ\000P\224"}} req = 0xb09308e8 __func__ = "ipsec_forward" s = 0x0 buf = "\345\211M\000\024'ҿxx\\267\034[U\267\240\200M\000\027\264\260\364h/\362\313\000\000\000\000p'ҿ\020\300H\267\320%ҿ\004`[\267\067WH\267\000\000\000\000\034[U\267\220\230\\267\000#c\027\000\000\200\000l\270j\000\064\031\017\000\000P\224\000<8ҿ\024*ҿ\030'ҿB"e\000\364\063ҿ\034[U\267p'ҿ\244\225T\267\000P\224\000\034[U\267xx\\267\024'ҿ\000P\224\000\000\000\000\000\377\377\377\377(\354H\267\311\000\000\000\000P\224\000\350&ҿ", '\000' <repeats 16 times>, "h\332d\000\364\063ҿ\220\241T\267\034[U\267N\331d\000\001\000\000\000\000\000\000\000\370&ҿ\000\000\000\000"... buf_len = 0 client_sock = 0x0 dst_info = {send_sock = 0x5f1b92 <pv_value_destroy+187>, to = {s = {sa_family = 9620, sa_data = "ҿ\000\000\000\000\020\000\000\000\000#c\027"}, sin = {sin_family = 9620, sin_port = 49106, sin_addr = {s_addr = 0}, sin_zero = "\020\000\000\000\000#c\027"}, sin6 = {sin6_family = 9620, sin6_port = 49106, sin6_flowinfo = 0, sin6_addr = {__in6_u = { __u6_addr8 = "\020\000\000\000\000#c\027\000P\224\000<8ҿ", __u6_addr16 = {16, 0, 8960, 5987, 20480, 148, 14396, 49106}, __u6_addr32 = {16, 392372992, 9719808, 3218225212}}}, sin6_scope_id = 3218220520}, sas = {ss_family = 9620, __ss_padding = "ҿ\000\000\000\000\020\000\000\000\000#c\027\000P\224\000<8ҿ\350%ҿh\332d\000\364\063ҿP\223T\267\034[U\267\000#c\027 `[\267\214=ҿx%ҿ\000P\224\000<8ҿ\024*ҿ8&ҿ\241\221M\000\204%ҿ\220\241T\267\034[U\267\004\221M\000\200%ҿ\034[U\267\220\241T\267\000#c\027\200%ҿլݷ\000P\224", __ss_align = 9719808}}, id = -1076742084, send_flags = {f = 20480, blst_imask = 148}, proto = -40 '\330', proto_pad0 = 38 '&', proto_pad1 = -16430} #2 0xb6f1a793 in w_forward (_m=0xb7555b1c, _d=0xb08e6b88 "\004k\216\260", _cflags=0x0) at ims_ipsec_pcscf_mod.c:421 No locals. #3 0x004cb1fd in do_action (h=0xbfd23264, a=0xb75c918c, msg=0xb7555b1c) at core/action.c:1082 ret = -5 v = 0 dst = {send_sock = 0x7757ce <qm_info+12>, to = {s = {sa_family = 49168, sa_data = "H\267\000P\224\000H.ҿ\020\300H\267"}, sin = {sin_family = 49168, sin_port = 46920, sin_addr = { s_addr = 9719808}, sin_zero = "H.ҿ\020\300H\267"}, sin6 = {sin6_family = 49168, sin6_port = 46920, sin6_flowinfo = 9719808, sin6_addr = {__in6_u = { __u6_addr8 = "H.ҿ\020\300H\267\005\000\000\000\000\260H\267", __u6_addr16 = {11848, 49106, 49168, 46920, 5, 0, 45056, 46920}, __u6_addr32 = {3218222664, 3074998288, 5, 3074994176}}}, sin6_scope_id = 3218222584}, sas = {ss_family = 49168, __ss_padding = "H\267\000P\224\000H.ҿ\020\300H\267\005\000\000\000\000\260H\267\370-ҿ\252WH\267\020\300H\267\320-ҿ\330\310\\267\067WH\267\020.ҿ|.ҿx.ҿ6\365R\000\000\000\200\000\224\270j\000\330\030\017\000pЎ\000\350\203\025\000\004\000\000\000\020\000\000\000\000#c\027b\000\000\000\224z5\267\330.ҿ\a\266}\000pЎ\000\000\000\000\000\000\004\000", __ss_align = 3218222648}}, id = -1221231911, send_flags = {f = 24448, blst_imask = 45177}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} tmp = 0xb7d4ea7f <__strftime_internal+1359> "\203\304\020\003l$\030\003|$\b\351`\373\377\377\213\$\004\203\376E\017\205\310\375\377\377\351\063\375\377\377\211\$\004\061\366\213D$ \205\300\017DD$\034\211D$\034\203\376E\017\204\021\375\377\377\213D$$\213X\020\211ڃ\376O\017\204\350\024" new_uri = 0xc859 <error: Cannot access memory at address 0xc859> end = 0x1dad7a8 "" crt = 0x1dab7b2 "45 /usr/sbin/kamailio[32593]: ERROR: <script>: NOTIFY (sip:711122330@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:711122330@ims.mnc001.mcc001.3gppnetwork.org, bddc50a5-f6da-67cd-0222"... --Type <RET> for more, q to quit, c to continue without paging-- cmd = 0xb7526698 len = -1076745526 user = 2 uri = {user = {s = 0x17632300 <error: Cannot access memory at address 0x17632300>, len = 31111072}, passwd = {s = 0xe9 <error: Cannot access memory at address 0xe9>, len = -1211037445}, host = {s = 0xb7e74000 "lM\036", len = 21}, port = {s = 0xb7e74000 "lM\036", len = -1076745032}, params = {s = 0xb7d8e41d <__vsyslog_internal+733> "\203\304\020\213E\344e+\005\024", len = 31111072}, sip_params = { s = 0x1dab7a0 "\260\222\332\001\020 \314\001 28 20:05:45 /usr/sbin/kamailio[32593]: ERROR: <script>: NOTIFY (sip:711122330@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:711122330@ims.mnc001.mcc001.3gppnetwork.org, bddc5"..., len = 233}, headers = {s = 0x4000 <error: Cannot access memory at address 0x4000>, len = -1219295780}, port_no = 11572, proto = 49106, type = 31119264, flags = (unknown: 0xbfd22ce8), transport = {s = 0xb70ca50a "%s: %.*s%s%s%s%.*s", len = 11}, ttl = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 31042192}, user_param = {s = 0xbfd22d50 "\350\203\025", len = -1218584096}, maddr = {s = 0x61cb43e9 <error: Cannot access memory at address 0x61cb43e9>, len = 31111072}, method = {s = 0xe9 <error: Cannot access memory at address 0xe9>, len = 45}, lr = {s = 0x5 <error: Cannot access memory at address 0x5>, len = 20}, r2 = { s = 0x1c <error: Cannot access memory at address 0x1c>, len = 11}, gr = {s = 0x79 <error: Cannot access memory at address 0x79>, len = 2}, transport_val = { s = 0x169 <error: Cannot access memory at address 0x169>, len = 0}, ttl_val = {s = 0x2a30 <error: Cannot access memory at address 0x2a30>, len = 30972216}, user_param_val = { s = 0x4 <error: Cannot access memory at address 0x4>, len = -1210209067}, maddr_val = {s = 0x945000 "\260\236K", len = 7821290}, method_val = {s = 0xbfd22cc0 "\203", len = 0}, lr_val = {s = 0x1c <error: Cannot access memory at address 0x1c>, len = 5516737}, r2_val = {s = 0xbfd22ca8 "\370-ҿ\345\211M", len = -1210204419}, gr_val = {s = 0x945000 "\260\236K", len = 392372992}} next_hop = {user = {s = 0xb744b841 "tm: t_reply.c", len = -1220128768}, passwd = {s = 0xbfd22ed8 "h0", len = 9719808}, host = {s = 0xbfd2383c "", len = 9719808}, port = { s = 0xbfd22df8 "\330.ҿ\a\266}", len = 5081573}, params = {s = 0xbfd233f4 "", len = -1219190040}, sip_params = {s = 0xb7555b1c "\n", len = 5079200}, headers = { s = 0x83 <error: Cannot access memory at address 0x83>, len = -1223908086}, port_no = 11496, proto = 49106, type = ERROR_URI_T, flags = (unknown: 0xbfd2383c), transport = { s = 0xbfd22fd4 "ب\\267\001", len = -1076744968}, ttl = {s = 0xb70c385b <xlog_helper+3092> "\203\304\060\213u\234\213\206\364\377\377\377\213", len = 131}, user_param = { s = 0xb7ddacd5 <__memset_sse2_rep+37> "\201\303;\206\004", len = 9719808}, maddr = {s = 0x7757ea <qm_info+40> "\203\304\020\213E\364\213P\004\213E\f\211\020\213E\f\307@\024\004", len = -1076744896}, method = {s = 0x0, len = 28}, lr = {s = 0x7757ce <qm_info+12> "\005\062\370\034", len = 168}, r2 = { s = 0xb75de1e0 "NOTIFY (sip:711122330@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:711122330@ims.mnc001.mcc001.3gppnetwork.org, bddc50a5-f6da-67cd-0222-9eef1e619cdc)\n", len = -1218584096}, gr = {s = 0xb748c010 "\001", len = -1076744864}, transport_val = {s = 0xb748b000 "\b\016\001", len = -1076744856}, ttl_val = { s = 0xb74857aa <pkg_proc_update_stats+127> "\203\304\020\213\223h\f", len = -1219969008}, user_param_val = {s = 0xbfd22d40 "\372\030", len = -1218673468}, maddr_val = { s = 0x787ef8 <parse_headers+17> "\201\303\b\321\033", len = 0}, method_val = {s = 0x0, len = -1076744728}, lr_val = {s = 0x0, len = 6394}, r2_val = {s = 0x0, len = 8}, gr_val = { s = 0x0, len = 1410024}} u = 0x0 port = 0 dst_host = 0xb7e74740 <main_arena> i = 8180 flags = -1218559940 avp = 0xb7e1b0ee st = {flags = 3085385728, id = 18240, name = {n = 240, s = {s = 0xf0 <error: Cannot access memory at address 0xf0>, len = 0}, re = 0xf0}, avp = 0x5} sct = 0x1dab7ac sjt = 0x17632300 rve = 0xb75c7ebc mct = 0xf rv = 0x3 rv1 = 0x2 c1 = {cache_type = 392372992, val_type = RV_NONE, c = {avp_val = {n = 9719808, s = {s = 0x945000 "\260\236K", len = -1076744808}, re = 0x945000}, pval = {rs = {s = 0x945000 "\260\236K", len = -1076744808}, ri = 5517120, flags = 0}}, i2s = "լݷ\000P\224\000\352Ww\000\320-ҿ\000\000\000\000\034"} s = {s = 0xb7d0e2e9 <__GI__IO_str_seekoff+9> "\201\303\027]\026", len = -1209581568} srevp = {0x4, 0x12} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #4 0x004d89e5 in run_actions (h=0xbfd23264, a=0xb75c90e0, msg=0xb7555b1c) at core/action.c:1581 --Type <RET> for more, q to quit, c to continue without paging-- t = 0xb75c918c ret = 1 tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} tz = {tz_minuteswest = 10220721, tz_dsttime = 32} tdiff = 0 __func__ = "run_actions" #5 0x004d92ab in run_top_route (a=0xb75c90e0, msg=0xb7555b1c, c=0xbfd23264) at core/action.c:1666 ctx = {rec_lev = 8018553, run_flags = -1076743968, last_retcode = -1219142884, jmp_env = {{__jmpbuf = {10220616, -1218664232, 1, 10220670, 9298587, -1220742057}, __mask_was_saved = 9299260, __saved_mask = {__val = {3074998288, 3076311284, 0, 1024, 3218223144, 3074225352, 2962419868, 0, 1, 3, 9334117, 5, 0, 3074838528, 3218225212, 3218223176, 3074229440, 2962419868, 3218223184, 3076303036, 3074229424, 3218225212, 3074225146, 3218223224, 5436726, 2962419868, 3074838528, 5462975, 3218223224, 5463391, 0, 3218223464}}}}} p = 0xbfd23264 ret = 2 sfbk = 0 #6 0xb73ebb7b in reply_received (p_msg=0xb7555b1c) at t_reply.c:2541 msg_status = 183 last_uac_status = 100 ack = 0xb75ca8d0 "H\001" ack_len = 7821262 branch = 0 reply_status = 9299260 onreply_route = 3 cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 0}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 0}}}} uac = 0xb092f0e4 t = 0xb092ef0c lack_dst = {send_sock = 0x9bf4aa <buf+394>, to = {s = {sa_family = 43188, sa_data = "\\267\000#c\027+\000\000\000լݷ"}, sin = {sin_family = 43188, sin_port = 46940, sin_addr = { s_addr = 392372992}, sin_zero = "+\000\000\000լݷ"}, sin6 = {sin6_family = 43188, sin6_port = 46940, sin6_flowinfo = 392372992, sin6_addr = {__in6_u = { __u6_addr8 = "+\000\000\000լݷ\000P\224\000\352Ww", __u6_addr16 = {43, 0, 44245, 47069, 20480, 148, 22506, 119}, __u6_addr32 = {43, 3084758229, 9719808, 7821290}}}, sin6_scope_id = 3218223680}, sas = {ss_family = 43188, __ss_padding = "\\267\000#c\027+\000\000\000լݷ\000P\224\000\352Ww\000@2ҿլݷ\372\030\000\000\000\000\000\000P2ҿ\000\000\000\000\034\000\000\000\316Ww\000\020\300H\267լݷ\000P\224\000\020\300H\267p2ҿլݷ\000P\224\000\352Ww\000\200\062ҿ\000\000\000\000\034\000\000\000\316Ww\000\000\000\200\000\000\260H\267\230\062ҿ\370~x\000\000\000\000", __ss_align = 0}}, id = -1076743512, send_flags = {f = 22442, blst_imask = 46920}, proto = -6 '\372', proto_pad0 = 24 '\030', proto_pad1 = 0} backup_user_from = 0x9cf330 <def_list+8> backup_user_to = 0x9cf334 <def_list+12> backup_domain_from = 0x9cf338 <def_list+16> backup_domain_to = 0x9cf33c <def_list+20> backup_uri_from = 0x9cf328 <def_list> backup_uri_to = 0x9cf32c <def_list+4> backup_xavps = 0x9cf3d4 <_xavp_list_head> backup_xavus = 0x9cf3d8 <_xavu_list_head> backup_xavis = 0x9cf3dc <_xavi_list_head> replies_locked = 1 branch_ret = -1332853704 prev_branch = -1219973120 blst_503_timeout = -1218656180 hf = 0x1 onsend_params = {req = 0x8de29b, rpl = 0xbfd231cc, param = 0x18fa, code = 0, flags = 2, branch = 0, t_rbuf = 0x0, dst = 0xb748c010, send_buf = {s = 0xb75ca884 "\f", len = -1218651916}} ctx = {rec_lev = 1, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {9719808, -1076742084, -1219142412, -1076744312, -190565353, -873320600}, __mask_was_saved = 0, __saved_mask = {__val = {1394140, 1410024, 4, 23, 3075824412, 8, 9719808, 3218223832, 5517120, 0, 0, 0, 5516737, 6392, 0, 2296, 0, 8299492, 3084758229, 9719808, 5436726, 10221705, 3218223880, 152, 0, 16, 0, 248, 0, 24, 0, 3218224088}}}}} bctx = 0x0 keng = 0x0 --Type <RET> for more, q to quit, c to continue without paging-- ret = 9719808 evname = {s = 0xb744e8bb "on_sl_reply", len = 11} __func__ = "reply_received" #7 0x00556110 in do_forward_reply (msg=0xb7555b1c, mode=0) at core/forward.c:764 new_buf = 0x0 dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 0, __ss_padding = '\000' <repeats 121 times>, __ss_align = 0}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} new_len = 4 r = 2 ip = {af = 7783723, len = 0, u = {addrl = {9719808, 3074937938, 3074994176, 3218224120}, addr32 = {9719808, 3074937938, 3074994176, 3218224120}, addr16 = {20480, 148, 54354, 46919, 45056, 46920, 13304, 49106}, addr = "\000P\224\000R\324G\267\000\260H\267\370\063ҿ"}} s = 0xbfd23390 "\376\377\377\377" len = -1219142912 __func__ = "do_forward_reply" #8 0x00558062 in forward_reply (msg=0xb7555b1c) at core/forward.c:865 No locals. #9 0x00609def in receive_msg ( buf=0x9bf320 <buf> "SIP/2.0 183 Session in Progress\r\nFrom: sip:711122330@ims.mnc001.mcc001.3gppnetwork.org;tag=527524296\r\nTo: sip:711122331@ims.mnc001.mcc001.3gppnetwork.org;tag=18016628\r\nContact: <sip:711122331@192."..., len=1385, rcv_info=0xbfd23774) at core/receive.c:587 msg = 0xb7555b1c ctx = {rec_lev = 4763648, run_flags = -1076742504, last_retcode = -1209552940, jmp_env = {{__jmpbuf = {-1208194672, -1211497464, 9242371, -1076742596, -1208097047, 9719808}, __mask_was_saved = -1218656232, __saved_mask = {__val = {3218225200, 3218224808, 3086895344, 3218225020, 3084528336, 56335, 3086981504, 744, 7583757, 44055, 7583287, 3086774304, 7583694, 1, 0, 1024, 12, 1385, 10220320, 3076310936, 3076310936, 0, 0, 3218224756, 3085481001, 9719808, 9, 10220320, 5436726, 0, 3076310936, 3218224868}}}}} bctx = 0x0 ret = 0 tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} diff = 0 inb = { s = 0x9bf320 <buf> "SIP/2.0 183 Session in Progress\r\nFrom: sip:711122330@ims.mnc001.mcc001.3gppnetwork.org;tag=527524296\r\nTo: sip:711122331@ims.mnc001.mcc001.3gppnetwork.org;tag=18016628\r\nContact: <sip:711122331@192."..., len = 1385} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0} keng = 0x0 evp = {data = 0xbfd23580, obuf = {s = 0x0, len = 0}, rcv = 0xbfd23774, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 exectime = 0 __func__ = "receive_msg" #10 0x00744552 in udp_rcv_loop () at core/udp_server.c:543 len = 1385 buf = "SIP/2.0 183 Session in Progress\r\nFrom: sip:711122330@ims.mnc001.mcc001.3gppnetwork.org;tag=527524296\r\nTo: sip:711122331@ims.mnc001.mcc001.3gppnetwork.org;tag=18016628\r\nContact: <sip:711122331@192."... tmp = 0x0 fromaddr = 0xb75cc798 fromaddrlen = 16 rcvi = {src_ip = {af = 2, len = 4, u = {addrl = {1782098112, 0, 0, 0}, addr32 = {1782098112, 0, 0, 0}, addr16 = {43200, 27192, 0, 0, 0, 0, 0, 0}, addr = "\300\250\070j", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {2016979136, 0, 0, 0}, addr32 = {2016979136, 0, 0, 0}, addr16 = {43200, 30776, 0, 0, 0, 0, 0, 0}, addr = "\300\250\070x", '\000' <repeats 11 times>}}, src_port = 6060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\027\254\300\250\070j\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 44055, sin_addr = {s_addr = 1782098112}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 44055, sin6_flowinfo = 1782098112, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, --Type <RET> for more, q to quit, c to continue without paging-- __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, __ss_padding = "\027\254\300\250\070j", '\000' <repeats 115 times>, __ss_align = 0}}, bind_address = 0xb7517f20, rflags = 0, proto = 1 '\001', proto_pad0 = 0 '\000', proto_pad1 = 0} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} printbuf = '\000' <repeats 16 times>, "\066\365R", '\000' <repeats 14 times>, "#c\027\000\000\000\000\360!\235\000\300!\235\000\000P\224\000\000P\224\000\214=ҿ\030\071ҿ\025\273j\000\020\000\000\000\004\000\000\000\326&\373\323F\267j\000(\336cE%\371ЩJO\004a\000#c\027jπ\000j\354sY\310\070ҿn\251\200\000`+\235\000`,\235\000\000P\224\000\364\233R\267\000\000\000\000\000P\224\000\350\070ҿA\251\200\000X+\235\000\340!\235\000 \000\000\000\001\000\000\000\000\000\000\000\000P\224\000\b9ҿ6\365R\000X+\235\000\340!\235\000 \000\000\000\000#c\027\000\000\000\000\000\000\000\000\070\071ҿ"... i = 15 j = 0 l = 0 __func__ = "udp_rcv_loop" #11 0x004b5d52 in main_loop () at main.c:1730 i = 15 pid = 0 si = 0xb7517f20 si_desc = "udp receiver child=15 sock=192.168.56.120:5060\000\277\000P\224\000;\177\000\000\214=ҿ\310;ҿ\a7c\000\210\254V\267\000\004\000\000\060\252Q\267\316\066c\000Ie\213\000\226A\213\000\005\000\000\000\000\000\000\000܅Tue De\370;ҿ\365\067c\000䨗\000ڋK\000\062\061\n" nrprocs = 16 woneinit = 1 __func__ = "main_loop" #12 0x004c1bd0 in main (argc=9, argv=0xbfd24064) at main.c:3053 cfg_stream = 0x1cc21c0 c = -1 r = 0 tmp = 0xbfd24e7a "" tmp_len = 1 port = 0 proto = -1209437896 ahost = 0x0 aport = 0 options = 0x871ef0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2926354237 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xbfd23e5f p = 0x1 <error: Cannot access memory at address 0x1> st = {st_dev = 22, __pad1 = 0, st_ino = 952, st_mode = 16888, st_nlink = 2, st_uid = 111, st_gid = 115, st_rdev = 0, __pad2 = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1640710116, tv_nsec = 690868980}, st_mtim = {tv_sec = 1640710116, tv_nsec = 690868980}, st_ctim = {tv_sec = 1640710116, tv_nsec = 690868980}, __glibc_reserved4 = 0, __glibc_reserved5 = 0} tbuf = "\000\000\000\000\000\000\000\000\004\000\000\020\000\360\377\377Linux", '\000' <repeats 60 times>, "pcscf.ims.mnc001.mcc001.3gppnetwork.org", '\000' <repeats 26 times>, "\065.10.0-9-686-pae", '\000' <repeats 22 times>, "\f\207\375\267\225jʷO\375I", '\000' <repeats 16 times>... option_index = 0 long_options = {{name = 0x87426a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x86f4d6 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x87426f "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x874275 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x87427b "substdef", has_arg = 1, flag = 0x0, val = 1026}, { name = 0x874284 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x87428e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x874298 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x8742a3 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x8742ac "log-engine", has_arg = 1, flag = 0x0, val = 1031}, { name = 0x8742b7 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x8742bd "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x8742c7 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" (gdb) ```
Can this bt help with the concept of the problem? Or is it better to take the dump again, attach the log and bt?
Hello, it crashed during the processing of an 183 message. Crashed function was fill_contact(..) in frame 0. You could investigate the passed data in this function to see why it crashed. Another idea would be to look to the SIP message that was processed to see if there was something unusual in it.
@alexyosifov If I do not misread it, I think the second backtrace was already done with debug symbols installed.
Hi Eugeniy, You need to install debug symbols as Henning mentioned and provide the backtrace with symbols to investigate. If possible provide P-cscf logs and Wireshark trace. BR Aleksandar Yosifov […](#) On Tue, Dec 28, 2021 at 2:52 PM Eugeniy ***@***.***> wrote: Colleagues, there is an opportunity to give a hint why it crashes ims_ipsec_pcscf ? — Reply to this email directly, view it on GitHub <[#2970 (comment)](https://github.com/kamailio/kamailio/issues/2970#issuecomment-1002089365)%3E, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABO7UZO5JE2IPFKDMH4ESLLUTGXFNANCNFSM5JZDNT7Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> _______________________________________________ Kamailio (SER) - Development Mailing List ***@***.*** https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Hi, Aleksandar.
In the archive in the attachment a new dump, log and bt with debug symbols [pcscf_crashes.zip](https://github.com/kamailio/kamailio/files/7837838/pcscf_crashes.zip)
Hello, it crashed during the processing of an 183 message. Crashed function was fill_contact(..) in frame 0. You could investigate the passed data in this function to see why it crashed. Another idea would be to look to the SIP message that was processed to see if there was something unusual in it.
Hey! Thanks for the advice, but don't tell me where the fill_contact fields are filled in from the SIP packets
It seems that problem is not in contact parsing. I think that object are in failed some state
In core dump we have what it seems to be a valid Contact field
``` #0 0x00007f0b29e26979 in fill_contact (ci=0x7fffff6755a0, m=0x7f0b2a68e780) at cmd.c:271 ``` ``` "sip:alice@10.2.16.36:5060;gr=00F54E59-1172-EC11-9AC7-7D467C83D9C2 ---- INVITE sip:bob@ims.mnc001.mcc001.3gppnetwork.org SIP/2.0 Via: SIP/2.0/UDP 10.2.16.36:5060;branch=z9hG4bK8019dd952e72ec119bd47d467c83d9c2;rport From: "PhonerLite" sip:alice@ims.mnc001.mcc001.3gppnetwork.org;tag=4116302762 To: sip:bob@ims.mnc001.mcc001.3gppnetwork.org Call-ID: 8019DD95-2E72-EC11-9BD3-7D467C83D9C2@10.2.16.36 CSeq: 1 INVITE Contact: sip:alice@10.2.16.36:5060;gr=00F54E59-1172-EC11-9AC7-7D467C83D9C2 Content-Type: application/sdp Allow: INVITE, ACK, BYE, CANCEL, INFO, MESSAGE, NOTIFY, OPTIONS, REFER, UPDATE, PRACK Max-Forwards: 69 Supported: 100rel, replaces, from-change, gruu User-Agent: PhonerLite/2.97 P-Preferred-Identity: sip:alice@ims.mnc001.mcc001.3gppnetwork.org Content-Length: 340 ```
We have normal pointer to parsed object but object itself contains garbage. This is why no parsing is performed and [this](https://github.com/kamailio/kamailio/blob/master/src/modules/ims_ipsec_pcscf...) check passes. ``` (gdb) p req->contact.parsed $45 = (void *) 0x7f0b2a69b778 (gdb) p *(contact_body_t*)req->contact.parsed $46 = { star = 3 '\003', <---- this contacts = 0x38 <---- and this }
```
If we take ```next``` header than the ```parsed``` pointer will contain even stranger things ``` (gdb) p *req->contact.next $49 = { type = HDR_CONTENTTYPE_T, name = { s = 0x7f0b238ab6c3 "Content-Type: application/sdp\r\nAllow: INVITE, ACK, BYE, CANCEL, INFO, MESSAGE, NOTIFY, OPTIONS, REFER, UPDATE, PRACK\r\nMax-Forwards: 69\r\nSupported: 100rel, replaces, from-change, gruu\r\nUser-Agent: Phon"..., len = 12 }, body = { s = 0x7f0b238ab6d1 "application/sdp\r\nAllow: INVITE, ACK, BYE, CANCEL, INFO, MESSAGE, NOTIFY, OPTIONS, REFER, UPDATE, PRACK\r\nMax-Forwards: 69\r\nSupported: 100rel, replaces, from-change, gruu\r\nUser-Agent: PhonerLite/2.97\r\nP"..., len = 15 }, len = 31, parsed = 0x30003, <------- this next = 0x7f0b238ac088 } ```
I am not much into this code and unfortunately I am unable to find where this values are set. Are messages reused or created from scratch each time? The ```m``` variable (reply message) that passed into this method seems to be ok by the way.
Friends, do you have any ideas about the problem?
This issue is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks.
Closed #2970 as not planned.