descartin created an issue (kamailio/kamailio#4313)
<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* https://lists.kamailio.org/mailman3/postorius/lists/sr-users.lists.kamailio....
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* https://lists.kamailio.org/mailman3/postorius/lists/sr-dev.lists.kamailio.or...
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
Note that an issue report may be closed automatically after about 2 months if there is no interest from developers or community users on pursuing it, being considered expired. In such case, it can be reopened by writing a comment that includes the token `/notexpired`. About two weeks before considered expired, the issue is marked with the label `stale`, trying to notify the submitter and everyone else that might be interested in it. To remove the label `stale`, write a comment that includes the token `/notstale`. Also, any comment postpone the `expire` timeline, being considered that there is interest in pursuing the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description Hello how are you, we are deploying a kamailio instance on 5.5.6 version ``` version: kamailio 5.5.6 (x86_64/linux) 0125f8-dirty flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 32MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 0125f8 -dirty compiled on 12:01:31 May 21 2025 with gcc 12.2.0 ``` and after some minutes processing calls we are seeing some core like this ``` (gdb) bt #0 __strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:79 #1 0x00007f5753edc258 in __vfprintf_internal (s=s@entry=0x7ffc848f6aa0, format=format@entry=0x5570eb867da8 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n", ap=ap@entry=0x7ffc848f6cc8, mode_flags=mode_flags@entry=0) at ./stdio-common/vfprintf-process-arg.c:397 #2 0x00007f5753efc758 in __vsnprintf_internal ( string=0x7ffc848f6d40 "CRITICAL: <core> [core/mem/q_malloc.c:123]: qm_debug_check_frag(): BUG: qm: fragm. 0x7f569357aae0 (address 0x7f569357ab18) beginning overwritten (663d610a0d303030)! Memory allocator was called from co"..., maxlen=<optimized out>, maxlen@entry=960, format=format@entry=0x5570eb867da8 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n", args=args@entry=0x7ffc848f6cc8, mode_flags=mode_flags@entry=0) at ./libio/vsnprintf.c:114 #3 0x00007f5753f7ef2c in __vsyslog_internal (pri=138, fmt=<optimized out>, ap=ap@entry=0x7ffc848f7150, mode_flags=mode_flags@entry=0) at ./misc/syslog.c:218 #4 0x00007f5753f7f536 in __syslog (pri=<optimized out>, fmt=<optimized out>) at ./misc/syslog.c:91 #5 0x00005570eb6d8f25 in qm_debug_check_frag (qm=0x7f5692cd8000, f=0x7f569357aae0, file=0x5570eb861995 "core: core/usr_avp.c", line=627, efile=0x5570eb867c25 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:123 #6 0x00005570eb6dd50f in qm_free (qmp=0x7f5692cd8000, p=0x7f569357ab18, file=0x5570eb861995 "core: core/usr_avp.c", func=0x5570eb863260 <__func__.8> "destroy_avp_list_unsafe", line=627, mname=0x5570eb861990 "core") at core/mem/q_malloc.c:511 #7 0x00005570eb6b32c3 in destroy_avp_list_unsafe (list=0x7f5693545238) at core/usr_avp.c:627 #8 0x00007f57524b3ba9 in free_cell_helper (dead_cell=0x7f5693545038, silent=1, fname=0x7f57525b7bdf "h_table.c", fline=466) at h_table.c:255 #9 0x00007f57524b49a2 in free_hash_table () at h_table.c:466 #10 0x00007f57524e8772 in tm_shutdown () at t_funcs.c:88 #11 0x00005570eb6081a0 in destroy_modules () at core/sr_module.c:842 #12 0x00005570eb3d8161 in cleanup (show_status=1) at main.c:575 #13 0x00005570eb3d9da8 in shutdown_children (sig=15, show_status=1) at main.c:718 #14 0x00005570eb3dd0bc in handle_sigs () at main.c:816 #15 0x00005570eb3ea5d1 in main_loop () at main.c:1903 #16 0x00005570eb3f3f9c in main (argc=15, argv=0x7ffc848f8168) at main.c:3061 ``` we are using debian12, and we see this log on the system when the crash ``` 2025-07-08T06:21:55.874670+00:00 mad-proxy-4 kernel: [4744966.979706] traps: kamailio[3282034] general protection fault ip:7f18c2a719d8 sp:7ffde5c84d68 error:0 in libc.so.6[7f18c2930000+155000] 2025-07-08T06:21:55.910639+00:00 mad-proxy-4 kernel: [4744967.018138] traps: kamailio[3282015] general protection fault ip:561437233f64 sp:7ffde5c859a0 error:0 in kamailio[561436f31000+425000] 2025-07-08T06:21:55.919781+00:00 mad-proxy-4 systemd[1]: theseus.service: Main process exited, code=killed, status=11/SEGV 2025-07-08T06:21:55.919880+00:00 mad-proxy-4 systemd[1]: theseus.service: Failed with result 'signal'. ``` The traffic which seems is causing the issue may be using sipt module, functions sipt_set_calling and sipt_destination
the libc version we have in the system is ``` ii libc6:amd64 2.36-9+deb12u10 amd64 GNU C Library: Shared libraries ii libc6-dbg:amd64 2.36-9+deb12u10 amd64 GNU C Library: detached debugging symbols ii libc6-dev:amd64 2.36-9+deb12u10 amd64 GNU C Library: Development Libraries and Header Files ii libc6-i386 2.36-9+deb12u10 amd64 GNU C Library: 32-bit shared libraries for AMD64 ``` could be possible any issue with the module sipt related to the library we are using? I checked commits between 5.5.6 and 6.0 and I think the difference is related to kemi support and some format changes only
thanks a lot and regards david escartin <!-- Explain what you did, what you expected to happen, and what actually happened. -->
### Troubleshooting
#### Reproduction
<!-- If the issue can be reproduced, describe how it can be done. -->
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` #0 __strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:79 79 ../sysdeps/x86_64/multiarch/strlen-evex.S: No existe el fichero o el directorio. (gdb) bt #0 __strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:79 #1 0x00007f5753edc258 in __vfprintf_internal (s=s@entry=0x7ffc848f6aa0, format=format@entry=0x5570eb867da8 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n", ap=ap@entry=0x7ffc848f6cc8, mode_flags=mode_flags@entry=0) at ./stdio-common/vfprintf-process-arg.c:397 #2 0x00007f5753efc758 in __vsnprintf_internal ( string=0x7ffc848f6d40 "CRITICAL: <core> [core/mem/q_malloc.c:123]: qm_debug_check_frag(): BUG: qm: fragm. 0x7f569357aae0 (address 0x7f569357ab18) beginning overwritten (663d610a0d303030)! Memory allocator was called from co"..., maxlen=<optimized out>, maxlen@entry=960, format=format@entry=0x5570eb867da8 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n", args=args@entry=0x7ffc848f6cc8, mode_flags=mode_flags@entry=0) at ./libio/vsnprintf.c:114 #3 0x00007f5753f7ef2c in __vsyslog_internal (pri=138, fmt=<optimized out>, ap=ap@entry=0x7ffc848f7150, mode_flags=mode_flags@entry=0) at ./misc/syslog.c:218 #4 0x00007f5753f7f536 in __syslog (pri=<optimized out>, fmt=<optimized out>) at ./misc/syslog.c:91 #5 0x00005570eb6d8f25 in qm_debug_check_frag (qm=0x7f5692cd8000, f=0x7f569357aae0, file=0x5570eb861995 "core: core/usr_avp.c", line=627, efile=0x5570eb867c25 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:123 #6 0x00005570eb6dd50f in qm_free (qmp=0x7f5692cd8000, p=0x7f569357ab18, file=0x5570eb861995 "core: core/usr_avp.c", func=0x5570eb863260 <__func__.8> "destroy_avp_list_unsafe", line=627, mname=0x5570eb861990 "core") at core/mem/q_malloc.c:511 #7 0x00005570eb6b32c3 in destroy_avp_list_unsafe (list=0x7f5693545238) at core/usr_avp.c:627 #8 0x00007f57524b3ba9 in free_cell_helper (dead_cell=0x7f5693545038, silent=1, fname=0x7f57525b7bdf "h_table.c", fline=466) at h_table.c:255 #9 0x00007f57524b49a2 in free_hash_table () at h_table.c:466 #10 0x00007f57524e8772 in tm_shutdown () at t_funcs.c:88 #11 0x00005570eb6081a0 in destroy_modules () at core/sr_module.c:842 #12 0x00005570eb3d8161 in cleanup (show_status=1) at main.c:575 #13 0x00005570eb3d9da8 in shutdown_children (sig=15, show_status=1) at main.c:718 #14 0x00005570eb3dd0bc in handle_sigs () at main.c:816 #15 0x00005570eb3ea5d1 in main_loop () at main.c:1903 #16 0x00005570eb3f3f9c in main (argc=15, argv=0x7ffc848f8168) at main.c:3061 ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` (paste your log messages here) ```
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` (paste your sip traffic here) ```
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.5.6 (x86_64/linux) 0125f8-dirty flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 32MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 0125f8 -dirty compiled on 12:01:31 May 21 2025 with gcc 12.2.0 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) -->
``` No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm Linux mad-proxy-4.bts.io 6.1.0-30-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.124-1 (2025-01-12) x86_64 GNU/Linux ```