[sr-dev] [kamailio/kamailio] Kamailio hangs on get_profile_size method. (Issue #3947)

<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports. If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list: * https://lists.kamailio.org/mailman3/postorius/lists/sr-users.lists.kamailio… If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list: * https://lists.kamailio.org/mailman3/postorius/lists/sr-dev.lists.kamailio.o… Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue. Note that an issue report may be closed automatically after about 2 months if there is no interest from developers or community users on pursuing it, being considered expired. In such case, it can be reopened by writing a comment that includes the token `/notexpired`. About two weeks before considered expired, the issue is marked with the label `stale`, trying to notify the submitter and everyone else that might be interested in it. To remove the label `stale`, write a comment that includes the token `/notstale`. Also, any comment postpone the `expire` timeline, being considered that there is interest in pursuing the issue. If there is no content to be filled in a section, the entire section can be removed. You can delete the comments from the template sections when filling. You can delete next line and everything above before submitting (it is a comment). --> ### Description We are facing a hang issue with Kamailio for the get_profile_size method. Randomly, the Kamailio got hangs, and the CPU went 100%. During the hanging time, we generated the TRAP and we could see that the hang happened on the get_profile_size method. We are getting this hang randomly in dlg_profile.c of the following lines: The hang occurs, causing the CPU to spike to 100% at one of the following lines randomly: 1. if(value->len == ph->value.len && memcmp(value->s, ph->value.s, value->len) == 0) { 2. ph = ph->next; 3. while(ph != profile->entries[i].first); During our analysis, we found that PROCESS ID 7608 gets a spike of 100% utilization. We suspect there may be an issue with the processing of the 'ph' variable, but we couldn't identify the root cause. I've attached the Kamailio TRAP for further details. Could you please suggest what might be causing this problem and how we can resolve it? <!-- Explain what you did, what you expected to happen, and what actually happened. --> ### Troubleshooting We have analysed the coredump and TRAP, And found that the issue could be from the following lines: 1. if(value->len == ph->value.len && memcmp(value->s, ph->value.s, value->len) == 0) { 2. ph = ph->next; 3. while(ph != profile->entries[i].first); #### Reproduction This issue is occurring randomly. <!-- If the issue can be reproduced, describe how it can be done. --> #### Debugging Data I have attached the TRAP data. ---start 7608 ----------------------------------------------------- [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". get_profile_size (profile=0x7f7095698f50, value=0x7ffd5e21db20) at dlg_profile.c:860 860 }while( ph!=profile->entries[i].first ); #0 get_profile_size (profile=0x7f7095698f50, value=0x7ffd5e21db20) at dlg_profile.c:860 n = 0 i = 9 ph = 0x7f70958d3c98 #1 0x00007f7195585785 in w_get_profile_size_helper (msg=0x7f7198418558, profile=0x7f7095698f50, value=0x7ffd5e21db20, spd=0x7f71983a8cf0) at dialog.c:941 size = 32625 val = {rs = {s = 0x7ffd5e21da80 "\260\332!^\375\177", len = 591976160}, ri = xxxxxxxx77096, flags = 32765} __func__ = "w_get_profile_size_helper" #2 0x00007f7195585fd2 in w_get_profile_size3 (msg=0x7f7198418558, profile=0x7f7095698f50 "\200\220i\225p\177", value=0x7f71983b4b30 "80;\230q\177", result=0x7f71983a8cf0 "\004") at dialog.c:982 pve = 0x7f71983b4b30 val_s = {s = 0x7f7198346dd0 "2777995", len = 7} spd = 0x7f71983a8cf0 __func__ = "w_get_profile_size3" <!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location. gdb /path/to/kamailio /path/to/corefile bt full info locals list If you are familiar with gdb, feel free to attach more of what you consider to be relevant. --> #### Log Messages When Kamailio hangs, the logs are not written to the kamailio.log file. <!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> #### SIP Traffic I have attached the TRAP file. <!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> ### Possible Solutions We are expecting some kind of solution for it. <!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. --> ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` kamailio version is 5.5.5``` * **Operating System**: [gdb_kamailio_trap.zip](https://github.com/user-attachments/files/16581474/g… ``` "Debian GNU/Linux 10 (buster)"``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3947 You are receiving this because you are subscribed to this thread. Message ID: &lt;kamailio/kamailio/issues/3947(a)github.com&gt;