<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description
<!-- Explain what you did, what you expected to happen, and what actually happened. -->
I set up Kamailio IMS , in the end everything turned out great, calls are made, and I decided to check how Kamailio copes with the load. I am creating load using sipp application. There is no problem with a low call generation rate, but if you raise the call generation to more than 20 calls per second, some of the calls fail. ### Troubleshooting Challenges are always dropped sequentially  I compared the calls and saw that the ICSCF does not send a 180 Ringing message towards the SCSCF.  Below is an example of a successful call. 
CPU, MEM indicators are normal. I increased the shr_mem and pkg_mem parameters. S-CSCF and I-CSCF communicate over UDP, but reside on the same server. It doesn't look like the packets are dropping or suffering from collisions. 10.21.4.226 external IP P-CSCF 10.10.0.1 IP P-CSCF 10.10.0.2 IP S/I-CSCF #### Reproduction
<!-- If the issue can be reproduced, describe how it can be done. --> Using the sipp application, generate 100 calls at a generation rate greater than 20 calls per second. #### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` none ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` attached below ``` [logs.zip](https://github.com/kamailio/kamailio/files/8417659/logs.zip)
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` attached below
``` [call_rejected.zip](https://github.com/kamailio/kamailio/files/8417640/call_rejected.zip)
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.3.2 (x86_64/linux) 42be5d flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 42be5d compiled on 10:39:53 Dec 27 2021 with gcc 9.3.0 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) -->
``` Ubuntu 20.04.3 LTS Linux ims-core 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
```
I was thinking of trying to switch the transport between I-CSCF and S-CSCF to TCP instead of UDP. In addition to the settings on the DNS server, do you still need to make some changes to Kamailio so that it communicates between modules via UDP? (TCP ports Kamailio is already listening)
I think I found a message in the log for the problematic call:
```Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26162]: DEBUG: ims_icscf [scscf_list.c:361]: I_scscf_select(): I_scscf_select() for call-id 43-3057631@10.2.16.63 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26162]: DEBUG: ims_icscf [scscf_list.c:431]: take_scscf_entry(): Getting scscf entry from list Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26161]: DEBUG: <core> [core/mem/q_malloc.c:482]: qm_free(): qm_free(0x7fe66d684010, 0x7fe66d737b08), called from core: core/parser/parse_via.c: free_via_param_list(2725) Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26160]: DEBUG: <core> [core/mem/q_malloc.c:416]: qm_malloc(): qm_malloc(0x7fe66d684010, 64) returns address 0x7fe66d782a48 frag. 0x7fe66d782a10 (size=64) on 1 -th hit Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26162]: DEBUG: ims_icscf [scscf_list.c:461]: take_scscf_entry(): scscf has not been set so we just look for first match Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26161]: DEBUG: <core> [core/mem/q_malloc.c:525]: qm_free(): freeing frag. 0x7fe66d737ad0 alloc'ed from core: core/parser/parse_via.c: parse_via(2511) Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26162]: DEBUG: ims_icscf [scscf_list.c:368]: I_scscf_select(): no scscf entry for callid [43-3057631@10.2.16.63]```
And depend above, apparently due to the fact that the dialogue was deleted ahead of time, it remains to understand why. ``` Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_usrloc_scscf [contact_dlg_handlers.c:129]: contact_dlg_handler(): no dlg out... ignoring!!! for type [4] - usually happens on failure response in dialog Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 0 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): ref <=0 for dialog 0x7fc133ce61d8 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:196]: destroy_dlg(): destroying dialog 0x7fc133ce61d8 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:207]: destroy_dlg(): removed timer for dlg 0x7fc133ce61d8 [3791:1419] with clid '43-3057631@10.2.16.63' and tags '78885556600' Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:218]: destroy_dlg(): About to run dlg callback for destroy Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=8192 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_usrloc_scscf [contact_dlg_handlers.c:129]: contact_dlg_handler(): no dlg out... ignoring!!! for type [8192] - usually happens on failure response in dialog Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:220]: destroy_dlg(): DONE: About to run dlg callback for destroy Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:156]: destroy_entry_out(): Destroy dialog entry out ```
Can anyone tell me where you can read about the states that Kamailio writes about? ``` DEBUG: ims_dialog [dlg_hash.c:1293]: next_state_dlg(): dialog 0x7fc133ce61d8 changed from state 1 to state 6, due event 4 ```
henning@app01:~/repositories/kamailio$ ack DLG_STATE_EARLY src/modules/dialog/dlg_hash.h #define DLG_STATE_EARLY 2 /*!< early dialog */ ...
henning@app01:~/repositories/kamailio$ ack DLG_STATE_EARLY src/modules/dialog/dlg_hash.h #define DLG_STATE_EARLY 2 /*!< early dialog */ ...
Thanks for the answer. But perhaps I need the **ims_dialog** directory and not the **dialog** directory? src/modules/ims_dialog/dlg_hash.h ``` /* states of a dialog */ #define DLG_STATE_UNCONFIRMED 1 /*!< unconfirmed dialog */ #define DLG_STATE_EARLY 2 /*!< early dialog */ #define DLG_STATE_CONFIRMED 4 /*!< confirmed dialog */ #define DLG_STATE_CONFIRMED_NA 5 /*!< confirmed dialog without a ACK yet */ #define DLG_STATE_DELETED 6 /*!< deleted dialog */ #define DLG_STATE_CONCURRENTLY_CONFIRMED 7 /*!< confirmed concurrent dailogs */ ``` This message means not the cause but the consequence, there are suggestions why "deleted dialog" Perhaps the timer has gone off? below is a log sample by dialog 0x7fc133ce61d8 ``` Apr 5 08:24:33 ims-core /usr/local/sbin/kamailio[26061]: DEBUG: ims_dialog [dlg_handlers.c:284]: dlg_iuid_sfree(): freeing dlg iuid [465:10905] (0x7fc133ce61d8) Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1025]: link_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:251]: run_create_callbacks(): dialog=0x7fc133ce61d8 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_handlers.c:789]: dlg_onreq(): dialog [0x7fc133ce61d8] added to tm callbacks Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26059]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26059]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1293]: next_state_dlg(): dialog 0x7fc133ce61d8 changed from state 1 to state 6, due event 4 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_handlers.c:1489]: dlg_onreply(): dialog 0x7fc133ce61d8 failed (negative reply) Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=4 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 0 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): ref <=0 for dialog 0x7fc133ce61d8 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:196]: destroy_dlg(): destroying dialog 0x7fc133ce61d8 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:207]: destroy_dlg(): removed timer for dlg 0x7fc133ce61d8 [3791:1419] with clid '43-3057631@10.2.16.63' and tags '78885556600' Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=8192 ```
Yes, its the similar file in both directories. About the cause, I would suggest to analyse your dependencies. It could be some logging or database performance issue for example. For further hints how to debug this, please contact the sr-users list. If you found out more information, please add here. Otherwise this issue will be closed as its probably not a bug in kamailio.
I correctly understood that the problem will be addressed here https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users ?
Yes, this is our non-commercial users list. All issues with Kamailio usage or configuration can be discussed there.
Thank you very much, I'll go there. As soon as I find the problem and its solution, I will post it here.
The problem was partially solved by disabling debug redim and disabling logging. performance has increased, but still there are drop calls about high loads. There may be a problem with the database, I will look further. I am closing this thread.
Closed #3071.