[kamailio/kamailio] Application crash due to intensive "dispather.add" using (Issue #3424)
### Description I want to add addresses to dispatcher.list dynamically ``` modparam("dispatcher", "db_url", DBURL) modparam("dispatcher", "table_name", "dispatcher") modparam("dispatcher", "ds_ping_interval", 10) modparam("dispatcher", "ds_probing_mode", 1) modparam("dispatcher", "ds_inactive_threshold", 1) modparam("dispatcher", "ds_probing_threshold", 1) modparam("dispatcher", "reload_delta", 0) modparam("dispatcher", "ds_timer_mode", 1) ... if(!ds_is_from_list("1","0","$var(contact)")) { jsonrpc_exec( '{ "jsonrpc": "2.0", "method": "dispatcher.add", "params": [1, "$var(contact)"], "id": 1 }' ); } ``` this scheme works with low load(one or two UAs) but it doesn't work with dozens of requests, kamailio crashes
### Troubleshooting ``` Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: CRITICAL: {1 12016 REGISTER db88353a-66cf-4ca6-91e2-6f306e43cee9} <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f2f964578b8), called from dispatcher: dispatch.c: reindex_dests(790), first free dispatcher: dispatch.c: ds_avl_destroy(4013) - ignoring Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: INFO: {1 12016 REGISTER db88353a-66cf-4ca6-91e2-6f306e43cee9} dispatcher [dispatch.c:682]: dp_init_relative_weights(): extra rweight 100 for last active destination in group 1 Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6827]: INFO: {1 16898 REGISTER eebaeca9-c642-465f-bc09-67d1448ba6b7} dispatcher [dispatch.c:682]: dp_init_relative_weights(): extra rweight 100 for last active destination in group 1 Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: ALERT: <core> [main.c:774]: handle_sigs(): child process 6825 exited by a signal 11 Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: ALERT: <core> [main.c:777]: handle_sigs(): core was not generated Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: INFO: <core> [main.c:799]: handle_sigs(): terminating due to SIGCHLD Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6827]: INFO: <core> [main.c:854]: sig_usr(): signal 15 received Apr 22 22:18:10 kamailio /usr/sbin/kamailio[6808]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Main process exited, code=exited, status=1/FAILURE Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Failed with result 'exit-code'. Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Consumed 1.304s CPU time. ```
#### Reproduction use code from Description section on request route
#### Debugging Data
``` (gdb) bt full #0 0x000055945e0ccd20 in ?? () No symbol table info available. #1 0x000055945e0cd6ee in qm_malloc () No symbol table info available. #2 0x000055945e0da9c9 in qm_shm_malloc () No symbol table info available. #3 0x00007f325a05e921 in pack_dest () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #4 0x00007f325a060ad4 in add_dest2list () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #5 0x00007f325a087cf9 in ds_filter_dest_cb () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #6 0x00007f325a05affb in ds_iter_set () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #7 0x00007f325a088706 in ds_remove_dst () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #8 0x00007f325a0af7a8 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #9 0x00007f325e7e82f5 in jsonrpc_exec_ex () from /usr/lib/x86_64-linux-gnu/kamailio/modules/jsonrpcs.so No symbol table info available. #10 0x00007f325e7e875e in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/jsonrpcs.so No symbol table info available. #11 0x000055945ddce3f3 in do_action () No symbol table info available. #12 0x000055945dddd391 in run_actions () No symbol table info available. #13 0x000055945ddce2d3 in do_action () No symbol table info available. #14 0x000055945dddd391 in run_actions () No symbol table info available. #15 0x000055945ddce2d3 in do_action () No symbol table info available. #16 0x000055945dddd391 in run_actions () No symbol table info available. #17 0x000055945ddca244 in do_action () No symbol table info available. --Type <RET> for more, q to quit, c to continue without paging-- #18 0x000055945dddd391 in run_actions () No symbol table info available. #19 0x000055945ddce2d3 in do_action () No symbol table info available. #20 0x000055945dddd391 in run_actions () No symbol table info available. #21 0x000055945ddddc0b in run_top_route () No symbol table info available. #22 0x000055945df3b1b2 in receive_msg () No symbol table info available. #23 0x000055945e09ea5d in udp_rcv_loop () No symbol table info available. #24 0x000055945ddb709b in main_loop () No symbol table info available. #25 0x000055945ddc420c in main () No symbol table info available. ``` ```(gdb) info locals No symbol table info available.```
```(gdb) list 1 ../sysdeps/unix/sysv/linux/dl-vdso-setup.c: No such file or directory.```
#### SIP Traffic [kamailio.txt](https://github.com/kamailio/kamailio/files/11302151/kamailio.txt)
### Possible Solutions no idea ### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` root@kamailio:[/etc]: kamailio -v version: kamailio 5.6.3 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 10.2.1 ```
* **Operating System**:
``` root@kamailio:[/etc]: uname -a Linux kamailio 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux ```
What you are trying to achieve, using dispatcher on REGISTER messages to add the contact as destination? This is probably not a good idea at all. Why you are not using the usrloc module for this kind of functionality? As you are also using an old, unsupported version of Kamailio, I am closing this issue. If you have questions about how to use the usrloc module, please contact our sr-users mailing list.
Closed #3424 as completed.
Sorry, I misread the version - the 5.6.3 is of course supported. But anyway, please lets discuss on the users lists first, what you are trying to achieve, thank you.
I'm making a mid-registrar configuration, when reg comes I authenticate it on Kamailio at first, then save it to usrloc, then relay it to the main registrar making authentication with uac there. I use the dispatcher module to monitor registered users, usrloc's keep alive doesn't suit this purpose because there is no event route if the user went offline. I want to use the dispatcher module to trigger and send unregister to the main registrar when the user died.
Sorry, I misread the version - the 5.6.3 is of course supported. But anyway, please lets discuss on the users lists first, what you are trying to achieve, thank you.
In fact, my use case doesn't matter, if you add addresses to dispatcher list intensively, Kamailio crashes due to a memory leak. I reckon this is a bug
As it was mentioned above, this is for discussing on sr-users mailing list. There are limitations to use of rpc commands, some addressed in the latest version. If it proves something wrong in the design and code, then an issue can be opened for the right problem identified for it.
-
Daniel-Constantin Mierla -
Henning Westerholt
-
Nikolay Shakin