[sr-dev] [kamailio/kamailio] Kamailio memory leak - versions 5.1.3 (#1939)
ohammami
notifications at github.com
Fri Apr 26 15:45:42 CEST 2019
## Description
kamailio dispatcher cache is inaccessible and ds_select_domain is failing. all new calls are rejected.
```
# kamctl kamcmd dispatcher.list
error: 500 - no destination sets
```
## Troubleshooting
This is happened is production after a dispatcher reload and we are unable to reproduced it again. we restarted kamailio service and that fixed the issue.
## Reproduction
I think this happened after we did a 10 simultaneous RPC command to reload the dispatcher cache.
## Log Messages
```
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Host)=<XX.XX.XX.7:5060> hdr(Via)=<SIP/2.0/TCP XX.XX.XX.62:1900>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> = <yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: Successful authentication
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: http_rpc detected
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Host)=<XX.XX.XX.7:5060> hdr(Via)=<SIP/2.0/TCP XX.XX.XX.54:3314>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> = <yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: Successful authentication
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69c949b0), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69ae3840), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f696c5458), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69ce8610), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698e3cd0), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f697498c8), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69662058), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699b1990), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663320), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69a4f418), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663928), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699125a0), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698b4c30), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69acbee8), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69716f28), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69978d18), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69662d38), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698c4e70), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69b13130), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699027a0), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f6987f500), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663f10), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69664400), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69667bd0), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69661a50), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69857670), called from dispatcher: dispatch.c: reindex_dests(648), first free dispatcher: dispatch.c: reindex_dests(648) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f6987f500), called from dispatcher: dispatch.c: reindex_dests(648), first free dispatcher: dispatch.c: reindex_dests(648) - aborting
Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32182]: INFO: <script>: ===== redis sip code insert: SIP Code: 183 * time: 1556274535 * Rebtel-CID: TJI4V81TAP24NCN0DHHJG54NKG at YY.YY.YY.107 * Call-Id: 4b6b9685-c797-4e13-8cf5-cd6f7d23a39f
Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32191]: INFO: <script>: ===== redis sip code insert: SIP Code: 181 * time: 1556274535 * Rebtel-CID: UBQ1HL47T51HH51BESVJIOE66K at YY.YY.YY.107 * Call-Id: d74592af-d240-481f-92bc-6b6f1010887c
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>: ===== redis sip code insert: SIP Code: 487 * time: 1556274536 * Rebtel-CID: 12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id: b8078da6-7dff-4689-a28b-76268200a01a
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>: ===== redis sip code insert: SIP Code: 487 * time: 1556274523 * Rebtel-CID: 12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id: b8078da6-7dff-4689-a28b-76268200a01a
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32206]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32210]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32207]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32195]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32209]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32202]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
```
## Possible Solutions
Restarting Kamailio
## Additional Information
Kamailio Version - output of kamailio -v
```
version: kamailio 5.1.3 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 13:05:41 Apr 24 2018 with gcc 4.8.5
```
## Operating System:
CentOS Linux release 7.4.1708 (Core)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1939
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20190426/e06cc4d2/attachment-0001.html>
More information about the sr-dev
mailing list