<h2>Description</h2>
<p>kamailio dispatcher cache is inaccessible and ds_select_domain is failing. all new calls are rejected.</p>
<pre><code># kamctl kamcmd dispatcher.list
error: 500 - no destination sets
</code></pre>
<h2>Troubleshooting</h2>
<p>This is happened is production after a dispatcher reload and we are unable to reproduced it again. we restarted kamailio service and that fixed the issue.</p>
<h2>Reproduction</h2>
<p>I think this happened after we did a 10 simultaneous RPC command to reload the dispatcher cache.</p>
<h2>Log Messages</h2>
<pre><code>Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Host)=<XX.XX.XX.7:5060> hdr(Via)=<SIP/2.0/TCP XX.XX.XX.62:1900>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> = <yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: Successful authentication
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: http_rpc detected
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Host)=<XX.XX.XX.7:5060> hdr(Via)=<SIP/2.0/TCP XX.XX.XX.54:3314>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> = <yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: Successful authentication
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69c949b0), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69ae3840), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f696c5458), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69ce8610), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698e3cd0), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f697498c8), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69662058), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699b1990), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663320), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69a4f418), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663928), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699125a0), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698b4c30), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69acbee8), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69716f28), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69978d18), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69662d38), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698c4e70), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69b13130), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699027a0), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f6987f500), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663f10), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69664400), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69667bd0), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69661a50), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69857670), called from dispatcher: dispatch.c: reindex_dests(648), first free dispatcher: dispatch.c: reindex_dests(648) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f6987f500), called from dispatcher: dispatch.c: reindex_dests(648), first free dispatcher: dispatch.c: reindex_dests(648) - aborting
Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32182]: INFO: <script>: ===== redis sip code insert: SIP Code: 183 * time: 1556274535 * Rebtel-CID: TJI4V81TAP24NCN0DHHJG54NKG@YY.YY.YY.107 * Call-Id: 4b6b9685-c797-4e13-8cf5-cd6f7d23a39f
Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32191]: INFO: <script>: ===== redis sip code insert: SIP Code: 181 * time: 1556274535 * Rebtel-CID: UBQ1HL47T51HH51BESVJIOE66K@YY.YY.YY.107 * Call-Id: d74592af-d240-481f-92bc-6b6f1010887c
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>: ===== redis sip code insert: SIP Code: 487 * time: 1556274536 * Rebtel-CID: 12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id: b8078da6-7dff-4689-a28b-76268200a01a
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>: ===== redis sip code insert: SIP Code: 487 * time: 1556274523 * Rebtel-CID: 12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id: b8078da6-7dff-4689-a28b-76268200a01a
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32206]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32210]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32207]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32195]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32209]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32202]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets
</code></pre>
<h2>Possible Solutions</h2>
<p>Restarting Kamailio</p>
<h2>Additional Information</h2>
<p>Kamailio Version - output of kamailio -v</p>
<pre><code>version: kamailio 5.1.3 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 13:05:41 Apr 24 2018 with gcc 4.8.5
</code></pre>
<h2>Operating System:</h2>
<p>CentOS Linux release 7.4.1708 (Core)</p>

<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/1939">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/ABO7UZJMYUVNWWD7PFMW5U3PSMBQNANCNFSM4HIVX7DA">mute the thread</a>.<img src="https://github.com/notifications/beacon/ABO7UZNRSCTJKYMXNVFEYGDPSMBQNANCNFSM4HIVX7DA.gif" height="1" width="1" alt="" /></p>
<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Kamailio memory leak - versions 5.1.3 (#1939)"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1939"}}}</script>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/kamailio/kamailio/issues/1939",
"url": "https://github.com/kamailio/kamailio/issues/1939",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>