26 Apr
2019
26 Apr
'19
9:45 a.m.
## Description
kamailio dispatcher cache is inaccessible and ds_select_domain is failing. all new calls
are rejected.
```
# kamctl kamcmd dispatcher.list
error: 500 - no destination sets
```
## Troubleshooting
This is happened is production after a dispatcher reload and we are unable to reproduced
it again. we restarted kamailio service and that fixed the issue.
## Reproduction
I think this happened after we did a 10 simultaneous RPC command to reload the dispatcher
cache.
## Log Messages
```
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>:
[XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg=
hdr(Host)=<XX.XX.XX.7:5060> hdr(Via)=<SIP/2.0/TCP XX.XX.XX.62:1900>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>:
[XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg=
hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> =
<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>:
[XX.XX.XX.62:1900]: Successful authentication
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>:
[XX.XX.XX.54:3314]: http_rpc detected
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>:
[XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg=
hdr(Host)=<XX.XX.XX.7:5060> hdr(Via)=<SIP/2.0/TCP XX.XX.XX.54:3314>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>:
[XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg=
hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> =
<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww>
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>:
[XX.XX.XX.54:3314]: Successful authentication
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69c949b0),
called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher:
dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69ae3840),
called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher:
dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f696c5458),
called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher:
dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69ce8610),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698e3cd0),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f697498c8),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69662058),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699b1990),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663320),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69a4f418),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663928),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699125a0),
called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher:
dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698b4c30),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69acbee8),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69716f28),
called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher:
dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69978d18),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69662d38),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698c4e70),
called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher:
dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69b13130),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699027a0),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f6987f500),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663f10),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69664400),
called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher:
dispatch.c: ds_avl_destroy(3032) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69667bd0),
called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher:
dispatch.c: ds_avl_destroy(3037) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69661a50),
called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher:
dispatch.c: ds_avl_destroy(3038) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69857670),
called from dispatcher: dispatch.c: reindex_dests(648), first free dispatcher: dispatch.c:
reindex_dests(648) - aborting
Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core>
[core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f6987f500),
called from dispatcher: dispatch.c: reindex_dests(648), first free dispatcher: dispatch.c:
reindex_dests(648) - aborting
Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32182]: INFO: <script>: =====
redis sip code insert: SIP Code: 183 * time: 1556274535 * Rebtel-CID:
TJI4V81TAP24NCN0DHHJG54NKG(a)YY.YY.YY.107 * Call-Id: 4b6b9685-c797-4e13-8cf5-cd6f7d23a39f
Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32191]: INFO: <script>: =====
redis sip code insert: SIP Code: 181 * time: 1556274535 * Rebtel-CID:
UBQ1HL47T51HH51BESVJIOE66K(a)YY.YY.YY.107 * Call-Id: d74592af-d240-481f-92bc-6b6f1010887c
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>: =====
redis sip code insert: SIP Code: 487 * time: 1556274536 * Rebtel-CID:
12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id: b8078da6-7dff-4689-a28b-76268200a01a
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>: =====
redis sip code insert: SIP Code: 487 * time: 1556274523 * Rebtel-CID:
12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id: b8078da6-7dff-4689-a28b-76268200a01a
Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32206]: ERROR: dispatcher
[dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32210]: ERROR: dispatcher
[dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32207]: ERROR: dispatcher
[dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32195]: ERROR: dispatcher
[dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32209]: ERROR: dispatcher
[dispatch.c:1808]: ds_select_dst_limit(): no destination sets
Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32202]: ERROR: dispatcher
[dispatch.c:1808]: ds_select_dst_limit(): no destination sets
```
## Possible Solutions
Restarting Kamailio
## Additional Information
Kamailio Version - output of kamailio -v
```
version: kamailio 5.1.3 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 13:05:41 Apr 24 2018 with gcc 4.8.5
```
## Operating System:
CentOS Linux release 7.4.1708 (Core)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1939
29 Apr
29 Apr
11:13 a.m.
New subject: [kamailio/kamailio] BUG: qm_free: freeing already freed pointer - versions 5.1.3 (#1939)
Probably it requires a mechanism to detect if a reload operation is ongoing -- there are
two structures caching the records from the database, one active and one to build the new
list on reload, rotating once the reload is finished. But if many reload operations happen
at the same time, then it can be a conflict. I have to check the code, I don't recall
who written that part and what safety checks are in place.
Normally, if the rpc reload command is sent via fifo file, it cannot be more than one
executed at a time, but if you do it over http, then it can be a race.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1939#issuecomment-487620038
12:58 p.m.
New subject: [kamailio/kamailio] BUG: qm_free: freeing already freed pointer - versions 5.1.3 (#1939)
Yes we do reload it over http at the same time.
Meanwhile, I will fix my deployment code to run it at the end of the deployment instead of
do it for every instance.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1939#issuecomment-487659440
20 May
20 May
5:36 a.m.
New subject: [kamailio/kamailio] BUG: qm_free: freeing already freed pointer - versions 5.1.3 (#1939)
I added a parameter to control how often the rpc reload commands can be done. It is in
master branch.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1939#issuecomment-493911665
5:36 a.m.
New subject: [kamailio/kamailio] BUG: qm_free: freeing already freed pointer - versions 5.1.3 (#1939)
Closed #1939.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1939#event-2352512645
1949
days inactive
1973
days old
4 comments
2 participants
participants (2)
-
Daniel-Constantin Mierla -
ohammami