[SR-Users] DMQ Module Crash

Daniel-Constantin Mierla miconda at gmail.com
Mon May 11 13:42:57 CEST 2020


Hello,

can you open a bug on the github issue tracker, almost forgot about it
and with some things that popped up and the lockdown specific tasks, I
can't say if I can look at them today or tomorrow and then I may forget
again. Having it on tracker is easier to spot it. You can add a link to
mailing list archive there to jump to this discussion for details.

Cheers,
Daniel

On 08.05.20 21:13, Ilie Soltanici wrote:
> Daniel,
>
> There are 2 core dump files. Please see attached full back trace from
> both files.
> Thanks
>
> În vin., 8 mai 2020 la 19:53, Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> a scris:
>
>     Hello,
>
>     you have to provide the full output of gdb command from the core file:
>
>     bt full
>
>     To see the entire execution stack that cause the crash.
>
>     Cheers,
>     Daniel
>
>     On 08.05.20 19:09, Ilie Soltanici wrote:
>>     Hello,
>>     I'm trying to replicate a hash table to another 2 nodes by using
>>     DMQ Module. That's the related configuration:
>>
>>     loadmodule  "tm.so"
>>     loadmodule  "tmx.so"
>>     loadmodule  "dialog.so"
>>     loadmodule  "dmq.so"
>>     loadmodule  "htable.so"
>>
>>     modparam("dmq", "server_address", "sip:MY_INTERNAL_IP_ADDR:5062")
>>     modparam("dmq", "notification_address", "sip:dmq.domain.local:5062")
>>     modparam("dmq", "multi_notify", 1)
>>     modparam("dmq", "num_workers", 4)
>>     modparam("dmq", "ping_interval", 90)
>>
>>     modparam("htable", "enable_dmq", 1)
>>     modparam("htable", "dmq_init_sync", 1)
>>     modparam("htable", "htable",
>>     "test=>size=10;initval=0;dmqreplicate=1;")
>>
>>       if(is_method("KDMQ")) {
>>         dmq_handle_message();
>>         exit;
>>       }
>>
>>     After starting Kamailio - everything seems to be working fine,
>>     Kamailio is able to discover other nodes and replications is
>>     working fine, until after a few minutes I'm getting such errors
>>     in the Kamailio Log file:
>>
>>     May  8 17:14:26 kamailio kamailio[28064]: CRITICAL: <core>
>>     [core/mem/q_malloc.c:521]: qm_free(): BUG: freeing already freed
>>     pointer (0x7f0ac62493e0), called from dialog: dlg_var.c:
>>     free_local_varlist(145), first free dialog: dlg_var.c:
>>     set_dlg_variable_unsafe(205) - ignoring
>>
>>     Then, if I'm getting any SIP messages from any Device - Kamailio
>>     is crashing with the following info in the log file:
>>
>>     May  8 16:03:19 kamailio kam[10963]: CRITICAL: {REGISTER (1) |
>>     Seq=45534 | Source IP=192.168.1.5 (tcp) | Call
>>     ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core>
>>     [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev.
>>     fragm. tail overwritten(0, 0)[0x7f62cc746a10:0x7f62cc746a48]!
>>      Memory allocator was called from dialog: dlg_var.c:147. Fragment
>>     marked by dialog: dlg_var.c:108. Exec from core/mem/q_malloc.c:511.
>>     May  8 16:03:21 kamailio kam[10966]: CRITICAL: <core>
>>     [core/pass_fd.c:277]: receive_fd(): EOF on 59
>>     May  8 16:03:21 kamailio kam[10966]: DEBUG: <core>
>>     [core/tcp_main.c:3825]: handle_ser_child(): dead child 37, pid
>>     10963 (shutting down?)
>>     May  8 16:03:21 kamailio kam[10966]: DEBUG: <core>
>>     [core/io_wait.h:599]: io_watch_del(): DBG: io_watch_del
>>     (0xa9e1c0, 59, -1, 0x0) fd_no=48 called
>>     May  8 16:03:21 kamailio kam[10966]: DEBUG: <core>
>>     [core/tcp_main.c:3593]: handle_tcp_child(): dead tcp child 1 (pid
>>     10963, no 37) (shutting down?)
>>     May  8 16:03:21 kamailio kam[10966]: DEBUG: <core>
>>     [core/io_wait.h:599]: io_watch_del(): DBG: io_watch_del
>>     (0xa9e1c0, 62, -1, 0x0) fd_no=47 called
>>     May  8 16:03:21 kamailio kam[10919]: ALERT: <core> [main.c:767]:
>>     handle_sigs(): child process 10963 exited by a signal 6
>>     May  8 16:03:21 kamailio kam[10919]: ALERT: <core> [main.c:770]:
>>     handle_sigs(): core was generated
>>     May  8 16:03:21 kamailio kam[10919]: INFO: <core> [main.c:792]:
>>     handle_sigs(): terminating due to SIGCHLD
>>     May  8 16:03:21 kamailio kam[10919]: DEBUG: <core> [main.c:794]:
>>     handle_sigs(): terminating due to SIGCHLD
>>     May  8 16:03:21 kamailio kam[10966]: INFO: <core> [main.c:847]:
>>     sig_usr(): signal 15 received
>>
>>
>>     This is what i got from the dump core file:
>>
>>     uri = {user = {s = 0x3 <Address 0x3 out of bounds>, len =
>>     -264573566}, passwd = {s = 0x788 <Address 0x788 out of bounds>,
>>     len = 8208}, host = {s = 0x393100000000 <Address 0x393100000000
>>     out of bounds>, len = 1588950199}, port = {s = 0x2 <Address 0x2
>>     out of bounds>,
>>                 len = 24086112}, params = {s = 0x7ffcbb1600c0 "\200",
>>     len = -261173312}, sip_params = {s = 0x7ffcbb1601a8
>>     "\300\317n\360b\177", len = 24086112}, headers = {s = 0x0, len =
>>     -261173312}, port_no = 424, proto = 47894, type = 32764,
>>               flags = (URI_SIP_USER_PHONE | unknown: 4031456108),
>>     transport = {
>>                 s = 0x16f88cc "16:03:19 kamailio[10963]: CRITICAL:
>>     {REGISTER (1) | Seq=45534 | Source IP=192.168.1.5 (tcp) | Call
>>     ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core>
>>     [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm:
>>     pr"..., len = -261163168}, ttl = {
>>                 s = 0x8 <Address 0x8 out of bounds>, len =
>>     -263517049}, user_param = {
>>                 s = 0x16f88cc "16:03:19 kamailio[10963]: CRITICAL:
>>     {REGISTER (1) | Seq=45534 | Source IP=192.168.1.5 (tcp) | Call
>>     ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core>
>>     [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm:
>>     pr"..., len = -264350686}, maddr = {
>>                 s = 0x3336393031000080 <Address 0x3336393031000080
>>     out of bounds>, len = 8187}, method = {s = 0x0, len = 24086112},
>>     lr = {s = 0x820ce8 "%s: %s%s(): cseq <%.*s>: <%.*s> <%.*s>\n",
>>     len = -1156185384}, r2 = {s = 0xb <Address 0xb out of bounds>,
>>     len = 21}, gr = {
>>                 s = 0xffffffff <Address 0xffffffff out of bounds>,
>>     len = -264780793}, transport_val = {s = 0x1ff4 <Address 0x1ff4
>>     out of bounds>, len = 24086720}, ttl_val = {s = 0x7f62f06f0760
>>     <main_arena> "", len = 128}, user_param_val = {s = 0x16f88b0
>>     "`\002", len = 144},
>>               maddr_val = {s = 0x15 <Address 0x15 out of bounds>, len
>>     = -264573566}, method_val = {s = 0x788 <Address 0x788 out of
>>     bounds>, len = 8208}, lr_val = {s = 0x0, len = 1588950199},
>>     r2_val = {s = 0x2 <Address 0x2 out of bounds>, len = 24086112},
>>     gr_val = {
>>                 s = 0x7ffcbb1601f0 "\006", len = -261173312}}
>>             next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x0,
>>     len = 0}, host = {s = 0x7f62f06eef20 <_nl_C_LC_NUMERIC+32> "",
>>     len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x7f620000000b
>>     <Address 0x7f620000000b out of bounds>, len = 0}, sip_params = {
>>                 s = 0x7f62f03762cd <vfprintf+19661>
>>     "\200\275\360\372\377\377", len = 0}, headers = {s = 0x0, len =
>>     -1}, port_no = 25293, proto = 61495, type = 32610, flags =
>>     (URI_USER_NORMALIZE | URI_SIP_USER_PHONE), transport = {s =
>>     0x7ffcbb160038 "\267t\265^", len = 24086112},
>>               ttl = {s = 0xffffffff <Address 0xffffffff out of
>>     bounds>, len = 0}, user_param = {s = 0x7d62e8 "%s: %.*s%s%s():
>>     preparing to run routing scripts...\n", len = 11}, maddr = {s =
>>     0xbd <Address 0xbd out of bounds>, len = 3440}, method = {s =
>>     0x7d631c "", len = -263511186},
>>               lr = {s = 0x16f88cc "16:03:19 kamailio[10963]:
>>     CRITICAL: {REGISTER (1) | Seq=45534 | Source IP=192.168.1.5 (tcp)
>>     | Call ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core>
>>     [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm:
>>     pr"..., len = -261163168}, r2 = {
>>                 s = 0x8 <Address 0x8 out of bounds>, len =
>>     -263517049}, gr = {
>>                 s = 0x16f88cc "16:03:19 kamailio[10963]: CRITICAL:
>>     {REGISTER (1) | Seq=45534 | Source IP=192.168.1.5 (tcp) | Call
>>     ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core>
>>     [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm:
>>     pr"..., len = -264350686}, transport_val = {
>>                 s = 0x7d62ef "s%s%s(): preparing to run routing
>>     scripts...\n", len = 8187}, ttl_val = {s = 0x0, len =
>>     -264345823}, user_param_val = {s = 0x3000000010 <Address
>>     0x3000000010 out of bounds>, len = -1156184608}, maddr_val = {
>>                 s = 0x7ffcbb160520
>>     "\341+:\213'H\027\203\341+\342ޙ>\356|", len = 21}, method_val =
>>     {s = 0x7ffcbb160250 "\023", len = -1156186080}, lr_val = {s =
>>     0x7f62f06eef80 <_nl_C_LC_TIME> "r\036K\360b\177", len =
>>     -1156185648}, r2_val = {s = 0x17ff710 "IST", len = -1156186048},
>>               gr_val = {s = 0x0, len = -1156186058}}
>>             u = 0x400001ff4
>>
>>     By disabling DMQ Module i haven't noticed any crashes or issues
>>     like that, so i suppose that Kamailio is crashing because of this
>>     module, but i can be wrong. I've tried with 2 versions of
>>     Kamailio (5.2.6 and 5.3.4), both of them having the same issue.
>>
>>     Thanks
>>
>>
>>     _______________________________________________
>>     Kamailio (SER) - Users Mailing List
>>     sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>     https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>     -- 
>     Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>     www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>     Funding: https://www.paypal.me/dcmierla
>
-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200511/e7b2add6/attachment.html>


More information about the sr-users mailing list