<div dir="ltr">Thanks Daniel.<div>I'll upgrade to the latest stable version once it is out.<br><div><br></div><div>- Jayesh</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 2, 2018 at 10:55 AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <p>Hello,</p>
    <p>I looked at the core files when you reported, but due to busy
      times traveling, I didn't get the chance to follow up. However,
      the core files are truncated, so no useful details there.</p>
    <p>We will release 5.1.6 these days, it has few more bits that might
      help troubleshooting here, if there will be another good core dump
      file.</p>
    <p>Cheers,<br>
      Daniel<br>
    </p>
    <br>
    <div class="m_-4772721693493464624moz-cite-prefix">On 26.09.18 09:25, Jayesh Nambiar
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">Hello Daniel,
        <div><br>
          <div>
            <div class="gmail_quote">
              <div dir="ltr">
                <div dir="ltr">
                  <div dir="ltr">
                    <div>Just a quick update on this. After running from
                      the 5.1 branch twice it happened that Server
                      became unresponsive. It never crashed and on
                      forceful shutdown it only generated a core with
                      shutdown routine. Here is what the syslog
                      contained off while the calls were being sent to
                      kamailio but kamailio was not processing any
                      calls:</div>
                    <div><br>
                    </div>
                    <div>
                      <div>Sep 22 16:09:04 triviamedia
                        /usr/local/kamailio/sbin/kamailio[16006]:
                        CRITICAL: <core>
                        [core/mem/q_malloc.c:502]: qm_free(): BUG:
                        freeing already freed pointer (0x7fcd23ce5648),
                        called from tm: h_table.c:
                        free_cell_helper(235), first free tm: h_table.c:
                        free_cell_helper(235) - aborting</div>
                      <div>Sep 22 16:09:04 triviamedia
                        /usr/local/kamailio/sbin/kamailio[16006]:
                        CRITICAL: <core>
                        [core/mem/q_malloc.c:502]: qm_free(): BUG:
                        freeing already freed pointer (0x7fcd24a38ea8),
                        called from tm: h_table.c:
                        free_cell_helper(234), first free tm: h_table.c:
                        free_cell_helper(234) - aborting</div>
                      <div>Sep 22 16:09:04 triviamedia
                        /usr/local/kamailio/sbin/kamailio[16006]:
                        CRITICAL: <core>
                        [core/mem/q_malloc.c:502]: qm_free(): BUG:
                        freeing already freed pointer (0x7fcd23ce5648),
                        called from tm: h_table.c:
                        free_cell_helper(235), first free tm: h_table.c:
                        free_cell_helper(235) - aborting</div>
                      <div>Sep 22 16:09:04 triviamedia
                        /usr/local/kamailio/sbin/kamailio[16006]:
                        CRITICAL: <core>
                        [core/mem/q_malloc.c:502]: qm_free(): BUG:
                        freeing already freed pointer (0x7fcd24a38ea8),
                        called from tm: h_table.c:
                        free_cell_helper(234), first free tm: h_table.c:
                        free_cell_helper(234) - aborting</div>
                      <div>Sep 22 16:09:04 triviamedia
                        /usr/local/kamailio/sbin/kamailio[16006]:
                        CRITICAL: <core>
                        [core/mem/q_malloc.c:502]: qm_free(): BUG:
                        freeing already freed pointer (0x7fcd23ce5648),
                        called from tm: h_table.c:
                        free_cell_helper(235), first free tm: h_table.c:
                        free_cell_helper(235) - aborting</div>
                      <div>Sep 22 16:09:04 triviamedia
                        /usr/local/kamailio/sbin/kamailio[16006]:
                        CRITICAL: <core>
                        [core/mem/q_malloc.c:502]: qm_free(): BUG:
                        freeing already freed pointer (0x7fcd24a38ea8),
                        called from tm: h_table.c:
                        free_cell_helper(234), first free tm: h_table.c:
                        free_cell_helper(234) - aborting</div>
                      <div>Sep 22 16:09:04 triviamedia
                        /usr/local/kamailio/sbin/kamailio[16006]:
                        CRITICAL: <core>
                        [core/mem/q_malloc.c:502]: qm_free(): BUG:
                        freeing already freed pointer (0x7fcd23ce5648),
                        called from tm: h_table.c:
                        free_cell_helper(235), first fre</div>
                    </div>
                    <div><br>
                    </div>
                    <div>And here's the "bt full" of the core file:</div>
                    <div>
                      <div>[New LWP 16003]</div>
                      <div>Cannot access memory at address
                        0x7fcd64e1a148</div>
                      <div>Cannot access memory at address
                        0x7fcd64e1a140</div>
                      <div>Failed to read a valid object file image from
                        memory.</div>
                      <div>Core was generated by
                        `/usr/local/kamailio/sbin/kamailio -P
                        /var/run/siptrunk.pid -f /usr/local/carrie'.</div>
                      <div>Program terminated with signal SIGABRT,
                        Aborted.</div>
                      <div>#0  0x00007fcd6443f428 in ?? ()</div>
                      <div>(gdb) bt full</div>
                      <div>#0  0x00007fcd6443f428 in ?? ()</div>
                      <div>No symbol table info available.</div>
                      <div>Backtrace stopped: Cannot access memory at
                        address 0x7ffed40e9e68</div>
                    </div>
                    <div><br>
                    </div>
                    <div>Thanks,</div>
                    <div><br>
                    </div>
                    <div>- Jayesh</div>
                  </div>
                </div>
                <br>
                <div class="gmail_quote">
                  <div dir="ltr">On Sat, Sep 15, 2018 at 7:09 PM Jayesh
                    Nambiar <<a href="mailto:jayesh1017@gmail.com" target="_blank">jayesh1017@gmail.com</a>>
                    wrote:<br>
                  </div>
                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                    <div dir="ltr">Hello Daniel,
                      <div>I have now git cloned from 5.1 branch and
                        restarted kamailio today. Also an observation;
                        that when one particular customer with around
                        40-50cps capacity starts his dialing is when the
                        kamailio kept crashing. I will observe in the
                        coming week and report of any crashes
                        accordingly.</div>
                      <div><br>
                      </div>
                      <div>Thanks,</div>
                      <div><br>
                      </div>
                      <div>- Jayesh</div>
                    </div>
                    <br>
                    <div class="gmail_quote">
                      <div dir="ltr">On Wed, Sep 12, 2018 at 3:11 AM
                        Jayesh Nambiar <<a href="mailto:jayesh1017@gmail.com" target="_blank">jayesh1017@gmail.com</a>>
                        wrote:<br>
                      </div>
                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                        <div dir="ltr">
                          <div>Thank you Daniel. I will deploy from 5.1
                            branch. I will be able to update it in next
                            two days. Since this is a production system,
                            I will have to have find a proper time slot
                            to get this updated. Will let you know once
                            that is done.</div>
                          <div>Thanks again.</div>
                          <div><br>
                          </div>
                          <div>- Jayesh</div>
                        </div>
                        <br>
                        <div class="gmail_quote">
                          <div dir="ltr">On Tue, Sep 11, 2018 at 2:16 PM
                            Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
                            wrote:<br>
                          </div>
                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                            <div text="#000000" bgcolor="#FFFFFF">
                              <p>Hi Jayesh,</p>
                              <p>I looked over the weekend and seems
                                like a race or a double free. I
                                backported some patches to tm module
                                from master to 5.1 branch (a counter to
                                see if there is more than one free).
                                Would you be able to deploy from branch
                                5.1 or nightly builds for 5.1? With the
                                next crash I can evaluate (rule out or
                                go for it) if it is double free or not.</p>
                              <p>I haven't written you yet, because I
                                planned to dig more into libev (which is
                                used by evapi), to see if it creates
                                some threads that can mess up with some
                                kamailio globals, although I expect it
                                is event loop in a single thread. Other
                                work related project took a lot of time
                                these days, so not there yet ...<br>
                              </p>
                              Cheers,<br>
                              Daniel<br>
                              <br>
                              <div class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-cite-prefix">On
                                11.09.18 09:56, Jayesh Nambiar wrote:<br>
                              </div>
                              <blockquote type="cite">
                                <div dir="ltr">Hi Daniel,
                                  <div>Just a gentle follow up on this.
                                    Could you find anything useful?</div>
                                  <div><br>
                                  </div>
                                  <div>Thanks,</div>
                                  <div><br>
                                  </div>
                                  <div>- Jayesh</div>
                                </div>
                                <br>
                                <div class="gmail_quote">
                                  <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                    <div class="gmail_quote">
                                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                        <div text="#000000" bgcolor="#FFFFFF">
                                          <blockquote type="cite">
                                            <div class="gmail_quote">
                                              <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                <div text="#000000" bgcolor="#FFFFFF"><br>
                                                  <div class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-cite-prefix">On
                                                    07.09.18 09:56,
                                                    Jayesh Nambiar
                                                    wrote:<br>
                                                  </div>
                                                  <blockquote type="cite">
                                                    <div dir="ltr">
                                                      <div dir="ltr">Hi
                                                        Daniel,
                                                        <div>This
                                                          happens
                                                          randomly. It
                                                          is not a
                                                          testbed and
                                                          never
                                                          reproducible
                                                          on the test
                                                          server. I can
                                                          still give you
                                                          access if you
                                                          wish to
                                                          analyse the
                                                          core files to
                                                          check whats
                                                          happening here
                                                          exactly.</div>
                                                        <div>It simply
                                                          feels like on
                                                          heavy load the
                                                          evapi socket
                                                          gives up.
                                                          Earlier I was
                                                          running 5.0.2
                                                          and in that
                                                          kamailio would
                                                          get stuck by
                                                          not sending
                                                          any async
                                                          events on
                                                          evapi socket.
                                                          Not even the
                                                          evapi:connection-closed
                                                          event gets
                                                          triggered.
                                                          After I
                                                          upgraded to
                                                          latest stable
                                                          version
                                                          (5.1.5), it at
                                                          least started
                                                          crashing. Here
                                                          are two core
                                                          dumps put onto
                                                          pastebin.</div>
                                                        <div>
                                                          <div><a href="https://pastebin.com/nn6gJapm" target="_blank">https://pastebin.com/nn6gJapm</a></div>
                                                          <div><a href="https://pastebin.com/ph7b8vFH" target="_blank">https://pastebin.com/ph7b8vFH</a></div>
                                                        </div>
                                                        <div><br>
                                                        </div>
                                                        <div>Thanks for
                                                          all the
                                                          support,</div>
                                                        <div><br>
                                                        </div>
                                                        <div>- Jayesh</div>
                                                      </div>
                                                    </div>
                                                    <br>
                                                    <div class="gmail_quote">
                                                      <div dir="ltr">On
                                                        Fri, Sep 7, 2018
                                                        at 1:55 AM
                                                        Daniel-Constantin
                                                        Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<br>
                                                      </div>
                                                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                        <div text="#000000" bgcolor="#FFFFFF">
                                                          <p>Hello,</p>
                                                          <p>are you
                                                          able to
                                                          reproduce it
                                                          somehow? Or
                                                          just happens
                                                          randomly?</p>
                                                          <p>Is it on a
                                                          testbed where
                                                          I could get
                                                          access to
                                                          investigate
                                                          the corefiles?
                                                          If not, then I
                                                          will ask for
                                                          more details
                                                          from cores
                                                          over email,
                                                          first would be
                                                          'bt full' for
                                                          both cores and
                                                          'p *dead_cell'
                                                          for the second
                                                          one.<br>
                                                          </p>
                                                          Cheers,<br>
                                                          Daniel<br>
                                                          <br>
                                                          <div class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-cite-prefix">On
                                                          05.09.18
                                                          17:47, Jayesh
                                                          Nambiar wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">Hi
                                                          Daniel,
                                                          <div>Have got
                                                          these core
                                                          dumps. Let me
                                                          know if I
                                                          should be
                                                          doing a bt
                                                          full. I'll
                                                          pastebin and
                                                          send. Thanks,</div>
                                                          <div><br>
                                                          </div>
                                                          <div>
                                                          <div>Core was
                                                          generated by
                                                          `/usr/local/kamailio/sbin/kamailio
                                                          -P
                                                          /var/run/siptrunk.pid
                                                          -f
                                                          /usr/local/carrie'.</div>
                                                          <div>Program
                                                          terminated
                                                          with signal
                                                          SIGSEGV,
                                                          Segmentation
                                                          fault.</div>
                                                          <div>#0 
                                                          0x0000000000505d30
                                                          in
                                                          sip_msg_shm_clone
(org_msg=0x7f6ab0d9f618, sip_msg_len=0x7ffdddb2e8bc, clone_lumps=1) at
                                                          core/sip_msg_clone.c:491</div>
                                                          <div>491<span style="white-space:pre-wrap">                     </span>LUMP_LIST_LEN(len,
org_msg->add_rm);</div>
                                                          <div>(gdb) bt</div>
                                                          <div>#0 
                                                          0x0000000000505d30
                                                          in
                                                          sip_msg_shm_clone
(org_msg=0x7f6ab0d9f618, sip_msg_len=0x7ffdddb2e8bc, clone_lumps=1) at
                                                          core/sip_msg_clone.c:491</div>
                                                          <div>#1 
                                                          0x00007f6af0bdf68d
                                                          in fake_req
                                                          (shmem_msg=0x7f6ab0d9f618,
                                                          extra_flags=0,
uac=0x7f6ab1738980, len=0x7ffdddb2e8bc) at t_reply.c:854</div>
                                                          <div>#2 
                                                          0x00007f6af0c3aa27
                                                          in
                                                          t_continue_helper
(hash_index=58039, label=413633661, rtact=0x7f6af10500f0, cbname=0x0,
                                                          cbparam=0x0)
                                                          at
                                                          t_suspend.c:293</div>
                                                          <div>#3 
                                                          0x00007f6af0c3eed4
                                                          in t_continue
(hash_index=58039, label=413633661, route=0x7f6af10500f0) at
                                                          t_suspend.c:583</div>
                                                          <div>#4 
                                                          0x00007f6aae4dd010
                                                          in
                                                          w_t_continue
                                                          (msg=0x7ffdddb2fa60,
idx=0x7f6af1098e90 "8\306\t\361j\177", lbl=0x7f6af1098ff0
                                                          "\240\264\t\361j\177",
rtn=0x7f6af1099150 "0\275\t\361j\177") at tmx_mod.c:760</div>
                                                          <div>#5 
                                                          0x000000000045b477
                                                          in do_action
                                                          (h=0x7ffdddb2f850,
a=0x7f6af109ab38, msg=0x7ffdddb2fa60) at core/action.c:1085</div>
                                                          <div>#6 
                                                          0x0000000000467fd5
                                                          in run_actions
(h=0x7ffdddb2f850, a=0x7f6af1096630, msg=0x7ffdddb2fa60) at
                                                          core/action.c:1565</div>
                                                          <div>#7 
                                                          0x000000000045b234
                                                          in do_action
                                                          (h=0x7ffdddb2f850,
a=0x7f6af10a0f80, msg=0x7ffdddb2fa60) at core/action.c:1058</div>
                                                          <div>#8 
                                                          0x0000000000467fd5
                                                          in run_actions
(h=0x7ffdddb2f850, a=0x7f6af10a0f80, msg=0x7ffdddb2fa60) at
                                                          core/action.c:1565</div>
                                                          <div>#9 
                                                          0x0000000000468797
                                                          in
                                                          run_top_route
(a=0x7f6af10a0f80, msg=0x7ffdddb2fa60, c=0x0) at core/action.c:1654</div>
                                                          <div>#10
                                                          0x00007f6aabe79370
                                                          in
                                                          evapi_run_cfg_route
(evenv=0x7ffdddb30250, rt=3, rtname=0x7f6aac08cb18
                                                          <_evapi_rts+56>)
                                                          at
                                                          evapi_dispatch.c:161</div>
                                                          <div>#11
                                                          0x00007f6aabe7f271
                                                          in
                                                          evapi_recv_client
(loop=0x7f6aabe698e0, watcher=0x27af5e0, revents=1) at
                                                          evapi_dispatch.c:467</div>
                                                          <div>#12
                                                          0x00007f6aabc5fd73
                                                          in
                                                          ev_invoke_pending
                                                          () from
                                                          /usr/lib/x86_64-linux-gnu/libev.so.4</div>
                                                          <div>#13
                                                          0x00007f6aabc633de
                                                          in ev_run ()
                                                          from
                                                          /usr/lib/x86_64-linux-gnu/libev.so.4</div>
                                                          <div>#14
                                                          0x00007f6aabe7867c
                                                          in ev_loop
                                                          (loop=0x7f6aabe698e0,
                                                          flags=0) at
                                                          /usr/include/ev.h:835</div>
                                                          <div>#15
                                                          0x00007f6aabe83fc6
                                                          in
                                                          evapi_run_dispatcher
(laddr=0x7f6af0f72300 "127.0.0.1", lport=8060) at evapi_dispatch.c:705</div>
                                                          <div>#16
                                                          0x00007f6aabe6e262
                                                          in child_init
                                                          (rank=0) at
                                                          evapi_mod.c:213</div>
                                                          <div>#17
                                                          0x0000000000542cad
                                                          in
                                                          init_mod_child
(m=0x7f6af0f71b70, rank=0) at core/sr_module.c:943</div>
                                                          <div>#18
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f72968, rank=0) at core/sr_module.c:939</div>
                                                          <div>#19
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f73d38, rank=0) at core/sr_module.c:939</div>
                                                          <div>#20
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f74670, rank=0) at core/sr_module.c:939</div>
                                                          <div>#21
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f76708, rank=0) at core/sr_module.c:939</div>
                                                          <div>#22
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f76c08, rank=0) at core/sr_module.c:939</div>
                                                          <div>#23
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f770d0, rank=0) at core/sr_module.c:939</div>
                                                          <div>#24
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f77cf0, rank=0) at core/sr_module.c:939</div>
                                                          <div>#25
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f78808, rank=0) at core/sr_module.c:939</div>
                                                          <div>#26
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f78bd8, rank=0) at core/sr_module.c:939</div>
                                                          <div>#27
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f794c8, rank=0) at core/sr_module.c:939</div>
                                                          <div>#28
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f79920, rank=0) at core/sr_module.c:939</div>
                                                          <div>#29
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f7a330, rank=0) at core/sr_module.c:939</div>
                                                          <div>#30
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f7afd0, rank=0) at core/sr_module.c:939</div>
                                                          <div>#31
                                                          0x0000000000542971
                                                          in
                                                          init_mod_child
(m=0x7f6af0f7bc80, rank=0) at core/sr_module.c:939</div>
                                                          <div>#32
                                                          0x000000000054303d
                                                          in init_child
                                                          (rank=0) at
                                                          core/sr_module.c:970</div>
                                                          <div>#33
                                                          0x0000000000425399
                                                          in main_loop
                                                          () at
                                                          main.c:1701</div>
                                                          <div>#34
                                                          0x000000000042bd5c
                                                          in main
                                                          (argc=13,
                                                          argv=0x7ffdddb31088)
                                                          at main.c:2638</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>And this:</div>
                                                          <div>
                                                          <div>[New LWP
                                                          15804]</div>
                                                          <div>[Thread
                                                          debugging
                                                          using
                                                          libthread_db
                                                          enabled]</div>
                                                          <div>Using
                                                          host
                                                          libthread_db
                                                          library
                                                          "/lib/x86_64-linux-gnu/libthread_db.so.1".</div>
                                                          <div>Core was
                                                          generated by
                                                          `/usr/local/kamailio/sbin/kamailio
                                                          -P
                                                          /var/run/siptrunk.pid
                                                          -f
                                                          /usr/local/carrie'.</div>
                                                          <div>Program
                                                          terminated
                                                          with signal
                                                          SIGSEGV,
                                                          Segmentation
                                                          fault.</div>
                                                          <div>#0 
                                                          0x00007f6af0b90b70
                                                          in
                                                          free_cell_helper
(dead_cell=0x7f6ab0a6baa8, silent=0, fname=0x7f6af0c8f630 "timer.c",
                                                          fline=654) at
                                                          h_table.c:230</div>
                                                          <div>230<span style="white-space:pre-wrap">                     </span>foo
                                                          = tt->next;</div>
                                                          <div>(gdb) bt</div>
                                                          <div>#0 
                                                          0x00007f6af0b90b70
                                                          in
                                                          free_cell_helper
(dead_cell=0x7f6ab0a6baa8, silent=0, fname=0x7f6af0c8f630 "timer.c",
                                                          fline=654) at
                                                          h_table.c:230</div>
                                                          <div>#1 
                                                          0x00007f6af0c24409
                                                          in
                                                          wait_handler
                                                          (ti=932640643,
wait_tl=0x7f6ab0a6bb28, data=0x7f6ab0a6baa8) at timer.c:654</div>
                                                          <div>#2 
                                                          0x00000000004bb445
                                                          in
                                                          timer_list_expire
                                                          (t=932640643,
h=0x7f6ab03ad158, slow_l=0x7f6ab03ae480, slow_mark=271) at
                                                          core/timer.c:874</div>
                                                          <div>#3 
                                                          0x00000000004bb8ab
                                                          in
                                                          timer_handler
                                                          () at
                                                          core/timer.c:939</div>
                                                          <div>#4 
                                                          0x00000000004bbd30
                                                          in timer_main
                                                          () at
                                                          core/timer.c:978</div>
                                                          <div>#5 
                                                          0x00000000004250f9
                                                          in main_loop
                                                          () at
                                                          main.c:1691</div>
                                                          <div>#6 
                                                          0x000000000042bd5c
                                                          in main
                                                          (argc=13,
                                                          argv=0x7ffdddb31088)
                                                          at main.c:2638</div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <div class="gmail_quote">
                                                          <div dir="ltr">On
                                                          Wed, Sep 5,
                                                          2018 at 3:13
                                                          PM
                                                          Daniel-Constantin
                                                          Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
                                                          wrote:<br>
                                                          </div>
                                                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                                                          <div text="#000000" bgcolor="#FFFFFF">
                                                          <p>Hello,</p>
                                                          <p>the
                                                          backtrace
                                                          doesn't show
                                                          any hint about
                                                          kamailio, only
                                                          from
                                                          closelog() up.</p>
                                                          <p>It may be
                                                          the core
                                                          generated by
                                                          shutdown
                                                          procedure,
                                                          have you
                                                          enabled one
                                                          core file per
                                                          pid/process?
                                                          If not, do it
                                                          and reproduce
                                                          the issue
                                                          again, you may
                                                          get two core
                                                          files, one
                                                          being the
                                                          runtime issue
                                                          and the other
                                                          one from
                                                          shutdown
                                                          procedure,
                                                          which likely
                                                          is an effect
                                                          of the other
                                                          one. The one
                                                          from the
                                                          runtime is
                                                          more relevant.</p>
                                                          <p>Cheers,<br>
                                                          Daniel<br>
                                                          </p>
                                                          <br>
                                                          <div class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-cite-prefix">On
                                                          05.09.18
                                                          10:09, Jayesh
                                                          Nambiar wrote:<br>
                                                          </div>
                                                          <blockquote type="cite">
                                                          <div dir="ltr">
                                                          <div dir="ltr">Hello,
                                                          <div>I'm using
                                                          kamailio 5.1.5
                                                          with evapi. I
                                                          have a node.js
                                                          connecting
                                                          with kamailio
                                                          evapi to which
                                                          I send events
                                                          and also
                                                          consume events
                                                          based on which
                                                          I do the
                                                          routing. I
                                                          have 8 evapi
                                                          workers
                                                          defined in the
                                                          config. </div>
                                                          <div>The
                                                          problem is
                                                          that kamailio
                                                          randomly
                                                          crashes on
                                                          high load. I'm
                                                          assuming that
                                                          it is related
                                                          to the evapi
                                                          module as rest
                                                          of the config
                                                          is pretty
                                                          straight
                                                          forward. I
                                                          could get a
                                                          core file and
                                                          here's the
                                                          core dump:</div>
                                                          <div>
                                                          <div>[New LWP
                                                          14042]</div>
                                                          <div>[Thread
                                                          debugging
                                                          using
                                                          libthread_db
                                                          enabled]</div>
                                                          <div>Using
                                                          host
                                                          libthread_db
                                                          library
“/lib/x86_64-linux-gnu/libthread_db.so.1”.</div>
                                                          <div>Core was
                                                          generated by
                                                          `/usr/local/kamailio/sbin/kamailio
                                                          -P
                                                          /var/run/siptrunk.pid
                                                          -f
                                                          /usr/local/carrie’.</div>
                                                          <div>Program
                                                          terminated
                                                          with signal
                                                          SIGABRT,
                                                          Aborted.</div>
                                                          <div>#0 
                                                          0x00007f9995283428
                                                          in __GI_raise
(sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54</div>
                                                          <div>54   
                                                          ../sysdeps/unix/sysv/linux/raise.c:
                                                          No such file
                                                          or directory.</div>
                                                          <div>(gdb) bt</div>
                                                          <div>#0 
                                                          0x00007f9995283428
                                                          in __GI_raise
(sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54</div>
                                                          <div>#1 
                                                          0x00007f999528502a
                                                          in __GI_abort
                                                          () at
                                                          abort.c:89</div>
                                                          <div>#2 
                                                          0x000000000041a029
                                                          in
                                                          sig_alarm_abort
                                                          (signo=14) at
                                                          main.c:646</div>
                                                          <div>#3 
                                                          <signal
                                                          handler
                                                          called></div>
                                                          <div>#4 
                                                          0x00007f999534f497
                                                          in
                                                          __libc_cleanup_routine
(f=<optimized out>) at ../sysdeps/nptl/libc-lockP.h:291</div>
                                                          <div>#5 
                                                          closelog () at
../misc/syslog.c:415</div>
                                                          <div>#6 
                                                          0x0000000000000000
                                                          in ?? ()</div>
                                                          </div>
                                                          <div><br>
                                                          </div>
                                                          <div>Any help
                                                          in this
                                                          regards is
                                                          would allow me
                                                          to identify
                                                          the reason of
                                                          the crash.
                                                          Thanks for the
                                                          support.</div>
                                                          <div><br>
                                                          </div>
                                                          <div>- Jayesh</div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <fieldset class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478mimeAttachmentHeader"></fieldset>
                                                          <br>
                                                          <pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-signature" cols="72">-- 
Daniel-Constantin Mierla -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          <pre class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-signature" cols="72">-- 
Daniel-Constantin Mierla -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
                                                        </div>
                                                      </blockquote>
                                                    </div>
                                                  </blockquote>
                                                  <br>
                                                  <pre class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-signature" cols="72">-- 
Daniel-Constantin Mierla -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
                                                </div>
                                              </blockquote>
                                            </div>
                                          </blockquote>
                                          <br>
                                          <pre class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-signature" cols="72">-- 
Daniel-Constantin Mierla -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
                                        </div>
                                      </blockquote>
                                    </div>
                                  </blockquote>
                                </div>
                              </blockquote>
                              <br>
                              <pre class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-signature" cols="72">-- 
Daniel-Constantin Mierla -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-4772721693493464624m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
                            </div>
                          </blockquote>
                        </div>
                      </blockquote>
                    </div>
                  </blockquote>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <fieldset class="m_-4772721693493464624mimeAttachmentHeader"></fieldset>
      <br>
      <pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="m_-4772721693493464624moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a class="m_-4772721693493464624moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
    </blockquote>
    <br>
    <pre class="m_-4772721693493464624moz-signature" cols="72">-- 
Daniel-Constantin Mierla -- <a class="m_-4772721693493464624moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-4772721693493464624moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-4772721693493464624moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-4772721693493464624moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-4772721693493464624moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
  </div>

</blockquote></div>