<div dir="ltr"><div><div><div>Thanks Daniel,<br><br></div>The system has been stable for the last week.<br><br></div>We will apply the patch and report back if there is any problems.<br><br></div>/Morten<br></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Tue, Nov 5, 2013 at 10:42 AM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    Hello,<br>
    <br>
    I added a patch to fix allocation of 0 size in dialog, can you give
    it a try:<br>
    <br>
    -
<a href="http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=082a6c43938cf8e3839d46fd070e391bd522d4ed" target="_blank">http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=082a6c43938cf8e3839d46fd070e391bd522d4ed</a><br>

    <br>
    It is rather small, should be easy to backport.<br>
    <br>
    I will look further in memory manager, but for now I thought worth
    fixing dialog. If the issues doesn't appear any longer, then I know
    where to look properly.<br>
    <br>
    Cheers,<br>
    Daniel<div><div class="h5"><br>
    <br>
    <div>On 11/4/13 11:52 PM, Daniel-Constantin
      Mierla wrote:<br>
    </div>
    <blockquote type="cite">
      
      Hello,<br>
      <br>
      it might be related to the malloc of a size 0 done for one of the
      cseq value (which doesn't exist at the respective time). I need to
      investigate a bit more, I will come with updates as I get.<br>
      <br>
      Cheers,<br>
      Daniel<br>
      <br>
      <div>On 10/29/13 10:33 AM, Morten Isaksen
        wrote:<br>
      </div>
      <blockquote type="cite">
        <div dir="ltr">
          <div>
            <div>Hi Daniel,<br>
              <br>
            </div>
            I still have the core file. You can find the output of p
            *shm_block here <a href="http://pastebin.com/mgh4ed44" target="_blank">http://pastebin.com/mgh4ed44</a>
            - it was too big to poste on the list.<br>
            <br>
          </div>
          /Morten<br>
        </div>
        <div class="gmail_extra"><br>
          <br>
          <div class="gmail_quote">On Tue, Oct 29, 2013 at 9:56 AM,
            Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span> wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
                <br>
                going to look over it. Do you still have the core file?
                If yes, provide the output of:<br>
                <br>
                p *shm_block<br>
                <br>
                from inside gdb.<br>
                <br>
                Cheers,<br>
                Daniel
                <div>
                  <div><br>
                    <br>
                    <div>On 10/28/13 1:00 PM, Morten Isaksen wrote:<br>
                    </div>
                    <blockquote type="cite">
                      <div dir="ltr">
                        <div>Hi,<br>
                          <br>
                          We upgraded to 4.0.4 and now we got a new core
                          dump in q_malloc.c:269. It has been stable for
                          about 5 days after the upgrade.<br>
                          <br>
                        </div>
                        /Morten<br>
                        <div><br>
                          The full backtrace is here:<br>
                          <br>
                          Loaded symbols for /lib64/libnss_files.so.2<br>
                          Core was generated by
                          `/usr/local/sbin/kamailio -P
                          /var/run/kamailio/kamailio.pid -m 256 -M 8 -u
                          kamai'.<br>
                          Program terminated with signal 11,
                          Segmentation fault.<br>
                          #0 qm_detach_free (qm=0x7f5394de9000, size=0)
                          at mem/q_malloc.c:269<br>
                          269
                          FRAG_END(next)->prev_free=prev;<br>
                          Missing separate debuginfos, use:
                          debuginfo-install glibc-2.12-1.107.el6.x86_64
                          hiredis-0.10.1-3.el6.x86_64
                          keyutils-libs-1.4-4.el6.x86_64
                          krb5-libs-1.10.3-10.el6.x86_64
                          libcom_err-1.41.12-14.el6.x86_64
                          libselinux-2.0.94-5.3.el6.x86_64
                          libxml2-2.7.6-12.el6_4.1.x86_64
                          mysql-libs-5.1.67-1.el6_3.x86_64
                          nss-softokn-freebl-3.12.9-11.el6.x86_64
                          openssl-1.0.0-27.el6_4.2.x86_64
                          zlib-1.2.3-29.el6.x86_64<br>
                          (gdb) bt full<br>
                          #0 qm_detach_free (qm=0x7f5394de9000, size=0)
                          at mem/q_malloc.c:269<br>
                           prev = 0x7f539524ed60<br>
                           next = 0x0<br>
                          #1 qm_malloc (qm=0x7f5394de9000, size=0) at
                          mem/q_malloc.c:386<br>
                           f = 0x7f539524ed60<br>
                           hash = 0<br>
                          #2 0x00007f53a5662950 in shm_malloc (size=0)
                          at ../../mem/shm_mem.h:262<br>
                           p = <value optimized out><br>
                          #3 0x00007f53a566b49b in dlg_set_leg_info
                          (dlg=<value optimized out>,
                          tag=0x7ffff680b9a0, rr=0x7ffff680b860,
                          contact=0x7ffff680b870, cseq=0x7ffff680b880,
                          leg=1) at dlg_hash.c:498<br>
                           p = <value optimized out><br>
                           __FUNCTION__ = "dlg_set_leg_info"<br>
                          #4 0x00007f53a5659e88 in populate_leg_info
                          (dlg=0x7f53952312b0, msg=<value optimized
                          out>, t=<value optimized out>, leg=1,
                          tag=0x7ffff680b9a0) at dlg_handlers.c:276<br>
                           skip_recs = 4<br>
                           cseq = {s = 0x0, len = 0}<br>
                           contact = {<br>
                           s = 0x8da2c0 "<a href="http://sip:65993777@178.21.249.22:5060" target="_blank">sip:65993777@178.21.249.22:5060</a>>\r\nContent-Type:


                          application/sdp\r\nRequire:
                          timer\r\nContent-Length: 252\r\n\r\nv=0\r\no=-
                          <a href="tel:2042480911" value="+12042480911" target="_blank">2042480911</a> <a href="tel:2042480911" value="+12042480911" target="_blank">2042480911</a>
                          IN IP4 178.21.249.22\r\ns=Hosted PBX\r\nc=IN
                          IP4 178.21.249.22\r\nt=0 0"..., len = 31}<br>
                           rr_set = {s = 0x7f53ab927c20
                          <incomplete sequence \361\237\215>, len
                          = 0}<br>
                           __FUNCTION__ = "populate_leg_info"<br>
                          #5 0x00007f53a565b949 in dlg_onreply
                          (t=0x7f5395245030, type=<value optimized
                          out>, param=<value optimized out>) at
                          dlg_handlers.c:500<br>
                           dlg = 0x7f53952312b0<br>
                           iuid = <value optimized out><br>
                           new_state = 3<br>
                           old_state = 1<br>
                           unref = 0<br>
                           event = 3<br>
                           tag = {<br>
                           s = 0x8da1bc "as77c791c9\r\nCall-ID:
                          0dd71e51-ba5d-1231-c79f-00095200eb1e\r\nCSeq:
                          1206140 INVITE\r\nServer:
                          one-connect\r\nAllow: INVITE, ACK, CANCEL,
                          OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY, INFO,
                          PUBLISH\r\nSupported: repl"..., len = 10}<br>
                           req = 0x7f5395640090<br>
                           rpl = 0x7f53ab8cd7a0<br>
                           __FUNCTION__ = "dlg_onreply"<br>
                          #6 0x00007f53aa1f58ad in
                          run_trans_callbacks_internal (cb_lst=<value
                          optimized out>, type=1048576,
                          trans=0x7f5395245030, params=0x7ffff680baa0)
                          at t_hooks.c:290<br>
                           cbp = 0x7f5395311260<br>
                           backup_from = 0x8e9ed0<br>
                           backup_to = 0x8e9ed8<br>
                           backup_dom_from = 0x8e9ee0<br>
                           backup_dom_to = 0x8e9ee8<br>
                           backup_uri_from = 0x8e9ec0<br>
                           backup_uri_to = 0x8e9ec8<br>
                           backup_xavps = 0x8e9fd0<br>
                           __FUNCTION__ =
                          "run_trans_callbacks_internal"<br>
                          #7 0x00007f53aa1f5a94 in
                          run_trans_callbacks_with_buf (type=<value
                          optimized out>, rbuf=0x7f53952450f0,
                          req=<value optimized out>,
                          repl=<value optimized out>,
                          flags=<value optimized out>) at
                          t_hooks.c:336<br>
                           params = {req = 0x7f5395640090, rpl =
                          0x7f53ab8cd7a0, param = 0x7f5395311270, code =
                          200, flags = 200, branch = 0, t_rbuf =
                          0x7f53952450f0, dst = 0x7f5395245140, send_buf
                          = {<br>
                           s = 0x7f53953ac460 "SIP/2.0 200
                          OK\r\nVia: SIP/2.0/UDP
                          178.21.248.30;rport=5060;branch=z9hG4bKd24f.32134635.0\r\nVia:
                          SIP/2.0/UDP
                          178.21.248.8;rport=5060;branch=z9hG4bKd24f.b20f4bb3.0\r\nVia:
                          SIP/2.0/UDP 178.21.248.20;branch=z"..., len =
                          1341}}<br>
                           trans = <value optimized out><br>
                          #8 0x00007f53aa22728c in relay_reply
                          (t=<value optimized out>,
                          p_msg=<value optimized out>, branch=0,
                          msg_status=200, cancel_data=0x7ffff680be40,
                          do_put_on_wait=1) at t_reply.c:1867<br>
                           relay = 0<br>
                           save_clone = 0<br>
                           buf = 0x7f53ab9361a0 "SIP/2.0 200
                          OK\r\nVia: SIP/2.0/UDP
                          178.21.248.30;rport=5060;branch=z9hG4bKd24f.32134635.0\r\nVia:
                          SIP/2.0/UDP
                          178.21.248.8;rport=5060;branch=z9hG4bKd24f.b20f4bb3.0\r\nVia:
                          SIP/2.0/UDP 178.21.248.20;branch=z"...<br>
                           res_len = 1341<br>
                           relayed_code = 200<br>
                           relayed_msg = 0x7f53ab8cd7a0<br>
                           reply_bak = <value optimized
                          out><br>
                           bm = {to_tag_val = {s = 0x7f5395600c50
                          "P\233\365\224S\177", len = -1440728525}}<br>
                           totag_retr = 0<br>
                           reply_status = RPS_COMPLETED<br>
                           uas_rb = <value optimized out><br>
                           to_tag = <value optimized out><br>
                           reason = {s = 0x3 <Address 0x3 out
                          of bounds>, len = -159334788}<br>
                           onsend_params = {req = 0x7f53aa45e594,
                          rpl = 0x7ffff680be78, param = 0x0, code = 0,
                          flags = 0, branch = 0, t_rbuf = 0x10, dst =
                          0x7f53aa200a91, send_buf = {s = 0x7f5395601f38
                          "", len = -1440601726}}<br>
                          ---Type <return> to continue, or q
                          <return> to quit---<br>
                           __FUNCTION__ = "relay_reply"<br>
                          #9 0x00007f53aa227da6 in reply_received
                          (p_msg=0x7f53ab8cd7a0) at t_reply.c:2374<br>
                           msg_status = 200<br>
                           last_uac_status = 100<br>
                           ack = 0x1 <Address 0x1 out of
                          bounds><br>
                           ack_len = 0<br>
                           branch = 0<br>
                           reply_status = <value optimized
                          out><br>
                           onreply_route = <value optimized
                          out><br>
                           cancel_data = {cancel_bitmap = 0,
                          reason = {cause = 200, u = {text = {s = 0x0,
                          len = -1416571248}, e2e_cancel = 0x0,
                          packed_hdrs = {s = 0x0, len = -1416571248}}}}<br>
                           uac = 0x7f5395245198<br>
                           t = 0x7f5395245030<br>
                           lack_dst = {send_sock = 0x0, to = {s =
                          {sa_family = 48720, sa_data =
                          "\200\366\377\177\000\000\000\000\000\000\000\000\000"},
                          sin = {sin_family = 48720, sin_port = 63104,
                          sin_addr = {<br>
                           s_addr = 32767}, sin_zero =
                          "\000\000\000\000\000\000\000"}, sin6 =
                          {sin6_family = 48720, sin6_port = 63104,
                          sin6_flowinfo = 32767, sin6_addr = {__in6_u =
                          {<br>
                           __u6_addr8 =
                          "\000\000\000\000\000\000\000\000\260kq\253S\177\000",
                          __u6_addr16 = {0, 0, 0, 0, 27568, 43889,
                          32595, 0}, __u6_addr32 = {0, 0, 2876337072,
                          32595}}},<br>
                           sin6_scope_id = 2878134176}}, id
                          = 32595, proto = -76 '\264', send_flags = {f =
                          5 '\005', blst_imask = 0 '\000'}}<br>
                           backup_user_from = <value optimized
                          out><br>
                           backup_user_to = <value optimized
                          out><br>
                           backup_domain_from = <value
                          optimized out><br>
                           backup_domain_to = <value optimized
                          out><br>
                           backup_uri_from = <value optimized
                          out><br>
                           backup_uri_to = <value optimized
                          out><br>
                           backup_xavps = <value optimized
                          out><br>
                           replies_locked = <value optimized
                          out><br>
                           branch_ret = <value optimized
                          out><br>
                           prev_branch = <value optimized
                          out><br>
                           blst_503_timeout = <value optimized
                          out><br>
                           hf = <value optimized out><br>
                           onsend_params = {req = 0x8da424, rpl =
                          0x543274, param = 0x10, code = -1416833120,
                          flags = 32595, branch = 0, t_rbuf =
                          0x7ffff680be50, dst = 0x7f53a566690b, send_buf
                          = {<br>
                           s = 0x4ff1e7b00000000 <Address
                          0x4ff1e7b00000000 out of bounds>, len = 0}}<br>
                           ctx = {rec_lev = 0, run_flags = 0,
                          last_retcode = 1, jmp_env = {{__jmpbuf =
                          {140737329020144, -5577311537021373234,
                          139997336902320, 139997337147296, 200,
                          139996961202968, 5577292203380126926,<br>
                           -5577312076254333746},
                          __mask_was_saved = 0, __saved_mask = {__val =
                          {139997337420128, 9281488, 139997335445253, 1,
                          140737329020496, 6191275, 5978073, 8591616,
                          9281252, 68728758110,<br>
                           5548121, 9281656, 9281173,
                          139997337409168, 9282596, 139997337519904}}}}}<br>
                           __FUNCTION__ = "reply_received"<br>
                          #10 0x0000000000456d54 in do_forward_reply
                          (msg=0x7f53ab8cd7a0, mode=<value optimized
                          out>) at forward.c:799<br>
                           new_buf = 0x0<br>
                           dst = {send_sock = 0x0, to = {s =
                          {sa_family = 0, sa_data = '\000' <repeats
                          13 times>}, sin = {sin_family = 0, sin_port
                          = 0, sin_addr = {s_addr = 0}, sin_zero =
                          "\000\000\000\000\000\000\000"},<br>
                           sin6 = {sin6_family = 0, sin6_port
                          = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u =
                          {__u6_addr8 = '\000' <repeats 15 times>,
                          __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
                          __u6_addr32 = {0, 0, 0, 0}}},<br>
                           sin6_scope_id = 0}}, id = 0,
                          proto = 0 '\000', send_flags = {f = 0 '\000',
                          blst_imask = 0 '\000'}}<br>
                           new_len = <value optimized out><br>
                           r = <value optimized out><br>
                           s = <value optimized out><br>
                           len = <value optimized out><br>
                           __FUNCTION__ = "do_forward_reply"<br>
                          #11 0x000000000049f4be in receive_msg (<br>
                           buf=0x8d9e80 "SIP/2.0 200 OK\r\nVia:
                          SIP/2.0/UDP
                          178.21.249.20;branch=z9hG4bKd24f.20eeb764.0;received=178.21.249.20;rport=5060\r\nVia:


                          SIP/2.0/UDP
                          178.21.248.30;rport=5060;branch=z9hG4bKd24f.32134635.0\r\nVia:
                          SIP/2.0/UD"..., len=<value optimized
                          out>, rcv_info=0x7ffff680c100) at
                          receive.c:272<br>
                           msg = 0x7f53ab8cd7a0<br>
                           ctx = {rec_lev = 12, run_flags = 0,
                          last_retcode = -1517756399, jmp_env =
                          {{__jmpbuf = {139996960369776, 12,
                          219309716216, 139997343567872,
                          140737329021424, 4294967295, 140737329021551,
                          1},<br>
                           __mask_was_saved = 8581896,
                          __saved_mask = {__val = {0, 28, 16, 0,
                          219305533392, 1, 0, 139997335309296,
                          219309716216, 139996960369776,
                          139997337398864, 139997343565480,
                          139997337398872,<br>
                           140737329021320,
                          219305559701, 140737329021192}}}}}<br>
                           ret = <value optimized out><br>
                           inb = {<br>
                           s = 0x8d9e80 "SIP/2.0 200 OK\r\nVia:
                          SIP/2.0/UDP
                          178.21.249.20;branch=z9hG4bKd24f.20eeb764.0;received=178.21.249.20;rport=5060\r\nVia:


                          SIP/2.0/UDP
                          178.21.248.30;rport=5060;branch=z9hG4bKd24f.32134635.0\r\nVia:
                          SIP/2.0/UD"..., len = 1444}<br>
                           __FUNCTION__ = "receive_msg"<br>
                          #12 0x0000000000532306 in udp_rcv_loop () at
                          udp_server.c:557<br>
                           len = 1444<br>
                           buf = "SIP/2.0 200 OK\r\nVia:
                          SIP/2.0/UDP
                          178.21.249.20;branch=z9hG4bKd24f.20eeb764.0;received=178.21.249.20;rport=5060\r\nVia:


                          SIP/2.0/UDP
                          178.21.248.30;rport=5060;branch=z9hG4bKd24f.32134635.0\r\nVia:
                          SIP/2.0/UD"...<br>
                           from = 0x7f53ab90ae50<br>
                           fromlen = 16<br>
                           ri = {src_ip = {af = 2, len = 4, u =
                          {addrl = {385422770, 0}, addr32 = {385422770,
                          0, 0, 0}, addr16 = {5554, 5881, 0, 0, 0, 0, 0,
                          0}, addr = "\262\025\371\026", '\000'
                          <repeats 11 times>}},<br>
                          ---Type <return> to continue, or q
                          <return> to quit---<br>
                           dst_ip = {af = 2, len = 4, u =
                          {addrl = {351868338, 0}, addr32 = {351868338,
                          0, 0, 0}, addr16 = {5554, 5369, 0, 0, 0, 0, 0,
                          0}, addr = "\262\025\371\024", '\000'
                          <repeats 11 times>}},<br>
                           src_port = 5060, dst_port = 5060,
                          proto_reserved1 = 0, proto_reserved2 = 0,
                          src_su = {s = {sa_family = 2, sa_data =
                          "\023IJ\025\371\026\000\000\000\000\000\000\000"},
                          sin = {sin_family = 2,<br>
                           sin_port = 50195, sin_addr =
                          {s_addr = 385422770}, sin_zero =
                          "\000\000\000\000\000\000\000"}, sin6 =
                          {sin6_family = 2, sin6_port = 50195,
                          sin6_flowinfo = 385422770, sin6_addr =
                          {__in6_u = {<br>
                           __u6_addr8 = '\000'
                          <repeats 15 times>, __u6_addr16 = {0, 0,
                          0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
                          0}}}, sin6_scope_id = 0}}, bind_address =
                          0x7f53ab8cec10, proto = 1 '\001'}<br>
                           __FUNCTION__ = "udp_rcv_loop"<br>
                          #13 0x0000000000467a7a in main_loop () at
                          main.c:1638<br>
                           i = <value optimized out><br>
                           pid = <value optimized out><br>
                           si = <value optimized out><br>
                           si_desc = "udp receiver child=3 sock=<a href="http://178.21.249.20:5060" target="_blank">178.21.249.20:5060</a>\000\000\000\000@\\\220\253S\177\000\000\000\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000\001\000\000\000\001\000\000\000\300\375\216\000\000\000\000\000\001\000\000\000\000\000\000\000\000\376\216\000\000\000\000\000\000\000\200\020",


                          '\000' <repeats 12 times>,
                          "\005\000\000\000\000\000\000"<br>
                           nrprocs = <value optimized out><br>
                           __FUNCTION__ = "main_loop"<br>
                          #14 0x000000000046a912 in main (argc=<value
                          optimized out>, argv=<value optimized
                          out>) at main.c:2566<br>
                           cfg_stream = <value optimized
                          out><br>
                           c = <value optimized out><br>
                           r = <value optimized out><br>
                           tmp = 0x7ffff680d89a ""<br>
                           tmp_len = 0<br>
                           options = 0x5c1de8
                          ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"<br>
                           ret = -1<br>
                           seed = 1043769636<br>
                           rfd = <value optimized out><br>
                           debug_save = <value optimized
                          out><br>
                           debug_flag = <value optimized
                          out><br>
                           dont_fork_cnt = <value optimized
                          out><br>
                           n_lst = <value optimized out><br>
                           p = <value optimized out><br>
                           __FUNCTION__ = "main"<br>
                          <br>
                        </div>
                      </div>
                      <div class="gmail_extra"><br>
                        <br>
                        <div class="gmail_quote">On Mon, Oct 21, 2013 at
                          8:15 PM, Daniel-Constantin Mierla <span dir="ltr"><<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>></span>
                          wrote:<br>
                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                            <div bgcolor="#FFFFFF" text="#000000">
                              Hello,<br>
                              <br>
                              can you upgrade to 4.0.4? there were some
                              fixes related to new fields in sip message
                              structure that resulted in some out bound
                              memory inside transactions.<br>
                              <br>
                              The config file and database is the same,
                              so just re-install over the previous
                              installation.<br>
                              <br>
                              Cheers,<br>
                              Daniel
                              <div>
                                <div><br>
                                  <br>
                                  <div>On 10/21/13 11:10 AM, Morten
                                    Isaksen wrote:<br>
                                  </div>
                                </div>
                              </div>
                              <blockquote type="cite">
                                <div>
                                  <div>
                                    <div dir="ltr">
                                      <div>
                                        <div>
                                          <div>Hi,<br>
                                            <br>
                                          </div>
                                          Our Kamailio has stopped with
                                          a segmentation fault 4 times
                                          the last week.<br>
                                          <br>
                                        </div>
                                        There has been no changes to the
                                        configuration file the last 15
                                        days, so I suspect a SIP phone
                                        is sending a SIP packet that
                                        Kamailio does not like.<br>
                                        <br>
                                      </div>
                                      <div>We have a core dump file but
                                        I cannot read anything usefull
                                        from the backtrace. Can you see
                                        what is wrong from the
                                        backtrace?<br>
                                        <br>
                                      </div>
                                      <div>Regards<br>
                                      </div>
                                      <div>Morten<br>
                                      </div>
                                      <div><br>
                                      </div>
                                      The output from gdb (bt full):<br>
                                      <br>
                                      Core was generated by
                                      `/usr/local/sbin/kamailio -P
                                      /var/run/kamailio/kamailio.pid -m
                                      256 -M 8 -u kamai'.<br>
                                      Program terminated with signal 11,
                                      Segmentation fault.<br>
                                      #0 0x00007f0e10de17b2 in
                                      cancel_branch (t=0x7f0dfbf38e10,
                                      branch=0, reason=<value
                                      optimized out>, flags=4) at
                                      t_cancel.c:284<br>
                                      284 if (cfg_get(tm,
                                      tm_cfg, reparse_invite) ||<br>
                                      Missing separate debuginfos, use:
                                      debuginfo-install
                                      glibc-2.12-1.107.el6.x86_64
                                      hiredis-0.10.1-3.el6.x86_64
                                      keyutils-libs-1.4-4.el6.x86_64
                                      krb5-libs-1.10.3-10.el6.x86_64
                                      libcom_err-1.41.12-14.el6.x86_64
                                      libselinux-2.0.94-5.3.el6.x86_64
                                      libxml2-2.7.6-12.el6_4.1.x86_64
                                      mysql-libs-5.1.67-1.el6_3.x86_64
                                      nss-softokn-freebl-3.12.9-11.el6.x86_64
                                      openssl-1.0.0-27.el6_4.2.x86_64
                                      zlib-1.2.3-29.el6.x86_64<br>
                                      (gdb) bt full<br>
                                      #0 0x00007f0e10de17b2 in
                                      cancel_branch (t=0x7f0dfbf38e10,
                                      branch=0, reason=<value
                                      optimized out>, flags=4) at
                                      t_cancel.c:284<br>
                                       cancel = <value
                                      optimized out><br>
                                       len = <value optimized
                                      out><br>
                                       crb = 0x7f0dfbf39008<br>
                                       irb = 0x7f0dfbf38f80<br>
                                       ret = 1<br>
                                       tmp_cd = {cancel_bitmap =
                                      0, reason = {cause = 0, u = {text
                                      = {s = 0x0, len = 0}, e2e_cancel =
                                      0x0, packed_hdrs = {s = 0x0, len =
                                      0}}}}<br>
                                       pcbuf = <value
                                      optimized out><br>
                                       __FUNCTION__ =
                                      "cancel_branch"<br>
                                      #1 0x00007f0e10e298ab in
                                      reply_received
                                      (p_msg=0x7f0e124ce760) at
                                      t_reply.c:2194<br>
                                       msg_status = <value
                                      optimized out><br>
                                       last_uac_status = 408<br>
                                       ack = 0x7f0dfbf38e10 "<br>
                                      
                                      \267\373\r\177"<br>
                                       ack_len = <value
                                      optimized out><br>
                                       branch = 0<br>
                                       reply_status = <value
                                      optimized out><br>
                                       onreply_route = <value
                                      optimized out><br>
                                       cancel_data =
                                      {cancel_bitmap = 0, reason =
                                      {cause = 408, u = {text = {s =
                                      0x0, len = 307468800}, e2e_cancel
                                      = 0x0, packed_hdrs = {s = 0x0, len
                                      = 307468800}}}}<br>
                                       uac = <value optimized
                                      out><br>
                                       t = 0x7f0dfbf38e10<br>
                                       lack_dst = {send_sock =
                                      0x0, to = {s = {sa_family = 6704,
                                      sa_data =
                                      "\\#\377\177\000\000\000\000\000\000\000\000\000"},
                                      sin = {sin_family = 6704, sin_port
                                      = 9052, sin_addr = {s_addr =
                                      32767},<br>
                                       sin_zero =
                                      "\000\000\000\000\000\000\000"},
                                      sin6 = {sin6_family = 6704,
                                      sin6_port = 9052, sin6_flowinfo =
                                      32767, sin6_addr = {__in6_u = {<br>
                                       __u6_addr8 =
                                      "\000\000\000\000\000\000\000\000p{1\022\016\177\000",
                                      __u6_addr16 = {0, 0, 0, 0, 31600,
                                      4657, 32526, 0}, __u6_addr32 = {0,
                                      0, 305232752, 32526}}},<br>
                                       sin6_scope_id =
                                      307029856}}, id = 32526, proto =
                                      28 '\034', send_flags = {f = 0
                                      '\000', blst_imask = 0 '\000'}}<br>
                                       backup_user_from =
                                      <value optimized out><br>
                                       backup_user_to = <value
                                      optimized out><br>
                                       backup_domain_from =
                                      <value optimized out><br>
                                       backup_domain_to =
                                      <value optimized out><br>
                                       backup_uri_from =
                                      <value optimized out><br>
                                       backup_uri_to = <value
                                      optimized out><br>
                                       backup_xavps = <value
                                      optimized out><br>
                                       replies_locked = 0<br>
                                       branch_ret = <value
                                      optimized out><br>
                                       prev_branch = <value
                                      optimized out><br>
                                       blst_503_timeout =
                                      <value optimized out><br>
                                       hf = <value optimized
                                      out><br>
                                       onsend_params = {req =
                                      0x8d8a39, rpl = 0x541db4, param =
                                      0x7f0e1253e528, code = 307029856,
                                      flags = 32526, branch = 0, t_rbuf
                                      = 0x7fff235c1a30, dst =
                                      0x7f0e12317b70, send_buf = {<br>
                                       s = 0x375311000000000
                                      <Address 0x375311000000000 out
                                      of bounds>, len = 0}}<br>
                                       ctx = {rec_lev =
                                      307491008, run_flags = 32526,
                                      last_retcode = 5674412, jmp_env =
                                      {{__jmpbuf = {140733786626256,
                                      63331951475841423,
                                      139698413054576, 139698413299552,
                                      9276465, 139698039855608,<br>
                                      
                                      -63254168797292145,
                                      63332490682325391},
                                      __mask_was_saved = 0, __saved_mask
                                      = {__val = {139698413734928, 0,
                                      139698411522821, 1,
                                      140733786626608, 6185835, 5972697,
                                      8586176, 9275699,<br>
                                       69026945952, 3,
                                      9276465, 9275673, 139698413738496,
                                      9275961, 139698413760704}}}}}<br>
                                       __FUNCTION__ =
                                      "reply_received"<br>
                                      #2 0x0000000000456444 in
                                      do_forward_reply
                                      (msg=0x7f0e124ce760,
                                      mode=<value optimized out>)
                                      at forward.c:799<br>
                                       new_buf = 0x0<br>
                                       dst = {send_sock = 0x0, to
                                      = {s = {sa_family = 0, sa_data =
                                      '\000' <repeats 13 times>},
                                      sin = {sin_family = 0, sin_port =
                                      0, sin_addr = {s_addr = 0},
                                      sin_zero =
                                      "\000\000\000\000\000\000\000"},<br>
                                       sin6 = {sin6_family =
                                      0, sin6_port = 0, sin6_flowinfo =
                                      0, sin6_addr = {__in6_u =
                                      {__u6_addr8 = '\000' <repeats
                                      15 times>, __u6_addr16 = {0, 0,
                                      0, 0, 0, 0, 0, 0}, __u6_addr32 =
                                      {0, 0, 0, 0}}},<br>
                                       sin6_scope_id = 0}},
                                      id = 0, proto = 0 '\000',
                                      send_flags = {f = 0 '\000',
                                      blst_imask = 0 '\000'}}<br>
                                       new_len = <value
                                      optimized out><br>
                                       r = <value optimized
                                      out><br>
                                       s = <value optimized
                                      out><br>
                                       len = <value optimized
                                      out><br>
                                       __FUNCTION__ =
                                      "do_forward_reply"<br>
                                      #3 0x000000000049e15e in
                                      receive_msg (buf=<value
                                      optimized out>, len=313,
                                      rcv_info=0x7fff235c1cd0) at
                                      receive.c:270<br>
                                       msg = 0x7f0e124ce760<br>
                                       ctx = {rec_lev = 11,
                                      run_flags = 0, last_retcode =
                                      206110737, jmp_env = {{__jmpbuf =
                                      {139698036884436, 11,
                                      219309716216, 139698419720192,
                                      140733786627520, 4294967295,
                                      140733786627647, 1},<br>
                                       __mask_was_saved =
                                      8576456, __saved_mask = {__val =
                                      {0, 28, 16, 0, 219305533392, 1, 0,
                                      139698411461552, 219309716216,
                                      139698036884436, 139698413732672,
                                      139698419717800, 139698413732680,<br>
                                       140733786627416,
                                      219305559701, 140733786627288}}}}}<br>
                                       ret = <value optimized
                                      out><br>
                                       inb = {<br>
                                       s = 0x8d8900 "SIP/2.0
                                      100 Trying\r\nVia: SIP/2.0/UDP
                                      178.21.249.20;branch=z9hG4bK8149.c6575a95.0\r\nTo:
                                      <a href="mailto:sip%3A201@78799865.pbx.one-connect.dk" target="_blank">sip:201@78799865.pbx.one-connect.dk</a>;tag=07c44e68\r\nFrom:



                                      <a href="mailto:sip%3A201@78799865.pbx.one-connect.dk" target="_blank">sip:201@78799865.pbx.one-connect.dk</a>;tag=a6a1c5f60faecf035a"...,



                                      len = 313}<br>
                                       __FUNCTION__ =
                                      "receive_msg"<br>
                                      #4 0x0000000000530e46 in
                                      udp_rcv_loop () at
                                      udp_server.c:557<br>
                                      ---Type <return> to
                                      continue, or q <return> to
                                      quit---<br>
                                       len = 313<br>
                                       buf = "SIP/2.0 100
                                      Trying\r\nVia: SIP/2.0/UDP
                                      178.21.249.20;branch=z9hG4bK8149.c6575a95.0\r\nTo:
                                      <a href="mailto:sip%3A201@78799865.pbx.one-connect.dk" target="_blank">sip:201@78799865.pbx.one-connect.dk</a>;tag=07c44e68\r\nFrom:



                                      <a href="mailto:sip%3A201@78799865.pbx.one-connect.dk" target="_blank">sip:201@78799865.pbx.one-connect.dk</a>;tag=a6a1c5f60faecf035a"...<br>
                                       from = 0x7f0e12538340<br>
                                       fromlen = 16<br>
                                       ri = {src_ip = {af = 2,
                                      len = 4, u = {addrl = {2993962576,
                                      0}, addr32 = {2993962576, 0, 0,
                                      0}, addr16 = {15952, 45684, 0, 0,
                                      0, 0, 0, 0}, addr = "P>t\262",
                                      '\000' <repeats 11 times>}},
                                      dst_ip = {<br>
                                       af = 2, len = 4, u =
                                      {addrl = {351868338, 0}, addr32 =
                                      {351868338, 0, 0, 0}, addr16 =
                                      {5554, 5369, 0, 0, 0, 0, 0, 0},
                                      addr = "\262\025\371\024", '\000'
                                      <repeats 11 times>}},
                                      src_port = 35754,<br>
                                       dst_port = 5060,
                                      proto_reserved1 = 0,
                                      proto_reserved2 = 0, src_su = {s =
                                      {sa_family = 2, sa_data =
                                      "\213\252P>t\262\000\000\000\000\000\000\000"},
                                      sin = {sin_family = 2, sin_port =
                                      43659,<br>
                                       sin_addr = {s_addr =
                                      2993962576}, sin_zero =
                                      "\000\000\000\000\000\000\000"},
                                      sin6 = {sin6_family = 2, sin6_port
                                      = 43659, sin6_flowinfo =
                                      2993962576, sin6_addr = {__in6_u =
                                      {<br>
                                       __u6_addr8 =
                                      '\000' <repeats 15 times>,
                                      __u6_addr16 = {0, 0, 0, 0, 0, 0,
                                      0, 0}, __u6_addr32 = {0, 0, 0,
                                      0}}}, sin6_scope_id = 0}},
                                      bind_address = 0x7f0e124cfbd0,
                                      proto = 1 '\001'}<br>
                                       __FUNCTION__ =
                                      "udp_rcv_loop"<br>
                                      #5 0x000000000046716a in
                                      main_loop () at main.c:1638<br>
                                       i = <value optimized
                                      out><br>
                                       pid = <value optimized
                                      out><br>
                                       si = <value optimized
                                      out><br>
                                       si_desc = "udp receiver
                                      child=2 sock=<a href="http://178.21.249.20:5060" target="_blank">178.21.249.20:5060</a>\000\000\000\000\200\303P\022\016\177\000\000\000\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000\001\000\000\000\001\000\000\000@\350\216\000\000\000\000\000\001\000\000\000\000\000\000\000\200\350\216\000\000\000\000\000\000\000\200\020",



                                      '\000' <repeats 12 times>,
                                      "\005\000\000\000\000\000\000"<br>
                                       nrprocs = <value
                                      optimized out><br>
                                       __FUNCTION__ = "main_loop"<br>
                                      #6 0x000000000046a002 in main
                                      (argc=<value optimized out>,
                                      argv=<value optimized out>)
                                      at main.c:2566<br>
                                       cfg_stream = <value
                                      optimized out><br>
                                       c = <value optimized
                                      out><br>
                                       r = <value optimized
                                      out><br>
                                       tmp = 0x7fff235c377f ""<br>
                                       tmp_len = 0<br>
                                       options = 0x5c08c8
                                      ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"<br>
                                       ret = -1<br>
                                       seed = 1722854551<br>
                                       rfd = <value optimized
                                      out><br>
                                       debug_save = <value
                                      optimized out><br>
                                       debug_flag = <value
                                      optimized out><br>
                                       dont_fork_cnt = <value
                                      optimized out><br>
                                       n_lst = <value
                                      optimized out><br>
                                       p = <value optimized
                                      out><br>
                                       __FUNCTION__ = "main"<br>
                                      (gdb)<br>
                                      <br>
                                      <br>
                                      <br>
                                      <div>
                                        <div><br clear="all">
                                          <div>
                                            <div>
                                              <div><br>
                                                -- <br>
                                                Morten Isaksen<br>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                    <br>
                                    <fieldset></fieldset>
                                    <br>
                                  </div>
                                </div>
                                <pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><span><font color="#888888">
</font></span></pre>
                                <span><font color="#888888"> </font></span></blockquote>
                              <span><font color="#888888"> <br>
                                  <pre cols="72">-- 
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
  - more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
                                </font></span></div>
                            <br>
_______________________________________________<br>
                            SIP Express Router (SER) and Kamailio
                            (OpenSER) - sr-users mailing list<br>
                            <a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a><br>
                            <a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
                            <br>
                          </blockquote>
                        </div>
                        <br>
                        <br clear="all">
                        <br>
                        -- <br>
                        Morten Isaksen<br>
                      </div>
                      <br>
                      <fieldset></fieldset>
                      <br>
                      <pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
                    </blockquote>
                    <br>
                    <pre cols="72">-- 
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28
  - more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
                  </div>
                </div>
              </div>
              <br>
              _______________________________________________<br>
              SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
              mailing list<br>
              <a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a><br>
              <a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
              <br>
            </blockquote>
          </div>
          <br>
          <br clear="all">
          <br>
          -- <br>
          Morten Isaksen<br>
        </div>
        <br>
        <fieldset></fieldset>
        <br>
        <pre>_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
<a href="mailto:sr-users@lists.sip-router.org" target="_blank">sr-users@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
      </blockquote>
      <br>
      <pre cols="72">-- 
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28
  - more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
    </blockquote>
    <br>
    <pre cols="72">-- 
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Trainings - Berlin, Nov 25-28
  - more details about Kamailio trainings at <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a> -
</pre>
  </div></div></div>

<br>_______________________________________________<br>
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list<br>
<a href="mailto:sr-users@lists.sip-router.org">sr-users@lists.sip-router.org</a><br>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>Morten Isaksen<br>
</div>