[Kamailio-Devel] [ openser-Bugs-2649470 ] Kamailio 1.5 often segfaults

SourceForge.net noreply at sourceforge.net
Mon Mar 30 15:06:00 CEST 2009


Bugs item #2649470, was opened at 2009-02-28 19:44
Message generated for change (Comment added) made by henningw
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=743020&aid=2649470&group_id=139143

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Kamailio 1.5 often segfaults

Initial Comment:
I run openser 1.3.4 for about 1000 users and I can see a lot of segfaults (1-5 times a day) in last few days...

OS: CentOS 5 x86_64
Openser: 1.3.4-notls

Backtrace is attached.

Let me know, please, if some other informations are needed.

Best regards,

kokoska.rokoska


----------------------------------------------------------------------

>Comment By: Henning Westerholt (henningw)
Date: 2009-03-30 13:06

Message:
If the abort() call can be caused from some external events, like a invalid
message then i think the error handling needs to improved here.

----------------------------------------------------------------------

Comment By: Daniel-Constantin Mierla (miconda)
Date: 2009-03-30 12:46

Message:
And the first solution is available via sanity module. From the readme:

"The following checks are available: 
...
- content length - (128) - checks if the size of the body matches with the
value from the content length header. "

See:
http://kamailio.org/docs/modules/1.5.x/sanity.html

----------------------------------------------------------------------

Comment By: Daniel-Constantin Mierla (miconda)
Date: 2009-03-30 12:41

Message:
The issue is with content-length. It is set wrong by the softphone.

What would be the right approach here, send back 400 Bad Message or
correct the C-L header value? I will send an email on devel.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-29 18:09

Message:
Thank you very much, Ovidiu, for your help!

I have slimed the config a little bit yesterday, but reinspect it once
more...

At second, I think that thees crashes could be related to some user
(conmbination of user agent and router/NAT), but I can't find any clue.
As about call spikes - they aren't there :-) And if (difference in between
cps during the time is very small), the server woks fine during them...

Thanks once more for suggestions! If you have any other recommendations I
be very happy :-)

Best regards,

kokoska.rokoska

----------------------------------------------------------------------

Comment By: Ovidiu Sas (osas)
Date: 2009-03-29 14:16

Message:
> Is there any way I could help to resolve the situation, i.e. avoid
crashes?

If the crashes are not related to malformed messages, try to incrementally
slim down the config by removing what is not strictly necessary.  Maybe you
will arrive to a stable configuration and by inspecting the config deltas
you will figure out what's wrong.

Also, you mentioned that sometimes the server is working ok for a few
hours and all of a sudden you have crashes.  Inspect the traffic and see if
there's a specific user/user agent that is making or receiving the call or
the check if the crashes are related to cps spikes.  Try to find a pattern
in the traffic that is causing crashes.

Regards,
Ovidiu Sas

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-29 10:17

Message:
Hi all, now kamailio crashes every few seconds =>
1. I have to disable core dumps - don't have so much space on HDD
2. I'm totaly in the oven: a lot of angry users, and SIP server down most
of the time :-)

Is there any way I could help to resolve the situation, i.e. avoid
crashes?

Best regards,

kokoska.rokoska

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-29 07:49

Message:
Hi, I found INVITEs for mentioned crash in the sip_trace table, if it make
sense.

BTW: It is interesting that many hours of rest are always followed by a
long series of crashes.

Best regards,

kokoska.rokoska

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-29 07:04

Message:
Hi, saturday ended without crash, but now i have 8 crashes in an hour...

In the lgo I can see "CRITICAL:core:anchor_lump: offset exceeds message
size (1400 > 1399) aborting..." and sometimes Kamailio crashes without any
message in the log.

I have core files, but don't have pcap files. One of them is included
below. Should I post other ones?

Best regards,

kokoska.rokoska
-----------------------------------

#0  0x0000003bd8830155 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000003bd8831bf0 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00000000004190f9 in anchor_lump (msg=0x7fb4f8, offset=1395, len=0,
    type=HDR_OTHER_T) at data_lump.c:345
        tmp = <value optimized out>
        prev = <value optimized out>
        t = <value optimized out>
        list = <value optimized out>
        __FUNCTION__ = "anchor_lump"
#3  0x00002b344a8a13ee in force_rtp_proxy (msg=0x7fb4f8,
    str1=<value optimized out>, str2=<value optimized out>,
    offer=<value optimized out>) at nathelper.c:2913
        body = {
  s = 0x719d7b "v=0\r\no=- 635 635 IN IP4 9.8.7.6\r\ns=-\r\nc=IN IP4
9.8.7.6\r\nt=0 0\r\nm=audio 6972 RTP/AVP 8 0 2 4 18 96 97 98 100
101\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:2
G726-32/8000\r\na=rt"..., len = 440}
        body1 = {
  s = 0x719da8 "c=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6972 RTP/AVP 8 0 2 4
18 96 97 98 100 101\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:0
PCMU/8000\r\na=rtpmap:2 G726-32/8000\r\na=rtpmap:4 G723/8000\r\na=rtpmap:18
G729a/8000\r\na=r"...,
  len = 395}
        oldport = {
  s = 0x719dd0 "6972 RTP/AVP 8 0 2 4 18 96 97 98 100 101\r\na=rtpmap:8
PCMA/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:2 G726-32/8000\r\na=rtpmap:4
G723/8000\r\na=rtpmap:18 G729a/8000\r\na=rtpmap:96
G726-40/8000\r\na=rtpmap:97 G726-"...,
  len = 4}
        oldip = {
  s = 0x719db1 "9.8.7.6\r\nt=0 0\r\nm=audio 6972 RTP/AVP 8 0 2 4 18 96 97
98 100 101\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:2
G726-32/8000\r\na=rtpmap:4 G723/8000\r\na=rtpmap:18
G729a/8000\r\na=rtpmap:96 "...,
  len = 14}
        newport = {s = 0x73ac00 "46418", len = 5}
        newip = {s = 0x2b344aabd98e "1.2.3.4", len = 14}
        callid = {
  s = 0x719b39 "1491b79e-daeb1f4 at 9.8.7.6\r\nCSeq: 102
INVITE\r\nMax-Forwards: 68\r\nProxy-Authorization: Digest
username=\"987654321\",realm=\"company.cz\",nonce=\"49cf0b190000eb641dfdc137f70582dd18d2ea650dcd2417\",uri=\"sip:6"...,
  len = 31}
        from_tag = {
  s = 0x719afc "d54b0b964cc8ae5co0\r\nTo:
<sip:1234567896 at sip.company.tld>\r\nCall-ID:
1491b79e-daeb1f4 at 9.8.7.6\r\nCSeq: 102 INVITE\r\nMax-Forwards:
68\r\nProxy-Authorization: Digest
username=\"987654321\",realm=\"company.cz\",nonce="...,
  len = 18}
        to_tag = {s = 0x0, len = 0}
        create = 1
        port = 46418
        len = <value optimized out>
        asymmetric = 0
        flookup = 0
        argc = 395
        proxied = 0
        real = 1
        orgip = 0
        commip = 1
        oidx = 1
        pf = 2
        pf1 = <value optimized out>
        force = 0
        swap = 1326629816
        rep_oidx = 0
        opts =
"U▒Oa▒\177\000\000@▒\210▒;\000\000\000Šr\000\000\000\000\000▒▒Oa▒\177\000"
        rep_opts = "▒\177", '\0' <repeats 12 times>
        cp = <value optimized out>
        cp1 = <value optimized out>
        cpend = <value optimized out>
        next = <value optimized out>
        ap = <value optimized out>
        argv = {0x2b344aabd988 "46418", 0x2b344aabd98e "1.2.3.4", 0x0,
  0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
        anchor = <value optimized out>
        node = <value optimized out>
        v = {{iov_base = 0x2b344aabdaa0, iov_len = 8}, {
    iov_base = 0x7fff614fa750, iov_len = 1}, {iov_base = 0x2b344a8abab4,
    iov_len = 1}, {iov_base = 0x719b39, iov_len = 31}, {
    iov_base = 0x2b344a8abab4, iov_len = 1}, {iov_base = 0x719db1,
    iov_len = 14}, {iov_base = 0x2b344a8abab4, iov_len = 1}, {
    iov_base = 0x719dd0, iov_len = 4}, {iov_base = 0x2b344a8abab4,
    iov_len = 1}, {iov_base = 0x719afc, iov_len = 18}, {
    iov_base = 0x2b344a8a936f, iov_len = 1}, {iov_base = 0x7fff614fa770,
    iov_len = 1}, {iov_base = 0x2b344a8abab4, iov_len = 1}, {iov_base =
0x0,
    iov_len = 0}, {iov_base = 0x2b344a8a936f, iov_len = 1}, {
    iov_base = 0x7fff614fa770, iov_len = 1}}
        v1p = <value optimized out>
        v2p = 0x719f33 ""
        c1p = 0x719da8 "c=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6972 RTP/AVP
8 0 2 4 18 96 97 98 100 101\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:0
PCMU/8000\r\na=rtpmap:2 G726-32/8000\r\na=rtpmap:4 G723/8000\r\na=rtpmap:18
G729a/8000\r\na=r"...
        c2p = <value optimized out>
        m1p = <value optimized out>
        m2p = 0x719dc8 "m=audio 6972 RTP/AVP 8 0 2 4 18 96 97 98 100
101\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:2
G726-32/8000\r\na=rtpmap:4 G723/8000\r\na=rtpmap:18
G729a/8000\r\na=rtpmap:96 G726-40/8000\r\na=rtpmap:"...
        bodylimit = 0x719f33 ""
        o1p = <value optimized out>
        medianum_buf =
"1\000\000\000▒\023\000\0006\000\000\000\000\000\000\000\000\000\000"
        medianum = 1
        tmpstr1 = {
  s = 0x719dc8 "m=audio 6972 RTP/AVP 8 0 2 4 18 96 97 98 100
101\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:2
G726-32/8000\r\na=rtpmap:4 G723/8000\r\na=rtpmap:18
G729a/8000\r\na=rtpmap:96 G726-40/8000\r\na=rtpmap:"...,
  len = 363}
        c1p_altered = 1
        swap_warned = 0
        __FUNCTION__ = "force_rtp_proxy"
#4  0x00002b344a8a2025 in force_rtp_proxy1_f (msg=0x7fb4f8,
    str1=0x77eb40 "rc", str2=<value optimized out>) at nathelper.c:2430
        cp = <value optimized out>
        newip =
"1.2.3.4\000\000▒Oa▒\177\000\000(▒w\000\000\000\000\000\-
r\000\000\000\000\000\000"
#5  0x000000000040d0ba in do_action (a=0x77ebd0, msg=0x7fb4f8) at
action.c:874
        ret = <value optimized out>
        v = <value optimized out>
        to = <value optimized out>
        p = <value optimized out>
        tmp = <value optimized out>
        new_uri = <value optimized out>
        end = <value optimized out>
        crt = <value optimized out>
        len = dwarf2_read_address: Corrupted DWARF expression.


----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 19:32

Message:
pcap file has been sent. Thanks.

kokoska.rokoska

----------------------------------------------------------------------

Comment By: Daniel-Constantin Mierla (miconda)
Date: 2009-03-27 18:23

Message:
filter the sip messages belonging to this call (by callid) and send the
pcap file to me.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 18:13

Message:
Thank you, Daniel, for your analysis!

I'm sorry, but I don't know how to get exact size of body :-)
Wireshark reports 424 bytes (when I select message body) - if better
method I should use, please tell me...

Thank once more!

Best regards,

kokoska.rokoska

----------------------------------------------------------------------

Comment By: Daniel-Constantin Mierla (miconda)
Date: 2009-03-27 17:58

Message:
As you have the PCAP, can you get the right size of the body? I saved it in
a file and got 394. I added 16 -- a \r for each line, so the Content-Length
value is wrong in headers, but I want to be sure I didn't computed badly.

The crash is because a call of abort() function. That can be changed in
return NULL -- not sure why this solution (i.e., abort()) was chosen in
this case.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 17:19

Message:
Thanks for you reply, Daniel!

Here are both INVITES - first one without credentials, and second one with
them (after we respond 407).
BTW: If you need packets without "rewrited" IP addresses, usernames and
phone numbers I could send them directly to you...

Thanks once more!

Best regards,

kokoska.rokoska
----------------------------------
INVITE sip:123456789 at sip.company.tld SIP/2.0
Record-Route: <sip:1.2.3.3;lr=on;ftag=e467b916>
Via: SIP/2.0/UDP 1.2.3.3;branch=z9hG4bK3c62.263afe07.0
Via: SIP/2.0/UDP
9.8.7.6:14349;received=9.8.7.6;branch=z9hG4bK-d8754z-a0220b087834bf13-1---d8754z-;rport=14349
Max-Forwards: 69
Contact: <sip:987654321 at 9.8.7.6:14349>
To: "First Last"<sip:123456789 at sip.company.tld>
From: "987654321"<sip:987654321 at sip.company.tld>;tag=e467b916
Call-ID: MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.
CSeq: 1 INVITE
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE,
SUBSCRIBE, INFO
Content-Type: application/sdp
User-Agent: X-Lite release 1100l stamp 47546
Content-Length: 425
X-Received: ip=9.8.7.6;port=14349

v=0
o=- 8 2 IN IP4 9.8.7.6
s=CounterPath X-Lite 3.0
c=IN IP4 9.8.7.6
t=0 0
m=audio 6978 RTP/AVP 107 119 100 106 0 105 98 8 3 101
a=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688
a=fmtp:101 0-15
a=rtpmap:107 BV32/16000
a=rtpmap:119 BV32-FEC/16000
a=rtpmap:100 SPEEX/16000
a=rtpmap:106 SPEEX-FEC/16000
a=rtpmap:105 SPEEX-FEC/8000
a=rtpmap:98 iLBC/8000
a=rtpmap:101 telephone-event/8000
a=sendrecv





INVITE sip:123456789 at sip.company.tld SIP/2.0
Record-Route: <sip:1.2.3.3;lr=on;ftag=e467b916>
Via: SIP/2.0/UDP 1.2.3.3;branch=z9hG4bK0c62.26dcdb51.0
Via: SIP/2.0/UDP
9.8.7.6:14349;received=9.8.7.6;branch=z9hG4bK-d8754z-e16c876e6457e20b-1---d8754z-;rport=14349
Max-Forwards: 69
Contact: <sip:987654321 at 9.8.7.6:14349>
To: "First Last"<sip:123456789 at sip.company.tld>
From: "987654321"<sip:987654321 at sip.comapny.tld>;tag=e467b916
Call-ID: MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.
CSeq: 2 INVITE
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE,
SUBSCRIBE, INFO
Content-Type: application/sdp
Proxy-Authorization: Digest
username="987654321",realm="company.tld",nonce="49cceee300001e6d384c66511508c0ee043e8e6d5944ae8c",uri="sip:123456789 at sip.company.tld",response="a371e652fa9555001c15bf89a69e0aaf",algorithm=MD5
User-Agent: X-Lite release 1100l stamp 47546
Content-Length: 425
X-Received: ip=9.8.7.6;port=14349

v=0
o=- 8 2 IN IP4 9.8.7.6
s=CounterPath X-Lite 3.0
c=IN IP4 9.8.7.6
t=0 0
m=audio 6980 RTP/AVP 107 119 100 106 0 105 98 8 3 101
a=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688
a=fmtp:101 0-15
a=rtpmap:107 BV32/16000
a=rtpmap:119 BV32-FEC/16000
a=rtpmap:100 SPEEX/16000
a=rtpmap:106 SPEEX-FEC/16000
a=rtpmap:105 SPEEX-FEC/8000
a=rtpmap:98 iLBC/8000
a=rtpmap:101 telephone-event/8000
a=sendrecv


----------------------------------------------------------------------

Comment By: Daniel-Constantin Mierla (miconda)
Date: 2009-03-27 16:44

Message:
Seems there is an issue computing the length of the body. Can you get from
the pcap file the invite that matches. Call-Id:
MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E

Will help to see what is broken. Thanks

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 16:06

Message:
Sorry, I can't attache file :-) So, here is bt full in-line:
--------------------------
#0  0x0000003bd8830155 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x0000003bd8831bf0 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00000000004190f9 in anchor_lump (msg=0x7f8e38, offset=1385, len=0,
    type=HDR_OTHER_T) at data_lump.c:345
        tmp = <value optimized out>
        prev = <value optimized out>
        t = <value optimized out>
        list = <value optimized out>
        __FUNCTION__ = "anchor_lump"
#3  0x00002b8e4c7cc3ee in force_rtp_proxy (msg=0x7f8e38,
    str1=<value optimized out>, str2=<value optimized out>,
    offer=<value optimized out>) at nathelper.c:2913
        body = {
  s = 0x719d80 "v=0\r\no=- 8 2 IN IP4 9.8.7.6\r\ns=CounterPath X-Lite
3.0\r\nc=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119 100 106 0
105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na="...,
  len = 425}
        body1 = {
  s = 0x719dbe "c=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119
100 106 0 105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na=fmtp:101 0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/"...,
  len = 363}
        oldport = {
  s = 0x719de6 "6980 RTP/AVP 107 119 100 106 0 105 98 8 3 101\r\na=alt:1 1
: MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101 0-15\r\na=rtpmap:107
BV32/16000\r\na=rtpmap:119 BV32-FEC/16000\r\na=rtpmap:100
SPEEX/16000\r\na=rtpma"...,
  len = 4}
        oldip = {
  s = 0x719dc7 "9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119 100 106 0
105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na=fmtp:101 0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na="...,
  len = 14}
        newport = {s = 0x73ac00 "38924", len = 5}
        newip = {s = 0x2b8e4c9e898f "1.2.3.4", len = 14}
        callid = {
  s = 0x719b8e "MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.\r\nCSeq: 2
INVITE\r\nAllow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE,
SUBSCRIBE, INFO\r\nContent-Type: application/sdp\r\nProxy-Authorization:
Dig"..., len = 44}
        from_tag = {
  s = 0x719b7b "e467b916\r\nCall-ID:
MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.\r\nCSeq: 2 INVITE\r\nAllow:
INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE, SUBSCRIBE,
INFO\r\nContent-Type: application/sdp\r\nProxy"...,
  len = 8}
        to_tag = {s = 0x0, len = 0}
        create = 1
        port = 38924
        len = <value optimized out>
        asymmetric = 0
        flookup = 0
        argc = 363
        proxied = 0
        real = 1
        orgip = 0
        commip = 1
        oidx = 1
        pf = 2
        pf1 = <value optimized out>
        force = 0
        swap = 1359311800
        rep_oidx = 0
        opts =
"Uů\\_˙\177\000\000\030ý\\_˙\177\000\000\001\000\000\000\000\000\000\000\200ů\\_˙\177\000"
        rep_opts = "8\216\177", '\0' <repeats 12 times>
        cp = <value optimized out>
        cp1 = <value optimized out>
        cpend = <value optimized out>
        next = <value optimized out>
        ap = <value optimized out>
        argv = {0x2b8e4c9e8989 "38924", 0x2b8e4c9e898f "1.2.3.4", 0x0,
  0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
        anchor = <value optimized out>
        node = <value optimized out>
        v = {{iov_base = 0x2b8e4c9e8aa0, iov_len = 9}, {
    iov_base = 0x7fff5f5cf830, iov_len = 1}, {iov_base = 0x2b8e4c7d6ab4,
    iov_len = 1}, {iov_base = 0x719b8e, iov_len = 44}, {
    iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {iov_base = 0x719dc7,
    iov_len = 14}, {iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {
    iov_base = 0x719de6, iov_len = 4}, {iov_base = 0x2b8e4c7d6ab4,
    iov_len = 1}, {iov_base = 0x719b7b, iov_len = 8}, {
    iov_base = 0x2b8e4c7d436f, iov_len = 1}, {iov_base = 0x7fff5f5cf850,
    iov_len = 1}, {iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {iov_base =
0x0,
    iov_len = 0}, {iov_base = 0x2b8e4c7d436f, iov_len = 1}, {
    iov_base = 0x7fff5f5cf850, iov_len = 1}}
        v1p = <value optimized out>
        v2p = 0x719f29 "8000/1\r\na=rtpmap:0 PCMU/8000/1\r\na=rtpmap:8
PCMA/8000/1\r\na=rtpmap:97 iLBC/8000/1\r\na=rtpmap:18
G729/8000/1\r\na=rtpmap:98 telephone-event/8000/1\r\na=rtpmap:13
CN/8000/1\r\n"
        c1p = 0x719dbe "c=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP
107 119 100 106 0 105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM
192.168.1.7 27688\r\na=fmtp:101 0-15\r\na=rtpmap:107
BV32/16000\r\na=rtpmap:119 BV32-FEC/"...
        
        c2p = <value optimized out>
        m1p = <value optimized out>
        m2p = 0x719dde "m=audio 6980 RTP/AVP 107 119 100 106 0 105 98 8 3
101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101
0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na=rtpmap:100 SPEEX/16000\r"...
        bodylimit = 0x719f29 "8000/1\r\na=rtpmap:0
PCMU/8000/1\r\na=rtpmap:8 PCMA/8000/1\r\na=rtpmap:97
iLBC/8000/1\r\na=rtpmap:18 G729/8000/1\r\na=rtpmap:98
telephone-event/8000/1\r\na=rtpmap:13 CN/8000/1\r\n"
        o1p = <value optimized out>
        medianum_buf =
"1\000O\000\000\000\000\0007\000\000\000\000\000\000\000\000\000\000"
        medianum = 1
        tmpstr1 = {
  s = 0x719dde "m=audio 6980 RTP/AVP 107 119 100 106 0 105 98 8 3
101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101
0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na=rtpmap:100 SPEEX/16000\r"...,
  len = 331}
        c1p_altered = 1
        swap_warned = 0
        __FUNCTION__ = "force_rtp_proxy"
#4  0x00002b8e4c7cd025 in force_rtp_proxy1_f (msg=0x7f8e38,
    str1=0x77df00 "rc", str2=<value optimized out>) at nathelper.c:2430
        cp = <value optimized out>
        newip =
"1.2.3.4\000\000 ů\\_˙\177\000\000čŘw\000\000\000\000\000\r\000\000\000\000\000\000"
#5  0x000000000040d0ba in do_action (a=0x77df90, msg=0x7f8e38) at
action.c:874
        ret = <value optimized out>
        v = <value optimized out>
        to = <value optimized out>
        p = <value optimized out>
        tmp = <value optimized out>
        new_uri = <value optimized out>
        end = <value optimized out>
        crt = <value optimized out>
        len = dwarf2_read_address: Corrupted DWARF expression.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 16:03

Message:
I have core dump :-) But now there is no CRITICAL message in the log.

I have pcap.dump too, so let me know, please, what shoul I look for. I
didn't see any abnormality...

Backtrace (bt full) is attached.

Best regards,

kokoska.rokoska

----------------------------------------------------------------------

Comment By: Klaus Darilion (klaus_darilion)
Date: 2009-03-27 14:09

Message:
changed subject to "1.5"

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 13:50

Message:
I have upgraded to Kamailio 1.5 and after few days without troubles
Kamailio starts to segfaults very often (about every 10 minutes).
I'm using nearly same config as for previous Openser 1.3.4.

In the log I see before every crash:
CRITICAL:core:anchor_lump: offset exceeds message size (1084 > 1080)
aborting...
only reported sizes differs.

I don't have core files due to big RAM and lack of HDD space, but I try to
find some and than submit backtrace...

BTW: Should I open new Bug if it is for new version of server (1.5)?

Best regards,

kokoska.rokoska

----------------------------------------------------------------------

Comment By: Henning Westerholt (henningw)
Date: 2009-03-16 12:46

Message:
Hi Kokoska,

any updates on this problem? Were you able to isolate the "strange"
behaviour of your clients?

Henning

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-02 21:43

Message:
Hi Henning,
Hi Henning,

I'm sorry, but I'm on vacation now, so I can't say much about it :-)
I have tried to look at logs, but I havent seen anything interesting - for
me it looks like if this happed whed OpenSER tris tu dump stored messages
after successful registration.
I have sip_trace "armed" and thus I try to look what goes through the
server.

Please give me some time to look deeper into the problem (saturday,
sunday).

BTW: Config file is very complex, but I didn't made any changes before
OpenSER starts to segfaults => I thik it is something like "strange"
behaviour of one of the new clients.

Thank you very much for your interest!

Best regards,

kokoska.rokoska

----------------------------------------------------------------------

Comment By: Henning Westerholt (henningw)
Date: 2009-03-02 12:33

Message:
Hi Kokoska,

have you done any notable changes on the configuration or the setup before
it started to crash?

Have you tried to further examine the core dump, e.g. look at the SIP
message content, or the contact_body structure?

backtrace from the file for convinience:

Core was generated by `/usr/local/sbin/openser -m 1024'.
Program terminated with signal 11, Segmentation fault.
#0  free_contacts (_c=0x7cc030) at parser/contact/contact.c:287
287                     *_c = (*_c)->next;
(gdb) bt
#0  free_contacts (_c=0x7cc030) at parser/contact/contact.c:287
#1  0x00000000004b54da in free_contact (_c=0x2ba8f6fce2e0)
    at parser/contact/parse_contact.c:101
#2  0x000000000048fbc3 in clean_hdr_field (hf=0x2ba8f6fce2b0) at
parser/hf.c:78
#3  0x00002ba8f2436a29 in run_trans_callbacks (type=<value optimized
out>,
    trans=<value optimized out>, req=0x2ba8f6fcda08, rpl=0x7cbe68,
    code=<value optimized out>) at sip_msg.h:49
#4  0x00002ba8f2440112 in t_reply_matching (p_msg=0x7cbe68,
    p_branch=<value optimized out>) at t_lookup.c:840
#5  0x00002ba8f244056f in t_check (p_msg=0x7cbe68,
param_branch=0x7fffb8cd52ec)
    at t_lookup.c:911
#6  0x00002ba8f244fe16 in reply_received (p_msg=0x7cc030) at
t_reply.c:1285
#7  0x000000000041d1ac in forward_reply (msg=0x7cbe68) at forward.c:499
#8  0x000000000044c87c in receive_msg (
    buf=0x726920 "SIP/2.0 100 Trying\r\nVia: SIP/2.0/UDP
1.2.3.4;branch=z9hG4bK3553.caa63f57.1;received=1.2.3.4\r\nVia: SIP/2.0/UDP
4.3.2.1;rport=5060;branch=z9hG4bK-49aa467849a97dc2123\r\nFrom:
\"123456798"...,
    len=557, rcv_info=0x7fffb8cd53d0) at receive.c:194
#9  0x00000000004892de in udp_rcv_loop () at udp_server.c:438
#10 0x000000000042561f in main (argc=3, argv=0x7fffb8cd55e8) at
main.c:834


----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=743020&aid=2649470&group_id=139143



More information about the Devel mailing list