[Kamailio-Devel] [ openser-Bugs-2649470 ] Kamailio 1.5 often segfaults
SourceForge.net
noreply at sourceforge.net
Fri Mar 27 19:23:49 CET 2009
Bugs item #2649470, was opened at 2009-02-28 21:44
Message generated for change (Comment added) made by miconda
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=743020&aid=2649470&group_id=139143
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Kamailio 1.5 often segfaults
Initial Comment:
I run openser 1.3.4 for about 1000 users and I can see a lot of segfaults (1-5 times a day) in last few days...
OS: CentOS 5 x86_64
Openser: 1.3.4-notls
Backtrace is attached.
Let me know, please, if some other informations are needed.
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Daniel-Constantin Mierla (miconda)
Date: 2009-03-27 20:23
Message:
filter the sip messages belonging to this call (by callid) and send the
pcap file to me.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 20:13
Message:
Thank you, Daniel, for your analysis!
I'm sorry, but I don't know how to get exact size of body :-)
Wireshark reports 424 bytes (when I select message body) - if better
method I should use, please tell me...
Thank once more!
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Daniel-Constantin Mierla (miconda)
Date: 2009-03-27 19:58
Message:
As you have the PCAP, can you get the right size of the body? I saved it in
a file and got 394. I added 16 -- a \r for each line, so the Content-Length
value is wrong in headers, but I want to be sure I didn't computed badly.
The crash is because a call of abort() function. That can be changed in
return NULL -- not sure why this solution (i.e., abort()) was chosen in
this case.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 19:19
Message:
Thanks for you reply, Daniel!
Here are both INVITES - first one without credentials, and second one with
them (after we respond 407).
BTW: If you need packets without "rewrited" IP addresses, usernames and
phone numbers I could send them directly to you...
Thanks once more!
Best regards,
kokoska.rokoska
----------------------------------
INVITE sip:123456789 at sip.company.tld SIP/2.0
Record-Route: <sip:1.2.3.3;lr=on;ftag=e467b916>
Via: SIP/2.0/UDP 1.2.3.3;branch=z9hG4bK3c62.263afe07.0
Via: SIP/2.0/UDP
9.8.7.6:14349;received=9.8.7.6;branch=z9hG4bK-d8754z-a0220b087834bf13-1---d8754z-;rport=14349
Max-Forwards: 69
Contact: <sip:987654321 at 9.8.7.6:14349>
To: "First Last"<sip:123456789 at sip.company.tld>
From: "987654321"<sip:987654321 at sip.company.tld>;tag=e467b916
Call-ID: MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.
CSeq: 1 INVITE
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE,
SUBSCRIBE, INFO
Content-Type: application/sdp
User-Agent: X-Lite release 1100l stamp 47546
Content-Length: 425
X-Received: ip=9.8.7.6;port=14349
v=0
o=- 8 2 IN IP4 9.8.7.6
s=CounterPath X-Lite 3.0
c=IN IP4 9.8.7.6
t=0 0
m=audio 6978 RTP/AVP 107 119 100 106 0 105 98 8 3 101
a=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688
a=fmtp:101 0-15
a=rtpmap:107 BV32/16000
a=rtpmap:119 BV32-FEC/16000
a=rtpmap:100 SPEEX/16000
a=rtpmap:106 SPEEX-FEC/16000
a=rtpmap:105 SPEEX-FEC/8000
a=rtpmap:98 iLBC/8000
a=rtpmap:101 telephone-event/8000
a=sendrecv
INVITE sip:123456789 at sip.company.tld SIP/2.0
Record-Route: <sip:1.2.3.3;lr=on;ftag=e467b916>
Via: SIP/2.0/UDP 1.2.3.3;branch=z9hG4bK0c62.26dcdb51.0
Via: SIP/2.0/UDP
9.8.7.6:14349;received=9.8.7.6;branch=z9hG4bK-d8754z-e16c876e6457e20b-1---d8754z-;rport=14349
Max-Forwards: 69
Contact: <sip:987654321 at 9.8.7.6:14349>
To: "First Last"<sip:123456789 at sip.company.tld>
From: "987654321"<sip:987654321 at sip.comapny.tld>;tag=e467b916
Call-ID: MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.
CSeq: 2 INVITE
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE,
SUBSCRIBE, INFO
Content-Type: application/sdp
Proxy-Authorization: Digest
username="987654321",realm="company.tld",nonce="49cceee300001e6d384c66511508c0ee043e8e6d5944ae8c",uri="sip:123456789 at sip.company.tld",response="a371e652fa9555001c15bf89a69e0aaf",algorithm=MD5
User-Agent: X-Lite release 1100l stamp 47546
Content-Length: 425
X-Received: ip=9.8.7.6;port=14349
v=0
o=- 8 2 IN IP4 9.8.7.6
s=CounterPath X-Lite 3.0
c=IN IP4 9.8.7.6
t=0 0
m=audio 6980 RTP/AVP 107 119 100 106 0 105 98 8 3 101
a=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688
a=fmtp:101 0-15
a=rtpmap:107 BV32/16000
a=rtpmap:119 BV32-FEC/16000
a=rtpmap:100 SPEEX/16000
a=rtpmap:106 SPEEX-FEC/16000
a=rtpmap:105 SPEEX-FEC/8000
a=rtpmap:98 iLBC/8000
a=rtpmap:101 telephone-event/8000
a=sendrecv
----------------------------------------------------------------------
Comment By: Daniel-Constantin Mierla (miconda)
Date: 2009-03-27 18:44
Message:
Seems there is an issue computing the length of the body. Can you get from
the pcap file the invite that matches. Call-Id:
MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E
Will help to see what is broken. Thanks
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 18:06
Message:
Sorry, I can't attache file :-) So, here is bt full in-line:
--------------------------
#0 0x0000003bd8830155 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003bd8831bf0 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00000000004190f9 in anchor_lump (msg=0x7f8e38, offset=1385, len=0,
type=HDR_OTHER_T) at data_lump.c:345
tmp = <value optimized out>
prev = <value optimized out>
t = <value optimized out>
list = <value optimized out>
__FUNCTION__ = "anchor_lump"
#3 0x00002b8e4c7cc3ee in force_rtp_proxy (msg=0x7f8e38,
str1=<value optimized out>, str2=<value optimized out>,
offer=<value optimized out>) at nathelper.c:2913
body = {
s = 0x719d80 "v=0\r\no=- 8 2 IN IP4 9.8.7.6\r\ns=CounterPath X-Lite
3.0\r\nc=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119 100 106 0
105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na="...,
len = 425}
body1 = {
s = 0x719dbe "c=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119
100 106 0 105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na=fmtp:101 0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/"...,
len = 363}
oldport = {
s = 0x719de6 "6980 RTP/AVP 107 119 100 106 0 105 98 8 3 101\r\na=alt:1 1
: MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101 0-15\r\na=rtpmap:107
BV32/16000\r\na=rtpmap:119 BV32-FEC/16000\r\na=rtpmap:100
SPEEX/16000\r\na=rtpma"...,
len = 4}
oldip = {
s = 0x719dc7 "9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119 100 106 0
105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na=fmtp:101 0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na="...,
len = 14}
newport = {s = 0x73ac00 "38924", len = 5}
newip = {s = 0x2b8e4c9e898f "1.2.3.4", len = 14}
callid = {
s = 0x719b8e "MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.\r\nCSeq: 2
INVITE\r\nAllow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE,
SUBSCRIBE, INFO\r\nContent-Type: application/sdp\r\nProxy-Authorization:
Dig"..., len = 44}
from_tag = {
s = 0x719b7b "e467b916\r\nCall-ID:
MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.\r\nCSeq: 2 INVITE\r\nAllow:
INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE, SUBSCRIBE,
INFO\r\nContent-Type: application/sdp\r\nProxy"...,
len = 8}
to_tag = {s = 0x0, len = 0}
create = 1
port = 38924
len = <value optimized out>
asymmetric = 0
flookup = 0
argc = 363
proxied = 0
real = 1
orgip = 0
commip = 1
oidx = 1
pf = 2
pf1 = <value optimized out>
force = 0
swap = 1359311800
rep_oidx = 0
opts =
"Uů\\_˙\177\000\000\030ý\\_˙\177\000\000\001\000\000\000\000\000\000\000\200ů\\_˙\177\000"
rep_opts = "8\216\177", '\0' <repeats 12 times>
cp = <value optimized out>
cp1 = <value optimized out>
cpend = <value optimized out>
next = <value optimized out>
ap = <value optimized out>
argv = {0x2b8e4c9e8989 "38924", 0x2b8e4c9e898f "1.2.3.4", 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
anchor = <value optimized out>
node = <value optimized out>
v = {{iov_base = 0x2b8e4c9e8aa0, iov_len = 9}, {
iov_base = 0x7fff5f5cf830, iov_len = 1}, {iov_base = 0x2b8e4c7d6ab4,
iov_len = 1}, {iov_base = 0x719b8e, iov_len = 44}, {
iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {iov_base = 0x719dc7,
iov_len = 14}, {iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {
iov_base = 0x719de6, iov_len = 4}, {iov_base = 0x2b8e4c7d6ab4,
iov_len = 1}, {iov_base = 0x719b7b, iov_len = 8}, {
iov_base = 0x2b8e4c7d436f, iov_len = 1}, {iov_base = 0x7fff5f5cf850,
iov_len = 1}, {iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {iov_base =
0x0,
iov_len = 0}, {iov_base = 0x2b8e4c7d436f, iov_len = 1}, {
iov_base = 0x7fff5f5cf850, iov_len = 1}}
v1p = <value optimized out>
v2p = 0x719f29 "8000/1\r\na=rtpmap:0 PCMU/8000/1\r\na=rtpmap:8
PCMA/8000/1\r\na=rtpmap:97 iLBC/8000/1\r\na=rtpmap:18
G729/8000/1\r\na=rtpmap:98 telephone-event/8000/1\r\na=rtpmap:13
CN/8000/1\r\n"
c1p = 0x719dbe "c=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP
107 119 100 106 0 105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM
192.168.1.7 27688\r\na=fmtp:101 0-15\r\na=rtpmap:107
BV32/16000\r\na=rtpmap:119 BV32-FEC/"...
c2p = <value optimized out>
m1p = <value optimized out>
m2p = 0x719dde "m=audio 6980 RTP/AVP 107 119 100 106 0 105 98 8 3
101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101
0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na=rtpmap:100 SPEEX/16000\r"...
bodylimit = 0x719f29 "8000/1\r\na=rtpmap:0
PCMU/8000/1\r\na=rtpmap:8 PCMA/8000/1\r\na=rtpmap:97
iLBC/8000/1\r\na=rtpmap:18 G729/8000/1\r\na=rtpmap:98
telephone-event/8000/1\r\na=rtpmap:13 CN/8000/1\r\n"
o1p = <value optimized out>
medianum_buf =
"1\000O\000\000\000\000\0007\000\000\000\000\000\000\000\000\000\000"
medianum = 1
tmpstr1 = {
s = 0x719dde "m=audio 6980 RTP/AVP 107 119 100 106 0 105 98 8 3
101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101
0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na=rtpmap:100 SPEEX/16000\r"...,
len = 331}
c1p_altered = 1
swap_warned = 0
__FUNCTION__ = "force_rtp_proxy"
#4 0x00002b8e4c7cd025 in force_rtp_proxy1_f (msg=0x7f8e38,
str1=0x77df00 "rc", str2=<value optimized out>) at nathelper.c:2430
cp = <value optimized out>
newip =
"1.2.3.4\000\000 ů\\_˙\177\000\000čŘw\000\000\000\000\000\r\000\000\000\000\000\000"
#5 0x000000000040d0ba in do_action (a=0x77df90, msg=0x7f8e38) at
action.c:874
ret = <value optimized out>
v = <value optimized out>
to = <value optimized out>
p = <value optimized out>
tmp = <value optimized out>
new_uri = <value optimized out>
end = <value optimized out>
crt = <value optimized out>
len = dwarf2_read_address: Corrupted DWARF expression.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 18:03
Message:
I have core dump :-) But now there is no CRITICAL message in the log.
I have pcap.dump too, so let me know, please, what shoul I look for. I
didn't see any abnormality...
Backtrace (bt full) is attached.
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Klaus Darilion (klaus_darilion)
Date: 2009-03-27 16:09
Message:
changed subject to "1.5"
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 15:50
Message:
I have upgraded to Kamailio 1.5 and after few days without troubles
Kamailio starts to segfaults very often (about every 10 minutes).
I'm using nearly same config as for previous Openser 1.3.4.
In the log I see before every crash:
CRITICAL:core:anchor_lump: offset exceeds message size (1084 > 1080)
aborting...
only reported sizes differs.
I don't have core files due to big RAM and lack of HDD space, but I try to
find some and than submit backtrace...
BTW: Should I open new Bug if it is for new version of server (1.5)?
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Henning Westerholt (henningw)
Date: 2009-03-16 14:46
Message:
Hi Kokoska,
any updates on this problem? Were you able to isolate the "strange"
behaviour of your clients?
Henning
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-02 23:43
Message:
Hi Henning,
Hi Henning,
I'm sorry, but I'm on vacation now, so I can't say much about it :-)
I have tried to look at logs, but I havent seen anything interesting - for
me it looks like if this happed whed OpenSER tris tu dump stored messages
after successful registration.
I have sip_trace "armed" and thus I try to look what goes through the
server.
Please give me some time to look deeper into the problem (saturday,
sunday).
BTW: Config file is very complex, but I didn't made any changes before
OpenSER starts to segfaults => I thik it is something like "strange"
behaviour of one of the new clients.
Thank you very much for your interest!
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Henning Westerholt (henningw)
Date: 2009-03-02 14:33
Message:
Hi Kokoska,
have you done any notable changes on the configuration or the setup before
it started to crash?
Have you tried to further examine the core dump, e.g. look at the SIP
message content, or the contact_body structure?
backtrace from the file for convinience:
Core was generated by `/usr/local/sbin/openser -m 1024'.
Program terminated with signal 11, Segmentation fault.
#0 free_contacts (_c=0x7cc030) at parser/contact/contact.c:287
287 *_c = (*_c)->next;
(gdb) bt
#0 free_contacts (_c=0x7cc030) at parser/contact/contact.c:287
#1 0x00000000004b54da in free_contact (_c=0x2ba8f6fce2e0)
at parser/contact/parse_contact.c:101
#2 0x000000000048fbc3 in clean_hdr_field (hf=0x2ba8f6fce2b0) at
parser/hf.c:78
#3 0x00002ba8f2436a29 in run_trans_callbacks (type=<value optimized
out>,
trans=<value optimized out>, req=0x2ba8f6fcda08, rpl=0x7cbe68,
code=<value optimized out>) at sip_msg.h:49
#4 0x00002ba8f2440112 in t_reply_matching (p_msg=0x7cbe68,
p_branch=<value optimized out>) at t_lookup.c:840
#5 0x00002ba8f244056f in t_check (p_msg=0x7cbe68,
param_branch=0x7fffb8cd52ec)
at t_lookup.c:911
#6 0x00002ba8f244fe16 in reply_received (p_msg=0x7cc030) at
t_reply.c:1285
#7 0x000000000041d1ac in forward_reply (msg=0x7cbe68) at forward.c:499
#8 0x000000000044c87c in receive_msg (
buf=0x726920 "SIP/2.0 100 Trying\r\nVia: SIP/2.0/UDP
1.2.3.4;branch=z9hG4bK3553.caa63f57.1;received=1.2.3.4\r\nVia: SIP/2.0/UDP
4.3.2.1;rport=5060;branch=z9hG4bK-49aa467849a97dc2123\r\nFrom:
\"123456798"...,
len=557, rcv_info=0x7fffb8cd53d0) at receive.c:194
#9 0x00000000004892de in udp_rcv_loop () at udp_server.c:438
#10 0x000000000042561f in main (argc=3, argv=0x7fffb8cd55e8) at
main.c:834
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=743020&aid=2649470&group_id=139143
More information about the Devel
mailing list