[Kamailio-Devel] [ openser-Bugs-2649470 ] Kamailio 1.5 often segfaults
SourceForge.net
noreply at sourceforge.net
Fri Mar 27 17:06:30 CET 2009
Bugs item #2649470, was opened at 2009-02-28 19:44
Message generated for change (Comment added) made by nobody
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=743020&aid=2649470&group_id=139143
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Nobody/Anonymous (nobody)
Assigned to: Nobody/Anonymous (nobody)
Summary: Kamailio 1.5 often segfaults
Initial Comment:
I run openser 1.3.4 for about 1000 users and I can see a lot of segfaults (1-5 times a day) in last few days...
OS: CentOS 5 x86_64
Openser: 1.3.4-notls
Backtrace is attached.
Let me know, please, if some other informations are needed.
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 16:06
Message:
Sorry, I can't attache file :-) So, here is bt full in-line:
--------------------------
#0 0x0000003bd8830155 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003bd8831bf0 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00000000004190f9 in anchor_lump (msg=0x7f8e38, offset=1385, len=0,
type=HDR_OTHER_T) at data_lump.c:345
tmp = <value optimized out>
prev = <value optimized out>
t = <value optimized out>
list = <value optimized out>
__FUNCTION__ = "anchor_lump"
#3 0x00002b8e4c7cc3ee in force_rtp_proxy (msg=0x7f8e38,
str1=<value optimized out>, str2=<value optimized out>,
offer=<value optimized out>) at nathelper.c:2913
body = {
s = 0x719d80 "v=0\r\no=- 8 2 IN IP4 9.8.7.6\r\ns=CounterPath X-Lite
3.0\r\nc=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119 100 106 0
105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na="...,
len = 425}
body1 = {
s = 0x719dbe "c=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119
100 106 0 105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na=fmtp:101 0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/"...,
len = 363}
oldport = {
s = 0x719de6 "6980 RTP/AVP 107 119 100 106 0 105 98 8 3 101\r\na=alt:1 1
: MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101 0-15\r\na=rtpmap:107
BV32/16000\r\na=rtpmap:119 BV32-FEC/16000\r\na=rtpmap:100
SPEEX/16000\r\na=rtpma"...,
len = 4}
oldip = {
s = 0x719dc7 "9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP 107 119 100 106 0
105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7
27688\r\na=fmtp:101 0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na="...,
len = 14}
newport = {s = 0x73ac00 "38924", len = 5}
newip = {s = 0x2b8e4c9e898f "1.2.3.4", len = 14}
callid = {
s = 0x719b8e "MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.\r\nCSeq: 2
INVITE\r\nAllow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE,
SUBSCRIBE, INFO\r\nContent-Type: application/sdp\r\nProxy-Authorization:
Dig"..., len = 44}
from_tag = {
s = 0x719b7b "e467b916\r\nCall-ID:
MDY2OTc4MTJmNDA3NDExYzUwMDBmZjE5OWIxOTMzN2E.\r\nCSeq: 2 INVITE\r\nAllow:
INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE, SUBSCRIBE,
INFO\r\nContent-Type: application/sdp\r\nProxy"...,
len = 8}
to_tag = {s = 0x0, len = 0}
create = 1
port = 38924
len = <value optimized out>
asymmetric = 0
flookup = 0
argc = 363
proxied = 0
real = 1
orgip = 0
commip = 1
oidx = 1
pf = 2
pf1 = <value optimized out>
force = 0
swap = 1359311800
rep_oidx = 0
opts =
"Uů\\_˙\177\000\000\030ý\\_˙\177\000\000\001\000\000\000\000\000\000\000\200ů\\_˙\177\000"
rep_opts = "8\216\177", '\0' <repeats 12 times>
cp = <value optimized out>
cp1 = <value optimized out>
cpend = <value optimized out>
next = <value optimized out>
ap = <value optimized out>
argv = {0x2b8e4c9e8989 "38924", 0x2b8e4c9e898f "1.2.3.4", 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
anchor = <value optimized out>
node = <value optimized out>
v = {{iov_base = 0x2b8e4c9e8aa0, iov_len = 9}, {
iov_base = 0x7fff5f5cf830, iov_len = 1}, {iov_base = 0x2b8e4c7d6ab4,
iov_len = 1}, {iov_base = 0x719b8e, iov_len = 44}, {
iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {iov_base = 0x719dc7,
iov_len = 14}, {iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {
iov_base = 0x719de6, iov_len = 4}, {iov_base = 0x2b8e4c7d6ab4,
iov_len = 1}, {iov_base = 0x719b7b, iov_len = 8}, {
iov_base = 0x2b8e4c7d436f, iov_len = 1}, {iov_base = 0x7fff5f5cf850,
iov_len = 1}, {iov_base = 0x2b8e4c7d6ab4, iov_len = 1}, {iov_base =
0x0,
iov_len = 0}, {iov_base = 0x2b8e4c7d436f, iov_len = 1}, {
iov_base = 0x7fff5f5cf850, iov_len = 1}}
v1p = <value optimized out>
v2p = 0x719f29 "8000/1\r\na=rtpmap:0 PCMU/8000/1\r\na=rtpmap:8
PCMA/8000/1\r\na=rtpmap:97 iLBC/8000/1\r\na=rtpmap:18
G729/8000/1\r\na=rtpmap:98 telephone-event/8000/1\r\na=rtpmap:13
CN/8000/1\r\n"
c1p = 0x719dbe "c=IN IP4 9.8.7.6\r\nt=0 0\r\nm=audio 6980 RTP/AVP
107 119 100 106 0 105 98 8 3 101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM
192.168.1.7 27688\r\na=fmtp:101 0-15\r\na=rtpmap:107
BV32/16000\r\na=rtpmap:119 BV32-FEC/"...
c2p = <value optimized out>
m1p = <value optimized out>
m2p = 0x719dde "m=audio 6980 RTP/AVP 107 119 100 106 0 105 98 8 3
101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101
0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na=rtpmap:100 SPEEX/16000\r"...
bodylimit = 0x719f29 "8000/1\r\na=rtpmap:0
PCMU/8000/1\r\na=rtpmap:8 PCMA/8000/1\r\na=rtpmap:97
iLBC/8000/1\r\na=rtpmap:18 G729/8000/1\r\na=rtpmap:98
telephone-event/8000/1\r\na=rtpmap:13 CN/8000/1\r\n"
o1p = <value optimized out>
medianum_buf =
"1\000O\000\000\000\000\0007\000\000\000\000\000\000\000\000\000\000"
medianum = 1
tmpstr1 = {
s = 0x719dde "m=audio 6980 RTP/AVP 107 119 100 106 0 105 98 8 3
101\r\na=alt:1 1 : MMYx0loG Fvh0mkwM 192.168.1.7 27688\r\na=fmtp:101
0-15\r\na=rtpmap:107 BV32/16000\r\na=rtpmap:119
BV32-FEC/16000\r\na=rtpmap:100 SPEEX/16000\r"...,
len = 331}
c1p_altered = 1
swap_warned = 0
__FUNCTION__ = "force_rtp_proxy"
#4 0x00002b8e4c7cd025 in force_rtp_proxy1_f (msg=0x7f8e38,
str1=0x77df00 "rc", str2=<value optimized out>) at nathelper.c:2430
cp = <value optimized out>
newip =
"1.2.3.4\000\000 ů\\_˙\177\000\000čŘw\000\000\000\000\000\r\000\000\000\000\000\000"
#5 0x000000000040d0ba in do_action (a=0x77df90, msg=0x7f8e38) at
action.c:874
ret = <value optimized out>
v = <value optimized out>
to = <value optimized out>
p = <value optimized out>
tmp = <value optimized out>
new_uri = <value optimized out>
end = <value optimized out>
crt = <value optimized out>
len = dwarf2_read_address: Corrupted DWARF expression.
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 16:03
Message:
I have core dump :-) But now there is no CRITICAL message in the log.
I have pcap.dump too, so let me know, please, what shoul I look for. I
didn't see any abnormality...
Backtrace (bt full) is attached.
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Klaus Darilion (klaus_darilion)
Date: 2009-03-27 14:09
Message:
changed subject to "1.5"
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-27 13:50
Message:
I have upgraded to Kamailio 1.5 and after few days without troubles
Kamailio starts to segfaults very often (about every 10 minutes).
I'm using nearly same config as for previous Openser 1.3.4.
In the log I see before every crash:
CRITICAL:core:anchor_lump: offset exceeds message size (1084 > 1080)
aborting...
only reported sizes differs.
I don't have core files due to big RAM and lack of HDD space, but I try to
find some and than submit backtrace...
BTW: Should I open new Bug if it is for new version of server (1.5)?
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Henning Westerholt (henningw)
Date: 2009-03-16 12:46
Message:
Hi Kokoska,
any updates on this problem? Were you able to isolate the "strange"
behaviour of your clients?
Henning
----------------------------------------------------------------------
Comment By: Nobody/Anonymous (nobody)
Date: 2009-03-02 21:43
Message:
Hi Henning,
Hi Henning,
I'm sorry, but I'm on vacation now, so I can't say much about it :-)
I have tried to look at logs, but I havent seen anything interesting - for
me it looks like if this happed whed OpenSER tris tu dump stored messages
after successful registration.
I have sip_trace "armed" and thus I try to look what goes through the
server.
Please give me some time to look deeper into the problem (saturday,
sunday).
BTW: Config file is very complex, but I didn't made any changes before
OpenSER starts to segfaults => I thik it is something like "strange"
behaviour of one of the new clients.
Thank you very much for your interest!
Best regards,
kokoska.rokoska
----------------------------------------------------------------------
Comment By: Henning Westerholt (henningw)
Date: 2009-03-02 12:33
Message:
Hi Kokoska,
have you done any notable changes on the configuration or the setup before
it started to crash?
Have you tried to further examine the core dump, e.g. look at the SIP
message content, or the contact_body structure?
backtrace from the file for convinience:
Core was generated by `/usr/local/sbin/openser -m 1024'.
Program terminated with signal 11, Segmentation fault.
#0 free_contacts (_c=0x7cc030) at parser/contact/contact.c:287
287 *_c = (*_c)->next;
(gdb) bt
#0 free_contacts (_c=0x7cc030) at parser/contact/contact.c:287
#1 0x00000000004b54da in free_contact (_c=0x2ba8f6fce2e0)
at parser/contact/parse_contact.c:101
#2 0x000000000048fbc3 in clean_hdr_field (hf=0x2ba8f6fce2b0) at
parser/hf.c:78
#3 0x00002ba8f2436a29 in run_trans_callbacks (type=<value optimized
out>,
trans=<value optimized out>, req=0x2ba8f6fcda08, rpl=0x7cbe68,
code=<value optimized out>) at sip_msg.h:49
#4 0x00002ba8f2440112 in t_reply_matching (p_msg=0x7cbe68,
p_branch=<value optimized out>) at t_lookup.c:840
#5 0x00002ba8f244056f in t_check (p_msg=0x7cbe68,
param_branch=0x7fffb8cd52ec)
at t_lookup.c:911
#6 0x00002ba8f244fe16 in reply_received (p_msg=0x7cc030) at
t_reply.c:1285
#7 0x000000000041d1ac in forward_reply (msg=0x7cbe68) at forward.c:499
#8 0x000000000044c87c in receive_msg (
buf=0x726920 "SIP/2.0 100 Trying\r\nVia: SIP/2.0/UDP
1.2.3.4;branch=z9hG4bK3553.caa63f57.1;received=1.2.3.4\r\nVia: SIP/2.0/UDP
4.3.2.1;rport=5060;branch=z9hG4bK-49aa467849a97dc2123\r\nFrom:
\"123456798"...,
len=557, rcv_info=0x7fffb8cd53d0) at receive.c:194
#9 0x00000000004892de in udp_rcv_loop () at udp_server.c:438
#10 0x000000000042561f in main (argc=3, argv=0x7fffb8cd55e8) at
main.c:834
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=743020&aid=2649470&group_id=139143
More information about the Devel
mailing list