[sr-dev] [Fwd: Re: [SR-Users] Kamailio 1.5 stops processing messages]

[Santiago Gimeno]

Hi,

2010/6/14 Andrei Pelinescu-Onciul <andrei at iptel.org>
>
> If it's using 100% cpu it might mean the list that search_htable()
>> iterates on is corrupted and has become cyclic.
>>
>> You could also try compiling with debugging options, e.g.:
>> make config mode=debug; make all
>> or
>> make config CC_EXTRA_OPTS="-O0" ; make all
>>
>>
I have finally performed a test (7 calls per second / duration of call:
8secs) and couldn't reproduce the locking, but after a few hours kamailio
crashed. The backtrace of the core file is:

#0  0xb7f50424 in __kernel_vsyscall ()
#1  0xb7df9640 in raise () from /lib/i686/cmov/libc.so.6
#2  0xb7dfb018 in abort () from /lib/i686/cmov/libc.so.6
#3  0x08131857 in qm_free (qm=0xb5745000, p=0xb6b41804, file=0xb776a40f
"pua: send_publish.c", func=0xb776a584 "publ_cback_func", line=382)
    at mem/q_malloc.c:447
#4  0xb775ead6 in publ_cback_func (t=0xb6b07b4c, type=256, ps=0xbfd51690) at
send_publish.c:382
#5  0xb7a716a1 in run_trans_callbacks_internal (cb_lst=0xb6b07b8c, type=256,
trans=0xb6b07b4c, params=0xbfd51690) at t_hooks.c:290
#6  0xb7a717a7 in run_trans_callbacks (type=256, trans=0xb6b07b4c, req=0x0,
rpl=0x82c5074, code=412) at t_hooks.c:317
#7  0xb7a96a5c in local_reply (t=0xb6b07b4c, p_msg=0x82c5074, branch=0,
msg_status=412, cancel_bitmap=0xbfd518a8) at t_reply.c:1882
#8  0xb7a97e12 in reply_received (p_msg=0x82c5074) at t_reply.c:2217
#9  0x08086b7d in forward_reply (msg=0x82c5074) at forward.c:751
#10 0x080b9b70 in receive_msg (
    buf=0x8256200 "SIP/2.0 412 Conditional request failed\r\nVia:
SIP/2.0/UDP 192.168.222.203;branch=z9hG4bK2fb5.a8a0e8e3.0\r\nTo:
sip:service at 192.168.222.203:5060;tag=d3a667e9ba3dcfdbb46cfb20f8a24cf9-3e9e\r\nFrom:
sip:servic"..., len=391, rcv_info=0xbfd51adc) at receive.c:266
#11 0x08129890 in udp_rcv_loop () at udp_server.c:527
#12 0x0809246d in main_loop () at main.c:1462
#13 0x08094e12 in main (argc=1, argv=0xbfd51e14) at main.c:2315


And the log shows this:

Jul  8 15:38:05 devserver kamailio[27367]: WARNING: dialog
[dlg_handlers.c:815]: unable to find dialog for BYE with route param
'fbc.923ee663' [3263:913236777]
Jul  8 15:38:05 devserver kamailio[27367]: WARNING: dialog
[dlg_handlers.c:815]: unable to find dialog for BYE with route param
'1f7.f91bb257' [2033:1965797791]
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [uac.c:281]: t_uac:
short of cell shmem
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: pua [send_publish.c:565]:
in t_request tm module function
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: pua_dialoginfo
[dialog_publish.c:351]: while sending publish
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [t_lookup.c:1347]:
ERROR: new_t: out of mem:
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [t_lookup.c:1487]:
ERROR: t_newtran: new_t failed
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [uac.c:281]: t_uac:
short of cell shmem
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: pua [send_publish.c:565]:
in t_request tm module function
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: pua_dialoginfo
[dialog_publish.c:351]: while sending publish
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [t_lookup.c:1347]:
ERROR: new_t: out of mem:
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [t_lookup.c:1487]:
ERROR: t_newtran: new_t failed
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [t_lookup.c:1347]:
ERROR: new_t: out of mem:
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [t_lookup.c:1487]:
ERROR: t_newtran: new_t failed
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [t_lookup.c:1347]:
ERROR: new_t: out of mem:
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [t_lookup.c:1487]:
ERROR: t_newtran: new_t failed
Jul  8 15:38:05 devserver kamailio[27367]: WARNING: dialog
[dlg_handlers.c:815]: unable to find dialog for BYE with route param
'2b6.b125ba63' [1714:917197339]
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: tm [uac.c:281]: t_uac:
short of cell shmem
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: pua [send_publish.c:565]:
in t_request tm module function
Jul  8 15:38:05 devserver kamailio[27367]: ERROR: pua [send_publish.c:225]:
when trying to send PUBLISH
Jul  8 15:38:05 devserver kamailio[27367]: : <core> [mem/q_malloc.c:446]:
BUG: qm_free: freeing already freed pointer, first free: pua: hash.c:
delete_htable(264) - aborting
Jul  8 15:38:06 devserver kamailio[27381]: : <core> [pass_fd.c:293]: ERROR:
receive_fd: EOF on 9
Jul  8 15:38:07 devserver kamailio[27364]: ALERT: <core> [main.c:737]: child
process 27367 exited by a signal 6
Jul  8 15:38:07 devserver kamailio[27364]: ALERT: <core> [main.c:740]: core
was generated

The log shows lots of "out of mem" warnings, but I don't know if it is
related to the crahs.

Any ideas?

Best regards,

Santi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-dev/attachments/20100709/e945d5f8/attachment-0001.htm>