[SR-Users] Kamailio 1.5 stops processing messages

[marius zbihlei]

Santiago Gimeno wrote:
> Hi,
>
> I have experienced a new lock. I have seen that 4 of the UDP worker 
> processes, but not all, have gone 100% cpu. Kamailio was still able to 
> process messages through the rest of UDP worker processes. By checking 
> the logs I can see that the 4 blocked processes have stopped handling 
> messages at the same time.
>
> I have attached to the blocked processes and the backtrace looks 
> similar to the previous one except that it seems that the lock is not 
> exactly related to t_replicate.
>
> #0  0xb7f24410 in ?? ()
> #1  0xbf9cdeb8 in ?? ()
> #2  0x00000001 in ?? ()
> #3  0xa72eb318 in ?? ()
> #4  0xb7e804ac in sched_yield () from /lib/tls/i686/cmov/libc.so.6
> #5  0xb7aee463 in lock_hash (i=13692) at ../../mem/../fastlock.h:182
> #6  0xb7b09587 in t_lookup_request (p_msg=0x81d9708, 
> leave_new_locked=1) at t_lookup.c:468
> #7  0xb7b0a4ae in t_newtran (p_msg=0x81d9708) at t_lookup.c:1124
> #8  0xb7afc40c in t_relay_to (p_msg=0x81d9708, proxy=0x0, flags=8) at 
> t_funcs.c:212
> #9  0xb7b0fac7 in w_t_relay (p_msg=0x81d9708, proxy=0x0, flags=0x8 
> <Address 0x8 out of bounds>) at tm.c:1002
> #10 0x0805301c in do_action (a=0x818c370, msg=0x81d9708) at action.c:874
> #11 0x080557aa in run_action_list (a=0x818c370, msg=0x81d9708) at 
> action.c:145
> #12 0x0809c304 in eval_expr (e=0x818c3d8, msg=0x81d9708, val=0x0) at 
> route.c:1171
> #13 0x0809bd80 in eval_expr (e=0x818c400, msg=0x81d9708, val=0x0) at 
> route.c:1488
> #14 0x0809bd16 in eval_expr (e=0x818c428, msg=0x81d9708, val=0x0) at 
> route.c:1493
> #15 0x080527ed in do_action (a=0x818c740, msg=0x81d9708) at action.c:729
> #16 0x080557aa in run_action_list (a=0x818be08, msg=0x81d9708) at 
> action.c:145
> #17 0x08053efb in do_action (a=0x81a6a48, msg=0x81d9708) at action.c:120
> #18 0x080557aa in run_action_list (a=0x81a14e0, msg=0x81d9708) at 
> action.c:145
> #19 0x08053efb in do_action (a=0x81c29f0, msg=0x81d9708) at action.c:120
> #20 0x080557aa in run_action_list (a=0x81c2698, msg=0x81d9708) at 
> action.c:145
> #21 0x08054491 in do_action (a=0x81c2b48, msg=0x81d9708) at action.c:746
> #22 0x080557aa in run_action_list (a=0x81c2030, msg=0x81d9708) at 
> action.c:145
> #23 0x08053efb in do_action (a=0x8193278, msg=0x81d9708) at action.c:120
> #24 0x080557aa in run_action_list (a=0x8193278, msg=0x81d9708) at 
> action.c:145
> #25 0x08054491 in do_action (a=0x8193418, msg=0x81d9708) at action.c:746
> #26 0x080557aa in run_action_list (a=0x8193418, msg=0x81d9708) at 
> action.c:145
> #27 0x08054f2d in do_action (a=0x8193480, msg=0x81d9708) at action.c:752
> #28 0x080557aa in run_action_list (a=0x81922f0, msg=0x81d9708) at 
> action.c:145
> #29 0x08054491 in do_action (a=0x81934e8, msg=0x81d9708) at action.c:746
> #30 0x080557aa in run_action_list (a=0x818ee78, msg=0x81d9708) at 
> action.c:145
> #31 0x08053efb in do_action (a=0x81b3448, msg=0x81d9708) at action.c:120
> #32 0x080557aa in run_action_list (a=0x81b0ed0, msg=0x81d9708) at 
> action.c:145
> #33 0x08054491 in do_action (a=0x81b5f68, msg=0x81d9708) at action.c:746
> #34 0x080557aa in run_action_list (a=0x81b5f68, msg=0x81d9708) at 
> action.c:145
> #35 0x08054f2d in do_action (a=0x81b5fd0, msg=0x81d9708) at action.c:752
> #36 0x080557aa in run_action_list (a=0x81aefd0, msg=0x81d9708) at 
> action.c:145
> #37 0x08053efb in do_action (a=0x818bc08, msg=0x81d9708) at action.c:120
> #38 0x080557aa in run_action_list (a=0x8187910, msg=0x81d9708) at 
> action.c:145
> #39 0x08055b43 in run_top_route (a=0x8187910, msg=0x81d9708) at 
> action.c:120
> #40 0x0808c659 in receive_msg (
>     buf=0x8158040 "INVITE sip:xxxxxxxx at xxxxxxx.com 
> <mailto:sip%3Axxxxxxxx at xxxxxxx.com> SIP/2.0\r\nRecord-Route: 
> <sip:10.100.2.252;lr=on;ftag=as6897d90b>\r\nVia: SIP/2.0/UDP 
> 10.100.2.252;branch=z9hG4bKc753.2c8885d4.0\r\nVia: SIP/2.0/UDP 
> 10.100.2.253:5060;bra"..., len=896, rcv_info=0xbf9d0aa4) at receive.c:175
> #41 0x080c3ea3 in udp_rcv_loop () at udp_server.c:449
> #42 0x0806e394 in main (argc=9, argv=0xbf9d0c84) at main.c:774
>
> I hope this provides some new hints on what might be happening.


Hello,

It also might help if you can do a bt "full"  Also try printing the 
whole message  x/s 0x8158040 should do the trick.

If I recall this correctly you said something about only t_replicating 
REGISTERs. The message is an INVITE so it also might be that you have a  
cfg error somewhere

Cheers
Marius