[sr-dev] [SR-Users] Very high load average after upgrade to Kamailio v4.3.2

[M S]

OK, let me test it and get back to you.

Thank you.



On Mon, Sep 14, 2015 at 12:18 PM, Daniel-Constantin Mierla <
miconda at gmail.com> wrote:

> I found a regression when detecting spiraled dialogs with state deleted.
> That would be the case when you create a dialog for an INVITE, but you
> reply with a negative response code from kamailio and then quickly a new
> invite with same callid and from tag arrives.
>
> Can you upgrade to latest version in branch 4.3, try and see if works fine
> now?
>
> Cheers,
> Daniel
>
>
> On 14/09/15 11:55, M S wrote:
>
> OK runtime debugging shows something is wrong with dialog module, all 6
> processes are locked in trying to access dialog module when 200 OK is
> received for the call. For example, here is the BT of one the processes,
>
> --
> #0  0xb77a0424 in __kernel_vsyscall ()
> #1  0xb76c140c in sched_yield () from
> /lib/i386-linux-gnu/i686/cmov/libc.so.6
> #2  0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277
> #3  0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610
> #4  0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643
> #5  0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac) at
> dlg_handlers.c:429
> #6  0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398, type=2,
> trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268
> #7  0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358,
> req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295
> #8  0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8, p_branch=0xbff3a5a4)
> at t_lookup.c:966
> #9  0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4)
> at t_lookup.c:1069
> #10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at
> t_lookup.c:1111
> #11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134
> #12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at
> forward.c:747
> #13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849
> #14 0x08138aab in receive_msg (
>     buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
> X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia:
> SIP/2.0/UDP
> 192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"...,
> len=1103, rcv_info=0xbff3a93c) at receive.c:255
> #15 0x082188d0 in udp_rcv_loop () at udp_server.c:495
> #16 0x080dec65 in main_loop () at main.c:1573
> #17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
> --
>
> Here is BT of second process,
>
> --
> #0  0xb77a0424 in __kernel_vsyscall ()
> #1  0xb76c140c in sched_yield () from
> /lib/i386-linux-gnu/i686/cmov/libc.so.6
> #2  0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277
> #3  0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610
> #4  0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643
> #5  0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072,
> param=0xbff3a7ac) at dlg_handlers.c:398
> #6  0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464,
> type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268
> #7  0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424,
> req=0x0, rpl=0x0, code=0) at t_hooks.c:295
> #8  0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128
> #9  0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c,
> data=0xa5985424) at timer.c:648
> #10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718,
> slow_l=0xa530e7ec, slow_mark=6) at timer.c:873
> #11 0x0820dcc0 in timer_handler () at timer.c:938
> #12 0x0820e0d8 in timer_main () at timer.c:977
> #13 0x080df4e0 in main_loop () at main.c:1644
> #14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
> --
>
> Does this makes any sense to you? Let me know if you need anything else.
>
> Thank you.
>
>
>
> On Mon, Sep 14, 2015 at 11:36 AM, M S <shaheryarkh at gmail.com> wrote:
>
>> Sure, here is the list of module in given order,
>>
>> --
>> loadmodule "db_mysql.so"
>> loadmodule "mi_fifo.so"
>> loadmodule "mi_datagram.so"
>> loadmodule "kex.so"
>> loadmodule "corex.so"
>> loadmodule "tm.so"
>> loadmodule "tmx.so"
>> loadmodule "sl.so"
>> loadmodule "outbound.so"
>> loadmodule "rr.so"
>> loadmodule "path.so"
>> loadmodule "pv.so"
>> loadmodule "maxfwd.so"
>> loadmodule "usrloc.so"
>> loadmodule "registrar.so"
>> loadmodule "sdpops.so"
>> loadmodule "textops.so"
>> loadmodule "textopsx.so"
>> loadmodule "siputils.so"
>> loadmodule "xlog.so"
>> loadmodule "sanity.so"
>> loadmodule "ctl.so"
>> loadmodule "cfg_rpc.so"
>> loadmodule "mi_rpc.so"
>> loadmodule "dialog.so"
>> loadmodule "acc.so"
>> loadmodule "uac.so"
>> loadmodule "rtimer.so"
>> loadmodule "sqlops.so"
>> loadmodule "ndb_redis.so"
>> loadmodule "app_perl.so"
>> loadmodule "permissions.so"
>> loadmodule "domain.so"
>> loadmodule "async.so"
>> loadmodule "stun.so"
>> loadmodule "auth.so"
>> loadmodule "auth_db.so"
>> loadmodule "alias_db.so"
>> loadmodule "speeddial.so"
>> loadmodule "presence.so"
>> loadmodule "presence_mwi.so"
>> loadmodule "presence_xml.so"
>> loadmodule "presence_profile.so"
>> loadmodule "nathelper.so"
>> loadmodule "rtpengine.so"
>> loadmodule "tls.so"
>> loadmodule "htable.so"
>> loadmodule "pike.so"
>> loadmodule "xmlrpc.so"
>> loadmodule "debugger.so"
>> loadmodule "xhttp.so"
>> loadmodule "xhttp_rpc.so"
>> loadmodule "xhttp_pi.so"
>> loadmodule "xcap_server.so"
>> loadmodule "pua.so"
>> loadmodule "pua_mi.so"
>> loadmodule "rls.so"
>> loadmodule "cfgutils.so"
>> loadmodule "htable.so"
>> loadmodule "msrp.so"
>> loadmodule "websocket.so"
>> loadmodule "msilo.so"
>> loadmodule "siptrace.so"
>> --
>>
>> In "top" i see at least 6 kamailio processes using very high cpu (perhaps
>> these are the 6 child processes involved in that single call processing).
>>
>> Thank you.
>>
>>
>>
>> On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla <
>> <miconda at gmail.com>miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I will review the changes pushed to 4.3.2 vs 4.3.1. Can you send here
>>> the list of modules you are using? The loadmodule lines in kamailio.cfg.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 14/09/15 10:51, M S wrote:
>>>
>>> Hi,
>>>
>>> Over last weekend i upgraded one of my test servers from Kamailio
>>> v4.3.1-4d1b65 to latest stable release v4.3.2-07690f and now kamailio goes
>>> crazy even with single call (I am using same db and configuration of
>>> course).
>>>
>>> As soon as call establishes system load average (as seen in top command)
>>> starts increasing and soon it increases beyond 6.0 and system becomes
>>> completely unresponsive, sip messages are no longer being processed by
>>> kamailio service. Even after call hangup, system remains under high load.
>>> The "htop" indicates that IO Wait time has increased substantially.
>>>
>>> Any idea what is causing this? For now i have reverted by to
>>> v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need further info or
>>> testing.
>>>
>>> Thank you.
>>>
>>>
>>> _______________________________________________
>>> sr-dev mailing listsr-dev at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>>>
>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users at lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-dev/attachments/20150914/43009369/attachment-0001.html>