[SR-Users] [sr-dev] Very high load average after upgrade to Kamailio v4.3.2

Mon Sep 14 19:19:56 CEST 2015

Hi,

After doing all the test cases, it seems the problem has been resolved in
v4.3.2-3763f6.

Thank you so much for your help in this timely fix.

On Mon, Sep 14, 2015 at 12:52 PM, M S <shaheryarkh at gmail.com> wrote:

> OK, let me test it and get back to you.
>
> Thank you.
>
>
>
> On Mon, Sep 14, 2015 at 12:18 PM, Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>> I found a regression when detecting spiraled dialogs with state deleted.
>> That would be the case when you create a dialog for an INVITE, but you
>> reply with a negative response code from kamailio and then quickly a new
>> invite with same callid and from tag arrives.
>>
>> Can you upgrade to latest version in branch 4.3, try and see if works
>> fine now?
>>
>> Cheers,
>> Daniel
>>
>>
>> On 14/09/15 11:55, M S wrote:
>>
>> OK runtime debugging shows something is wrong with dialog module, all 6
>> processes are locked in trying to access dialog module when 200 OK is
>> received for the call. For example, here is the BT of one the processes,
>>
>> --
>> #0  0xb77a0424 in __kernel_vsyscall ()
>> #1  0xb76c140c in sched_yield () from
>> /lib/i386-linux-gnu/i686/cmov/libc.so.6
>> #2  0xb5a20206 in get_lock (lock=0xa55058e4) at
>> ../../mem/../fastlock.h:277
>> #3  0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610
>> #4  0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643
>> #5  0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac) at
>> dlg_handlers.c:429
>> #6  0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398,
>> type=2, trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268
>> #7  0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358,
>> req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295
>> #8  0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8,
>> p_branch=0xbff3a5a4) at t_lookup.c:966
>> #9  0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4)
>> at t_lookup.c:1069
>> #10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at
>> t_lookup.c:1111
>> #11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134
>> #12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at
>> forward.c:747
>> #13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849
>> #14 0x08138aab in receive_msg (
>>     buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
>> X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia:
>> SIP/2.0/UDP
>> 192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"...,
>> len=1103, rcv_info=0xbff3a93c) at receive.c:255
>> #15 0x082188d0 in udp_rcv_loop () at udp_server.c:495
>> #16 0x080dec65 in main_loop () at main.c:1573
>> #17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
>> --
>>
>> Here is BT of second process,
>>
>> --
>> #0  0xb77a0424 in __kernel_vsyscall ()
>> #1  0xb76c140c in sched_yield () from
>> /lib/i386-linux-gnu/i686/cmov/libc.so.6
>> #2  0xb5a20206 in get_lock (lock=0xa55058e4) at
>> ../../mem/../fastlock.h:277
>> #3  0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610
>> #4  0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643
>> #5  0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072,
>> param=0xbff3a7ac) at dlg_handlers.c:398
>> #6  0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464,
>> type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268
>> #7  0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424,
>> req=0x0, rpl=0x0, code=0) at t_hooks.c:295
>> #8  0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128
>> #9  0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c,
>> data=0xa5985424) at timer.c:648
>> #10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718,
>> slow_l=0xa530e7ec, slow_mark=6) at timer.c:873
>> #11 0x0820dcc0 in timer_handler () at timer.c:938
>> #12 0x0820e0d8 in timer_main () at timer.c:977
>> #13 0x080df4e0 in main_loop () at main.c:1644
>> #14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
>> --
>>
>> Does this makes any sense to you? Let me know if you need anything else.
>>
>> Thank you.
>>
>>
>>
>> On Mon, Sep 14, 2015 at 11:36 AM, M S <shaheryarkh at gmail.com> wrote:
>>
>>> Sure, here is the list of module in given order,
>>>
>>> --
>>> loadmodule "db_mysql.so"
>>> loadmodule "mi_fifo.so"
>>> loadmodule "mi_datagram.so"
>>> loadmodule "kex.so"
>>> loadmodule "corex.so"
>>> loadmodule "tm.so"
>>> loadmodule "tmx.so"
>>> loadmodule "sl.so"
>>> loadmodule "outbound.so"
>>> loadmodule "rr.so"
>>> loadmodule "path.so"
>>> loadmodule "pv.so"
>>> loadmodule "maxfwd.so"
>>> loadmodule "usrloc.so"
>>> loadmodule "registrar.so"
>>> loadmodule "sdpops.so"
>>> loadmodule "textops.so"
>>> loadmodule "textopsx.so"
>>> loadmodule "siputils.so"
>>> loadmodule "xlog.so"
>>> loadmodule "sanity.so"
>>> loadmodule "ctl.so"
>>> loadmodule "cfg_rpc.so"
>>> loadmodule "mi_rpc.so"
>>> loadmodule "dialog.so"
>>> loadmodule "acc.so"
>>> loadmodule "uac.so"
>>> loadmodule "rtimer.so"
>>> loadmodule "sqlops.so"
>>> loadmodule "ndb_redis.so"
>>> loadmodule "app_perl.so"
>>> loadmodule "permissions.so"
>>> loadmodule "domain.so"
>>> loadmodule "async.so"
>>> loadmodule "stun.so"
>>> loadmodule "auth.so"
>>> loadmodule "auth_db.so"
>>> loadmodule "alias_db.so"
>>> loadmodule "speeddial.so"
>>> loadmodule "presence.so"
>>> loadmodule "presence_mwi.so"
>>> loadmodule "presence_xml.so"
>>> loadmodule "presence_profile.so"
>>> loadmodule "nathelper.so"
>>> loadmodule "rtpengine.so"
>>> loadmodule "tls.so"
>>> loadmodule "htable.so"
>>> loadmodule "pike.so"
>>> loadmodule "xmlrpc.so"
>>> loadmodule "debugger.so"
>>> loadmodule "xhttp.so"
>>> loadmodule "xhttp_rpc.so"
>>> loadmodule "xhttp_pi.so"
>>> loadmodule "xcap_server.so"
>>> loadmodule "pua.so"
>>> loadmodule "pua_mi.so"
>>> loadmodule "rls.so"
>>> loadmodule "cfgutils.so"
>>> loadmodule "htable.so"
>>> loadmodule "msrp.so"
>>> loadmodule "websocket.so"
>>> loadmodule "msilo.so"
>>> loadmodule "siptrace.so"
>>> --
>>>
>>> In "top" i see at least 6 kamailio processes using very high cpu
>>> (perhaps these are the 6 child processes involved in that single call
>>> processing).
>>>
>>> Thank you.
>>>
>>>
>>>
>>> On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla <
>>> <miconda at gmail.com>miconda at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I will review the changes pushed to 4.3.2 vs 4.3.1. Can you send here
>>>> the list of modules you are using? The loadmodule lines in kamailio.cfg.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 14/09/15 10:51, M S wrote:
>>>>
>>>> Hi,
>>>>
>>>> Over last weekend i upgraded one of my test servers from Kamailio
>>>> v4.3.1-4d1b65 to latest stable release v4.3.2-07690f and now kamailio goes
>>>> crazy even with single call (I am using same db and configuration of
>>>> course).
>>>>
>>>> As soon as call establishes system load average (as seen in top
>>>> command) starts increasing and soon it increases beyond 6.0 and system
>>>> becomes completely unresponsive, sip messages are no longer being processed
>>>> by kamailio service. Even after call hangup, system remains under high
>>>> load. The "htop" indicates that IO Wait time has increased substantially.
>>>>
>>>> Any idea what is causing this? For now i have reverted by to
>>>> v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need further info or
>>>> testing.
>>>>
>>>> Thank you.
>>>>
>>>>
>>>> _______________________________________________
>>>> sr-dev mailing listsr-dev at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>> sr-users at lists.sip-router.org
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>
>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20150914/1de3ecde/attachment.html>