[SR-Users] [sr-dev] Very high load average after upgrade to Kamailio v4.3.2

Mon Sep 14 11:55:11 CEST 2015

OK runtime debugging shows something is wrong with dialog module, all 6
processes are locked in trying to access dialog module when 200 OK is
received for the call. For example, here is the BT of one the processes,

--
#0  0xb77a0424 in __kernel_vsyscall ()
#1  0xb76c140c in sched_yield () from
/lib/i386-linux-gnu/i686/cmov/libc.so.6
#2  0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277
#3  0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610
#4  0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643
#5  0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac) at
dlg_handlers.c:429
#6  0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398, type=2,
trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268
#7  0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358,
req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295
#8  0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8, p_branch=0xbff3a5a4)
at t_lookup.c:966
#9  0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4)
at t_lookup.c:1069
#10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at
t_lookup.c:1111
#11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134
#12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at forward.c:747
#13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849
#14 0x08138aab in receive_msg (
    buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia:
SIP/2.0/UDP
192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"...,
len=1103, rcv_info=0xbff3a93c) at receive.c:255
#15 0x082188d0 in udp_rcv_loop () at udp_server.c:495
#16 0x080dec65 in main_loop () at main.c:1573
#17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
--

Here is BT of second process,

--
#0  0xb77a0424 in __kernel_vsyscall ()
#1  0xb76c140c in sched_yield () from
/lib/i386-linux-gnu/i686/cmov/libc.so.6
#2  0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277
#3  0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610
#4  0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643
#5  0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072,
param=0xbff3a7ac) at dlg_handlers.c:398
#6  0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464,
type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268
#7  0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424,
req=0x0, rpl=0x0, code=0) at t_hooks.c:295
#8  0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128
#9  0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c,
data=0xa5985424) at timer.c:648
#10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718,
slow_l=0xa530e7ec, slow_mark=6) at timer.c:873
#11 0x0820dcc0 in timer_handler () at timer.c:938
#12 0x0820e0d8 in timer_main () at timer.c:977
#13 0x080df4e0 in main_loop () at main.c:1644
#14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
--

Does this makes any sense to you? Let me know if you need anything else.

Thank you.

On Mon, Sep 14, 2015 at 11:36 AM, M S <shaheryarkh at gmail.com> wrote:

> Sure, here is the list of module in given order,
>
> --
> loadmodule "db_mysql.so"
> loadmodule "mi_fifo.so"
> loadmodule "mi_datagram.so"
> loadmodule "kex.so"
> loadmodule "corex.so"
> loadmodule "tm.so"
> loadmodule "tmx.so"
> loadmodule "sl.so"
> loadmodule "outbound.so"
> loadmodule "rr.so"
> loadmodule "path.so"
> loadmodule "pv.so"
> loadmodule "maxfwd.so"
> loadmodule "usrloc.so"
> loadmodule "registrar.so"
> loadmodule "sdpops.so"
> loadmodule "textops.so"
> loadmodule "textopsx.so"
> loadmodule "siputils.so"
> loadmodule "xlog.so"
> loadmodule "sanity.so"
> loadmodule "ctl.so"
> loadmodule "cfg_rpc.so"
> loadmodule "mi_rpc.so"
> loadmodule "dialog.so"
> loadmodule "acc.so"
> loadmodule "uac.so"
> loadmodule "rtimer.so"
> loadmodule "sqlops.so"
> loadmodule "ndb_redis.so"
> loadmodule "app_perl.so"
> loadmodule "permissions.so"
> loadmodule "domain.so"
> loadmodule "async.so"
> loadmodule "stun.so"
> loadmodule "auth.so"
> loadmodule "auth_db.so"
> loadmodule "alias_db.so"
> loadmodule "speeddial.so"
> loadmodule "presence.so"
> loadmodule "presence_mwi.so"
> loadmodule "presence_xml.so"
> loadmodule "presence_profile.so"
> loadmodule "nathelper.so"
> loadmodule "rtpengine.so"
> loadmodule "tls.so"
> loadmodule "htable.so"
> loadmodule "pike.so"
> loadmodule "xmlrpc.so"
> loadmodule "debugger.so"
> loadmodule "xhttp.so"
> loadmodule "xhttp_rpc.so"
> loadmodule "xhttp_pi.so"
> loadmodule "xcap_server.so"
> loadmodule "pua.so"
> loadmodule "pua_mi.so"
> loadmodule "rls.so"
> loadmodule "cfgutils.so"
> loadmodule "htable.so"
> loadmodule "msrp.so"
> loadmodule "websocket.so"
> loadmodule "msilo.so"
> loadmodule "siptrace.so"
> --
>
> In "top" i see at least 6 kamailio processes using very high cpu (perhaps
> these are the 6 child processes involved in that single call processing).
>
> Thank you.
>
>
>
> On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>> Hello,
>>
>> I will review the changes pushed to 4.3.2 vs 4.3.1. Can you send here the
>> list of modules you are using? The loadmodule lines in kamailio.cfg.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 14/09/15 10:51, M S wrote:
>>
>> Hi,
>>
>> Over last weekend i upgraded one of my test servers from Kamailio
>> v4.3.1-4d1b65 to latest stable release v4.3.2-07690f and now kamailio goes
>> crazy even with single call (I am using same db and configuration of
>> course).
>>
>> As soon as call establishes system load average (as seen in top command)
>> starts increasing and soon it increases beyond 6.0 and system becomes
>> completely unresponsive, sip messages are no longer being processed by
>> kamailio service. Even after call hangup, system remains under high load.
>> The "htop" indicates that IO Wait time has increased substantially.
>>
>> Any idea what is causing this? For now i have reverted by to
>> v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need further info or
>> testing.
>>
>> Thank you.
>>
>>
>> _______________________________________________
>> sr-dev mailing listsr-dev at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>>
>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20150914/a2f09298/attachment.html>