[SR-Users] [sr-dev] Very high load average after upgrade to Kamailio v4.3.2
Daniel-Constantin Mierla
miconda at gmail.com
Mon Sep 14 12:18:31 CEST 2015
I found a regression when detecting spiraled dialogs with state deleted.
That would be the case when you create a dialog for an INVITE, but you
reply with a negative response code from kamailio and then quickly a new
invite with same callid and from tag arrives.
Can you upgrade to latest version in branch 4.3, try and see if works
fine now?
Cheers,
Daniel
On 14/09/15 11:55, M S wrote:
> OK runtime debugging shows something is wrong with dialog module, all
> 6 processes are locked in trying to access dialog module when 200 OK
> is received for the call. For example, here is the BT of one the
> processes,
>
> --
> #0 0xb77a0424 in __kernel_vsyscall ()
> #1 0xb76c140c in sched_yield () from
> /lib/i386-linux-gnu/i686/cmov/libc.so.6
> #2 0xb5a20206 in get_lock (lock=0xa55058e4) at
> ../../mem/../fastlock.h:277
> #3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610
> #4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643
> #5 0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac)
> at dlg_handlers.c:429
> #6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398,
> type=2, trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268
> #7 0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358,
> req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295
> #8 0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8,
> p_branch=0xbff3a5a4) at t_lookup.c:966
> #9 0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8,
> param_branch=0xbff3a5a4) at t_lookup.c:1069
> #10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4)
> at t_lookup.c:1111
> #11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134
> #12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at
> forward.c:747
> #13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849
> #14 0x08138aab in receive_msg (
> buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
> X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia:
> SIP/2.0/UDP
> 192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"...,
> len=1103, rcv_info=0xbff3a93c) at receive.c:255
> #15 0x082188d0 in udp_rcv_loop () at udp_server.c:495
> #16 0x080dec65 in main_loop () at main.c:1573
> #17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
> --
>
> Here is BT of second process,
>
> --
> #0 0xb77a0424 in __kernel_vsyscall ()
> #1 0xb76c140c in sched_yield () from
> /lib/i386-linux-gnu/i686/cmov/libc.so.6
> #2 0xb5a20206 in get_lock (lock=0xa55058e4) at
> ../../mem/../fastlock.h:277
> #3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610
> #4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643
> #5 0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072,
> param=0xbff3a7ac) at dlg_handlers.c:398
> #6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464,
> type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268
> #7 0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424,
> req=0x0, rpl=0x0, code=0) at t_hooks.c:295
> #8 0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128
> #9 0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c,
> data=0xa5985424) at timer.c:648
> #10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718,
> slow_l=0xa530e7ec, slow_mark=6) at timer.c:873
> #11 0x0820dcc0 in timer_handler () at timer.c:938
> #12 0x0820e0d8 in timer_main () at timer.c:977
> #13 0x080df4e0 in main_loop () at main.c:1644
> #14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
> --
>
> Does this makes any sense to you? Let me know if you need anything else.
>
> Thank you.
>
>
>
> On Mon, Sep 14, 2015 at 11:36 AM, M S <shaheryarkh at gmail.com
> <mailto:shaheryarkh at gmail.com>> wrote:
>
> Sure, here is the list of module in given order,
>
> --
> loadmodule "db_mysql.so"
> loadmodule "mi_fifo.so"
> loadmodule "mi_datagram.so"
> loadmodule "kex.so"
> loadmodule "corex.so"
> loadmodule "tm.so"
> loadmodule "tmx.so"
> loadmodule "sl.so"
> loadmodule "outbound.so"
> loadmodule "rr.so"
> loadmodule "path.so"
> loadmodule "pv.so"
> loadmodule "maxfwd.so"
> loadmodule "usrloc.so"
> loadmodule "registrar.so"
> loadmodule "sdpops.so"
> loadmodule "textops.so"
> loadmodule "textopsx.so"
> loadmodule "siputils.so"
> loadmodule "xlog.so"
> loadmodule "sanity.so"
> loadmodule "ctl.so"
> loadmodule "cfg_rpc.so"
> loadmodule "mi_rpc.so"
> loadmodule "dialog.so"
> loadmodule "acc.so"
> loadmodule "uac.so"
> loadmodule "rtimer.so"
> loadmodule "sqlops.so"
> loadmodule "ndb_redis.so"
> loadmodule "app_perl.so"
> loadmodule "permissions.so"
> loadmodule "domain.so"
> loadmodule "async.so"
> loadmodule "stun.so"
> loadmodule "auth.so"
> loadmodule "auth_db.so"
> loadmodule "alias_db.so"
> loadmodule "speeddial.so"
> loadmodule "presence.so"
> loadmodule "presence_mwi.so"
> loadmodule "presence_xml.so"
> loadmodule "presence_profile.so"
> loadmodule "nathelper.so"
> loadmodule "rtpengine.so"
> loadmodule "tls.so"
> loadmodule "htable.so"
> loadmodule "pike.so"
> loadmodule "xmlrpc.so"
> loadmodule "debugger.so"
> loadmodule "xhttp.so"
> loadmodule "xhttp_rpc.so"
> loadmodule "xhttp_pi.so"
> loadmodule "xcap_server.so"
> loadmodule "pua.so"
> loadmodule "pua_mi.so"
> loadmodule "rls.so"
> loadmodule "cfgutils.so"
> loadmodule "htable.so"
> loadmodule "msrp.so"
> loadmodule "websocket.so"
> loadmodule "msilo.so"
> loadmodule "siptrace.so"
> --
>
> In "top" i see at least 6 kamailio processes using very high cpu
> (perhaps these are the 6 child processes involved in that single
> call processing).
>
> Thank you.
>
>
>
> On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
> I will review the changes pushed to 4.3.2 vs 4.3.1. Can you
> send here the list of modules you are using? The loadmodule
> lines in kamailio.cfg.
>
> Cheers,
> Daniel
>
>
> On 14/09/15 10:51, M S wrote:
>> Hi,
>>
>> Over last weekend i upgraded one of my test servers from
>> Kamailio v4.3.1-4d1b65 to latest stable release v4.3.2-07690f
>> and now kamailio goes crazy even with single call (I am using
>> same db and configuration of course).
>>
>> As soon as call establishes system load average (as seen in
>> top command) starts increasing and soon it increases beyond
>> 6.0 and system becomes completely unresponsive, sip messages
>> are no longer being processed by kamailio service. Even after
>> call hangup, system remains under high load. The "htop"
>> indicates that IO Wait time has increased substantially.
>>
>> Any idea what is causing this? For now i have reverted by to
>> v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need
>> further info or testing.
>>
>> Thank you.
>>
>>
>> _______________________________________________
>> sr-dev mailing list
>> sr-dev at lists.sip-router.org <mailto:sr-dev at lists.sip-router.org>
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
> mailing list
> sr-users at lists.sip-router.org
> <mailto:sr-users at lists.sip-router.org>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20150914/e52c663f/attachment.html>
More information about the sr-users
mailing list