[SR-Users] [sr-dev] Very high load average after upgrade to Kamailio v4.3.2

Daniel-Constantin Mierla miconda at gmail.com
Mon Sep 14 12:18:31 CEST 2015


I found a regression when detecting spiraled dialogs with state deleted.
That would be the case when you create a dialog for an INVITE, but you
reply with a negative response code from kamailio and then quickly a new
invite with same callid and from tag arrives.

Can you upgrade to latest version in branch 4.3, try and see if works
fine now?

Cheers,
Daniel

On 14/09/15 11:55, M S wrote:
> OK runtime debugging shows something is wrong with dialog module, all
> 6 processes are locked in trying to access dialog module when 200 OK
> is received for the call. For example, here is the BT of one the
> processes,
>
> --
> #0  0xb77a0424 in __kernel_vsyscall ()
> #1  0xb76c140c in sched_yield () from
> /lib/i386-linux-gnu/i686/cmov/libc.so.6
> #2  0xb5a20206 in get_lock (lock=0xa55058e4) at
> ../../mem/../fastlock.h:277
> #3  0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610
> #4  0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643
> #5  0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac)
> at dlg_handlers.c:429
> #6  0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398,
> type=2, trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268
> #7  0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358,
> req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295
> #8  0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8,
> p_branch=0xbff3a5a4) at t_lookup.c:966
> #9  0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8,
> param_branch=0xbff3a5a4) at t_lookup.c:1069
> #10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4)
> at t_lookup.c:1111
> #11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134
> #12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at
> forward.c:747
> #13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849
> #14 0x08138aab in receive_msg (
>     buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
> X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia:
> SIP/2.0/UDP
> 192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"...,
> len=1103, rcv_info=0xbff3a93c) at receive.c:255
> #15 0x082188d0 in udp_rcv_loop () at udp_server.c:495
> #16 0x080dec65 in main_loop () at main.c:1573
> #17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
> --
>
> Here is BT of second process,
>
> --
> #0  0xb77a0424 in __kernel_vsyscall ()
> #1  0xb76c140c in sched_yield () from
> /lib/i386-linux-gnu/i686/cmov/libc.so.6
> #2  0xb5a20206 in get_lock (lock=0xa55058e4) at
> ../../mem/../fastlock.h:277
> #3  0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610
> #4  0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643
> #5  0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072,
> param=0xbff3a7ac) at dlg_handlers.c:398
> #6  0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464,
> type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268
> #7  0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424,
> req=0x0, rpl=0x0, code=0) at t_hooks.c:295
> #8  0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128
> #9  0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c,
> data=0xa5985424) at timer.c:648
> #10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718,
> slow_l=0xa530e7ec, slow_mark=6) at timer.c:873
> #11 0x0820dcc0 in timer_handler () at timer.c:938
> #12 0x0820e0d8 in timer_main () at timer.c:977
> #13 0x080df4e0 in main_loop () at main.c:1644
> #14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
> --
>
> Does this makes any sense to you? Let me know if you need anything else.
>
> Thank you.
>
>
>
> On Mon, Sep 14, 2015 at 11:36 AM, M S <shaheryarkh at gmail.com
> <mailto:shaheryarkh at gmail.com>> wrote:
>
>     Sure, here is the list of module in given order,
>
>     --
>     loadmodule "db_mysql.so"
>     loadmodule "mi_fifo.so"
>     loadmodule "mi_datagram.so"
>     loadmodule "kex.so"
>     loadmodule "corex.so"
>     loadmodule "tm.so"
>     loadmodule "tmx.so"
>     loadmodule "sl.so"
>     loadmodule "outbound.so"
>     loadmodule "rr.so"
>     loadmodule "path.so"
>     loadmodule "pv.so"
>     loadmodule "maxfwd.so"
>     loadmodule "usrloc.so"
>     loadmodule "registrar.so"
>     loadmodule "sdpops.so"
>     loadmodule "textops.so"
>     loadmodule "textopsx.so"
>     loadmodule "siputils.so"
>     loadmodule "xlog.so"
>     loadmodule "sanity.so"
>     loadmodule "ctl.so"
>     loadmodule "cfg_rpc.so"
>     loadmodule "mi_rpc.so"
>     loadmodule "dialog.so"
>     loadmodule "acc.so"
>     loadmodule "uac.so"
>     loadmodule "rtimer.so"
>     loadmodule "sqlops.so"
>     loadmodule "ndb_redis.so"
>     loadmodule "app_perl.so"
>     loadmodule "permissions.so"
>     loadmodule "domain.so"
>     loadmodule "async.so"
>     loadmodule "stun.so"
>     loadmodule "auth.so"
>     loadmodule "auth_db.so"
>     loadmodule "alias_db.so"
>     loadmodule "speeddial.so"
>     loadmodule "presence.so"
>     loadmodule "presence_mwi.so"
>     loadmodule "presence_xml.so"
>     loadmodule "presence_profile.so"
>     loadmodule "nathelper.so"
>     loadmodule "rtpengine.so"
>     loadmodule "tls.so"
>     loadmodule "htable.so"
>     loadmodule "pike.so"
>     loadmodule "xmlrpc.so"
>     loadmodule "debugger.so"
>     loadmodule "xhttp.so"
>     loadmodule "xhttp_rpc.so"
>     loadmodule "xhttp_pi.so"
>     loadmodule "xcap_server.so"
>     loadmodule "pua.so"
>     loadmodule "pua_mi.so"
>     loadmodule "rls.so"
>     loadmodule "cfgutils.so"
>     loadmodule "htable.so"
>     loadmodule "msrp.so"
>     loadmodule "websocket.so"
>     loadmodule "msilo.so"
>     loadmodule "siptrace.so"
>     --
>
>     In "top" i see at least 6 kamailio processes using very high cpu
>     (perhaps these are the 6 child processes involved in that single
>     call processing).
>
>     Thank you.
>
>
>
>     On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla
>     <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
>         Hello,
>
>         I will review the changes pushed to 4.3.2 vs 4.3.1. Can you
>         send here the list of modules you are using? The loadmodule
>         lines in kamailio.cfg.
>
>         Cheers,
>         Daniel
>
>
>         On 14/09/15 10:51, M S wrote:
>>         Hi,
>>
>>         Over last weekend i upgraded one of my test servers from
>>         Kamailio v4.3.1-4d1b65 to latest stable release v4.3.2-07690f
>>         and now kamailio goes crazy even with single call (I am using
>>         same db and configuration of course).
>>
>>         As soon as call establishes system load average (as seen in
>>         top command) starts increasing and soon it increases beyond
>>         6.0 and system becomes completely unresponsive, sip messages
>>         are no longer being processed by kamailio service. Even after
>>         call hangup, system remains under high load. The "htop"
>>         indicates that IO Wait time has increased substantially.
>>
>>         Any idea what is causing this? For now i have reverted by to
>>         v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need
>>         further info or testing.
>>
>>         Thank you.
>>
>>
>>         _______________________________________________
>>         sr-dev mailing list
>>         sr-dev at lists.sip-router.org <mailto:sr-dev at lists.sip-router.org>
>>         http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
>
>         -- 
>         Daniel-Constantin Mierla
>         http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>         Book: SIP Routing With Kamailio - http://www.asipto.com
>         Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
>         _______________________________________________
>         SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>         mailing list
>         sr-users at lists.sip-router.org
>         <mailto:sr-users at lists.sip-router.org>
>         http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20150914/e52c663f/attachment.html>


More information about the sr-users mailing list