Re: [sr-dev] [SR-Users] Very high load average after upgrade to Kamailio v4.3.2

OK runtime debugging shows something is wrong with dialog module, all 6 processes are locked in trying to access dialog module when 200 OK is received for the call. For example, here is the BT of one the processes, -- #0 0xb77a0424 in __kernel_vsyscall () #1 0xb76c140c in sched_yield () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 #2 0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277 #3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610 #4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643 #5 0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac) at dlg_handlers.c:429 #6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398, type=2, trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268 #7 0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358, req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295 #8 0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8, p_branch=0xbff3a5a4) at t_lookup.c:966 #9 0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at t_lookup.c:1069 #10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at t_lookup.c:1111 #11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134 #12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at forward.c:747 #13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849 #14 0x08138aab in receive_msg ( buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia: SIP/2.0/UDP 192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"..., len=1103, rcv_info=0xbff3a93c) at receive.c:255 #15 0x082188d0 in udp_rcv_loop () at udp_server.c:495 #16 0x080dec65 in main_loop () at main.c:1573 #17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533 -- Here is BT of second process, -- #0 0xb77a0424 in __kernel_vsyscall () #1 0xb76c140c in sched_yield () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 #2 0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277 #3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610 #4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643 #5 0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072, param=0xbff3a7ac) at dlg_handlers.c:398 #6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464, type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268 #7 0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424, req=0x0, rpl=0x0, code=0) at t_hooks.c:295 #8 0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128 #9 0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c, data=0xa5985424) at timer.c:648 #10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718, slow_l=0xa530e7ec, slow_mark=6) at timer.c:873 #11 0x0820dcc0 in timer_handler () at timer.c:938 #12 0x0820e0d8 in timer_main () at timer.c:977 #13 0x080df4e0 in main_loop () at main.c:1644 #14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533 -- Does this makes any sense to you? Let me know if you need anything else. Thank you. On Mon, Sep 14, 2015 at 11:36 AM, M S &lt;shaheryarkh(a)gmail.com <mailto:shaheryarkh@gmail.com>> wrote: Sure, here is the list of module in given order, -- loadmodule "db_mysql.so" loadmodule "mi_fifo.so" loadmodule "mi_datagram.so" loadmodule "kex.so" loadmodule "corex.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "outbound.so" loadmodule "rr.so" loadmodule "path.so" loadmodule "pv.so" loadmodule "maxfwd.so" loadmodule "usrloc.so" loadmodule "registrar.so" loadmodule "sdpops.so" loadmodule "textops.so" loadmodule "textopsx.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "ctl.so" loadmodule "cfg_rpc.so" loadmodule "mi_rpc.so" loadmodule "dialog.so" loadmodule "acc.so" loadmodule "uac.so" loadmodule "rtimer.so" loadmodule "sqlops.so" loadmodule "ndb_redis.so" loadmodule "app_perl.so" loadmodule "permissions.so" loadmodule "domain.so" loadmodule "async.so" loadmodule "stun.so" loadmodule "auth.so" loadmodule "auth_db.so" loadmodule "alias_db.so" loadmodule "speeddial.so" loadmodule "presence.so" loadmodule "presence_mwi.so" loadmodule "presence_xml.so" loadmodule "presence_profile.so" loadmodule "nathelper.so" loadmodule "rtpengine.so" loadmodule "tls.so" loadmodule "htable.so" loadmodule "pike.so" loadmodule "xmlrpc.so" loadmodule "debugger.so" loadmodule "xhttp.so" loadmodule "xhttp_rpc.so" loadmodule "xhttp_pi.so" loadmodule "xcap_server.so" loadmodule "pua.so" loadmodule "pua_mi.so" loadmodule "rls.so" loadmodule "cfgutils.so" loadmodule "htable.so" loadmodule "msrp.so" loadmodule "websocket.so" loadmodule "msilo.so" loadmodule "siptrace.so" -- In "top" i see at least 6 kamailio processes using very high cpu (perhaps these are the 6 child processes involved in that single call processing). Thank you. On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote: Hello, I will review the changes pushed to 4.3.2 vs 4.3.1. Can you send here the list of modules you are using? The loadmodule lines in kamailio.cfg. Cheers, Daniel On 14/09/15 10:51, M S wrote: -- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users