OK runtime debugging shows something is wrong with dialog module, all 6 processes are locked in trying to access dialog module when 200 OK is received for the call. For example, here is the BT of one the processes,

--
#0 0xb77a0424 in __kernel_vsyscall ()
#1 0xb76c140c in sched_yield () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277
#3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610
#4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643
#5 0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac) at dlg_handlers.c:429
#6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398, type=2, trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268
#7 0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358, req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295
#8 0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8, p_branch=0xbff3a5a4) at t_lookup.c:966
#9 0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at t_lookup.c:1069
#10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at t_lookup.c:1111
#11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134
#12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at forward.c:747
#13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849
#14 0x08138aab in receive_msg (
buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia: SIP/2.0/UDP 192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"..., len=1103, rcv_info=0xbff3a93c) at receive.c:255
#15 0x082188d0 in udp_rcv_loop () at udp_server.c:495
#16 0x080dec65 in main_loop () at main.c:1573
#17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
--

Here is BT of second process,

--
#0 0xb77a0424 in __kernel_vsyscall ()
#1 0xb76c140c in sched_yield () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
#2 0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277
#3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610
#4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643
#5 0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072, param=0xbff3a7ac) at dlg_handlers.c:398
#6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464, type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268
#7 0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424, req=0x0, rpl=0x0, code=0) at t_hooks.c:295
#8 0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128
#9 0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c, data=0xa5985424) at timer.c:648
#10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718, slow_l=0xa530e7ec, slow_mark=6) at timer.c:873
#11 0x0820dcc0 in timer_handler () at timer.c:938
#12 0x0820e0d8 in timer_main () at timer.c:977
#13 0x080df4e0 in main_loop () at main.c:1644
#14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533
--

Does this makes any sense to you? Let me know if you need anything else.

Thank you.

On Mon, Sep 14, 2015 at 11:36 AM, M S <shaheryarkh@gmail.com> wrote:

Sure, here is the list of module in given order,

--
loadmodule "db_mysql.so"
loadmodule "mi_fifo.so"
loadmodule "mi_datagram.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "outbound.so"
loadmodule "rr.so"
loadmodule "path.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "usrloc.so"
loadmodule "registrar.so"
loadmodule "sdpops.so"
loadmodule "textops.so"
loadmodule "textopsx.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dialog.so"
loadmodule "acc.so"
loadmodule "uac.so"
loadmodule "rtimer.so"
loadmodule "sqlops.so"
loadmodule "ndb_redis.so"
loadmodule "app_perl.so"
loadmodule "permissions.so"
loadmodule "domain.so"
loadmodule "async.so"
loadmodule "stun.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "alias_db.so"
loadmodule "speeddial.so"
loadmodule "presence.so"
loadmodule "presence_mwi.so"
loadmodule "presence_xml.so"
loadmodule "presence_profile.so"
loadmodule "nathelper.so"
loadmodule "rtpengine.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
loadmodule "xmlrpc.so"
loadmodule "debugger.so"
loadmodule "xhttp.so"
loadmodule "xhttp_rpc.so"
loadmodule "xhttp_pi.so"
loadmodule "xcap_server.so"
loadmodule "pua.so"
loadmodule "pua_mi.so"
loadmodule "rls.so"
loadmodule "cfgutils.so"
loadmodule "htable.so"
loadmodule "msrp.so"
loadmodule "websocket.so"
loadmodule "msilo.so"
loadmodule "siptrace.so"
--

In "top" i see at least 6 kamailio processes using very high cpu (perhaps these are the 6 child processes involved in that single call processing).

Thank you.
On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,

I will review the changes pushed to 4.3.2 vs 4.3.1. Can you send here the list of modules you are using? The loadmodule lines in kamailio.cfg.

Cheers,
Daniel

On 14/09/15 10:51, M S wrote:
Hi,

Over last weekend i upgraded one of my test servers from Kamailio v4.3.1-4d1b65 to latest stable release v4.3.2-07690f and now kamailio goes crazy even with single call (I am using same db and configuration of course).

As soon as call establishes system load average (as seen in top command) starts increasing and soon it increases beyond 6.0 and system becomes completely unresponsive, sip messages are no longer being processed by kamailio service. Even after call hangup, system remains under high load. The "htop" indicates that IO Wait time has increased substantially.

Any idea what is causing this? For now i have reverted by to v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need further info or testing.

Thank you.
_______________________________________________
sr-dev mailing list
sr-dev@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users