I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html
Had someone found the solution or faced similar issue ?
Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine.
I will do what as has been advised : enabling pike & check with gdb when it appears.
thibault
Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module?
Cheers, Daniel
On 18/11/15 12:00, Thibault Gueslin wrote:
I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html
Had someone found the solution or faced similar issue ?
Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine.
I will do what as has been advised : enabling pike & check with gdb when it appears.
thibault
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
It seems to appear randomly.
I don't have database access (default config without db) I have a mqueue with rtimer to do asynchronous http request.
I have seen also sometimes that all tcp worker were busy even on a test server with 4 clients.
Regards,
Thibault
Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla miconda@gmail.com a écrit :
Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module?
Cheers, Daniel
On 18/11/15 12:00, Thibault Gueslin wrote: I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html
Had someone found the solution or faced similar issue ?
Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine.
I will do what as has been advised : enabling pike & check with gdb when it appears.
thibault
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
Are you doing the http requests for all sip requests? How many rtimer processes do you have consuming from mqueue?
What do you mean by "all tcp workers were busy"? They used lot of CPU?
Cheers, Daniel
On 19/11/15 01:30, Thibault Gueslin wrote:
It seems to appear randomly.
I don't have database access (default config without db) I have a mqueue with rtimer to do asynchronous http request.
I have seen also sometimes that all tcp worker were busy even on a test server with 4 clients.
Regards,
Thibault
Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> a écrit :
Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module?
Cheers, Daniel
On 18/11/15 12:00, Thibault Gueslin wrote:
I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html
Had someone found the solution or faced similar issue ?
Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine.
I will do what as has been advised : enabling pike & check with gdb when it appears.
thibault
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
I am adding the info in a mqueue. In a single rtimer, I am consuming the mqueue and calling a lua script.
(I have followed the example of twitter).
I have increase the number of tcp_children and I got issue for memory allocation on allocations new tcp connection after a few hours. Restart with SHMEM increased. (but shame:used_size seems to increase)
Regards
2015-11-19 11:55 GMT+01:00 Daniel-Constantin Mierla miconda@gmail.com:
Are you doing the http requests for all sip requests? How many rtimer processes do you have consuming from mqueue?
What do you mean by "all tcp workers were busy"? They used lot of CPU?
Cheers, Daniel
On 19/11/15 01:30, Thibault Gueslin wrote:
It seems to appear randomly.
I don't have database access (default config without db) I have a mqueue with rtimer to do asynchronous http request.
I have seen also sometimes that all tcp worker were busy even on a test server with 4 clients.
Regards,
Thibault
Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla < miconda@gmail.com miconda@gmail.com> a écrit :
Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module?
Cheers, Daniel
On 18/11/15 12:00, Thibault Gueslin wrote:
I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html
Had someone found the solution or faced similar issue ?
Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine.
I will do what as has been advised : enabling pike & check with gdb when it appears.
thibault
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
If you get the shm used size increasing, run following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Do it after some time of running, when you see that the increase has happened.
Then take the logs from syslog related to shm summary and send them over here.
On the other hand, I saw recently some article about latency spikes for tcp:
- https://http2.cloudflare.com/the-story-of-one-latency-spike/
It is about http, but actually relates to tcp setting of os. Maybe you can verify your values in OS.
Cheers, Daniel
On 20/11/15 12:40, Thibault Gueslin wrote:
I am adding the info in a mqueue. In a single rtimer, I am consuming the mqueue and calling a lua script.
(I have followed the example of twitter).
I have increase the number of tcp_children and I got issue for memory allocation on allocations new tcp connection after a few hours. Restart with SHMEM increased. (but shame:used_size seems to increase)
Regards
2015-11-19 11:55 GMT+01:00 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>:
Are you doing the http requests for all sip requests? How many rtimer processes do you have consuming from mqueue? What do you mean by "all tcp workers were busy"? They used lot of CPU? Cheers, Daniel On 19/11/15 01:30, Thibault Gueslin wrote:It seems to appear randomly. I don't have database access (default config without db) I have a mqueue with rtimer to do asynchronous http request. I have seen also sometimes that all tcp worker were busy even on a test server with 4 clients. Regards, Thibault Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> a écrit :Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module? Cheers, Daniel On 18/11/15 12:00, Thibault Gueslin wrote:I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html Had someone found the solution or faced similar issue ? Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine. I will do what as has been advised : enabling pike & check with gdb when it appears. thibault _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
It seems located in tcp connection:
Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: summarizing all alloc'ed. fragments: Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 800 bytes from <core>: cfg/cfg_struct.c: cfg_clone_global(626) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 9320 bytes from usrloc: urecord.c: new_urecord(65) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 250 size= 58000 bytes from usrloc: ucontact.c: new_ucontact(98) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1279 size= 51288 bytes from usrloc: ../../ut.h: shm_str_dup(691) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 2048 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 1088 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 15856 bytes from usrloc: urecord.c: new_urecord(58) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2555 size= 43782480 bytes from <core>: tcp_main.c: tcpconn_new(957) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 12800 bytes from <core>: counters.c: counters_prefork_init(207) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 3840 bytes from sl: sl_stats.c: init_sl_stats_child(125) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 5120 bytes from tm: t_stats.c: init_tm_stats_child(60) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 960 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40960 bytes from usrloc: udomain.c: new_udomain(93) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(556) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(548) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2800 bytes from <core>: pt.c: init_pt(104) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 24 bytes from app_lua: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(83) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from <core>: timer.c: register_timer(1011) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from htable: ht_api.c: ht_init_tables(336) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 65536 bytes from tsilo: ts_hash.c: init_ts_table(105) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8208 bytes from tsilo: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from tsilo: ts_hash.c: init_ts_table(75) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(920) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(914) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(906) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from rtpproxy: rtpproxy.c: insert_rtpp_node(563) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from rtpproxy: rtpproxy.c: get_rtpp_set(535) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from rtpproxy: rtpproxy.c: mod_init(911) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from nathelper: nathelper.c: mod_init(662) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: ul_callback.c: register_ulcb(94) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4635) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4629) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4621) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4614) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4607) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4601) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4589) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dst_blacklist.c: init_dst_blacklist(437) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dst_blacklist.c: init_dst_blacklist(430) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(283) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(281) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(280) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(269) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(237) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(219) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(206) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_child_cb_new(830) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from mqueue: mqueue_api.c: mq_head_add(168) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from htable: ht_api.c: ht_add_table(275) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: -----------------------------
2015-11-20 12:49 GMT+01:00 Daniel-Constantin Mierla miconda@gmail.com:
If you get the shm used size increasing, run following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Do it after some time of running, when you see that the increase has happened.
Then take the logs from syslog related to shm summary and send them over here.
On the other hand, I saw recently some article about latency spikes for tcp:
It is about http, but actually relates to tcp setting of os. Maybe you can verify your values in OS.
Cheers, Daniel
On 20/11/15 12:40, Thibault Gueslin wrote:
I am adding the info in a mqueue. In a single rtimer, I am consuming the mqueue and calling a lua script.
(I have followed the example of twitter).
I have increase the number of tcp_children and I got issue for memory allocation on allocations new tcp connection after a few hours. Restart with SHMEM increased. (but shame:used_size seems to increase)
Regards
2015-11-19 11:55 GMT+01:00 Daniel-Constantin Mierla < miconda@gmail.com miconda@gmail.com>:
Are you doing the http requests for all sip requests? How many rtimer processes do you have consuming from mqueue?
What do you mean by "all tcp workers were busy"? They used lot of CPU?
Cheers, Daniel
On 19/11/15 01:30, Thibault Gueslin wrote:
It seems to appear randomly.
I don't have database access (default config without db) I have a mqueue with rtimer to do asynchronous http request.
I have seen also sometimes that all tcp worker were busy even on a test server with 4 clients.
Regards,
Thibault
Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla miconda@gmail.com a écrit :
Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module?
Cheers, Daniel
On 18/11/15 12:00, Thibault Gueslin wrote:
I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html
Had someone found the solution or faced similar issue ?
Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine.
I will do what as has been advised : enabling pike & check with gdb when it appears.
thibault
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
What is the exact version are you using (kamailio -v)?
How many open connections do you expect to have at same time?
Or get:
kamcmd core.tcp_info
Cheers, Daniel
On 20/11/15 13:54, Thibault Gueslin wrote:
It seems located in tcp connection:
Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: summarizing all alloc'ed. fragments: Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 800 bytes from <core>: cfg/cfg_struct.c: cfg_clone_global(626) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 9320 bytes from usrloc: urecord.c: new_urecord(65) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 250 size= 58000 bytes from usrloc: ucontact.c: new_ucontact(98) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1279 size= 51288 bytes from usrloc: ../../ut.h: shm_str_dup(691) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 2048 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 1088 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 15856 bytes from usrloc: urecord.c: new_urecord(58) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2555 size= 43782480 bytes from <core>: tcp_main.c: tcpconn_new(957) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 12800 bytes from <core>: counters.c: counters_prefork_init(207) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 3840 bytes from sl: sl_stats.c: init_sl_stats_child(125) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 5120 bytes from tm: t_stats.c: init_tm_stats_child(60) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 960 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40960 bytes from usrloc: udomain.c: new_udomain(93) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(556) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(548) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2800 bytes from <core>: pt.c: init_pt(104) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 24 bytes from app_lua: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(83) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from <core>: timer.c: register_timer(1011) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from htable: ht_api.c: ht_init_tables(336) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 65536 bytes from tsilo: ts_hash.c: init_ts_table(105) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8208 bytes from tsilo: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from tsilo: ts_hash.c: init_ts_table(75) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(920) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(914) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(906) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from rtpproxy: rtpproxy.c: insert_rtpp_node(563) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from rtpproxy: rtpproxy.c: get_rtpp_set(535) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from rtpproxy: rtpproxy.c: mod_init(911) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from nathelper: nathelper.c: mod_init(662) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: ul_callback.c: register_ulcb(94) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4635) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4629) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4621) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4614) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4607) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4601) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4589) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dst_blacklist.c: init_dst_blacklist(437) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dst_blacklist.c: init_dst_blacklist(430) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(283) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(281) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(280) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(269) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(237) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(219) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(206) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_child_cb_new(830) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from mqueue: mqueue_api.c: mq_head_add(168) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from htable: ht_api.c: ht_add_table(275) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums:
2015-11-20 12:49 GMT+01:00 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>:
If you get the shm used size increasing, run following commands: kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary Do it after some time of running, when you see that the increase has happened. Then take the logs from syslog related to shm summary and send them over here. On the other hand, I saw recently some article about latency spikes for tcp: - https://http2.cloudflare.com/the-story-of-one-latency-spike/ It is about http, but actually relates to tcp setting of os. Maybe you can verify your values in OS. Cheers, Daniel On 20/11/15 12:40, Thibault Gueslin wrote:I am adding the info in a mqueue. In a single rtimer, I am consuming the mqueue and calling a lua script. (I have followed the example of twitter). I have increase the number of tcp_children and I got issue for memory allocation on allocations new tcp connection after a few hours. Restart with SHMEM increased. (but shame:used_size seems to increase) Regards 2015-11-19 11:55 GMT+01:00 Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>>: Are you doing the http requests for all sip requests? How many rtimer processes do you have consuming from mqueue? What do you mean by "all tcp workers were busy"? They used lot of CPU? Cheers, Daniel On 19/11/15 01:30, Thibault Gueslin wrote:It seems to appear randomly. I don't have database access (default config without db) I have a mqueue with rtimer to do asynchronous http request. I have seen also sometimes that all tcp worker were busy even on a test server with 4 clients. Regards, Thibault Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> a écrit :Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module? Cheers, Daniel On 18/11/15 12:00, Thibault Gueslin wrote:I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html Had someone found the solution or faced similar issue ? Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine. I will do what as has been advised : enabling pike & check with gdb when it appears. thibault _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
I am using a 4.3.3 version built myself from source. (deb package were not released at that time)
I am expecting a few hundred connections. (core.tcp_info or netstat ) Finally I have removed this line: modparam("usrloc", "close_expired_tcp", 1)
It seems it was causing the memory leak.
I am still monitoring everything.
2015-11-20 14:07 GMT+01:00 Daniel-Constantin Mierla miconda@gmail.com:
What is the exact version are you using (kamailio -v)?
How many open connections do you expect to have at same time?
Or get:
kamcmd core.tcp_info
Cheers, Daniel
On 20/11/15 13:54, Thibault Gueslin wrote:
It seems located in tcp connection:
Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: summarizing all alloc'ed. fragments: Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 800 bytes from <core>: cfg/cfg_struct.c: cfg_clone_global(626) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 9320 bytes from usrloc: urecord.c: new_urecord(65) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 250 size= 58000 bytes from usrloc: ucontact.c: new_ucontact(98) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1279 size= 51288 bytes from usrloc: ../../ut.h: shm_str_dup(691) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 2048 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 1088 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 15856 bytes from usrloc: urecord.c: new_urecord(58) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2555 size= 43782480 bytes from <core>: tcp_main.c: tcpconn_new(957) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 12800 bytes from <core>: counters.c: counters_prefork_init(207) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 3840 bytes from sl: sl_stats.c: init_sl_stats_child(125) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 5120 bytes from tm: t_stats.c: init_tm_stats_child(60) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 960 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40960 bytes from usrloc: udomain.c: new_udomain(93) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(556) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(548) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2800 bytes from <core>: pt.c: init_pt(104) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 24 bytes from app_lua: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(83) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from <core>: timer.c: register_timer(1011) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from htable: ht_api.c: ht_init_tables(336) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 65536 bytes from tsilo: ts_hash.c: init_ts_table(105) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8208 bytes from tsilo: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from tsilo: ts_hash.c: init_ts_table(75) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(920) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(914) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(906) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from rtpproxy: rtpproxy.c: insert_rtpp_node(563) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from rtpproxy: rtpproxy.c: get_rtpp_set(535) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from rtpproxy: rtpproxy.c: mod_init(911) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from nathelper: nathelper.c: mod_init(662) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: ul_callback.c: register_ulcb(94) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4635) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4629) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4621) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4614) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4607) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4601) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4589) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dst_blacklist.c: init_dst_blacklist(437) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dst_blacklist.c: init_dst_blacklist(430) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(283) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(281) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(280) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(269) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(237) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(219) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(206) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_child_cb_new(830) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from mqueue: mqueue_api.c: mq_head_add(168) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from htable: ht_api.c: ht_add_table(275) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums:
2015-11-20 12:49 GMT+01:00 Daniel-Constantin Mierla < miconda@gmail.com miconda@gmail.com>:
If you get the shm used size increasing, run following commands:
kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary
Do it after some time of running, when you see that the increase has happened.
Then take the logs from syslog related to shm summary and send them over here.
On the other hand, I saw recently some article about latency spikes for tcp:
It is about http, but actually relates to tcp setting of os. Maybe you can verify your values in OS.
Cheers, Daniel
On 20/11/15 12:40, Thibault Gueslin wrote:
I am adding the info in a mqueue. In a single rtimer, I am consuming the mqueue and calling a lua script.
(I have followed the example of twitter).
I have increase the number of tcp_children and I got issue for memory allocation on allocations new tcp connection after a few hours. Restart with SHMEM increased. (but shame:used_size seems to increase)
Regards
2015-11-19 11:55 GMT+01:00 Daniel-Constantin Mierla miconda@gmail.com:
Are you doing the http requests for all sip requests? How many rtimer processes do you have consuming from mqueue?
What do you mean by "all tcp workers were busy"? They used lot of CPU?
Cheers, Daniel
On 19/11/15 01:30, Thibault Gueslin wrote:
It seems to appear randomly.
I don't have database access (default config without db) I have a mqueue with rtimer to do asynchronous http request.
I have seen also sometimes that all tcp worker were busy even on a test server with 4 clients.
Regards,
Thibault
Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla < miconda@gmail.com miconda@gmail.com> a écrit :
Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module?
Cheers, Daniel
On 18/11/15 12:00, Thibault Gueslin wrote:
I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html
Had someone found the solution or faced similar issue ?
Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine.
I will do what as has been advised : enabling pike & check with gdb when it appears.
thibault
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
On 20/11/15 15:02, Thibault Gueslin wrote:
I am using a 4.3.3 version built myself from source. (deb package were not released at that time)
kamailio -v shows the last git commit id when building from sources, helping to identify exactly the snapshot of the source code that you built.
I am expecting a few hundred connections. (core.tcp_info or netstat ) Finally I have removed this line: modparam("usrloc", "close_expired_tcp", 1)
It seems it was causing the memory leak.
Thanks for troubleshooting and pointing to this, I will investigate that part of the code and see if I catch any reason for a leak there.
I am still monitoring everything.
Let us know the results, would be good to nail it done properly.
Cheers, Daniel
2015-11-20 14:07 GMT+01:00 Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>:
What is the exact version are you using (kamailio -v)? How many open connections do you expect to have at same time? Or get: kamcmd core.tcp_info Cheers, Daniel On 20/11/15 13:54, Thibault Gueslin wrote:It seems located in tcp connection: Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: summarizing all alloc'ed. fragments: Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 800 bytes from <core>: cfg/cfg_struct.c: cfg_clone_global(626) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 9320 bytes from usrloc: urecord.c: new_urecord(65) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 250 size= 58000 bytes from usrloc: ucontact.c: new_ucontact(98) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1279 size= 51288 bytes from usrloc: ../../ut.h: shm_str_dup(691) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 2048 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 8 size= 1088 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 246 size= 15856 bytes from usrloc: urecord.c: new_urecord(58) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2555 size= 43782480 bytes from <core>: tcp_main.c: tcpconn_new(957) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 12800 bytes from <core>: counters.c: counters_prefork_init(207) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 3840 bytes from sl: sl_stats.c: init_sl_stats_child(125) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 5120 bytes from tm: t_stats.c: init_tm_stats_child(60) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 960 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40960 bytes from usrloc: udomain.c: new_udomain(93) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(556) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(548) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2800 bytes from <core>: pt.c: init_pt(104) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 24 bytes from app_lua: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(83) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from <core>: timer.c: register_timer(1011) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from htable: ht_api.c: ht_init_tables(336) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 65536 bytes from tsilo: ts_hash.c: init_ts_table(105) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8208 bytes from tsilo: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from tsilo: ts_hash.c: init_ts_table(75) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(920) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(914) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(906) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from rtpproxy: rtpproxy.c: insert_rtpp_node(563) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 48 bytes from rtpproxy: rtpproxy.c: get_rtpp_set(535) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from rtpproxy: rtpproxy.c: mod_init(911) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from nathelper: nathelper.c: mod_init(662) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32 bytes from usrloc: ul_callback.c: register_ulcb(94) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4635) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4629) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4621) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4614) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4607) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4601) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4589) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dst_blacklist.c: init_dst_blacklist(437) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dst_blacklist.c: init_dst_blacklist(430) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(283) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(281) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(280) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(269) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(237) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(219) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: timer.c: init_timer(206) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_child_cb_new(830) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 72 bytes from mqueue: mqueue_api.c: mq_head_add(168) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 96 bytes from htable: ht_api.c: ht_add_table(275) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179) Nov 20 12:44:54 sip /usr/sbin/kamailio[30722]: NOTICE: qm_sums: ----------------------------- 2015-11-20 12:49 GMT+01:00 Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>>: If you get the shm used size increasing, run following commands: kamcmd cfg.set_now_int core memlog 1 kamcmd corex.shm_summary Do it after some time of running, when you see that the increase has happened. Then take the logs from syslog related to shm summary and send them over here. On the other hand, I saw recently some article about latency spikes for tcp: - https://http2.cloudflare.com/the-story-of-one-latency-spike/ It is about http, but actually relates to tcp setting of os. Maybe you can verify your values in OS. Cheers, Daniel On 20/11/15 12:40, Thibault Gueslin wrote:I am adding the info in a mqueue. In a single rtimer, I am consuming the mqueue and calling a lua script. (I have followed the example of twitter). I have increase the number of tcp_children and I got issue for memory allocation on allocations new tcp connection after a few hours. Restart with SHMEM increased. (but shame:used_size seems to increase) Regards 2015-11-19 11:55 GMT+01:00 Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>>: Are you doing the http requests for all sip requests? How many rtimer processes do you have consuming from mqueue? What do you mean by "all tcp workers were busy"? They used lot of CPU? Cheers, Daniel On 19/11/15 01:30, Thibault Gueslin wrote:It seems to appear randomly. I don't have database access (default config without db) I have a mqueue with rtimer to do asynchronous http request. I have seen also sometimes that all tcp worker were busy even on a test server with 4 clients. Regards, Thibault Le 18 nov. 2015 à 13:28, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> a écrit :Is it something repeating at constant interval of times, or are they random? Are you doing database queries in your config? Any of them via rtimer module? Cheers, Daniel On 18/11/15 12:00, Thibault Gueslin wrote:I am facing an issue with kamailio 4.3.3 that is very similar to http://lists.sip-router.org/pipermail/sr-users/2015-May/088195.html Had someone found the solution or faced similar issue ? Sometimes kamailio is receiving the message (check with wireshark, logs in tcp) but the message is not handled nor relayed. After a certain amount of time everything works fine. I will do what as has been advised : enabling pike & check with gdb when it appears. thibault _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
Le Fri, 20 Nov 2015 16:02:26 +0100, Daniel-Constantin Mierla miconda@gmail.com a écrit :
Finally I have removed this line: modparam("usrloc", "close_expired_tcp", 1)
It seems it was causing the memory leak.
Thanks for troubleshooting and pointing to this, I will investigate that part of the code and see if I catch any reason for a leak there.
Hi,
this close_expired_tcp option did not seem to cause any obvious trouble so far, but I wonder if the code I wrote to force-close the connection (the worker sends a message via the internal control socket to the main TCP process) can be replaced with what has been done in the websocket module's ws_conn.c wsconn_close_now() function, that looks to use only safe operations. The side effect if I understand correctly would be to raise a little bit the TCP socket close delay (max 5 seconds?), but it's definitely worth it if this method is safer.
On 20/11/15 17:01, Camille Oudot wrote:
Le Fri, 20 Nov 2015 16:02:26 +0100, Daniel-Constantin Mierla miconda@gmail.com a écrit :
Finally I have removed this line: modparam("usrloc", "close_expired_tcp", 1)
It seems it was causing the memory leak.
Thanks for troubleshooting and pointing to this, I will investigate that part of the code and see if I catch any reason for a leak there.
Hi,
this close_expired_tcp option did not seem to cause any obvious trouble so far, but I wonder if the code I wrote to force-close the connection (the worker sends a message via the internal control socket to the main TCP process) can be replaced with what has been done in the websocket module's ws_conn.c wsconn_close_now() function, that looks to use only safe operations. The side effect if I understand correctly would be to raise a little bit the TCP socket close delay (max 5 seconds?), but it's definitely worth it if this method is safer.
although I haven't had the time to dig in the code, I would second the option with safe operations for closing, in this way it follows the typically flow. Also the tm uses the wait timer for last unref, so everything emerges to a single system of destroying the structure.
Cheers, Daniel
Le Fri, 20 Nov 2015 15:02:24 +0100, Thibault Gueslin thibault.gueslin@gmail.com a écrit :
Finally I have removed this line: modparam("usrloc", "close_expired_tcp", 1)
It seems it was causing the memory leak.
Hi Thibault,
the latest commit on master and 4.3 branch should fix this memory leak. Thanks for spotting the issue.
Good I will apply again this parameter.
2015-11-25 12:33 GMT+01:00 Camille Oudot camille.oudot@orange.com:
Le Fri, 20 Nov 2015 15:02:24 +0100, Thibault Gueslin thibault.gueslin@gmail.com a écrit :
Finally I have removed this line: modparam("usrloc", "close_expired_tcp", 1)
It seems it was causing the memory leak.
Hi Thibault,
the latest commit on master and 4.3 branch should fix this memory leak. Thanks for spotting the issue.
-- Camille
-
Camille Oudot -
Daniel-Constantin Mierla -
Thibault Gueslin