<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list
* http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If you submit a feature request (or enhancement), you can delete the text of the template and only add the description of what you would like to be added.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description
<!-- Explain what you did, what you expected to happen, and what actually happened. -->
We are using lua base routing with app_lua. We are seeing that our memory-utilization on that system continuously getting increase. our route.lua will have all routing logic same as
https://github.com/kamailio/kamailio/blob/master/misc/examples/kemi/kamailio...
we are using following set of modules loadmodule "tm.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "maxfwd.so" loadmodule "nathelper.so" loadmodule "textops.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "path.so" loadmodule "jsonrpcs.so" loadmodule "app_lua.so" loadmodule "dispatcher.so" loadmodule "json.so" loadmodule "cfg_rpc.so" loadmodule "ctl.so" loadmodule "ipops.so" loadmodule "http_async_client.so" loadmodule "dialog.so"
# ----------------- setting module-specific parameters ---------------
# ----- rr params ----- # set next param to 1 to add value to ;lr param (helps with some UAs) modparam("rr", "enable_full_lr", 0) # do not append from tag to the RR (no need for this script) modparam("rr", "append_fromtag", 1)
# ----- dispatcher params ----- modparam("dispatcher", "list_file", DISPATCHER_FILE) modparam("dispatcher", "ds_probing_mode", 1) modparam("dispatcher", "flags", 2) modparam("dispatcher", "ds_ping_interval", 10) modparam("dispatcher", "ds_ping_from", "sip:ping@test.com")
# ----- nathelper params ---- modparam("nathelper", "received_avp", "$avp(s:rcv)")
# ----- http_async_client ------- # ---- number of worker process for async req and res modparam("http_async_client", "workers", 4)
# ----- ctl ------ modparam("ctl", "binrpc", "tcp:MY_IP_ADDR:2086")
# ----- app_lua params ---- modparam("app_lua", "reload", 1) modparam("app_lua", "load", ROUTE_LUA)
# ---- dialog params ---- modparam("dialog", "enable_stats", 1) modparam("dialog", "dlg_flag", 4) modparam("dialog", "event_callback", "ksr_dialog_event")
cfgengine "lua"
### Troubleshooting
#### Reproduction
We can easily reproduce by loading app_lua with kamailio-basic-kemi-lua.lua
https://github.com/kamailio/kamailio/blob/master/misc/examples/kemi/kamailio... <!-- If the issue can be reproduced, describe how it can be done. -->
#### Debugging Data
root@ip-172-31-4-160:~# kamcmd -s tcp:172.31.4.160:2046 tm.stats { current: 2220 waiting: 2198 total: 6113169 total_local: 1371 rpl_received: 11958486 rpl_generated: 6111711 rpl_sent: 12224635 6xx: 0 5xx: 6104789 4xx: 6831 3xx: 0 2xx: 1483 created: 6113169 freed: 6110949 delayed_free: 0 }
transaction are getting free as they are created so no transaction memory leak.
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` U 54.183.198.87:5060 -> 172.31.4.160:5060 INVITE sip:919980950111@13.57.122.212:5060 SIP/2.0. Record-Route: sip:54.183.198.87;lr;ftag=2186327. Via: SIP/2.0/UDP 54.183.198.87:5060;branch=z9hG4bK62f3.22dfb9a43503cc12dafdcdbc9ab1e17d.0. Via: SIP/2.0/UDP 172.31.13.202:5060;rport=5060;received=13.232.78.75;branch=z9hG4bK-20254-2186327-0. From: sip:919980950111@21979254854757081.zt.plivo.com;tag=2186327. To: sip:919980950111@21979254854757081.zt.plivo.com. Call-ID: 2186327-20254@172.31.13.202. CSeq: 1 INVITE. Contact: sip:sipp@13.126.252.148:5060;transport=UDP;alias=13.232.78.75~5060~1. Max-Forwards: 69. User-Agent: Sipp/Linux. Supported: replaces, outbound. Content-Type: application/sdp. Content-Length: 139. X-ZT-Cust-IP: 13.232.78.75. X-ZT-Trunk: 21979254854757081.zt.plivo.com. . v=0. o=user1 53655765 2353687637 IN IP4 13.126.252.148. s=-. c=IN IP4 13.126.252.148. t=0 0. m=audio 6000 RTP/AVP 0. a=rtpmap:0 PCMU/8000.
# U 172.31.4.160:5060 -> 54.183.198.87:5060 SIP/2.0 100 trying -- your call is important to us. Via: SIP/2.0/UDP 54.183.198.87:5060;branch=z9hG4bK62f3.22dfb9a43503cc12dafdcdbc9ab1e17d.0;rport=5060. Via: SIP/2.0/UDP 172.31.13.202:5060;rport=5060;received=13.232.78.75;branch=z9hG4bK-20254-2186327-0. From: sip:919980950111@21979254854757081.zt.plivo.com;tag=2186327. To: sip:919980950111@21979254854757081.zt.plivo.com. Call-ID: 2186327-20254@172.31.13.202. CSeq: 1 INVITE. Server: kamailio (5.1.3 (x86_64/linux)). Content-Length: 0. .
# U 172.31.4.160:5060 -> 54.183.198.87:5060 SIP/2.0 408 Request Timeout. Via: SIP/2.0/UDP 54.183.198.87:5060;branch=z9hG4bK223b.cd9c3cb1d2dc6adc346e6fb7acdee708.0;rport=5060. Via: SIP/2.0/UDP 172.31.13.202:5060;rport=5060;received=13.232.78.75;branch=z9hG4bK-20254-2185727-0. From: sip:919980950111@21979254854757081.zt.plivo.com;tag=2185727. To: sip:919980950111@21979254854757081.zt.plivo.com;tag=c99e9820d9036d36e5541ff825b95248-8c22. Call-ID: 2185727-20254@172.31.13.202. CSeq: 1 INVITE. Server: kamailio (5.1.3 (x86_64/linux)). Content-Length: 0. .
```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` Jun 29 19:18:14 ip-172-31-4-160 kamailio[8119]: ERROR: <core> [core/mem/q_malloc.c:287]: qm_find_free(): qm_find_free(0x7fe974b24000, 35992); Free fragment not found! Jun 29 19:18:14 ip-172-31-4-160 kamailio[8119]: ERROR: <core> [core/mem/q_malloc.c:421]: qm_malloc(): qm_malloc(0x7fe974b24000, 35992) called from http_async_client: http_multi.c: curl_shm_calloc(346), module: http_async_client; Free fragment not found! Jun 29 19:18:14 ip-172-31-4-160 kamailio[8119]: ERROR: http_async_client [http_multi.c:437]: new_request(): curl_easy_init() failed! ```
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` (paste your sip traffic here) ```
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` (paste your output here) ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) -->
``` root@ip-172-31-4-160:/etc/kamailio# uname -a Linux ip-172-31-4-160 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ```
@miconda rightnow our memory utilization are 97%.
script to memory count for kamailio process ``` #!/bin/bash free echo "----------------------------------------" for line in $(pgrep -f '/usr/local/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio.pid -m 16 -M 8 -u root -g root') do pms=$(pmap -x $line|grep total) echo "$pms pid:$line" done ```
more log for pkg_memory ``` 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: sr_lua_load_script(136) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: sr_lua_reload_script(451) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/counters.c: init_counters(116) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/counters.c: init_counters(122) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/dset.c: init_dst_set(83) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/modparam.c: set_mod_param_regex(121) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/name_alias.h: add_alias(91) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/name_alias.h: add_alias(93) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/nonsip_hooks.c: init_nonsip_hooks(43) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_init_buffer(2045) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/route.c: route_new_list(200) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/rpc_lookup.c: rpc_hash_add(146) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_hostname(1301) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_socket_list(1509) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_sock_str(420) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_sock_str(437) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(230) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(235) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(252) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(264) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(273) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/sr_module.c: init_modules(1026) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/timer_proc.c: sr_wtimer_init(308) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/udp_server.c: udp_rcv_loop(436) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_fifo.c: jsonrpc_fifo_mod_init(525) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_fifo.c: jsonrpc_init_fifo_server(123) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_sock.c: jsonrpc_dgram_mod_init(182) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from pv: pv_shv.c: pv_get_shvar(362) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from xlog: xlog.c: mod_init(214) 1 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: dumping all alloc'ed. fragments: 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: sr_lua_load_script(136) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: sr_lua_reload_script(451) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/counters.c: init_counters(116) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/counters.c: init_counters(122) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/dset.c: init_dst_set(83) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/modparam.c: set_mod_param_regex(121) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/name_alias.h: add_alias(91) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/name_alias.h: add_alias(93) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/nonsip_hooks.c: init_nonsip_hooks(43) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_init_buffer(2045) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/route.c: route_new_list(200) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/rpc_lookup.c: rpc_hash_add(146) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_hostname(1301) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_socket_list(1509) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_sock_str(420) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: fix_sock_str(437) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(230) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(235) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(252) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(264) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/socket_info.c: new_sock_info(273) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/sr_module.c: init_modules(1026) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/timer_proc.c: sr_wtimer_init(308) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/udp_server.c: udp_rcv_loop(436) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_fifo.c: jsonrpc_fifo_mod_init(525) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_fifo.c: jsonrpc_init_fifo_server(123) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from jsonrpcs: jsonrpcs_sock.c: jsonrpc_dgram_mod_init(182) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from pv: pv_shv.c: pv_get_shvar(362) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from xlog: xlog.c: mod_init(214) 1 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: dumping all alloc'ed. fragments: 2 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/cfg.y: yyparse(1732) 2 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/cfg.y: yyparse(642) 2 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/modparam.c: set_mod_param_regex(110) 2 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/re.c: subst_str(514) 2 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/select.c: register_select_table(458) 2 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: t_hooks.c: insert_tmcb(137) 2 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: t_msgbuilder.c: build_uac_req(1592) 2 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/cfg.y: yyparse(1732) 2 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/cfg.y: yyparse(642) 2 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/modparam.c: set_mod_param_regex(110) 2 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/re.c: subst_str(514) 2 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/select.c: register_select_table(458) 4 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/counters.c: grp_hash_add(234) 4 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/ppcfg.c: pp_subst_add(57) 4 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/re.c: subst_parser(274) 4 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/re.c: subst_parser(291) 4 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/re.c: subst_parser(301) 4 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/script_cb.c: add_callback(59) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(81) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from app_lua: app_lua_api.c: lua_sr_alloc_script_ver(88) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from app_lua: ../../core/lock_alloc.h: lock_set_alloc(70) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: cfg_clone_global(625) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: cfg_shmize(217) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: sr_cfg_init(322) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: sr_cfg_init(334) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: sr_cfg_init(346) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: sr_cfg_init(353) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: sr_cfg_init(360) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/counters.c: counters_prefork_init(207) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/dns_cache.c: init_dns_cache(345) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/dns_cache.c: init_dns_cache(354) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/dns_cache.c: init_dns_cache(361) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/dns_cache.c: init_dns_cache(369) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/dst_blacklist.c: init_dst_blacklist(430) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/dst_blacklist.c: init_dst_blacklist(437) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/kemi.c: sr_kemi_cbname_list_init(1433) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/kemi.c: sr_kemi_cbname_list_init(1443) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/kemi.c: sr_kemi_cbname_list_init(1452) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/mem/q_malloc.c: qm_shm_lock_init(1182) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/pt.c: init_pt(107) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/pt.c: init_pt(108) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/pt.c: init_pt(113) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/tcp_main.c: init_tcp(4654) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/tcp_main.c: init_tcp(4666) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/tcp_main.c: init_tcp(4672) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/tcp_main.c: init_tcp(4679) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/tcp_main.c: init_tcp(4686) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/tcp_main.c: init_tcp(4694) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/tcp_main.c: init_tcp(4700) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(207) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(220) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(221) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(238) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(270) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(281) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(282) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(283) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: init_timer(284) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: register_timer(1012) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/usr_avp.c: init_avps(89) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/usr_avp.c: init_avps(90) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from dispatcher: dispatch.c: ds_avl_insert(3064) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from dispatcher: dispatch.c: ds_ping_active_init(118) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from dispatcher: dispatch.c: init_data(218) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from dispatcher: dispatch.c: init_data(226) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from dispatcher: dispatch.c: pack_dest(355) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from dispatcher: dispatch.c: reindex_dests(629) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from dispatcher: dispatcher.c: mod_init(306) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from dispatcher: dispatcher.c: mod_init(308) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from pv: ../../core/lock_alloc.h: lock_set_alloc(70) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from pv: pv_shv.c: add_shvar(170) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from pv: pv_shv.c: add_shvar(177) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from pv: pv_shv.c: set_shvar_value(243) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from sl: sl_funcs.c: sl_startup(83) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from sl: sl_stats.c: init_sl_stats(110) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from sl: sl_stats.c: init_sl_stats_child(125) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: h_table.c: init_hash_table(464) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: t_hooks.c: init_tmcb_lists(72) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: t_hooks.c: init_tmcb_lists(74) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: t_stats.c: init_tm_stats_child(58) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tmx: tmx_pretran.c: tmx_init_pretran_table(90) 4 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: dumping all alloc'ed. fragments: 4 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/counters.c: grp_hash_add(234) 4 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/ppcfg.c: pp_subst_add(57) 4 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/re.c: subst_parser(274) 4 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/re.c: subst_parser(291) 4 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/re.c: subst_parser(301) 4 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/script_cb.c: add_callback(59) 5 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_parse_format(1150) 5 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/route.c: init_rlist(146) 5 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_parse_format(1150) 5 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/route.c: init_rlist(146) 7 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/cfg/cfg.c: cfg_declare(50) 7 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: cfg_new_group(79) 7 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/pvapi.c: tr_table_add(1875) 7 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/cfg/cfg.c: cfg_declare(50) 7 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: cfg_new_group(79) 7 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/pvapi.c: tr_table_add(1875) 8 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/route.c: route_add(124) 8 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_ctx.c: cfg_register_ctx(47) 8 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: cfg_child_cb_new(828) 8 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/cfg/cfg_struct.c: cfg_clone_str(130) 8 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/timer.c: timer_alloc(515) 8 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/route.c: route_add(124) 9 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/str_hash.h: str_hash_alloc(59) 9 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/str_hash.h: str_hash_alloc(59) 10 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_init_buffer(2055) 10 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_init_buffer(2055) 12 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347) 12 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347) 18 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/sr_module.c: sr_cmd_exports_convert(235) 18 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/sr_module.c: sr_cmd_exports_convert(235) 19 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/sr_module.c: load_module(496) 19 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/sr_module.c: register_module(280) 19 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/sr_module.c: load_module(496) 19 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/sr_module.c: register_module(280) 23 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/cfg.lex: pp_define(1767) 23 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/cfg.lex: pp_define(1767) 32 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tmx: tmx_pretran.c: tmx_check_pretran(250) 32 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tmx: tmx_pretran.c: tmx_check_pretran(271) 50 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/counters.c: cnt_hash_add(332) 50 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/counters.c: cnt_hash_add(332) 55 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/cfg.lex: addstr(1400) 55 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/cfg.lex: addstr(1400) 104 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/rpc_lookup.c: rpc_hash_add(100) 104 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: t_reply.c: _reply_light(489) 104 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/rpc_lookup.c: rpc_hash_add(100) 130 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: t_reply.c: relay_reply(1918) 184 ip-172-31-11-179 kamailio[11425]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_table_add(236) 184 ip-172-31-11-179 kamailio[11918]: ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_table_add(236) 234 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/msg_translator.c: build_req_buf_from_sip_req(2162) 234 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/sip_msg_clone.c: msg_lump_cloner(986) 234 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from core: core/sip_msg_clone.c: sip_msg_shm_clone(496) 236 ip-172-31-11-179 kamailio[11438]: ALERT: qm_status: alloc'd from tm: h_table.c: build_cell(320) ```
It is not clear what type of memory leaks? Is it the on managed by kamailio in pkg or shm? Or is the system memory? Because in the shell script you use system's `free`.
To watch kamailio's shm memory do:
``` kamctl stats shmem ```
For pkg:
``` kamctl rpc pkg.stats ```
If it is system memory, then the logs of kamailio related to qm_status are irrelevant.
So once it is clarified what memory type is leaking, then we can provide next steps to analyze the issue.
we are seeing the memory related system increasing to 97% so its might the case that system memory getting increase.
Are you loading and using any Lua modules/libraries?
Or it is just using the KSR module exported by Kamailio? Are you issuing often reloads of lua script?
The way to troubleshoot leak of system memory is to run kamailio via valgrind (http://valgrind.org/). Do not forget you have to track forked processes:
``` valgrind --trace-children=yes ... ```
yeah sure i will run that one. i am using both ways 1. using lua modules 2. without using lua modules both ways i can figure out that system memory getting increases.
@miconda please see this issue.
I do not see any new relevant information attached here, have you run with valgrind as I metioned in my previous comment? It should give the report if there are leaks for system memory.
i used this command /usr/bin/valgrind --tool=memcheck --log-file=vg.log --leak-check=full --trace-children=yes --show-leak-kinds=all -v /usr/local/sbin/kamailio -f /etc/kamailio/kamailio.cfg
[vg.log](https://github.com/kamailio/kamailio/files/2173037/vg.log) Please check the valgrind log.
Did you run and test it long enough that the memory increased a lot? I checked the logs for couple of PIDs and no relevant leak is reported.
Maybe you can make minimal kamailio.cfg and kemi lua script along with a sipp scenario that I can use and reproduce here.
An update -- I pushed a commit to restore the Lua execution stack. Try with latest master and if the issue is still expose, try to give the minimal configs and scripts to reproduce, as I wrote in my previous comment.
Updates: I am running test with master branch. i will report if i found memory leak with simple config so you can reproduce.
this can be reporduced by following lua config kamailio.cfg
``` #!KAMAILIO ####### Defined Values #########
#!substdef "!MY_IP_ADDR!<MY_IP>!g" #!substdef "!MY_EXTERNAL_IP!<MY_EXTERNAL_IP>!g" #!substdef "!MY_UDP_PORT!5060!g" #!substdef "!MY_UDP_ADDR!udp:MY_IP_ADDR:MY_UDP_PORT!g"
# Dispatcher File #!define DISPATCHER_FILE "/etc/kamailio/dispatcher.list"
# Routing File #!define ROUTE_LUA "/etc/kamailio/route.lua"
####### Global Parameters #########
## LOG Levels: 3=DBG, 2=INFO, 1=NOTICE, 0=WARN, -1=ERR #!ifdef WITH_DEBUG debug = 4 log_stderror = yes #!else debug = 2 log_stderror = no #!endif
memdbg = 5 memlog = 5
# log setting log_facility = LOG_LOCAL0
listen = MY_UDP_ADDR advertise MY_EXTERNAL_IP:MY_UDP_PORT
children = 8 async_workers=8
####### Modules Section ######## #!ifdef WITH_SRCPATH mpath = "modules/" #!else mpath = "/usr/local/lib64/kamailio/modules/" #!endif # ------------------ module loading ---------------------------------- loadmodule "tm.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "maxfwd.so" loadmodule "nathelper.so" loadmodule "textops.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "path.so" loadmodule "jsonrpcs.so" loadmodule "app_lua.so" loadmodule "dispatcher.so" loadmodule "json.so" loadmodule "cfg_rpc.so" loadmodule "ctl.so"
# ----------------- setting module-specific parameters ---------------
# ----- rr params ----- # set next param to 1 to add value to ;lr param (helps with some UAs) modparam("rr", "enable_full_lr", 0) # do not append from tag to the RR (no need for this script) modparam("rr", "append_fromtag", 1)
# ----- dispatcher params ----- modparam("dispatcher", "list_file", DISPATCHER_FILE) modparam("dispatcher", "ds_probing_mode", 1) modparam("dispatcher", "flags", 2) modparam("dispatcher", "ds_ping_interval", 10)
modparam("dispatcher", "ds_ping_from", "sip:ping@test.my.com")
# ----- nathelper params ---- modparam("nathelper", "received_avp", "$avp(s:rcv)")
# ----- ctl ------ modparam("ctl", "binrpc", "tcp:MY_IP_ADDR:2046")
# ----- app_lua params ---- modparam("app_lua", "reload", 1) modparam("app_lua", "load", ROUTE_LUA)
# ----- tm params ----- # auto-discard branches from previous serial forking leg modparam("tm", "failure_reply_mode", 3) modparam("tm", "fr_timer", 5000) modparam("tm", "fr_inv_timer", 120000) modparam("tm", "restart_fr_on_each_reply", 1) modparam("tm", "pass_provisional_replies", 1) modparam("tm", "contacts_avp", "tm_contacts"); modparam("tm", "contact_flows_avp", "tm_contact_flows");
cfgengine "lua"
route.lua ```
function ksr_request_route() local request_method = headers.get("$rm") or ""; local user_agent = headers.get("$ua") or ""; KSR.info("recieve request " .. request_method); KSR.exit(); return 1; end
```
So, I assume it is still there in the master branch, after my last commit realted to app_lua. I will try to reproduce here with the first available time frame.
yes i am using master branch only.
@miconda are you able to reproduce it?
@miconda please let me know if this issue is reproduced by the same config at your side or not
Didn't get the chance to look at it, being in a trip to USA (Cluecon). Hope to get back to it soon, after returning.
Thank you for the updates. :-)
@miconda please look into this one. because now a days developer already started to move for kamailio in lua. if there is memory leak then it will be great issue.
Back from traveling, I will get back to this one these days, need to setup the testbed for it.
I could not see any memory use increase when I ran a sipsak flood for more than 10min on a config based on the one you posted on the comment:
* https://github.com/kamailio/kamailio/issues/1577#issuecomment-404720655
I removed dispatcher and nathelper modules, becuase they were not used in the Lua ksr_request_route() at all.
There is no memory usage increase reported by `top` or by the shell script you provided in a previous comment.
Are you sure you tested with latest master that includes the commit eb96593 ?
i will again run the load test with different configs. and will post the result here. yeah i used the 5.1.4 with back-porting this commit to that branch
Any chance to test? I run few more tests and still no leak showed up.
i did not get a chance to test. but surely i will test by end of week. will you please also conclude about using modular approach with kemi lua. https://github.com/sipgate/lua-kamailio/blob/master/src/kamailio/init.lua .
Still i am seeing the memory continuously increasing. I tested with modular approach using app_lua mentioned by sipgate. I will test app_lua with LuaJIT. Also will be trying with one simple lua.
You haven't attached the valgrind output.
I tested with the example you provided at:
* https://github.com/kamailio/kamailio/issues/1577#issuecomment-404720655
and I could not spot any leak after sending thousands of requests.
I am not a Lua expert to comment on others libs/packages/approaches. There can be libs out there in the Lua world that can leak, it is not kamailio's role to fix them. Use kamailio with lua and functions exported by kamailio to lua, only then if there is a leak can be fixed in kamailio.
Closed #1577.
Closing, no follow up in one month, but also the code evolved since the issue was opened. Test with master branch and if still problems, open a new issue here with the details specific for it.
-
Daniel-Constantin Mierla -
Surendra Tiwari