[SR-Users] Kamailio out-of-mem
José Seabra
joseseabra4 at gmail.com
Sun Jul 3 20:35:57 CEST 2016
Hello,
I'm monitoring these testes with "corex.shm_summary" provided by kamcmd.
I'm noticing that the core module - xavp.c is consuming lot of memory and
it is increasing.
result of command corex.shm_summary is:
fm_status: summarizing all alloc'ed. fragments:
fm_status: count= 2 size= 1120 bytes from tm: t_reply.c:
_reply_light(542)
fm_status: count= 2 size= 896 bytes from tm: t_msgbuilder.c:
build_uac_req(1543)
fm_status: count= 2182 size= 231088 bytes from tm: t_fwd.c:
prepare_new_uac(479)
fm_status: count= 12488 size= 12050808 bytes from tm: t_reply.c:
relay_reply(1884)
fm_status: count= 12490 size= 14564656 bytes from core:
msg_translator.c: build_req_buf_from_sip_req(2149)
fm_status: count= 12492 size= 71366536 bytes from tm: h_table.c:
build_cell(317)
fm_status: count= 12490 size= 75086184 bytes from core: sip_msg_clone.c:
sip_msg_shm_clone(494)
fm_status: count= 2 size= 96 bytes from tm: t_hooks.c:
insert_tmcb(137)
fm_status: count= 2182 size= 17800 bytes from tm: t_fwd.c:
prepare_new_uac(524)
fm_status: count= 3 size= 512 bytes from htable: ht_api.c:
ht_cell_new(183)
fm_status: count= 12490 size= 10586712 bytes from core: sip_msg_clone.c:
msg_lump_cloner(978)
fm_status: count= 56508 size= 3457904 bytes from core: usr_avp.c:
create_avp(175)
fm_status: count= 4 size= 600 bytes from tmx: tmx_pretran.c:
tmx_check_pretran(250)
fm_status: count= 4 size= 1008 bytes from usrloc: ucontact.c:
new_ucontact(98)
fm_status: count= 4 size= 1320 bytes from tmx: tmx_pretran.c:
tmx_check_pretran(271)
fm_status: count= 4 size= 144 bytes from usrloc: urecord.c:
new_urecord(65)
fm_status: count= 4 size= 328 bytes from usrloc: urecord.c:
new_urecord(58)
fm_status: count= 24 size= 952 bytes from usrloc: ../../ut.h:
shm_str_dup(723)
fm_status: count= 2182 size= 53040 bytes from tm: t_fwd.c:
prepare_new_uac(509)
fm_status: count=12545409 size=1420184592 bytes from core: xavp.c:
xavp_new_value(94)
fm_status: count= 4 size= 128 bytes from dmq: worker.c:
alloc_job_queue(229)
fm_status: count= 1 size= 13440 bytes from core: counters.c:
counters_prefork_init(207)
fm_status: count= 1 size= 4032 bytes from sl: sl_stats.c:
init_sl_stats_child(125)
fm_status: count= 1 size= 256 bytes from tmx: tmx_pretran.c:
tmx_init_pretran_table(90)
fm_status: count= 1 size= 5376 bytes from tm: t_stats.c:
init_tm_stats_child(60)
fm_status: count= 1 size= 1008 bytes from kex: pkg_stats.c:
pkg_proc_stats_init(79)
fm_status: count= 3 size= 88 bytes from core:
cfg/cfg_struct.c: cfg_clone_str(130)
fm_status: count= 1 size= 744 bytes from core:
cfg/cfg_struct.c: cfg_shmize(217)
fm_status: count= 3 size= 64 bytes from usrloc: udomain.c:
build_stat_name(51)
fm_status: count= 1 size= 40960 bytes from usrloc: udomain.c:
new_udomain(93)
fm_status: count= 1 size= 48 bytes from usrloc: udomain.c:
new_udomain(86)
fm_status: count= 1 size= 16 bytes from usrloc: dlist.c:
new_dlist(573)
fm_status: count= 1 size= 32 bytes from usrloc: dlist.c:
new_dlist(565)
fm_status: count= 1 size= 8 bytes from core: pt.c:
init_pt(110)
fm_status: count= 1 size= 8 bytes from core: pt.c:
init_pt(105)
fm_status: count= 1 size= 2944 bytes from core: pt.c:
init_pt(104)
fm_status: count= 1 size= 32 bytes from usrloc: ul_callback.c:
register_ulcb(94)
fm_status: count= 1 size= 24 bytes from dmq: ../../ut.h:
shm_str_dup(723)
fm_status: count= 1 size= 448 bytes from dmq: dmqnode.c:
build_dmq_node(156)
fm_status: count= 2 size= 168 bytes from dmq: peer.c:
add_peer(67)
fm_status: count= 1 size= 8 bytes from dmq: dmq.c:
mod_init(243)
fm_status: count= 1 size= 96 bytes from dmq: dmq.c:
mod_init(237)
fm_status: count= 1 size= 24 bytes from dmq: dmqnode.c:
init_dmq_node_list(66)
fm_status: count= 1 size= 24 bytes from dmq: peer.c:
init_peer_list(33)
fm_status: count= 1 size= 16 bytes from usrloc: ul_callback.c:
init_ulcb_list(45)
fm_status: count= 1 size= 4112 bytes from usrloc:
../../lock_alloc.h: lock_set_alloc(70)
fm_status: count= 1 size= 8 bytes from carrierroute:
cr_data.c: rule_fixup_recursor(584)
fm_status: count= 5 size= 56 bytes from carrierroute:
../../ut.h: shm_str_dup(723)
fm_status: count= 1 size= 152 bytes from carrierroute:
cr_rule.c: add_route_rule(74)
fm_status: count= 3 size= 240 bytes from core: dtrie.c:
dtrie_insert(157)
fm_status: count= 3 size= 48 bytes from core: dtrie.c:
dtrie_insert(148)
fm_status: count= 1 size= 48 bytes from carrierroute:
cr_rule.c: add_route_flags(225)
fm_status: count= 2 size= 160 bytes from core: dtrie.c:
dtrie_init(60)
fm_status: count= 2 size= 32 bytes from core: dtrie.c:
dtrie_init(51)
fm_status: count= 1 size= 32 bytes from carrierroute:
cr_domain.c: create_domain_data(83)
fm_status: count= 1 size= 8 bytes from carrierroute:
cr_carrier.c: create_carrier_data(59)
fm_status: count= 1 size= 64 bytes from carrierroute:
cr_carrier.c: create_carrier_data(50)
fm_status: count= 1 size= 8 bytes from carrierroute: cr_db.c:
load_route_data_db(295)
fm_status: count= 5 size= 1008 bytes from dispatcher:
dispatch.c: reindex_dests(600)
fm_status: count= 1 size= 48 bytes from carrierroute: cr_db.c:
load_domain_map(182)
fm_status: count= 1 size= 24 bytes from carrierroute: cr_db.c:
load_domain_map(171)
fm_status: count= 1 size= 48 bytes from carrierroute: cr_db.c:
load_carrier_map(126)
fm_status: count= 1 size= 24 bytes from carrierroute: cr_db.c:
load_carrier_map(115)
fm_status: count= 7 size= 168 bytes from dispatcher:
dispatch.c: add_dest2list(350)
fm_status: count= 1 size= 64 bytes from carrierroute:
cr_data.c: reload_route_data(172)
fm_status: count= 1 size= 8 bytes from carrierroute:
cr_data.c: init_route_data(74)
fm_status: count= 5 size= 4200 bytes from dispatcher:
dispatch.c: add_dest2list(324)
fm_status: count= 1 size= 16 bytes from dispatcher:
dispatch.c: init_data(204)
fm_status: count= 1 size= 16 bytes from dispatcher:
dispatch.c: init_data(195)
fm_status: count= 1 size= 8 bytes from dispatcher:
dispatcher.c: mod_init(309)
fm_status: count= 1 size= 8 bytes from dispatcher:
dispatcher.c: mod_init(307)
fm_status: count= 1 size= 8 bytes from dispatcher:
dispatch.c: ds_ping_active_init(102)
fm_status: count= 1 size= 256 bytes from db_text: dbt_lib.c:
dbt_init_cache(81)
fm_status: count= 1 size= 8 bytes from db_text: dbt_lib.c:
dbt_init_cache(69)
fm_status: count= 1 size= 8 bytes from db_text: dbt_lib.c:
dbt_init_cache(54)
fm_status: count= 3 size= 288 bytes from core: timer.c:
register_timer(1011)
fm_status: count= 4 size= 8388608 bytes from htable: ht_api.c:
ht_init_tables(381)
fm_status: count= 1 size= 8 bytes from sl: sl_funcs.c:
sl_startup(83)
fm_status: count= 1 size= 8 bytes from sl: sl_stats.c:
init_sl_stats(110)
fm_status: count= 1 size= 16 bytes from tm: t_hooks.c:
init_tmcb_lists(74)
fm_status: count= 1 size= 16 bytes from tm: t_hooks.c:
init_tmcb_lists(72)
fm_status: count= 1 size= 2097152 bytes from tm: h_table.c:
init_hash_table(467)
fm_status: count= 2 size= 64 bytes from core: cfg/cfg_ctx.c:
cfg_register_ctx(47)
fm_status: count= 1 size= 80 bytes from db_postgres:
../../lock_alloc.h: lock_set_alloc(70)
fm_status: count= 1 size= 8192 bytes from core: tcp_main.c:
init_tcp(4635)
fm_status: count= 1 size= 32768 bytes from core: tcp_main.c:
init_tcp(4629)
fm_status: count= 1 size= 8 bytes from core: tcp_main.c:
init_tcp(4621)
fm_status: count= 1 size= 8 bytes from core: tcp_main.c:
init_tcp(4614)
fm_status: count= 1 size= 8 bytes from core: tcp_main.c:
init_tcp(4607)
fm_status: count= 1 size= 8 bytes from core: tcp_main.c:
init_tcp(4601)
fm_status: count= 1 size= 8 bytes from core: tcp_main.c:
init_tcp(4589)
fm_status: count= 1 size= 8 bytes from core: usr_avp.c:
init_avps(90)
fm_status: count= 1 size= 8 bytes from core: usr_avp.c:
init_avps(89)
fm_status: count= 1 size= 16384 bytes from core: dst_blacklist.c:
init_dst_blacklist(437)
fm_status: count= 1 size= 8 bytes from core: dst_blacklist.c:
init_dst_blacklist(430)
fm_status: count= 2 size= 96 bytes from core: timer.c:
timer_alloc(514)
fm_status: count= 1 size= 8 bytes from core: dns_cache.c:
init_dns_cache(366)
fm_status: count= 1 size= 16384 bytes from core: dns_cache.c:
init_dns_cache(358)
fm_status: count= 1 size= 16 bytes from core: dns_cache.c:
init_dns_cache(351)
fm_status: count= 1 size= 8 bytes from core: dns_cache.c:
init_dns_cache(345)
fm_status: count= 1 size= 8 bytes from core: timer.c:
init_timer(283)
fm_status: count= 1 size= 16384 bytes from core: timer.c:
init_timer(282)
fm_status: count= 1 size= 8 bytes from core: timer.c:
init_timer(281)
fm_status: count= 1 size= 8 bytes from core: timer.c:
init_timer(280)
fm_status: count= 1 size= 8 bytes from core: timer.c:
init_timer(269)
fm_status: count= 1 size= 8 bytes from core: timer.c:
init_timer(237)
fm_status: count= 1 size= 278544 bytes from core: timer.c:
init_timer(220)
fm_status: count= 1 size= 8 bytes from core: timer.c:
init_timer(219)
fm_status: count= 1 size= 8 bytes from core: timer.c:
init_timer(206)
fm_status: count= 1 size= 64 bytes from core:
cfg/cfg_struct.c: cfg_child_cb_new(830)
fm_status: count= 1 size= 8 bytes from core:
cfg/cfg_struct.c: sr_cfg_init(361)
fm_status: count= 1 size= 8 bytes from core:
cfg/cfg_struct.c: sr_cfg_init(354)
fm_status: count= 1 size= 8 bytes from core:
cfg/cfg_struct.c: sr_cfg_init(347)
fm_status: count= 1 size= 8 bytes from core:
cfg/cfg_struct.c: sr_cfg_init(335)
fm_status: count= 1 size= 8 bytes from core:
cfg/cfg_struct.c: sr_cfg_init(323)
fm_status: count= 4 size= 928 bytes from htable: ht_api.c:
ht_add_table(278)
fm_status: count= 1 size= 8 bytes from core: mem/shm.c:
shm_core_lock_init(153)
fm_status: -----------------------------
If i stop these tests and after few minutes, run again the command
"corex.shm_summary" the entry that contains the memory consume for xavp.c
doesn't decrease.
It seems that there is a memory leak in this module xavp.c.
Thank you for your help.
Regards
José
2016-07-02 11:45 GMT+01:00 José Seabra <joseseabra4 at gmail.com>:
> I'm running burning tests against kamailio and after some time ~ 2 days I
> see kamailio out of mem:
>
> ERROR: tm [t_lookup.c:1271]: new_t(): ERROR: new_t: out of mem:
> ERROR: tm [t_lookup.c:1411]: t_newtran(): ERROR: t_newtran: new_t failed
>
> The result of command kamcmd mod.stats all is:
>
> Module: core
> {
> sip_msg_shm_clone(494): 10653392
> build_req_buf_from_sip_req(2149): 2565480
> msg_lump_cloner(978): 2943584
> create_avp(175): 969736
> xavp_new_value(94): 858469320
> counters_prefork_init(207): 16896
> cfg_clone_str(130): 88
> cfg_shmize(217): 744
> init_pt(110): 8
> init_pt(105): 8
> init_pt(104): 3080
> dtrie_insert(157): 240
> dtrie_insert(148): 48
> dtrie_init(60): 160
> dtrie_init(51): 32
> register_timer(1011): 360
> cfg_register_ctx(47): 32
> init_tcp(4635): 8192
> init_tcp(4629): 32768
> init_tcp(4621): 8
> init_tcp(4614): 8
> init_tcp(4607): 8
> init_tcp(4601): 8
> init_tcp(4589): 8
> init_avps(90): 8
> init_avps(89): 8
> init_dst_blacklist(437): 16384
> init_dst_blacklist(430): 8
> timer_alloc(514): 96
> init_dns_cache(366): 8
> init_dns_cache(358): 16384
> init_dns_cache(351): 16
> init_dns_cache(345): 8
> init_timer(283): 8
> init_timer(282): 16384
> init_timer(281): 8
> init_timer(280): 8
> init_timer(269): 8
> init_timer(237): 8
> init_timer(220): 278544
> init_timer(219): 8
> init_timer(206): 8
> cfg_child_cb_new(830): 64
> sr_cfg_init(361): 8
> sr_cfg_init(354): 8
> sr_cfg_init(347): 8
> sr_cfg_init(335): 8
> sr_cfg_init(323): 8
> shm_core_lock_init(153): 8
> Total: 875992224
> }
>
> Module: dmq
> {
> alloc_job_queue(229): 128
> shm_str_dup(723): 24
> build_dmq_node(156): 448
> add_peer(67): 168
> mod_init(243): 8
> mod_init(237): 96
> init_dmq_node_list(66): 24
> init_peer_list(33): 24
> Total: 920
> }
>
> Module: auth_db
> {
> Total: 0
> }
>
> Module: dialog
> {
> init_dlg_table(284): 131088
> init_dlg_table(276): 8
> init_dlg_table(269): 8
> init_dlg_table(264): 8
> init_dlg_timer(57): 8
> init_dlg_timer(48): 32
> new_dlg_profile(187): 624
> Total: 131776
> }
>
> Module: rr
> {
> Total: 0
> }
>
> Module: xlog
> {
> Total: 0
> }
>
> Module: kex
> {
> pkg_proc_stats_init(79): 1056
> Total: 1056
> }
>
> Module: db_postgres
> {
> lock_set_alloc(70): 80
> Total: 80
> }
>
> Module: pv
> {
> Total: 0
> }
>
> Module: textops
> {
> Total: 0
> }
>
> Module: htable
> {
> ht_cell_new(183): 1104
> ht_init_tables(381): 8388608
> ht_add_table(278): 928
> Total: 8390640
> }
>
> Module: sqlops
> {
> Total: 0
> }
>
> Module: tm
> {
> build_uac_req(1543): 848
> prepare_new_uac(479): 28880
> _reply_light(542): 751504
> build_local_reparse(341): 1168568
> relay_reply(1884): 643088
> build_cell(317): 9826624
> prepare_new_uac(524): 8112
> prepare_new_uac(509): 10192
> init_tm_stats_child(60): 5632
> insert_tmcb(137): 136
> init_tmcb_lists(74): 16
> init_tmcb_lists(72): 16
> init_hash_table(467): 2097152
> Total: 14540768
> }
>
> Module: usrloc
> {
> new_urecord(65): 136
> new_ucontact(98): 1096
> new_urecord(58): 288
> shm_str_dup(723): 1200
> build_stat_name(51): 64
> new_udomain(93): 40960
> new_udomain(86): 48
> new_dlist(573): 16
> new_dlist(565): 32
> register_ulcb(94): 32
> init_ulcb_list(45): 16
> lock_set_alloc(70): 4112
> Total: 48000
> }
>
> Module: tmx
> {
> tmx_check_pretran(250): 864
> tmx_check_pretran(271): 1216
> tmx_init_pretran_table(90): 256
> Total: 2336
> }
>
> Module: sl
> {
> init_sl_stats_child(125): 4224
> sl_startup(83): 8
> init_sl_stats(110): 8
> Total: 4240
> }
>
> Module: carrierroute
> {
> rule_fixup_recursor(584): 8
> shm_str_dup(723): 56
> add_route_rule(74): 152
> add_route_flags(225): 48
> create_domain_data(83): 32
> create_carrier_data(59): 8
> create_carrier_data(50): 64
> load_route_data_db(295): 8
> load_domain_map(182): 48
> load_domain_map(171): 24
> load_carrier_map(126): 48
> load_carrier_map(115): 24
> reload_route_data(172): 64
> init_route_data(74): 8
> Total: 592
> }
>
> Module: dispatcher
> {
> reindex_dests(600): 1008
> add_dest2list(350): 168
> add_dest2list(324): 4200
> init_data(204): 16
> init_data(195): 16
> mod_init(309): 8
> mod_init(307): 8
> ds_ping_active_init(102): 8
> Total: 5432
> }
>
> Module: db_text
> {
> dbt_init_cache(81): 256
> dbt_init_cache(69): 8
> dbt_init_cache(54): 8
> Total: 272
> }
>
> The result of command kamctl shmem is:
>
> shmem:fragments = 6353
> shmem:free_size = 406208
> shmem:max_used_size = 8464105472
> shmem:real_used_size = 8463699264
> shmem:total_size = 8464105472
> shmem:used_size = 5190698208
>
> The server has 8G of shared memory allocated, and the issue happens more
> or less after 2 days running tests.
>
> I'm using sipp to generate traffic, it is sending 500CAPS.
>
> There is any possibility of memory leak on Kamailio 4.4.0?
>
> Best Regards
>
> --
> José Seabra
>
--
Cumprimentos
José Seabra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20160703/807dbd7d/attachment.html>
More information about the sr-users
mailing list