Hello,
I'm monitoring these testes with "corex.shm_summary" provided by kamcmd.

I'm noticing  that the core module  - xavp.c is consuming lot of memory and it is increasing.

result of command corex.shm_summary is:

 fm_status: summarizing all alloc'ed. fragments:
 fm_status:  count=     2 size=      1120 bytes from tm: t_reply.c: _reply_light(542)
 fm_status:  count=     2 size=       896 bytes from tm: t_msgbuilder.c: build_uac_req(1543)
 fm_status:  count=  2182 size=    231088 bytes from tm: t_fwd.c: prepare_new_uac(479)
 fm_status:  count= 12488 size=  12050808 bytes from tm: t_reply.c: relay_reply(1884)
 fm_status:  count= 12490 size=  14564656 bytes from core: msg_translator.c: build_req_buf_from_sip_req(2149)
 fm_status:  count= 12492 size=  71366536 bytes from tm: h_table.c: build_cell(317)
 fm_status:  count= 12490 size=  75086184 bytes from core: sip_msg_clone.c: sip_msg_shm_clone(494)
 fm_status:  count=     2 size=        96 bytes from tm: t_hooks.c: insert_tmcb(137)
 fm_status:  count=  2182 size=     17800 bytes from tm: t_fwd.c: prepare_new_uac(524)
 fm_status:  count=     3 size=       512 bytes from htable: ht_api.c: ht_cell_new(183)
 fm_status:  count= 12490 size=  10586712 bytes from core: sip_msg_clone.c: msg_lump_cloner(978)
 fm_status:  count= 56508 size=   3457904 bytes from core: usr_avp.c: create_avp(175)
 fm_status:  count=     4 size=       600 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250)
 fm_status:  count=     4 size=      1008 bytes from usrloc: ucontact.c: new_ucontact(98)
 fm_status:  count=     4 size=      1320 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271)
 fm_status:  count=     4 size=       144 bytes from usrloc: urecord.c: new_urecord(65)
 fm_status:  count=     4 size=       328 bytes from usrloc: urecord.c: new_urecord(58)
 fm_status:  count=    24 size=       952 bytes from usrloc: ../../ut.h: shm_str_dup(723)
 fm_status:  count=  2182 size=     53040 bytes from tm: t_fwd.c: prepare_new_uac(509)
 fm_status:  count=12545409 size=1420184592 bytes from core: xavp.c: xavp_new_value(94)
 fm_status:  count=     4 size=       128 bytes from dmq: worker.c: alloc_job_queue(229)
 fm_status:  count=     1 size=     13440 bytes from core: counters.c: counters_prefork_init(207)
 fm_status:  count=     1 size=      4032 bytes from sl: sl_stats.c: init_sl_stats_child(125)
 fm_status:  count=     1 size=       256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
 fm_status:  count=     1 size=      5376 bytes from tm: t_stats.c: init_tm_stats_child(60)
 fm_status:  count=     1 size=      1008 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)
 fm_status:  count=     3 size=        88 bytes from core: cfg/cfg_struct.c: cfg_clone_str(130)
 fm_status:  count=     1 size=       744 bytes from core: cfg/cfg_struct.c: cfg_shmize(217)
 fm_status:  count=     3 size=        64 bytes from usrloc: udomain.c: build_stat_name(51)
 fm_status:  count=     1 size=     40960 bytes from usrloc: udomain.c: new_udomain(93)
 fm_status:  count=     1 size=        48 bytes from usrloc: udomain.c: new_udomain(86)
 fm_status:  count=     1 size=        16 bytes from usrloc: dlist.c: new_dlist(573)
 fm_status:  count=     1 size=        32 bytes from usrloc: dlist.c: new_dlist(565)
 fm_status:  count=     1 size=         8 bytes from core: pt.c: init_pt(110)
 fm_status:  count=     1 size=         8 bytes from core: pt.c: init_pt(105)
 fm_status:  count=     1 size=      2944 bytes from core: pt.c: init_pt(104)
 fm_status:  count=     1 size=        32 bytes from usrloc: ul_callback.c: register_ulcb(94)
 fm_status:  count=     1 size=        24 bytes from dmq: ../../ut.h: shm_str_dup(723)
 fm_status:  count=     1 size=       448 bytes from dmq: dmqnode.c: build_dmq_node(156)
 fm_status:  count=     2 size=       168 bytes from dmq: peer.c: add_peer(67)
 fm_status:  count=     1 size=         8 bytes from dmq: dmq.c: mod_init(243)
 fm_status:  count=     1 size=        96 bytes from dmq: dmq.c: mod_init(237)
 fm_status:  count=     1 size=        24 bytes from dmq: dmqnode.c: init_dmq_node_list(66)
 fm_status:  count=     1 size=        24 bytes from dmq: peer.c: init_peer_list(33)
 fm_status:  count=     1 size=        16 bytes from usrloc: ul_callback.c: init_ulcb_list(45)
 fm_status:  count=     1 size=      4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70)
 fm_status:  count=     1 size=         8 bytes from carrierroute: cr_data.c: rule_fixup_recursor(584)
 fm_status:  count=     5 size=        56 bytes from carrierroute: ../../ut.h: shm_str_dup(723)
 fm_status:  count=     1 size=       152 bytes from carrierroute: cr_rule.c: add_route_rule(74)
 fm_status:  count=     3 size=       240 bytes from core: dtrie.c: dtrie_insert(157)
 fm_status:  count=     3 size=        48 bytes from core: dtrie.c: dtrie_insert(148)
 fm_status:  count=     1 size=        48 bytes from carrierroute: cr_rule.c: add_route_flags(225)
 fm_status:  count=     2 size=       160 bytes from core: dtrie.c: dtrie_init(60)
 fm_status:  count=     2 size=        32 bytes from core: dtrie.c: dtrie_init(51)
 fm_status:  count=     1 size=        32 bytes from carrierroute: cr_domain.c: create_domain_data(83)
 fm_status:  count=     1 size=         8 bytes from carrierroute: cr_carrier.c: create_carrier_data(59)
 fm_status:  count=     1 size=        64 bytes from carrierroute: cr_carrier.c: create_carrier_data(50)
 fm_status:  count=     1 size=         8 bytes from carrierroute: cr_db.c: load_route_data_db(295)
 fm_status:  count=     5 size=      1008 bytes from dispatcher: dispatch.c: reindex_dests(600)
 fm_status:  count=     1 size=        48 bytes from carrierroute: cr_db.c: load_domain_map(182)
 fm_status:  count=     1 size=        24 bytes from carrierroute: cr_db.c: load_domain_map(171)
 fm_status:  count=     1 size=        48 bytes from carrierroute: cr_db.c: load_carrier_map(126)
 fm_status:  count=     1 size=        24 bytes from carrierroute: cr_db.c: load_carrier_map(115)
 fm_status:  count=     7 size=       168 bytes from dispatcher: dispatch.c: add_dest2list(350)
 fm_status:  count=     1 size=        64 bytes from carrierroute: cr_data.c: reload_route_data(172)
 fm_status:  count=     1 size=         8 bytes from carrierroute: cr_data.c: init_route_data(74)
 fm_status:  count=     5 size=      4200 bytes from dispatcher: dispatch.c: add_dest2list(324)
 fm_status:  count=     1 size=        16 bytes from dispatcher: dispatch.c: init_data(204)
 fm_status:  count=     1 size=        16 bytes from dispatcher: dispatch.c: init_data(195)
 fm_status:  count=     1 size=         8 bytes from dispatcher: dispatcher.c: mod_init(309)
 fm_status:  count=     1 size=         8 bytes from dispatcher: dispatcher.c: mod_init(307)
 fm_status:  count=     1 size=         8 bytes from dispatcher: dispatch.c: ds_ping_active_init(102)
 fm_status:  count=     1 size=       256 bytes from db_text: dbt_lib.c: dbt_init_cache(81)
 fm_status:  count=     1 size=         8 bytes from db_text: dbt_lib.c: dbt_init_cache(69)
 fm_status:  count=     1 size=         8 bytes from db_text: dbt_lib.c: dbt_init_cache(54)
 fm_status:  count=     3 size=       288 bytes from core: timer.c: register_timer(1011)
 fm_status:  count=     4 size=   8388608 bytes from htable: ht_api.c: ht_init_tables(381)
 fm_status:  count=     1 size=         8 bytes from sl: sl_funcs.c: sl_startup(83)
 fm_status:  count=     1 size=         8 bytes from sl: sl_stats.c: init_sl_stats(110)
 fm_status:  count=     1 size=        16 bytes from tm: t_hooks.c: init_tmcb_lists(74)
 fm_status:  count=     1 size=        16 bytes from tm: t_hooks.c: init_tmcb_lists(72)
 fm_status:  count=     1 size=   2097152 bytes from tm: h_table.c: init_hash_table(467)
 fm_status:  count=     2 size=        64 bytes from core: cfg/cfg_ctx.c: cfg_register_ctx(47)
 fm_status:  count=     1 size=        80 bytes from db_postgres: ../../lock_alloc.h: lock_set_alloc(70)
 fm_status:  count=     1 size=      8192 bytes from core: tcp_main.c: init_tcp(4635)
 fm_status:  count=     1 size=     32768 bytes from core: tcp_main.c: init_tcp(4629)
 fm_status:  count=     1 size=         8 bytes from core: tcp_main.c: init_tcp(4621)
 fm_status:  count=     1 size=         8 bytes from core: tcp_main.c: init_tcp(4614)
 fm_status:  count=     1 size=         8 bytes from core: tcp_main.c: init_tcp(4607)
 fm_status:  count=     1 size=         8 bytes from core: tcp_main.c: init_tcp(4601)
 fm_status:  count=     1 size=         8 bytes from core: tcp_main.c: init_tcp(4589)
 fm_status:  count=     1 size=         8 bytes from core: usr_avp.c: init_avps(90)
 fm_status:  count=     1 size=         8 bytes from core: usr_avp.c: init_avps(89)
 fm_status:  count=     1 size=     16384 bytes from core: dst_blacklist.c: init_dst_blacklist(437)
 fm_status:  count=     1 size=         8 bytes from core: dst_blacklist.c: init_dst_blacklist(430)
 fm_status:  count=     2 size=        96 bytes from core: timer.c: timer_alloc(514)
 fm_status:  count=     1 size=         8 bytes from core: dns_cache.c: init_dns_cache(366)
 fm_status:  count=     1 size=     16384 bytes from core: dns_cache.c: init_dns_cache(358)
 fm_status:  count=     1 size=        16 bytes from core: dns_cache.c: init_dns_cache(351)
 fm_status:  count=     1 size=         8 bytes from core: dns_cache.c: init_dns_cache(345)
 fm_status:  count=     1 size=         8 bytes from core: timer.c: init_timer(283)
 fm_status:  count=     1 size=     16384 bytes from core: timer.c: init_timer(282)
 fm_status:  count=     1 size=         8 bytes from core: timer.c: init_timer(281)
 fm_status:  count=     1 size=         8 bytes from core: timer.c: init_timer(280)
 fm_status:  count=     1 size=         8 bytes from core: timer.c: init_timer(269)
 fm_status:  count=     1 size=         8 bytes from core: timer.c: init_timer(237)
 fm_status:  count=     1 size=    278544 bytes from core: timer.c: init_timer(220)
 fm_status:  count=     1 size=         8 bytes from core: timer.c: init_timer(219)
 fm_status:  count=     1 size=         8 bytes from core: timer.c: init_timer(206)
 fm_status:  count=     1 size=        64 bytes from core: cfg/cfg_struct.c: cfg_child_cb_new(830)
 fm_status:  count=     1 size=         8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(361)
 fm_status:  count=     1 size=         8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(354)
 fm_status:  count=     1 size=         8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(347)
 fm_status:  count=     1 size=         8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(335)
 fm_status:  count=     1 size=         8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(323)
 fm_status:  count=     4 size=       928 bytes from htable: ht_api.c: ht_add_table(278)
 fm_status:  count=     1 size=         8 bytes from core: mem/shm.c: shm_core_lock_init(153)
 fm_status: -----------------------------

If i stop these tests and after few minutes, run again the command "corex.shm_summary" the entry that contains the memory consume for xavp.c doesn't decrease.

It seems that there is a memory leak in this module xavp.c.

Thank you for your help.
Regards
José

2016-07-02 11:45 GMT+01:00 José Seabra <joseseabra4@gmail.com>:
I'm running burning tests against kamailio and after some time ~ 2 days I see kamailio out of mem:

ERROR: tm [t_lookup.c:1271]: new_t(): ERROR: new_t: out of mem:
ERROR: tm [t_lookup.c:1411]: t_newtran(): ERROR: t_newtran: new_t failed

The result of command  kamcmd mod.stats all is:

Module: core
{
    sip_msg_shm_clone(494): 10653392
    build_req_buf_from_sip_req(2149): 2565480
    msg_lump_cloner(978): 2943584
    create_avp(175): 969736
    xavp_new_value(94): 858469320
    counters_prefork_init(207): 16896
    cfg_clone_str(130): 88
    cfg_shmize(217): 744
    init_pt(110): 8
    init_pt(105): 8
    init_pt(104): 3080
    dtrie_insert(157): 240
    dtrie_insert(148): 48
    dtrie_init(60): 160
    dtrie_init(51): 32
    register_timer(1011): 360
    cfg_register_ctx(47): 32
    init_tcp(4635): 8192
    init_tcp(4629): 32768
    init_tcp(4621): 8
    init_tcp(4614): 8
    init_tcp(4607): 8
    init_tcp(4601): 8
    init_tcp(4589): 8
    init_avps(90): 8
    init_avps(89): 8
    init_dst_blacklist(437): 16384
    init_dst_blacklist(430): 8
    timer_alloc(514): 96
    init_dns_cache(366): 8
    init_dns_cache(358): 16384
    init_dns_cache(351): 16
    init_dns_cache(345): 8
    init_timer(283): 8
    init_timer(282): 16384
    init_timer(281): 8
    init_timer(280): 8
    init_timer(269): 8
    init_timer(237): 8
    init_timer(220): 278544
    init_timer(219): 8
    init_timer(206): 8
    cfg_child_cb_new(830): 64
    sr_cfg_init(361): 8
    sr_cfg_init(354): 8
    sr_cfg_init(347): 8
    sr_cfg_init(335): 8
    sr_cfg_init(323): 8
    shm_core_lock_init(153): 8
    Total: 875992224
}

Module: dmq
{
    alloc_job_queue(229): 128
    shm_str_dup(723): 24
    build_dmq_node(156): 448
    add_peer(67): 168
    mod_init(243): 8
    mod_init(237): 96
    init_dmq_node_list(66): 24
    init_peer_list(33): 24
    Total: 920
}

Module: auth_db
{
    Total: 0
}

Module: dialog
{
    init_dlg_table(284): 131088
    init_dlg_table(276): 8
    init_dlg_table(269): 8
    init_dlg_table(264): 8
    init_dlg_timer(57): 8
    init_dlg_timer(48): 32
    new_dlg_profile(187): 624
    Total: 131776
}

Module: rr
{
    Total: 0
}

Module: xlog
{
    Total: 0
}

Module: kex
{
    pkg_proc_stats_init(79): 1056
    Total: 1056
}

Module: db_postgres
{
    lock_set_alloc(70): 80
    Total: 80
}

Module: pv
{
    Total: 0
}

Module: textops
{
    Total: 0
}

Module: htable
{
    ht_cell_new(183): 1104
    ht_init_tables(381): 8388608
    ht_add_table(278): 928
    Total: 8390640
}

Module: sqlops
{
    Total: 0
}

Module: tm
{
    build_uac_req(1543): 848
    prepare_new_uac(479): 28880
    _reply_light(542): 751504
    build_local_reparse(341): 1168568
    relay_reply(1884): 643088
    build_cell(317): 9826624
    prepare_new_uac(524): 8112
    prepare_new_uac(509): 10192
    init_tm_stats_child(60): 5632
    insert_tmcb(137): 136
    init_tmcb_lists(74): 16
    init_tmcb_lists(72): 16
    init_hash_table(467): 2097152
    Total: 14540768
}

Module: usrloc
{
    new_urecord(65): 136
    new_ucontact(98): 1096
    new_urecord(58): 288
    shm_str_dup(723): 1200
    build_stat_name(51): 64
    new_udomain(93): 40960
    new_udomain(86): 48
    new_dlist(573): 16
    new_dlist(565): 32
    register_ulcb(94): 32
    init_ulcb_list(45): 16
    lock_set_alloc(70): 4112
    Total: 48000
}

Module: tmx
{
    tmx_check_pretran(250): 864
    tmx_check_pretran(271): 1216
    tmx_init_pretran_table(90): 256
    Total: 2336
}

Module: sl
{
    init_sl_stats_child(125): 4224
    sl_startup(83): 8
    init_sl_stats(110): 8
    Total: 4240
}

Module: carrierroute
{
    rule_fixup_recursor(584): 8
    shm_str_dup(723): 56
    add_route_rule(74): 152
    add_route_flags(225): 48
    create_domain_data(83): 32
    create_carrier_data(59): 8
    create_carrier_data(50): 64
    load_route_data_db(295): 8
    load_domain_map(182): 48
    load_domain_map(171): 24
    load_carrier_map(126): 48
    load_carrier_map(115): 24
    reload_route_data(172): 64
    init_route_data(74): 8
    Total: 592
}

Module: dispatcher
{
    reindex_dests(600): 1008
    add_dest2list(350): 168
    add_dest2list(324): 4200
    init_data(204): 16
    init_data(195): 16
    mod_init(309): 8
    mod_init(307): 8
    ds_ping_active_init(102): 8
    Total: 5432
}

Module: db_text
{
    dbt_init_cache(81): 256
    dbt_init_cache(69): 8
    dbt_init_cache(54): 8
    Total: 272
}

The result of command kamctl shmem is:

shmem:fragments = 6353
shmem:free_size = 406208
shmem:max_used_size = 8464105472
shmem:real_used_size = 8463699264
shmem:total_size = 8464105472
shmem:used_size = 5190698208

The server has 8G of shared memory allocated, and the issue happens more or less after 2 days running tests.

I'm using sipp to generate traffic, it is sending 500CAPS.

There is any possibility of memory leak on Kamailio 4.4.0?

Best Regards 

-- 
José Seabra



--
Cumprimentos
José Seabra