[SR-Users] Kamailio out-of-mem

Daniel-Constantin Mierla miconda at gmail.com
Wed Jul 6 11:11:34 CEST 2016


Hello,

maybe you can send me a simple kamailio.cfg that exposes the issue,
along with the details of how to reproduce.

Cheers,
Daniel


On 05/07/16 10:58, José Seabra wrote:
> Hello Daniel,
> Is there any possibility of you explain me what do you need from GDB
> and I execute these steps in the server?
> That's because  create a new environment will take me lot of time,
>
> If it isn't possible, I will create the new environment.
>
> Thank you for your help.
> Regards
> José
>
>
> 2016-07-04 17:59 GMT+01:00 José Seabra <joseseabra4 at gmail.com
> <mailto:joseseabra4 at gmail.com>>:
>
>     Hello,
>
>     It won't be ease because of the network firewall where the servers
>     are behind.
>
>     I will setup a new test environment in my laptop then give feedback.
>
>     Regards
>     José 
>
>     2016-07-04 15:46 GMT+01:00 Daniel-Constantin Mierla
>     <miconda at gmail.com <mailto:miconda at gmail.com>>:
>
>         Hello,
>
>         looking at the source code, it seems that the xavp is linked
>         to the root list and should be deleted with the rest of xavps
>         when the message/transaction structure is destroyed.
>
>         Would it be possible to allow me access to the test system in
>         order to check the memory of kamailio with gdb?
>
>         Cheers,
>         Daniel
>
>
>         On 04/07/16 15:48, José Seabra wrote:
>>         Hello
>>
>>         I'm using lookup() and registered().
>>
>>         Let me know if do you need more information or tests from my
>>         side.
>>
>>         Regards
>>
>>         2016-07-04 14:41 GMT+01:00 Daniel-Constantin Mierla
>>         <miconda at gmail.com <mailto:miconda at gmail.com>>:
>>
>>             Hello,
>>
>>             one more question: are you using only lookup(...) or
>>             using registered(...) (or other functions from registrar
>>             module) as well?
>>
>>             Cheers,
>>             Daniel
>>
>>
>>             On 04/07/16 15:21, José Seabra wrote:
>>>             Hello,
>>>             Right now I'm using registrar functions only in
>>>             request_route.
>>>
>>>             At first i had disabled both, after received your email,
>>>             I disabled one by one and I see that the only the
>>>             parameter that is affecting this, is
>>>             modparam("registrar", "xavp_rcd", "ulrcd")
>>>
>>>             Let me know if do you need more information.
>>>
>>>             Regards
>>>             José
>>>
>>>             2016-07-04 13:20 GMT+01:00 Daniel-Constantin Mierla
>>>             <miconda at gmail.com <mailto:miconda at gmail.com>>:
>>>
>>>                 Hello,
>>>
>>>                 very useful information -- are you using the
>>>                 registrar functions only inside request_route, or
>>>                 you have them also in failure_route or other type of
>>>                 routes?
>>>
>>>                 Were you able to detect if both cause the leak or
>>>                 you disabled both of them without trying each one?
>>>
>>>                 Cheers,
>>>                 Daniel
>>>
>>>
>>>                 On 04/07/16 14:01, José Seabra wrote:
>>>>                 Hello Daniel,
>>>>
>>>>                 I think that the issue is on registrar module.
>>>>
>>>>                 I'm using the following parameters:
>>>>
>>>>                   * modparam("registrar", "xavp_cfg", "regcfg")
>>>>                   * modparam("registrar", "xavp_rcd", "ulrcd")
>>>>
>>>>                 If I disable it there isn't  memory leak.
>>>>
>>>>                 Let me know if you need more information.
>>>>
>>>>                 Thank you
>>>>                 Regards
>>>>                 José
>>>>
>>>>                 2016-07-04 11:27 GMT+01:00 Daniel-Constantin Mierla
>>>>                 <miconda at gmail.com <mailto:miconda at gmail.com>>:
>>>>
>>>>                     Hello,
>>>>
>>>>                     can you provide more details about where you
>>>>                     use xavps in the configuration file? Do you use
>>>>                     them in some route block executed by rtimer or
>>>>                     via asyn framework (e.g., via t_continue() or
>>>>                     async module)?
>>>>
>>>>                     Cheers,
>>>>                     Daniel
>>>>
>>>>
>>>>                     On 03/07/16 20:35, José Seabra wrote:
>>>>>                     Hello,
>>>>>                     I'm monitoring these testes with
>>>>>                     "corex.shm_summary" provided by kamcmd.
>>>>>
>>>>>                     I'm noticing  that the core module  - xavp.c
>>>>>                     is consuming lot of memory and it is increasing.
>>>>>
>>>>>                     result of command corex.shm_summary is:
>>>>>
>>>>>                      fm_status: summarizing all alloc'ed. fragments:
>>>>>                      fm_status:  count=     2 size=      1120
>>>>>                     bytes from tm: t_reply.c: _reply_light(542)
>>>>>                      fm_status:  count=     2 size=       896
>>>>>                     bytes from tm: t_msgbuilder.c: build_uac_req(1543)
>>>>>                      fm_status:  count=  2182 size=    231088
>>>>>                     bytes from tm: t_fwd.c: prepare_new_uac(479)
>>>>>                      fm_status:  count= 12488 size=  12050808
>>>>>                     bytes from tm: t_reply.c: relay_reply(1884)
>>>>>                      fm_status:  count= 12490 size=  14564656
>>>>>                     bytes from core: msg_translator.c:
>>>>>                     build_req_buf_from_sip_req(2149)
>>>>>                      fm_status:  count= 12492 size=  71366536
>>>>>                     bytes from tm: h_table.c: build_cell(317)
>>>>>                      fm_status:  count= 12490 size=  75086184
>>>>>                     bytes from core: sip_msg_clone.c:
>>>>>                     sip_msg_shm_clone(494)
>>>>>                      fm_status:  count=     2 size=        96
>>>>>                     bytes from tm: t_hooks.c: insert_tmcb(137)
>>>>>                      fm_status:  count=  2182 size=     17800
>>>>>                     bytes from tm: t_fwd.c: prepare_new_uac(524)
>>>>>                      fm_status:  count=     3 size=       512
>>>>>                     bytes from htable: ht_api.c: ht_cell_new(183)
>>>>>                      fm_status:  count= 12490 size=  10586712
>>>>>                     bytes from core: sip_msg_clone.c:
>>>>>                     msg_lump_cloner(978)
>>>>>                      fm_status:  count= 56508 size=   3457904
>>>>>                     bytes from core: usr_avp.c: create_avp(175)
>>>>>                      fm_status:  count=     4 size=       600
>>>>>                     bytes from tmx: tmx_pretran.c:
>>>>>                     tmx_check_pretran(250)
>>>>>                      fm_status:  count=     4 size=      1008
>>>>>                     bytes from usrloc: ucontact.c: new_ucontact(98)
>>>>>                      fm_status:  count=     4 size=      1320
>>>>>                     bytes from tmx: tmx_pretran.c:
>>>>>                     tmx_check_pretran(271)
>>>>>                      fm_status:  count=     4 size=       144
>>>>>                     bytes from usrloc: urecord.c: new_urecord(65)
>>>>>                      fm_status:  count=     4 size=       328
>>>>>                     bytes from usrloc: urecord.c: new_urecord(58)
>>>>>                      fm_status:  count=    24 size=       952
>>>>>                     bytes from usrloc: ../../ut.h: shm_str_dup(723)
>>>>>                      fm_status:  count=  2182 size=     53040
>>>>>                     bytes from tm: t_fwd.c: prepare_new_uac(509)
>>>>>                      fm_status:  count=12545409 size=1420184592
>>>>>                     bytes from core: xavp.c: xavp_new_value(94)
>>>>>                      fm_status:  count=     4 size=       128
>>>>>                     bytes from dmq: worker.c: alloc_job_queue(229)
>>>>>                      fm_status:  count=     1 size=     13440
>>>>>                     bytes from core: counters.c:
>>>>>                     counters_prefork_init(207)
>>>>>                      fm_status:  count=     1 size=      4032
>>>>>                     bytes from sl: sl_stats.c:
>>>>>                     init_sl_stats_child(125)
>>>>>                      fm_status:  count=     1 size=       256
>>>>>                     bytes from tmx: tmx_pretran.c:
>>>>>                     tmx_init_pretran_table(90)
>>>>>                      fm_status:  count=     1 size=      5376
>>>>>                     bytes from tm: t_stats.c: init_tm_stats_child(60)
>>>>>                      fm_status:  count=     1 size=      1008
>>>>>                     bytes from kex: pkg_stats.c:
>>>>>                     pkg_proc_stats_init(79)
>>>>>                      fm_status:  count=     3 size=        88
>>>>>                     bytes from core: cfg/cfg_struct.c:
>>>>>                     cfg_clone_str(130)
>>>>>                      fm_status:  count=     1 size=       744
>>>>>                     bytes from core: cfg/cfg_struct.c: cfg_shmize(217)
>>>>>                      fm_status:  count=     3 size=        64
>>>>>                     bytes from usrloc: udomain.c: build_stat_name(51)
>>>>>                      fm_status:  count=     1 size=     40960
>>>>>                     bytes from usrloc: udomain.c: new_udomain(93)
>>>>>                      fm_status:  count=     1 size=        48
>>>>>                     bytes from usrloc: udomain.c: new_udomain(86)
>>>>>                      fm_status:  count=     1 size=        16
>>>>>                     bytes from usrloc: dlist.c: new_dlist(573)
>>>>>                      fm_status:  count=     1 size=        32
>>>>>                     bytes from usrloc: dlist.c: new_dlist(565)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: pt.c: init_pt(110)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: pt.c: init_pt(105)
>>>>>                      fm_status:  count=     1 size=      2944
>>>>>                     bytes from core: pt.c: init_pt(104)
>>>>>                      fm_status:  count=     1 size=        32
>>>>>                     bytes from usrloc: ul_callback.c:
>>>>>                     register_ulcb(94)
>>>>>                      fm_status:  count=     1 size=        24
>>>>>                     bytes from dmq: ../../ut.h: shm_str_dup(723)
>>>>>                      fm_status:  count=     1 size=       448
>>>>>                     bytes from dmq: dmqnode.c: build_dmq_node(156)
>>>>>                      fm_status:  count=     2 size=       168
>>>>>                     bytes from dmq: peer.c: add_peer(67)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from dmq: dmq.c: mod_init(243)
>>>>>                      fm_status:  count=     1 size=        96
>>>>>                     bytes from dmq: dmq.c: mod_init(237)
>>>>>                      fm_status:  count=     1 size=        24
>>>>>                     bytes from dmq: dmqnode.c: init_dmq_node_list(66)
>>>>>                      fm_status:  count=     1 size=        24
>>>>>                     bytes from dmq: peer.c: init_peer_list(33)
>>>>>                      fm_status:  count=     1 size=        16
>>>>>                     bytes from usrloc: ul_callback.c:
>>>>>                     init_ulcb_list(45)
>>>>>                      fm_status:  count=     1 size=      4112
>>>>>                     bytes from usrloc: ../../lock_alloc.h:
>>>>>                     lock_set_alloc(70)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from carrierroute: cr_data.c:
>>>>>                     rule_fixup_recursor(584)
>>>>>                      fm_status:  count=     5 size=        56
>>>>>                     bytes from carrierroute: ../../ut.h:
>>>>>                     shm_str_dup(723)
>>>>>                      fm_status:  count=     1 size=       152
>>>>>                     bytes from carrierroute: cr_rule.c:
>>>>>                     add_route_rule(74)
>>>>>                      fm_status:  count=     3 size=       240
>>>>>                     bytes from core: dtrie.c: dtrie_insert(157)
>>>>>                      fm_status:  count=     3 size=        48
>>>>>                     bytes from core: dtrie.c: dtrie_insert(148)
>>>>>                      fm_status:  count=     1 size=        48
>>>>>                     bytes from carrierroute: cr_rule.c:
>>>>>                     add_route_flags(225)
>>>>>                      fm_status:  count=     2 size=       160
>>>>>                     bytes from core: dtrie.c: dtrie_init(60)
>>>>>                      fm_status:  count=     2 size=        32
>>>>>                     bytes from core: dtrie.c: dtrie_init(51)
>>>>>                      fm_status:  count=     1 size=        32
>>>>>                     bytes from carrierroute: cr_domain.c:
>>>>>                     create_domain_data(83)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from carrierroute: cr_carrier.c:
>>>>>                     create_carrier_data(59)
>>>>>                      fm_status:  count=     1 size=        64
>>>>>                     bytes from carrierroute: cr_carrier.c:
>>>>>                     create_carrier_data(50)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from carrierroute: cr_db.c:
>>>>>                     load_route_data_db(295)
>>>>>                      fm_status:  count=     5 size=      1008
>>>>>                     bytes from dispatcher: dispatch.c:
>>>>>                     reindex_dests(600)
>>>>>                      fm_status:  count=     1 size=        48
>>>>>                     bytes from carrierroute: cr_db.c:
>>>>>                     load_domain_map(182)
>>>>>                      fm_status:  count=     1 size=        24
>>>>>                     bytes from carrierroute: cr_db.c:
>>>>>                     load_domain_map(171)
>>>>>                      fm_status:  count=     1 size=        48
>>>>>                     bytes from carrierroute: cr_db.c:
>>>>>                     load_carrier_map(126)
>>>>>                      fm_status:  count=     1 size=        24
>>>>>                     bytes from carrierroute: cr_db.c:
>>>>>                     load_carrier_map(115)
>>>>>                      fm_status:  count=     7 size=       168
>>>>>                     bytes from dispatcher: dispatch.c:
>>>>>                     add_dest2list(350)
>>>>>                      fm_status:  count=     1 size=        64
>>>>>                     bytes from carrierroute: cr_data.c:
>>>>>                     reload_route_data(172)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from carrierroute: cr_data.c:
>>>>>                     init_route_data(74)
>>>>>                      fm_status:  count=     5 size=      4200
>>>>>                     bytes from dispatcher: dispatch.c:
>>>>>                     add_dest2list(324)
>>>>>                      fm_status:  count=     1 size=        16
>>>>>                     bytes from dispatcher: dispatch.c: init_data(204)
>>>>>                      fm_status:  count=     1 size=        16
>>>>>                     bytes from dispatcher: dispatch.c: init_data(195)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from dispatcher: dispatcher.c: mod_init(309)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from dispatcher: dispatcher.c: mod_init(307)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from dispatcher: dispatch.c:
>>>>>                     ds_ping_active_init(102)
>>>>>                      fm_status:  count=     1 size=       256
>>>>>                     bytes from db_text: dbt_lib.c: dbt_init_cache(81)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from db_text: dbt_lib.c: dbt_init_cache(69)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from db_text: dbt_lib.c: dbt_init_cache(54)
>>>>>                      fm_status:  count=     3 size=       288
>>>>>                     bytes from core: timer.c: register_timer(1011)
>>>>>                      fm_status:  count=     4 size=   8388608
>>>>>                     bytes from htable: ht_api.c: ht_init_tables(381)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from sl: sl_funcs.c: sl_startup(83)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from sl: sl_stats.c: init_sl_stats(110)
>>>>>                      fm_status:  count=     1 size=        16
>>>>>                     bytes from tm: t_hooks.c: init_tmcb_lists(74)
>>>>>                      fm_status:  count=     1 size=        16
>>>>>                     bytes from tm: t_hooks.c: init_tmcb_lists(72)
>>>>>                      fm_status:  count=     1 size=   2097152
>>>>>                     bytes from tm: h_table.c: init_hash_table(467)
>>>>>                      fm_status:  count=     2 size=        64
>>>>>                     bytes from core: cfg/cfg_ctx.c:
>>>>>                     cfg_register_ctx(47)
>>>>>                      fm_status:  count=     1 size=        80
>>>>>                     bytes from db_postgres: ../../lock_alloc.h:
>>>>>                     lock_set_alloc(70)
>>>>>                      fm_status:  count=     1 size=      8192
>>>>>                     bytes from core: tcp_main.c: init_tcp(4635)
>>>>>                      fm_status:  count=     1 size=     32768
>>>>>                     bytes from core: tcp_main.c: init_tcp(4629)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: tcp_main.c: init_tcp(4621)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: tcp_main.c: init_tcp(4614)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: tcp_main.c: init_tcp(4607)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: tcp_main.c: init_tcp(4601)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: tcp_main.c: init_tcp(4589)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: usr_avp.c: init_avps(90)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: usr_avp.c: init_avps(89)
>>>>>                      fm_status:  count=     1 size=     16384
>>>>>                     bytes from core: dst_blacklist.c:
>>>>>                     init_dst_blacklist(437)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: dst_blacklist.c:
>>>>>                     init_dst_blacklist(430)
>>>>>                      fm_status:  count=     2 size=        96
>>>>>                     bytes from core: timer.c: timer_alloc(514)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: dns_cache.c: init_dns_cache(366)
>>>>>                      fm_status:  count=     1 size=     16384
>>>>>                     bytes from core: dns_cache.c: init_dns_cache(358)
>>>>>                      fm_status:  count=     1 size=        16
>>>>>                     bytes from core: dns_cache.c: init_dns_cache(351)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: dns_cache.c: init_dns_cache(345)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: timer.c: init_timer(283)
>>>>>                      fm_status:  count=     1 size=     16384
>>>>>                     bytes from core: timer.c: init_timer(282)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: timer.c: init_timer(281)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: timer.c: init_timer(280)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: timer.c: init_timer(269)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: timer.c: init_timer(237)
>>>>>                      fm_status:  count=     1 size=    278544
>>>>>                     bytes from core: timer.c: init_timer(220)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: timer.c: init_timer(219)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: timer.c: init_timer(206)
>>>>>                      fm_status:  count=     1 size=        64
>>>>>                     bytes from core: cfg/cfg_struct.c:
>>>>>                     cfg_child_cb_new(830)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: cfg/cfg_struct.c:
>>>>>                     sr_cfg_init(361)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: cfg/cfg_struct.c:
>>>>>                     sr_cfg_init(354)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: cfg/cfg_struct.c:
>>>>>                     sr_cfg_init(347)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: cfg/cfg_struct.c:
>>>>>                     sr_cfg_init(335)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: cfg/cfg_struct.c:
>>>>>                     sr_cfg_init(323)
>>>>>                      fm_status:  count=     4 size=       928
>>>>>                     bytes from htable: ht_api.c: ht_add_table(278)
>>>>>                      fm_status:  count=     1 size=         8
>>>>>                     bytes from core: mem/shm.c:
>>>>>                     shm_core_lock_init(153)
>>>>>                      fm_status: -----------------------------
>>>>>
>>>>>                     If i stop these tests and after few minutes,
>>>>>                     run again the command "corex.shm_summary" the
>>>>>                     entry that contains the memory consume for
>>>>>                     xavp.c doesn't decrease.
>>>>>
>>>>>                     It seems that there is a memory leak in this
>>>>>                     module xavp.c.
>>>>>
>>>>>                     Thank you for your help.
>>>>>                     Regards
>>>>>                     José
>>>>>
>>>>
>>>>
>>>>                     -- 
>>>>                     Daniel-Constantin Mierla
>>>>                     http://www.asipto.com - http://www.kamailio.org
>>>>                     http://twitter.com/#!/miconda
>>>>                     <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>>                     _______________________________________________
>>>>                     SIP Express Router (SER) and Kamailio (OpenSER)
>>>>                     - sr-users mailing list
>>>>                     sr-users at lists.sip-router.org
>>>>                     <mailto:sr-users at lists.sip-router.org>
>>>>                     http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>>
>>>>
>>>>                 -- 
>>>>                 Cumprimentos
>>>>                 José Seabra
>>>
>>>                 -- 
>>>                 Daniel-Constantin Mierla
>>>                 http://www.asipto.com - http://www.kamailio.org
>>>                 http://twitter.com/#!/miconda
>>>                 <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>>>
>>>
>>>
>>>
>>>             -- 
>>>             Cumprimentos
>>>             José Seabra
>>
>>             -- 
>>             Daniel-Constantin Mierla
>>             http://www.asipto.com - http://www.kamailio.org
>>             http://twitter.com/#!/miconda
>>             <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>>
>>
>>
>>
>>         -- 
>>         Cumprimentos
>>         José Seabra
>
>         -- 
>         Daniel-Constantin Mierla
>         http://www.asipto.com - http://www.kamailio.org
>         http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>
>
>
>
>     -- 
>     Cumprimentos
>     José Seabra
>
>
>
>
> -- 
> Cumprimentos
> José Seabra

-- 
Daniel-Constantin Mierla
http://www.asipto.com - http://www.kamailio.org
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20160706/15a40564/attachment.html>


More information about the sr-users mailing list