[SR-Users] Kamailio memory manager

[Daniel-Constantin Mierla]

The easier way would be to get the memory usage summary, it will give a
report ordered by the size of used memory based on code location,
allowing to assert if it is some startup (allocated once per lifetime of
kamailio) or runtime.

Some hints about how to do it:

  * https://www.kamailio.org/wiki/tutorials/troubleshooting/memory

Cheers,
Daniel

On 03.10.18 08:48, Igor Olhovskiy wrote:
> Thanks Daniel
>
> But how to find out what exact eats all memory?
>
> pkg.stats
> {
> entry: 0
> pid: 13
> rank: 0
> used: 3639400
> free: 3323464
> real_used: 5065144
> total_size: 8388608
> total_frags: 38
> }
> {
> entry: 1
> pid: 14
> rank: 1
> used: 3672088
> free: 3253128
> real_used: 5135480
> total_size: 8388608
> total_frags: 125
> }
> {
> entry: 2
> pid: 15
> rank: 2
> used: 3675992
> free: 3246312
> real_used: 5142296
> total_size: 8388608
> total_frags: 121
> }
> {
> entry: 3
> pid: 16
> rank: 3
> used: 3670912
> free: 3254512
> real_used: 5134096
> total_size: 8388608
> total_frags: 129
> }
> {
> entry: 4
> pid: 17
> rank: 4
> used: 3673712
> free: 3250152
> real_used: 5138456
> total_size: 8388608
> total_frags: 122
> }
> {
> entry: 5
> pid: 18
> rank: 5
> used: 3643776
> free: 3318152
> real_used: 5070456
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 6
> pid: 19
> rank: 6
> used: 3643776
> free: 3318152
> real_used: 5070456
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 7
> pid: 20
> rank: 7
> used: 3643776
> free: 3318152
> real_used: 5070456
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 8
> pid: 21
> rank: 8
> used: 3643776
> free: 3318152
> real_used: 5070456
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 9
> pid: 22
> rank: 9
> used: 3643776
> free: 3315032
> real_used: 5073576
> total_size: 8388608
> total_frags: 54
> }
> {
> entry: 10
> pid: 23
> rank: 10
> used: 3643776
> free: 3315032
> real_used: 5073576
> total_size: 8388608
> total_frags: 54
> }
> {
> entry: 11
> pid: 24
> rank: 11
> used: 3643776
> free: 3315032
> real_used: 5073576
> total_size: 8388608
> total_frags: 54
> }
> {
> entry: 12
> pid: 25
> rank: 12
> used: 3643776
> free: 3315032
> real_used: 5073576
> total_size: 8388608
> total_frags: 54
> }
> {
> entry: 13
> pid: 26
> rank: 13
> used: 3643776
> free: 3318152
> real_used: 5070456
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 14
> pid: 27
> rank: 14
> used: 3643776
> free: 3318152
> real_used: 5070456
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 15
> pid: 28
> rank: 15
> used: 3643776
> free: 3318152
> real_used: 5070456
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 16
> pid: 29
> rank: 16
> used: 3643776
> free: 3318152
> real_used: 5070456
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 17
> pid: 30
> rank: -1
> used: 3645488
> free: 3308328
> real_used: 5080280
> total_size: 8388608
> total_frags: 89
> }
> {
> entry: 18
> pid: 31
> rank: -1
> used: 3645920
> free: 3303528
> real_used: 5085080
> total_size: 8388608
> total_frags: 122
> }
> {
> entry: 19
> pid: 32
> rank: -1
> used: 3643600
> free: 3318536
> real_used: 5070072
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 20
> pid: 33
> rank: -2
> used: 3643744
> free: 3318288
> real_used: 5070320
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 21
> pid: 34
> rank: -2
> used: 3643744
> free: 3318288
> real_used: 5070320
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 22
> pid: 35
> rank: -2
> used: 3643744
> free: 3318288
> real_used: 5070320
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 23
> pid: 36
> rank: -2
> used: 3643744
> free: 3318288
> real_used: 5070320
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 24
> pid: 37
> rank: -128
> used: 0
> free: 0
> real_used: 0
> total_size: 8388608
> total_frags: 0
> }
> {
> entry: 25
> pid: 38
> rank: -128
> used: 0
> free: 0
> real_used: 0
> total_size: 8388608
> total_frags: 0
> }
> {
> entry: 26
> pid: 39
> rank: -128
> used: 0
> free: 0
> real_used: 0
> total_size: 8388608
> total_frags: 0
> }
> {
> entry: 27
> pid: 40
> rank: -128
> used: 0
> free: 0
> real_used: 0
> total_size: 8388608
> total_frags: 0
> }
> {
> entry: 28
> pid: 41
> rank: -2
> used: 3657016
> free: 3304912
> real_used: 5083696
> total_size: 8388608
> total_frags: 23
> }
> {
> entry: 29
> pid: 42
> rank: -2
> used: 5810992
> free: 3816
> real_used: 8384792
> total_size: 8388608
> total_frags: 93
> }
> {
> entry: 30
> pid: 43
> rank: -2
> used: 3643744
> free: 3318288
> real_used: 5070320
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 31
> pid: 44
> rank: -2
> used: 3643744
> free: 3318288
> real_used: 5070320
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 32
> pid: 45
> rank: -1
> used: 3643600
> free: 3318536
> real_used: 5070072
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 33
> pid: 46
> rank: -1
> used: 3643600
> free: 3318536
> real_used: 5070072
> total_size: 8388608
> total_frags: 24
> }
> {
> entry: 34
> pid: 47
> rank: 17
> used: 3725080
> free: 3234768
> real_used: 5153840
> total_size: 8388608
> total_frags: 39
> }
> {
> entry: 35
> pid: 48
> rank: 18
> used: 3725064
> free: 3234992
> real_used: 5153616
> total_size: 8388608
> total_frags: 38
> }
> {
> entry: 36
> pid: 49
> rank: 19
> used: 3725048
> free: 3235112
> real_used: 5153496
> total_size: 8388608
> total_frags: 37
> }
> {
> entry: 37
> pid: 50
> rank: 20
> used: 3725048
> free: 3234800
> real_used: 5153808
> total_size: 8388608
> total_frags: 40
> }
> {
> entry: 38
> pid: 51
> rank: -4
> used: 3723728
> free: 3238824
> real_used: 5149784
> total_size: 8388608
> total_frags: 37
> }
>
> Regards, Igor
> On Oct 2, 2018, 10:37 PM +0300, Daniel-Constantin Mierla
> <miconda at gmail.com>, wrote:
>>
>> Hello,
>>
>> pkg is out of free size, not shm. See the output for:
>>
>> kamctl rpc pkg.stats
>>
>> You may need to increase the pkg allocated pool with -M command line
>> parameter.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 02.10.18 20:02, Igor Olhovskiy wrote:
>>> Hi!
>>>
>>> What is the best way to debug messages like...
>>>
>>> 29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free():
>>> qm_find_free(0x7fad301cf010, 56); Free fragment not found!
>>> 29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc():
>>> qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c:
>>> dup_lump_list_r(537), module: core; Free fragment not found!
>>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
>>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
>>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
>>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
>>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
>>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
>>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
>>> 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
>>> 29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add
>>> branches
>>>
>>> Cause core.shmmem shows
>>>  total: 536870912
>>> free: 522859104
>>> used: 12224144
>>> real_used: 14011808
>>> max_used: 14479048
>>> fragments: 10560
>>>
>>>
>>> Also 
>>>
>>> mod.stats all pkg
>>> Module: core
>>> {
>>> init_io_wait(475): 6960
>>> init_io_wait(524): 6312
>>> new_db_id(272): 512
>>> dupl_string_name(72): 32
>>> db_do_init2(298): 488
>>> mk_switch_cond_table(43): 144
>>> sr_wtimer_add(330): 48
>>> as_asciiz(161): 24
>>> init_modules(1026): 16
>>> rpc_hash_add(146): 2048
>>> db_allocate_columns(150): 32
>>> db_new_result(114): 56
>>> get_hdr_field(116): 1160
>>> db_allocate_columns(160): 16
>>> cnt_hash_add(383): 1024
>>> register_select_table(458): 48
>>> add_callback(59): 320
>>> sr_wtimer_init(308): 136
>>> cfg_new_group(79): 704
>>> cfg_declare(50): 3360
>>> fix_sock_str(420): 480
>>> init_dst_set(83): 32208
>>> grp_hash_add(234): 808
>>> fix_hostname(1301): 152
>>> add_alias(91): 464
>>> add_alias(93): 256
>>> dupl_string(48): 176
>>> parse_headers(325): 432
>>> subst_parser(291): 144
>>> subst_parser(274): 64
>>> exp_optimize_right(411): 24
>>> fix_expr(541): 600
>>> mk_case_stm(3775): 400
>>> fixup_regexp_null(213): 72
>>> mk_elem(90): 560
>>> route_new_list(200): 1344
>>> fix_param(1196): 1336
>>> parse_select(204): 1488
>>> fix_param(1257): 536
>>> fix_match_rve(3028): 320
>>> mk_rval_expr1(2616): 142656
>>> tr_new(1646): 1464
>>> fixup_pvar_all(279): 1624
>>> fix_param(1162): 16144
>>> mk_action(118): 350168
>>> mk_rval_expr2(2674): 489552
>>> mk_rval_expr_v(2537): 1475600
>>> yyparse(2757): 8792
>>> pv_parse_format(1150): 89064
>>> pv_cache_add(347): 31456
>>> parse_params2(585): 1400
>>> set_mod_param_regex(110): 376
>>> set_mod_param_regex(121): 832
>>> tr_table_add(1875): 400
>>> sr_cmd_exports_convert(235): 36288
>>> register_module(280): 6624
>>> load_module(496): 2640
>>> new_sock_info(235): 224
>>> fix_socket_list(1509): 48
>>> new_sock_info(230): 3920
>>> add_alias(93): 56
>>> add_alias(91): 96
>>> yyparse(1774): 64
>>> yyparse(667): 152
>>> yyparse(2306): 128
>>> yyparse(648): 144
>>> sr_push_yy_state(1666): 32
>>> sr_push_yy_state(1656): 104
>>> sr_push_yy_state(1596): 184
>>> addstr(1413): 415752
>>> rpc_hash_add(100): 25656
>>> str_hash_alloc(59): 512
>>> pv_table_add(236): 17864
>>> pv_init_buffer(2055): 327680
>>> pv_init_buffer(2045): 320
>>> init_nonsip_hooks(43): 24
>>> init_rlist(146): 8
>>> mk_rval_expr_v(2548): 1264
>>> route_add(124): 5872
>>> str_hash_alloc(59): 768
>>> rval_get_str(1256): 56632
>>> pp_define(1788): 2248
>>> init_counters(122): 128
>>> cnt_hash_add(332): 20608
>>> str_hash_alloc(59): 1280
>>> Total: 3602152
>>> }
>>>
>>> Module: rtpengine
>>> {
>>> child_init(1812): 256
>>> Total: 256
>>> }
>>>
>>> Module: db_mysql
>>> {
>>> db_mysql_new_connection(65): 96
>>> db_mysql_new_connection(75): 3040
>>> Total: 3136
>>> }
>>>
>>> Module: siptrace
>>> {
>>> mod_init(344): 384
>>> Total: 384
>>> }
>>>
>>> Module: sl
>>> {
>>> sl_register_callback(460): 88
>>> Total: 88
>>> }
>>>
>>> Module: acc
>>> {
>>> parse_acc_extra(116): 1512
>>> Total: 1512
>>> }
>>>
>>> Module: dialog
>>> {
>>> dlg_bridge_init_hdrs(66): 80
>>> Total: 80
>>> }
>>>
>>> Module: rr
>>> {
>>> register_rrcb(61): 80
>>> Total: 80
>>> }
>>>
>>> Module: db_text
>>> {
>>> dbt_get_columns(67): 64
>>> Total: 64
>>> }
>>>
>>> Module: permissions
>>> {
>>> get_pathname(242): 64
>>> Total: 64
>>> }
>>>
>>> Module: auth
>>> {
>>> generate_random_secret(239): 32
>>> generate_random_secret(238): 32
>>> Total: 64
>>> }
>>>
>>> Module: xlog
>>> {
>>> mod_init(214): 4104
>>> xdbg_fixup_helper(500): 6032
>>> xlog_fixup_helper(535): 26992
>>> Total: 37128
>>> }
>>>
>>> Module: dmq
>>> {
>>> make_socket_str_from_uri(149): 24
>>> Total: 24
>>> }
>>>
>>> Module: sqlops
>>> {
>>> sqlops_tr_buffer_init(46): 2048
>>> sql_init_con(83): 240
>>> Total: 2288
>>> }
>>>
>>> Module: textops
>>> {
>>> tr_txt_parse_re(212): 32
>>> hname_fixup(3217): 768
>>> fixup_method(3287): 1384
>>> Total: 2184
>>> }
>>>
>>> Module: pv
>>> {
>>> tr_parse_string(2343): 120
>>> tr_parse_string(2332): 32
>>> tr_parse_string(2378): 96
>>> tr_parse_string(2368): 184
>>> tr_parse_string(2544): 576
>>> add_var(58): 3616
>>> add_var(65): 1112
>>> Total: 5736
>>> }
>>>
>>> Module: textopsx
>>> {
>>> fixup_hname_param(598): 352
>>> Total: 352
>>> }
>>>
>>> Module: avpops
>>> {
>>> avpops_parse_pvar(57): 416
>>> Total: 416
>>> }
>>>
>>> Module: userblacklist
>>> {
>>> check_blacklist_fixup(522): 16
>>> Total: 16
>>> }
>>>
>>> Module: htable
>>> {
>>> fixup_ht_key(348): 176
>>> pv_parse_ht_name(158): 520
>>> Total: 696
>>> }
>>>
>>> Module: dialplan
>>> {
>>> dp_trans_fixup(396): 296
>>> Total: 296
>>> }
>>>
>>> Module: dispatcher
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: tmx
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: tm
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: kex
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: usrloc
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: topos
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: utils
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: app_python
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: cfgutils
>>> {
>>> Total: 0
>>> }
>>>
>>> Module: nathelper
>>> {
>>> Total: 0
>>> }
>>>
>>> Where is the place to look at? Means what module can cause this?
>>>
>>> Regards, Igor
>>>
>>>
>>> _______________________________________________
>>> Kamailio (SER) - Users Mailing List
>>> sr-users at lists.kamailio.org
>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --  
>> Daniel-Constantin Mierla -- www.asipto.com
>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio World Conference -- www.kamailioworld.com
>> Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- www.asipto.com
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio World Conference -- www.kamailioworld.com
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- www.asipto.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20181005/8cc1ceaf/attachment.html>