[SR-Users] Kamailio memory manager

Wed Oct 3 08:48:28 CEST 2018

Thanks Daniel

But how to find out what exact eats all memory?

pkg.stats
{
entry: 0
pid: 13
rank: 0
used: 3639400
free: 3323464
real_used: 5065144
total_size: 8388608
total_frags: 38
}
{
entry: 1
pid: 14
rank: 1
used: 3672088
free: 3253128
real_used: 5135480
total_size: 8388608
total_frags: 125
}
{
entry: 2
pid: 15
rank: 2
used: 3675992
free: 3246312
real_used: 5142296
total_size: 8388608
total_frags: 121
}
{
entry: 3
pid: 16
rank: 3
used: 3670912
free: 3254512
real_used: 5134096
total_size: 8388608
total_frags: 129
}
{
entry: 4
pid: 17
rank: 4
used: 3673712
free: 3250152
real_used: 5138456
total_size: 8388608
total_frags: 122
}
{
entry: 5
pid: 18
rank: 5
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 6
pid: 19
rank: 6
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 7
pid: 20
rank: 7
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 8
pid: 21
rank: 8
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 9
pid: 22
rank: 9
used: 3643776
free: 3315032
real_used: 5073576
total_size: 8388608
total_frags: 54
}
{
entry: 10
pid: 23
rank: 10
used: 3643776
free: 3315032
real_used: 5073576
total_size: 8388608
total_frags: 54
}
{
entry: 11
pid: 24
rank: 11
used: 3643776
free: 3315032
real_used: 5073576
total_size: 8388608
total_frags: 54
}
{
entry: 12
pid: 25
rank: 12
used: 3643776
free: 3315032
real_used: 5073576
total_size: 8388608
total_frags: 54
}
{
entry: 13
pid: 26
rank: 13
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 14
pid: 27
rank: 14
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 15
pid: 28
rank: 15
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 16
pid: 29
rank: 16
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 17
pid: 30
rank: -1
used: 3645488
free: 3308328
real_used: 5080280
total_size: 8388608
total_frags: 89
}
{
entry: 18
pid: 31
rank: -1
used: 3645920
free: 3303528
real_used: 5085080
total_size: 8388608
total_frags: 122
}
{
entry: 19
pid: 32
rank: -1
used: 3643600
free: 3318536
real_used: 5070072
total_size: 8388608
total_frags: 24
}
{
entry: 20
pid: 33
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 21
pid: 34
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 22
pid: 35
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 23
pid: 36
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 24
pid: 37
rank: -128
used: 0
free: 0
real_used: 0
total_size: 8388608
total_frags: 0
}
{
entry: 25
pid: 38
rank: -128
used: 0
free: 0
real_used: 0
total_size: 8388608
total_frags: 0
}
{
entry: 26
pid: 39
rank: -128
used: 0
free: 0
real_used: 0
total_size: 8388608
total_frags: 0
}
{
entry: 27
pid: 40
rank: -128
used: 0
free: 0
real_used: 0
total_size: 8388608
total_frags: 0
}
{
entry: 28
pid: 41
rank: -2
used: 3657016
free: 3304912
real_used: 5083696
total_size: 8388608
total_frags: 23
}
{
entry: 29
pid: 42
rank: -2
used: 5810992
free: 3816
real_used: 8384792
total_size: 8388608
total_frags: 93
}
{
entry: 30
pid: 43
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 31
pid: 44
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 32
pid: 45
rank: -1
used: 3643600
free: 3318536
real_used: 5070072
total_size: 8388608
total_frags: 24
}
{
entry: 33
pid: 46
rank: -1
used: 3643600
free: 3318536
real_used: 5070072
total_size: 8388608
total_frags: 24
}
{
entry: 34
pid: 47
rank: 17
used: 3725080
free: 3234768
real_used: 5153840
total_size: 8388608
total_frags: 39
}
{
entry: 35
pid: 48
rank: 18
used: 3725064
free: 3234992
real_used: 5153616
total_size: 8388608
total_frags: 38
}
{
entry: 36
pid: 49
rank: 19
used: 3725048
free: 3235112
real_used: 5153496
total_size: 8388608
total_frags: 37
}
{
entry: 37
pid: 50
rank: 20
used: 3725048
free: 3234800
real_used: 5153808
total_size: 8388608
total_frags: 40
}
{
entry: 38
pid: 51
rank: -4
used: 3723728
free: 3238824
real_used: 5149784
total_size: 8388608
total_frags: 37
}

Regards, Igor
On Oct 2, 2018, 10:37 PM +0300, Daniel-Constantin Mierla <miconda at gmail.com>, wrote:
> Hello,
> pkg is out of free size, not shm. See the output for:
> kamctl rpc pkg.stats
> You may need to increase the pkg allocated pool with -M command line parameter.
> Cheers,
> Daniel
>
> On 02.10.18 20:02, Igor Olhovskiy wrote:
> > Hi!
> >
> > What is the best way to debug messages like...
> >
> > 29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free(): qm_find_free(0x7fad301cf010, 56); Free fragment not found!
> > 29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc(): qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c: dup_lump_list_r(537), module: core; Free fragment not found!
> > 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
> > 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
> > 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
> > 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
> > 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
> > 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
> > 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
> > 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
> > 29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add branches
> >
> > Cause core.shmmem shows
> >  total: 536870912
> > free: 522859104
> > used: 12224144
> > real_used: 14011808
> > max_used: 14479048
> > fragments: 10560
> >
> >
> > Also
> >
> > mod.stats all pkg
> > Module: core
> > {
> > init_io_wait(475): 6960
> > init_io_wait(524): 6312
> > new_db_id(272): 512
> > dupl_string_name(72): 32
> > db_do_init2(298): 488
> > mk_switch_cond_table(43): 144
> > sr_wtimer_add(330): 48
> > as_asciiz(161): 24
> > init_modules(1026): 16
> > rpc_hash_add(146): 2048
> > db_allocate_columns(150): 32
> > db_new_result(114): 56
> > get_hdr_field(116): 1160
> > db_allocate_columns(160): 16
> > cnt_hash_add(383): 1024
> > register_select_table(458): 48
> > add_callback(59): 320
> > sr_wtimer_init(308): 136
> > cfg_new_group(79): 704
> > cfg_declare(50): 3360
> > fix_sock_str(420): 480
> > init_dst_set(83): 32208
> > grp_hash_add(234): 808
> > fix_hostname(1301): 152
> > add_alias(91): 464
> > add_alias(93): 256
> > dupl_string(48): 176
> > parse_headers(325): 432
> > subst_parser(291): 144
> > subst_parser(274): 64
> > exp_optimize_right(411): 24
> > fix_expr(541): 600
> > mk_case_stm(3775): 400
> > fixup_regexp_null(213): 72
> > mk_elem(90): 560
> > route_new_list(200): 1344
> > fix_param(1196): 1336
> > parse_select(204): 1488
> > fix_param(1257): 536
> > fix_match_rve(3028): 320
> > mk_rval_expr1(2616): 142656
> > tr_new(1646): 1464
> > fixup_pvar_all(279): 1624
> > fix_param(1162): 16144
> > mk_action(118): 350168
> > mk_rval_expr2(2674): 489552
> > mk_rval_expr_v(2537): 1475600
> > yyparse(2757): 8792
> > pv_parse_format(1150): 89064
> > pv_cache_add(347): 31456
> > parse_params2(585): 1400
> > set_mod_param_regex(110): 376
> > set_mod_param_regex(121): 832
> > tr_table_add(1875): 400
> > sr_cmd_exports_convert(235): 36288
> > register_module(280): 6624
> > load_module(496): 2640
> > new_sock_info(235): 224
> > fix_socket_list(1509): 48
> > new_sock_info(230): 3920
> > add_alias(93): 56
> > add_alias(91): 96
> > yyparse(1774): 64
> > yyparse(667): 152
> > yyparse(2306): 128
> > yyparse(648): 144
> > sr_push_yy_state(1666): 32
> > sr_push_yy_state(1656): 104
> > sr_push_yy_state(1596): 184
> > addstr(1413): 415752
> > rpc_hash_add(100): 25656
> > str_hash_alloc(59): 512
> > pv_table_add(236): 17864
> > pv_init_buffer(2055): 327680
> > pv_init_buffer(2045): 320
> > init_nonsip_hooks(43): 24
> > init_rlist(146): 8
> > mk_rval_expr_v(2548): 1264
> > route_add(124): 5872
> > str_hash_alloc(59): 768
> > rval_get_str(1256): 56632
> > pp_define(1788): 2248
> > init_counters(122): 128
> > cnt_hash_add(332): 20608
> > str_hash_alloc(59): 1280
> > Total: 3602152
> > }
> >
> > Module: rtpengine
> > {
> > child_init(1812): 256
> > Total: 256
> > }
> >
> > Module: db_mysql
> > {
> > db_mysql_new_connection(65): 96
> > db_mysql_new_connection(75): 3040
> > Total: 3136
> > }
> >
> > Module: siptrace
> > {
> > mod_init(344): 384
> > Total: 384
> > }
> >
> > Module: sl
> > {
> > sl_register_callback(460): 88
> > Total: 88
> > }
> >
> > Module: acc
> > {
> > parse_acc_extra(116): 1512
> > Total: 1512
> > }
> >
> > Module: dialog
> > {
> > dlg_bridge_init_hdrs(66): 80
> > Total: 80
> > }
> >
> > Module: rr
> > {
> > register_rrcb(61): 80
> > Total: 80
> > }
> >
> > Module: db_text
> > {
> > dbt_get_columns(67): 64
> > Total: 64
> > }
> >
> > Module: permissions
> > {
> > get_pathname(242): 64
> > Total: 64
> > }
> >
> > Module: auth
> > {
> > generate_random_secret(239): 32
> > generate_random_secret(238): 32
> > Total: 64
> > }
> >
> > Module: xlog
> > {
> > mod_init(214): 4104
> > xdbg_fixup_helper(500): 6032
> > xlog_fixup_helper(535): 26992
> > Total: 37128
> > }
> >
> > Module: dmq
> > {
> > make_socket_str_from_uri(149): 24
> > Total: 24
> > }
> >
> > Module: sqlops
> > {
> > sqlops_tr_buffer_init(46): 2048
> > sql_init_con(83): 240
> > Total: 2288
> > }
> >
> > Module: textops
> > {
> > tr_txt_parse_re(212): 32
> > hname_fixup(3217): 768
> > fixup_method(3287): 1384
> > Total: 2184
> > }
> >
> > Module: pv
> > {
> > tr_parse_string(2343): 120
> > tr_parse_string(2332): 32
> > tr_parse_string(2378): 96
> > tr_parse_string(2368): 184
> > tr_parse_string(2544): 576
> > add_var(58): 3616
> > add_var(65): 1112
> > Total: 5736
> > }
> >
> > Module: textopsx
> > {
> > fixup_hname_param(598): 352
> > Total: 352
> > }
> >
> > Module: avpops
> > {
> > avpops_parse_pvar(57): 416
> > Total: 416
> > }
> >
> > Module: userblacklist
> > {
> > check_blacklist_fixup(522): 16
> > Total: 16
> > }
> >
> > Module: htable
> > {
> > fixup_ht_key(348): 176
> > pv_parse_ht_name(158): 520
> > Total: 696
> > }
> >
> > Module: dialplan
> > {
> > dp_trans_fixup(396): 296
> > Total: 296
> > }
> >
> > Module: dispatcher
> > {
> > Total: 0
> > }
> >
> > Module: tmx
> > {
> > Total: 0
> > }
> >
> > Module: tm
> > {
> > Total: 0
> > }
> >
> > Module: kex
> > {
> > Total: 0
> > }
> >
> > Module: usrloc
> > {
> > Total: 0
> > }
> >
> > Module: topos
> > {
> > Total: 0
> > }
> >
> > Module: utils
> > {
> > Total: 0
> > }
> >
> > Module: app_python
> > {
> > Total: 0
> > }
> >
> > Module: cfgutils
> > {
> > Total: 0
> > }
> >
> > Module: nathelper
> > {
> > Total: 0
> > }
> >
> > Where is the place to look at? Means what module can cause this?
> >
> > Regards, Igor
> >
> >
> > _______________________________________________
> > Kamailio (SER) - Users Mailing List
> > sr-users at lists.kamailio.org
> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.com
> www.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio World Conference -- www.kamailioworld.com
> Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- www.asipto.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20181003/612d4387/attachment.html>