Thanks Daniel
But how to find out what exact eats all memory?
pkg.stats { entry: 0 pid: 13 rank: 0 used: 3639400 free: 3323464 real_used: 5065144 total_size: 8388608 total_frags: 38 } { entry: 1 pid: 14 rank: 1 used: 3672088 free: 3253128 real_used: 5135480 total_size: 8388608 total_frags: 125 } { entry: 2 pid: 15 rank: 2 used: 3675992 free: 3246312 real_used: 5142296 total_size: 8388608 total_frags: 121 } { entry: 3 pid: 16 rank: 3 used: 3670912 free: 3254512 real_used: 5134096 total_size: 8388608 total_frags: 129 } { entry: 4 pid: 17 rank: 4 used: 3673712 free: 3250152 real_used: 5138456 total_size: 8388608 total_frags: 122 } { entry: 5 pid: 18 rank: 5 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 6 pid: 19 rank: 6 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 7 pid: 20 rank: 7 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 8 pid: 21 rank: 8 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 9 pid: 22 rank: 9 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 10 pid: 23 rank: 10 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 11 pid: 24 rank: 11 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 12 pid: 25 rank: 12 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 13 pid: 26 rank: 13 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 14 pid: 27 rank: 14 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 15 pid: 28 rank: 15 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 16 pid: 29 rank: 16 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 17 pid: 30 rank: -1 used: 3645488 free: 3308328 real_used: 5080280 total_size: 8388608 total_frags: 89 } { entry: 18 pid: 31 rank: -1 used: 3645920 free: 3303528 real_used: 5085080 total_size: 8388608 total_frags: 122 } { entry: 19 pid: 32 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 20 pid: 33 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 21 pid: 34 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 22 pid: 35 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 23 pid: 36 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 24 pid: 37 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 25 pid: 38 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 26 pid: 39 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 27 pid: 40 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 28 pid: 41 rank: -2 used: 3657016 free: 3304912 real_used: 5083696 total_size: 8388608 total_frags: 23 } { entry: 29 pid: 42 rank: -2 used: 5810992 free: 3816 real_used: 8384792 total_size: 8388608 total_frags: 93 } { entry: 30 pid: 43 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 31 pid: 44 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 32 pid: 45 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 33 pid: 46 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 34 pid: 47 rank: 17 used: 3725080 free: 3234768 real_used: 5153840 total_size: 8388608 total_frags: 39 } { entry: 35 pid: 48 rank: 18 used: 3725064 free: 3234992 real_used: 5153616 total_size: 8388608 total_frags: 38 } { entry: 36 pid: 49 rank: 19 used: 3725048 free: 3235112 real_used: 5153496 total_size: 8388608 total_frags: 37 } { entry: 37 pid: 50 rank: 20 used: 3725048 free: 3234800 real_used: 5153808 total_size: 8388608 total_frags: 40 } { entry: 38 pid: 51 rank: -4 used: 3723728 free: 3238824 real_used: 5149784 total_size: 8388608 total_frags: 37 }
Regards, Igor On Oct 2, 2018, 10:37 PM +0300, Daniel-Constantin Mierla miconda@gmail.com, wrote:
Hello, pkg is out of free size, not shm. See the output for: kamctl rpc pkg.stats You may need to increase the pkg allocated pool with -M command line parameter. Cheers, Daniel
On 02.10.18 20:02, Igor Olhovskiy wrote:
Hi!
What is the best way to debug messages like...
29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free(): qm_find_free(0x7fad301cf010, 56); Free fragment not found! 29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc(): qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c: dup_lump_list_r(537), module: core; Free fragment not found! 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add branches
Cause core.shmmem shows total: 536870912 free: 522859104 used: 12224144 real_used: 14011808 max_used: 14479048 fragments: 10560
Also
mod.stats all pkg Module: core { init_io_wait(475): 6960 init_io_wait(524): 6312 new_db_id(272): 512 dupl_string_name(72): 32 db_do_init2(298): 488 mk_switch_cond_table(43): 144 sr_wtimer_add(330): 48 as_asciiz(161): 24 init_modules(1026): 16 rpc_hash_add(146): 2048 db_allocate_columns(150): 32 db_new_result(114): 56 get_hdr_field(116): 1160 db_allocate_columns(160): 16 cnt_hash_add(383): 1024 register_select_table(458): 48 add_callback(59): 320 sr_wtimer_init(308): 136 cfg_new_group(79): 704 cfg_declare(50): 3360 fix_sock_str(420): 480 init_dst_set(83): 32208 grp_hash_add(234): 808 fix_hostname(1301): 152 add_alias(91): 464 add_alias(93): 256 dupl_string(48): 176 parse_headers(325): 432 subst_parser(291): 144 subst_parser(274): 64 exp_optimize_right(411): 24 fix_expr(541): 600 mk_case_stm(3775): 400 fixup_regexp_null(213): 72 mk_elem(90): 560 route_new_list(200): 1344 fix_param(1196): 1336 parse_select(204): 1488 fix_param(1257): 536 fix_match_rve(3028): 320 mk_rval_expr1(2616): 142656 tr_new(1646): 1464 fixup_pvar_all(279): 1624 fix_param(1162): 16144 mk_action(118): 350168 mk_rval_expr2(2674): 489552 mk_rval_expr_v(2537): 1475600 yyparse(2757): 8792 pv_parse_format(1150): 89064 pv_cache_add(347): 31456 parse_params2(585): 1400 set_mod_param_regex(110): 376 set_mod_param_regex(121): 832 tr_table_add(1875): 400 sr_cmd_exports_convert(235): 36288 register_module(280): 6624 load_module(496): 2640 new_sock_info(235): 224 fix_socket_list(1509): 48 new_sock_info(230): 3920 add_alias(93): 56 add_alias(91): 96 yyparse(1774): 64 yyparse(667): 152 yyparse(2306): 128 yyparse(648): 144 sr_push_yy_state(1666): 32 sr_push_yy_state(1656): 104 sr_push_yy_state(1596): 184 addstr(1413): 415752 rpc_hash_add(100): 25656 str_hash_alloc(59): 512 pv_table_add(236): 17864 pv_init_buffer(2055): 327680 pv_init_buffer(2045): 320 init_nonsip_hooks(43): 24 init_rlist(146): 8 mk_rval_expr_v(2548): 1264 route_add(124): 5872 str_hash_alloc(59): 768 rval_get_str(1256): 56632 pp_define(1788): 2248 init_counters(122): 128 cnt_hash_add(332): 20608 str_hash_alloc(59): 1280 Total: 3602152 }
Module: rtpengine { child_init(1812): 256 Total: 256 }
Module: db_mysql { db_mysql_new_connection(65): 96 db_mysql_new_connection(75): 3040 Total: 3136 }
Module: siptrace { mod_init(344): 384 Total: 384 }
Module: sl { sl_register_callback(460): 88 Total: 88 }
Module: acc { parse_acc_extra(116): 1512 Total: 1512 }
Module: dialog { dlg_bridge_init_hdrs(66): 80 Total: 80 }
Module: rr { register_rrcb(61): 80 Total: 80 }
Module: db_text { dbt_get_columns(67): 64 Total: 64 }
Module: permissions { get_pathname(242): 64 Total: 64 }
Module: auth { generate_random_secret(239): 32 generate_random_secret(238): 32 Total: 64 }
Module: xlog { mod_init(214): 4104 xdbg_fixup_helper(500): 6032 xlog_fixup_helper(535): 26992 Total: 37128 }
Module: dmq { make_socket_str_from_uri(149): 24 Total: 24 }
Module: sqlops { sqlops_tr_buffer_init(46): 2048 sql_init_con(83): 240 Total: 2288 }
Module: textops { tr_txt_parse_re(212): 32 hname_fixup(3217): 768 fixup_method(3287): 1384 Total: 2184 }
Module: pv { tr_parse_string(2343): 120 tr_parse_string(2332): 32 tr_parse_string(2378): 96 tr_parse_string(2368): 184 tr_parse_string(2544): 576 add_var(58): 3616 add_var(65): 1112 Total: 5736 }
Module: textopsx { fixup_hname_param(598): 352 Total: 352 }
Module: avpops { avpops_parse_pvar(57): 416 Total: 416 }
Module: userblacklist { check_blacklist_fixup(522): 16 Total: 16 }
Module: htable { fixup_ht_key(348): 176 pv_parse_ht_name(158): 520 Total: 696 }
Module: dialplan { dp_trans_fixup(396): 296 Total: 296 }
Module: dispatcher { Total: 0 }
Module: tmx { Total: 0 }
Module: tm { Total: 0 }
Module: kex { Total: 0 }
Module: usrloc { Total: 0 }
Module: topos { Total: 0 }
Module: utils { Total: 0 }
Module: app_python { Total: 0 }
Module: cfgutils { Total: 0 }
Module: nathelper { Total: 0 }
Where is the place to look at? Means what module can cause this?
Regards, Igor
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference -- www.kamailioworld.com Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- www.asipto.com