Hi!
What is the best way to debug messages like...
29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free(): qm_find_free(0x7fad301cf010, 56); Free fragment not found! 29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc(): qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c: dup_lump_list_r(537), module: core; Free fragment not found! 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add branches
Cause core.shmmem shows total: 536870912 free: 522859104 used: 12224144 real_used: 14011808 max_used: 14479048 fragments: 10560
Also
mod.stats all pkg Module: core { init_io_wait(475): 6960 init_io_wait(524): 6312 new_db_id(272): 512 dupl_string_name(72): 32 db_do_init2(298): 488 mk_switch_cond_table(43): 144 sr_wtimer_add(330): 48 as_asciiz(161): 24 init_modules(1026): 16 rpc_hash_add(146): 2048 db_allocate_columns(150): 32 db_new_result(114): 56 get_hdr_field(116): 1160 db_allocate_columns(160): 16 cnt_hash_add(383): 1024 register_select_table(458): 48 add_callback(59): 320 sr_wtimer_init(308): 136 cfg_new_group(79): 704 cfg_declare(50): 3360 fix_sock_str(420): 480 init_dst_set(83): 32208 grp_hash_add(234): 808 fix_hostname(1301): 152 add_alias(91): 464 add_alias(93): 256 dupl_string(48): 176 parse_headers(325): 432 subst_parser(291): 144 subst_parser(274): 64 exp_optimize_right(411): 24 fix_expr(541): 600 mk_case_stm(3775): 400 fixup_regexp_null(213): 72 mk_elem(90): 560 route_new_list(200): 1344 fix_param(1196): 1336 parse_select(204): 1488 fix_param(1257): 536 fix_match_rve(3028): 320 mk_rval_expr1(2616): 142656 tr_new(1646): 1464 fixup_pvar_all(279): 1624 fix_param(1162): 16144 mk_action(118): 350168 mk_rval_expr2(2674): 489552 mk_rval_expr_v(2537): 1475600 yyparse(2757): 8792 pv_parse_format(1150): 89064 pv_cache_add(347): 31456 parse_params2(585): 1400 set_mod_param_regex(110): 376 set_mod_param_regex(121): 832 tr_table_add(1875): 400 sr_cmd_exports_convert(235): 36288 register_module(280): 6624 load_module(496): 2640 new_sock_info(235): 224 fix_socket_list(1509): 48 new_sock_info(230): 3920 add_alias(93): 56 add_alias(91): 96 yyparse(1774): 64 yyparse(667): 152 yyparse(2306): 128 yyparse(648): 144 sr_push_yy_state(1666): 32 sr_push_yy_state(1656): 104 sr_push_yy_state(1596): 184 addstr(1413): 415752 rpc_hash_add(100): 25656 str_hash_alloc(59): 512 pv_table_add(236): 17864 pv_init_buffer(2055): 327680 pv_init_buffer(2045): 320 init_nonsip_hooks(43): 24 init_rlist(146): 8 mk_rval_expr_v(2548): 1264 route_add(124): 5872 str_hash_alloc(59): 768 rval_get_str(1256): 56632 pp_define(1788): 2248 init_counters(122): 128 cnt_hash_add(332): 20608 str_hash_alloc(59): 1280 Total: 3602152 }
Module: rtpengine { child_init(1812): 256 Total: 256 }
Module: db_mysql { db_mysql_new_connection(65): 96 db_mysql_new_connection(75): 3040 Total: 3136 }
Module: siptrace { mod_init(344): 384 Total: 384 }
Module: sl { sl_register_callback(460): 88 Total: 88 }
Module: acc { parse_acc_extra(116): 1512 Total: 1512 }
Module: dialog { dlg_bridge_init_hdrs(66): 80 Total: 80 }
Module: rr { register_rrcb(61): 80 Total: 80 }
Module: db_text { dbt_get_columns(67): 64 Total: 64 }
Module: permissions { get_pathname(242): 64 Total: 64 }
Module: auth { generate_random_secret(239): 32 generate_random_secret(238): 32 Total: 64 }
Module: xlog { mod_init(214): 4104 xdbg_fixup_helper(500): 6032 xlog_fixup_helper(535): 26992 Total: 37128 }
Module: dmq { make_socket_str_from_uri(149): 24 Total: 24 }
Module: sqlops { sqlops_tr_buffer_init(46): 2048 sql_init_con(83): 240 Total: 2288 }
Module: textops { tr_txt_parse_re(212): 32 hname_fixup(3217): 768 fixup_method(3287): 1384 Total: 2184 }
Module: pv { tr_parse_string(2343): 120 tr_parse_string(2332): 32 tr_parse_string(2378): 96 tr_parse_string(2368): 184 tr_parse_string(2544): 576 add_var(58): 3616 add_var(65): 1112 Total: 5736 }
Module: textopsx { fixup_hname_param(598): 352 Total: 352 }
Module: avpops { avpops_parse_pvar(57): 416 Total: 416 }
Module: userblacklist { check_blacklist_fixup(522): 16 Total: 16 }
Module: htable { fixup_ht_key(348): 176 pv_parse_ht_name(158): 520 Total: 696 }
Module: dialplan { dp_trans_fixup(396): 296 Total: 296 }
Module: dispatcher { Total: 0 }
Module: tmx { Total: 0 }
Module: tm { Total: 0 }
Module: kex { Total: 0 }
Module: usrloc { Total: 0 }
Module: topos { Total: 0 }
Module: utils { Total: 0 }
Module: app_python { Total: 0 }
Module: cfgutils { Total: 0 }
Module: nathelper { Total: 0 }
Where is the place to look at? Means what module can cause this?
Regards, Igor
Hello,
pkg is out of free size, not shm. See the output for:
kamctl rpc pkg.stats
You may need to increase the pkg allocated pool with -M command line parameter.
Cheers, Daniel
On 02.10.18 20:02, Igor Olhovskiy wrote:
Hi!
What is the best way to debug messages like...
29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free(): qm_find_free(0x7fad301cf010, 56); Free fragment not found! 29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc(): qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c: dup_lump_list_r(537), module: core; Free fragment not found! 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add branches
Cause core.shmmem shows total: 536870912 free: 522859104 used: 12224144 real_used: 14011808 max_used: 14479048 fragments: 10560
Also
mod.stats all pkg Module: core { init_io_wait(475): 6960 init_io_wait(524): 6312 new_db_id(272): 512 dupl_string_name(72): 32 db_do_init2(298): 488 mk_switch_cond_table(43): 144 sr_wtimer_add(330): 48 as_asciiz(161): 24 init_modules(1026): 16 rpc_hash_add(146): 2048 db_allocate_columns(150): 32 db_new_result(114): 56 get_hdr_field(116): 1160 db_allocate_columns(160): 16 cnt_hash_add(383): 1024 register_select_table(458): 48 add_callback(59): 320 sr_wtimer_init(308): 136 cfg_new_group(79): 704 cfg_declare(50): 3360 fix_sock_str(420): 480 init_dst_set(83): 32208 grp_hash_add(234): 808 fix_hostname(1301): 152 add_alias(91): 464 add_alias(93): 256 dupl_string(48): 176 parse_headers(325): 432 subst_parser(291): 144 subst_parser(274): 64 exp_optimize_right(411): 24 fix_expr(541): 600 mk_case_stm(3775): 400 fixup_regexp_null(213): 72 mk_elem(90): 560 route_new_list(200): 1344 fix_param(1196): 1336 parse_select(204): 1488 fix_param(1257): 536 fix_match_rve(3028): 320 mk_rval_expr1(2616): 142656 tr_new(1646): 1464 fixup_pvar_all(279): 1624 fix_param(1162): 16144 mk_action(118): 350168 mk_rval_expr2(2674): 489552 mk_rval_expr_v(2537): 1475600 yyparse(2757): 8792 pv_parse_format(1150): 89064 pv_cache_add(347): 31456 parse_params2(585): 1400 set_mod_param_regex(110): 376 set_mod_param_regex(121): 832 tr_table_add(1875): 400 sr_cmd_exports_convert(235): 36288 register_module(280): 6624 load_module(496): 2640 new_sock_info(235): 224 fix_socket_list(1509): 48 new_sock_info(230): 3920 add_alias(93): 56 add_alias(91): 96 yyparse(1774): 64 yyparse(667): 152 yyparse(2306): 128 yyparse(648): 144 sr_push_yy_state(1666): 32 sr_push_yy_state(1656): 104 sr_push_yy_state(1596): 184 addstr(1413): 415752 rpc_hash_add(100): 25656 str_hash_alloc(59): 512 pv_table_add(236): 17864 pv_init_buffer(2055): 327680 pv_init_buffer(2045): 320 init_nonsip_hooks(43): 24 init_rlist(146): 8 mk_rval_expr_v(2548): 1264 route_add(124): 5872 str_hash_alloc(59): 768 rval_get_str(1256): 56632 pp_define(1788): 2248 init_counters(122): 128 cnt_hash_add(332): 20608 str_hash_alloc(59): 1280 Total: 3602152 }
Module: rtpengine { child_init(1812): 256 Total: 256 }
Module: db_mysql { db_mysql_new_connection(65): 96 db_mysql_new_connection(75): 3040 Total: 3136 }
Module: siptrace { mod_init(344): 384 Total: 384 }
Module: sl { sl_register_callback(460): 88 Total: 88 }
Module: acc { parse_acc_extra(116): 1512 Total: 1512 }
Module: dialog { dlg_bridge_init_hdrs(66): 80 Total: 80 }
Module: rr { register_rrcb(61): 80 Total: 80 }
Module: db_text { dbt_get_columns(67): 64 Total: 64 }
Module: permissions { get_pathname(242): 64 Total: 64 }
Module: auth { generate_random_secret(239): 32 generate_random_secret(238): 32 Total: 64 }
Module: xlog { mod_init(214): 4104 xdbg_fixup_helper(500): 6032 xlog_fixup_helper(535): 26992 Total: 37128 }
Module: dmq { make_socket_str_from_uri(149): 24 Total: 24 }
Module: sqlops { sqlops_tr_buffer_init(46): 2048 sql_init_con(83): 240 Total: 2288 }
Module: textops { tr_txt_parse_re(212): 32 hname_fixup(3217): 768 fixup_method(3287): 1384 Total: 2184 }
Module: pv { tr_parse_string(2343): 120 tr_parse_string(2332): 32 tr_parse_string(2378): 96 tr_parse_string(2368): 184 tr_parse_string(2544): 576 add_var(58): 3616 add_var(65): 1112 Total: 5736 }
Module: textopsx { fixup_hname_param(598): 352 Total: 352 }
Module: avpops { avpops_parse_pvar(57): 416 Total: 416 }
Module: userblacklist { check_blacklist_fixup(522): 16 Total: 16 }
Module: htable { fixup_ht_key(348): 176 pv_parse_ht_name(158): 520 Total: 696 }
Module: dialplan { dp_trans_fixup(396): 296 Total: 296 }
Module: dispatcher { Total: 0 }
Module: tmx { Total: 0 }
Module: tm { Total: 0 }
Module: kex { Total: 0 }
Module: usrloc { Total: 0 }
Module: topos { Total: 0 }
Module: utils { Total: 0 }
Module: app_python { Total: 0 }
Module: cfgutils { Total: 0 }
Module: nathelper { Total: 0 }
Where is the place to look at? Means what module can cause this?
Regards, Igor
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Thanks Daniel
But how to find out what exact eats all memory?
pkg.stats { entry: 0 pid: 13 rank: 0 used: 3639400 free: 3323464 real_used: 5065144 total_size: 8388608 total_frags: 38 } { entry: 1 pid: 14 rank: 1 used: 3672088 free: 3253128 real_used: 5135480 total_size: 8388608 total_frags: 125 } { entry: 2 pid: 15 rank: 2 used: 3675992 free: 3246312 real_used: 5142296 total_size: 8388608 total_frags: 121 } { entry: 3 pid: 16 rank: 3 used: 3670912 free: 3254512 real_used: 5134096 total_size: 8388608 total_frags: 129 } { entry: 4 pid: 17 rank: 4 used: 3673712 free: 3250152 real_used: 5138456 total_size: 8388608 total_frags: 122 } { entry: 5 pid: 18 rank: 5 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 6 pid: 19 rank: 6 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 7 pid: 20 rank: 7 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 8 pid: 21 rank: 8 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 9 pid: 22 rank: 9 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 10 pid: 23 rank: 10 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 11 pid: 24 rank: 11 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 12 pid: 25 rank: 12 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 13 pid: 26 rank: 13 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 14 pid: 27 rank: 14 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 15 pid: 28 rank: 15 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 16 pid: 29 rank: 16 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 17 pid: 30 rank: -1 used: 3645488 free: 3308328 real_used: 5080280 total_size: 8388608 total_frags: 89 } { entry: 18 pid: 31 rank: -1 used: 3645920 free: 3303528 real_used: 5085080 total_size: 8388608 total_frags: 122 } { entry: 19 pid: 32 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 20 pid: 33 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 21 pid: 34 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 22 pid: 35 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 23 pid: 36 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 24 pid: 37 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 25 pid: 38 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 26 pid: 39 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 27 pid: 40 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 28 pid: 41 rank: -2 used: 3657016 free: 3304912 real_used: 5083696 total_size: 8388608 total_frags: 23 } { entry: 29 pid: 42 rank: -2 used: 5810992 free: 3816 real_used: 8384792 total_size: 8388608 total_frags: 93 } { entry: 30 pid: 43 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 31 pid: 44 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 32 pid: 45 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 33 pid: 46 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 34 pid: 47 rank: 17 used: 3725080 free: 3234768 real_used: 5153840 total_size: 8388608 total_frags: 39 } { entry: 35 pid: 48 rank: 18 used: 3725064 free: 3234992 real_used: 5153616 total_size: 8388608 total_frags: 38 } { entry: 36 pid: 49 rank: 19 used: 3725048 free: 3235112 real_used: 5153496 total_size: 8388608 total_frags: 37 } { entry: 37 pid: 50 rank: 20 used: 3725048 free: 3234800 real_used: 5153808 total_size: 8388608 total_frags: 40 } { entry: 38 pid: 51 rank: -4 used: 3723728 free: 3238824 real_used: 5149784 total_size: 8388608 total_frags: 37 }
Regards, Igor On Oct 2, 2018, 10:37 PM +0300, Daniel-Constantin Mierla miconda@gmail.com, wrote:
Hello, pkg is out of free size, not shm. See the output for: kamctl rpc pkg.stats You may need to increase the pkg allocated pool with -M command line parameter. Cheers, Daniel
On 02.10.18 20:02, Igor Olhovskiy wrote:
Hi!
What is the best way to debug messages like...
29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free(): qm_find_free(0x7fad301cf010, 56); Free fragment not found! 29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc(): qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c: dup_lump_list_r(537), module: core; Free fragment not found! 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add branches
Cause core.shmmem shows total: 536870912 free: 522859104 used: 12224144 real_used: 14011808 max_used: 14479048 fragments: 10560
Also
mod.stats all pkg Module: core { init_io_wait(475): 6960 init_io_wait(524): 6312 new_db_id(272): 512 dupl_string_name(72): 32 db_do_init2(298): 488 mk_switch_cond_table(43): 144 sr_wtimer_add(330): 48 as_asciiz(161): 24 init_modules(1026): 16 rpc_hash_add(146): 2048 db_allocate_columns(150): 32 db_new_result(114): 56 get_hdr_field(116): 1160 db_allocate_columns(160): 16 cnt_hash_add(383): 1024 register_select_table(458): 48 add_callback(59): 320 sr_wtimer_init(308): 136 cfg_new_group(79): 704 cfg_declare(50): 3360 fix_sock_str(420): 480 init_dst_set(83): 32208 grp_hash_add(234): 808 fix_hostname(1301): 152 add_alias(91): 464 add_alias(93): 256 dupl_string(48): 176 parse_headers(325): 432 subst_parser(291): 144 subst_parser(274): 64 exp_optimize_right(411): 24 fix_expr(541): 600 mk_case_stm(3775): 400 fixup_regexp_null(213): 72 mk_elem(90): 560 route_new_list(200): 1344 fix_param(1196): 1336 parse_select(204): 1488 fix_param(1257): 536 fix_match_rve(3028): 320 mk_rval_expr1(2616): 142656 tr_new(1646): 1464 fixup_pvar_all(279): 1624 fix_param(1162): 16144 mk_action(118): 350168 mk_rval_expr2(2674): 489552 mk_rval_expr_v(2537): 1475600 yyparse(2757): 8792 pv_parse_format(1150): 89064 pv_cache_add(347): 31456 parse_params2(585): 1400 set_mod_param_regex(110): 376 set_mod_param_regex(121): 832 tr_table_add(1875): 400 sr_cmd_exports_convert(235): 36288 register_module(280): 6624 load_module(496): 2640 new_sock_info(235): 224 fix_socket_list(1509): 48 new_sock_info(230): 3920 add_alias(93): 56 add_alias(91): 96 yyparse(1774): 64 yyparse(667): 152 yyparse(2306): 128 yyparse(648): 144 sr_push_yy_state(1666): 32 sr_push_yy_state(1656): 104 sr_push_yy_state(1596): 184 addstr(1413): 415752 rpc_hash_add(100): 25656 str_hash_alloc(59): 512 pv_table_add(236): 17864 pv_init_buffer(2055): 327680 pv_init_buffer(2045): 320 init_nonsip_hooks(43): 24 init_rlist(146): 8 mk_rval_expr_v(2548): 1264 route_add(124): 5872 str_hash_alloc(59): 768 rval_get_str(1256): 56632 pp_define(1788): 2248 init_counters(122): 128 cnt_hash_add(332): 20608 str_hash_alloc(59): 1280 Total: 3602152 }
Module: rtpengine { child_init(1812): 256 Total: 256 }
Module: db_mysql { db_mysql_new_connection(65): 96 db_mysql_new_connection(75): 3040 Total: 3136 }
Module: siptrace { mod_init(344): 384 Total: 384 }
Module: sl { sl_register_callback(460): 88 Total: 88 }
Module: acc { parse_acc_extra(116): 1512 Total: 1512 }
Module: dialog { dlg_bridge_init_hdrs(66): 80 Total: 80 }
Module: rr { register_rrcb(61): 80 Total: 80 }
Module: db_text { dbt_get_columns(67): 64 Total: 64 }
Module: permissions { get_pathname(242): 64 Total: 64 }
Module: auth { generate_random_secret(239): 32 generate_random_secret(238): 32 Total: 64 }
Module: xlog { mod_init(214): 4104 xdbg_fixup_helper(500): 6032 xlog_fixup_helper(535): 26992 Total: 37128 }
Module: dmq { make_socket_str_from_uri(149): 24 Total: 24 }
Module: sqlops { sqlops_tr_buffer_init(46): 2048 sql_init_con(83): 240 Total: 2288 }
Module: textops { tr_txt_parse_re(212): 32 hname_fixup(3217): 768 fixup_method(3287): 1384 Total: 2184 }
Module: pv { tr_parse_string(2343): 120 tr_parse_string(2332): 32 tr_parse_string(2378): 96 tr_parse_string(2368): 184 tr_parse_string(2544): 576 add_var(58): 3616 add_var(65): 1112 Total: 5736 }
Module: textopsx { fixup_hname_param(598): 352 Total: 352 }
Module: avpops { avpops_parse_pvar(57): 416 Total: 416 }
Module: userblacklist { check_blacklist_fixup(522): 16 Total: 16 }
Module: htable { fixup_ht_key(348): 176 pv_parse_ht_name(158): 520 Total: 696 }
Module: dialplan { dp_trans_fixup(396): 296 Total: 296 }
Module: dispatcher { Total: 0 }
Module: tmx { Total: 0 }
Module: tm { Total: 0 }
Module: kex { Total: 0 }
Module: usrloc { Total: 0 }
Module: topos { Total: 0 }
Module: utils { Total: 0 }
Module: app_python { Total: 0 }
Module: cfgutils { Total: 0 }
Module: nathelper { Total: 0 }
Where is the place to look at? Means what module can cause this?
Regards, Igor
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference -- www.kamailioworld.com Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- www.asipto.com
The easier way would be to get the memory usage summary, it will give a report ordered by the size of used memory based on code location, allowing to assert if it is some startup (allocated once per lifetime of kamailio) or runtime.
Some hints about how to do it:
* https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
Cheers, Daniel
On 03.10.18 08:48, Igor Olhovskiy wrote:
Thanks Daniel
But how to find out what exact eats all memory?
pkg.stats { entry: 0 pid: 13 rank: 0 used: 3639400 free: 3323464 real_used: 5065144 total_size: 8388608 total_frags: 38 } { entry: 1 pid: 14 rank: 1 used: 3672088 free: 3253128 real_used: 5135480 total_size: 8388608 total_frags: 125 } { entry: 2 pid: 15 rank: 2 used: 3675992 free: 3246312 real_used: 5142296 total_size: 8388608 total_frags: 121 } { entry: 3 pid: 16 rank: 3 used: 3670912 free: 3254512 real_used: 5134096 total_size: 8388608 total_frags: 129 } { entry: 4 pid: 17 rank: 4 used: 3673712 free: 3250152 real_used: 5138456 total_size: 8388608 total_frags: 122 } { entry: 5 pid: 18 rank: 5 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 6 pid: 19 rank: 6 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 7 pid: 20 rank: 7 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 8 pid: 21 rank: 8 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 9 pid: 22 rank: 9 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 10 pid: 23 rank: 10 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 11 pid: 24 rank: 11 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 12 pid: 25 rank: 12 used: 3643776 free: 3315032 real_used: 5073576 total_size: 8388608 total_frags: 54 } { entry: 13 pid: 26 rank: 13 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 14 pid: 27 rank: 14 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 15 pid: 28 rank: 15 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 16 pid: 29 rank: 16 used: 3643776 free: 3318152 real_used: 5070456 total_size: 8388608 total_frags: 24 } { entry: 17 pid: 30 rank: -1 used: 3645488 free: 3308328 real_used: 5080280 total_size: 8388608 total_frags: 89 } { entry: 18 pid: 31 rank: -1 used: 3645920 free: 3303528 real_used: 5085080 total_size: 8388608 total_frags: 122 } { entry: 19 pid: 32 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 20 pid: 33 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 21 pid: 34 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 22 pid: 35 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 23 pid: 36 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 24 pid: 37 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 25 pid: 38 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 26 pid: 39 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 27 pid: 40 rank: -128 used: 0 free: 0 real_used: 0 total_size: 8388608 total_frags: 0 } { entry: 28 pid: 41 rank: -2 used: 3657016 free: 3304912 real_used: 5083696 total_size: 8388608 total_frags: 23 } { entry: 29 pid: 42 rank: -2 used: 5810992 free: 3816 real_used: 8384792 total_size: 8388608 total_frags: 93 } { entry: 30 pid: 43 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 31 pid: 44 rank: -2 used: 3643744 free: 3318288 real_used: 5070320 total_size: 8388608 total_frags: 24 } { entry: 32 pid: 45 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 33 pid: 46 rank: -1 used: 3643600 free: 3318536 real_used: 5070072 total_size: 8388608 total_frags: 24 } { entry: 34 pid: 47 rank: 17 used: 3725080 free: 3234768 real_used: 5153840 total_size: 8388608 total_frags: 39 } { entry: 35 pid: 48 rank: 18 used: 3725064 free: 3234992 real_used: 5153616 total_size: 8388608 total_frags: 38 } { entry: 36 pid: 49 rank: 19 used: 3725048 free: 3235112 real_used: 5153496 total_size: 8388608 total_frags: 37 } { entry: 37 pid: 50 rank: 20 used: 3725048 free: 3234800 real_used: 5153808 total_size: 8388608 total_frags: 40 } { entry: 38 pid: 51 rank: -4 used: 3723728 free: 3238824 real_used: 5149784 total_size: 8388608 total_frags: 37 }
Regards, Igor On Oct 2, 2018, 10:37 PM +0300, Daniel-Constantin Mierla miconda@gmail.com, wrote:
Hello,
pkg is out of free size, not shm. See the output for:
kamctl rpc pkg.stats
You may need to increase the pkg allocated pool with -M command line parameter.
Cheers, Daniel
On 02.10.18 20:02, Igor Olhovskiy wrote:
Hi!
What is the best way to debug messages like...
29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free(): qm_find_free(0x7fad301cf010, 56); Free fragment not found! 29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc(): qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c: dup_lump_list_r(537), module: core; Free fragment not found! 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem 29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add branches
Cause core.shmmem shows total: 536870912 free: 522859104 used: 12224144 real_used: 14011808 max_used: 14479048 fragments: 10560
Also
mod.stats all pkg Module: core { init_io_wait(475): 6960 init_io_wait(524): 6312 new_db_id(272): 512 dupl_string_name(72): 32 db_do_init2(298): 488 mk_switch_cond_table(43): 144 sr_wtimer_add(330): 48 as_asciiz(161): 24 init_modules(1026): 16 rpc_hash_add(146): 2048 db_allocate_columns(150): 32 db_new_result(114): 56 get_hdr_field(116): 1160 db_allocate_columns(160): 16 cnt_hash_add(383): 1024 register_select_table(458): 48 add_callback(59): 320 sr_wtimer_init(308): 136 cfg_new_group(79): 704 cfg_declare(50): 3360 fix_sock_str(420): 480 init_dst_set(83): 32208 grp_hash_add(234): 808 fix_hostname(1301): 152 add_alias(91): 464 add_alias(93): 256 dupl_string(48): 176 parse_headers(325): 432 subst_parser(291): 144 subst_parser(274): 64 exp_optimize_right(411): 24 fix_expr(541): 600 mk_case_stm(3775): 400 fixup_regexp_null(213): 72 mk_elem(90): 560 route_new_list(200): 1344 fix_param(1196): 1336 parse_select(204): 1488 fix_param(1257): 536 fix_match_rve(3028): 320 mk_rval_expr1(2616): 142656 tr_new(1646): 1464 fixup_pvar_all(279): 1624 fix_param(1162): 16144 mk_action(118): 350168 mk_rval_expr2(2674): 489552 mk_rval_expr_v(2537): 1475600 yyparse(2757): 8792 pv_parse_format(1150): 89064 pv_cache_add(347): 31456 parse_params2(585): 1400 set_mod_param_regex(110): 376 set_mod_param_regex(121): 832 tr_table_add(1875): 400 sr_cmd_exports_convert(235): 36288 register_module(280): 6624 load_module(496): 2640 new_sock_info(235): 224 fix_socket_list(1509): 48 new_sock_info(230): 3920 add_alias(93): 56 add_alias(91): 96 yyparse(1774): 64 yyparse(667): 152 yyparse(2306): 128 yyparse(648): 144 sr_push_yy_state(1666): 32 sr_push_yy_state(1656): 104 sr_push_yy_state(1596): 184 addstr(1413): 415752 rpc_hash_add(100): 25656 str_hash_alloc(59): 512 pv_table_add(236): 17864 pv_init_buffer(2055): 327680 pv_init_buffer(2045): 320 init_nonsip_hooks(43): 24 init_rlist(146): 8 mk_rval_expr_v(2548): 1264 route_add(124): 5872 str_hash_alloc(59): 768 rval_get_str(1256): 56632 pp_define(1788): 2248 init_counters(122): 128 cnt_hash_add(332): 20608 str_hash_alloc(59): 1280 Total: 3602152 }
Module: rtpengine { child_init(1812): 256 Total: 256 }
Module: db_mysql { db_mysql_new_connection(65): 96 db_mysql_new_connection(75): 3040 Total: 3136 }
Module: siptrace { mod_init(344): 384 Total: 384 }
Module: sl { sl_register_callback(460): 88 Total: 88 }
Module: acc { parse_acc_extra(116): 1512 Total: 1512 }
Module: dialog { dlg_bridge_init_hdrs(66): 80 Total: 80 }
Module: rr { register_rrcb(61): 80 Total: 80 }
Module: db_text { dbt_get_columns(67): 64 Total: 64 }
Module: permissions { get_pathname(242): 64 Total: 64 }
Module: auth { generate_random_secret(239): 32 generate_random_secret(238): 32 Total: 64 }
Module: xlog { mod_init(214): 4104 xdbg_fixup_helper(500): 6032 xlog_fixup_helper(535): 26992 Total: 37128 }
Module: dmq { make_socket_str_from_uri(149): 24 Total: 24 }
Module: sqlops { sqlops_tr_buffer_init(46): 2048 sql_init_con(83): 240 Total: 2288 }
Module: textops { tr_txt_parse_re(212): 32 hname_fixup(3217): 768 fixup_method(3287): 1384 Total: 2184 }
Module: pv { tr_parse_string(2343): 120 tr_parse_string(2332): 32 tr_parse_string(2378): 96 tr_parse_string(2368): 184 tr_parse_string(2544): 576 add_var(58): 3616 add_var(65): 1112 Total: 5736 }
Module: textopsx { fixup_hname_param(598): 352 Total: 352 }
Module: avpops { avpops_parse_pvar(57): 416 Total: 416 }
Module: userblacklist { check_blacklist_fixup(522): 16 Total: 16 }
Module: htable { fixup_ht_key(348): 176 pv_parse_ht_name(158): 520 Total: 696 }
Module: dialplan { dp_trans_fixup(396): 296 Total: 296 }
Module: dispatcher { Total: 0 }
Module: tmx { Total: 0 }
Module: tm { Total: 0 }
Module: kex { Total: 0 }
Module: usrloc { Total: 0 }
Module: topos { Total: 0 }
Module: utils { Total: 0 }
Module: app_python { Total: 0 }
Module: cfgutils { Total: 0 }
Module: nathelper { Total: 0 }
Where is the place to look at? Means what module can cause this?
Regards, Igor
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference -- www.kamailioworld.com Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- www.asipto.com
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users