<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>The easier way would be to get the memory usage summary, it will
give a report ordered by the size of used memory based on code
location, allowing to assert if it is some startup (allocated once
per lifetime of kamailio) or runtime.<br>
</p>
Some hints about how to do it:<br>
<br>
* <a class="moz-txt-link-freetext" href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a><br>
<br>
Cheers,<br>
Daniel<br>
<br>
<div class="moz-cite-prefix">On 03.10.18 08:48, Igor Olhovskiy
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:934dbd99-a964-452d-9e16-3b65458c3324@Spark">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<title></title>
<div name="messageBodySection" style="font-size: 14px;
font-family: -apple-system, BlinkMacSystemFont, sans-serif;">Thanks
Daniel
<div><br>
</div>
<div>But how to find out what exact eats all memory?</div>
<div><br>
</div>
<div>pkg.stats<br>
{<br>
entry: 0<br>
pid: 13<br>
rank: 0<br>
used: 3639400<br>
free: 3323464<br>
real_used: 5065144<br>
total_size: 8388608<br>
total_frags: 38<br>
}<br>
{<br>
entry: 1<br>
pid: 14<br>
rank: 1<br>
used: 3672088<br>
free: 3253128<br>
real_used: 5135480<br>
total_size: 8388608<br>
total_frags: 125<br>
}<br>
{<br>
entry: 2<br>
pid: 15<br>
rank: 2<br>
used: 3675992<br>
free: 3246312<br>
real_used: 5142296<br>
total_size: 8388608<br>
total_frags: 121<br>
}<br>
{<br>
entry: 3<br>
pid: 16<br>
rank: 3<br>
used: 3670912<br>
free: 3254512<br>
real_used: 5134096<br>
total_size: 8388608<br>
total_frags: 129<br>
}<br>
{<br>
entry: 4<br>
pid: 17<br>
rank: 4<br>
used: 3673712<br>
free: 3250152<br>
real_used: 5138456<br>
total_size: 8388608<br>
total_frags: 122<br>
}<br>
{<br>
entry: 5<br>
pid: 18<br>
rank: 5<br>
used: 3643776<br>
free: 3318152<br>
real_used: 5070456<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 6<br>
pid: 19<br>
rank: 6<br>
used: 3643776<br>
free: 3318152<br>
real_used: 5070456<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 7<br>
pid: 20<br>
rank: 7<br>
used: 3643776<br>
free: 3318152<br>
real_used: 5070456<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 8<br>
pid: 21<br>
rank: 8<br>
used: 3643776<br>
free: 3318152<br>
real_used: 5070456<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 9<br>
pid: 22<br>
rank: 9<br>
used: 3643776<br>
free: 3315032<br>
real_used: 5073576<br>
total_size: 8388608<br>
total_frags: 54<br>
}<br>
{<br>
entry: 10<br>
pid: 23<br>
rank: 10<br>
used: 3643776<br>
free: 3315032<br>
real_used: 5073576<br>
total_size: 8388608<br>
total_frags: 54<br>
}<br>
{<br>
entry: 11<br>
pid: 24<br>
rank: 11<br>
used: 3643776<br>
free: 3315032<br>
real_used: 5073576<br>
total_size: 8388608<br>
total_frags: 54<br>
}<br>
{<br>
entry: 12<br>
pid: 25<br>
rank: 12<br>
used: 3643776<br>
free: 3315032<br>
real_used: 5073576<br>
total_size: 8388608<br>
total_frags: 54<br>
}<br>
{<br>
entry: 13<br>
pid: 26<br>
rank: 13<br>
used: 3643776<br>
free: 3318152<br>
real_used: 5070456<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 14<br>
pid: 27<br>
rank: 14<br>
used: 3643776<br>
free: 3318152<br>
real_used: 5070456<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 15<br>
pid: 28<br>
rank: 15<br>
used: 3643776<br>
free: 3318152<br>
real_used: 5070456<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 16<br>
pid: 29<br>
rank: 16<br>
used: 3643776<br>
free: 3318152<br>
real_used: 5070456<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 17<br>
pid: 30<br>
rank: -1<br>
used: 3645488<br>
free: 3308328<br>
real_used: 5080280<br>
total_size: 8388608<br>
total_frags: 89<br>
}<br>
{<br>
entry: 18<br>
pid: 31<br>
rank: -1<br>
used: 3645920<br>
free: 3303528<br>
real_used: 5085080<br>
total_size: 8388608<br>
total_frags: 122<br>
}<br>
{<br>
entry: 19<br>
pid: 32<br>
rank: -1<br>
used: 3643600<br>
free: 3318536<br>
real_used: 5070072<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 20<br>
pid: 33<br>
rank: -2<br>
used: 3643744<br>
free: 3318288<br>
real_used: 5070320<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 21<br>
pid: 34<br>
rank: -2<br>
used: 3643744<br>
free: 3318288<br>
real_used: 5070320<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 22<br>
pid: 35<br>
rank: -2<br>
used: 3643744<br>
free: 3318288<br>
real_used: 5070320<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 23<br>
pid: 36<br>
rank: -2<br>
used: 3643744<br>
free: 3318288<br>
real_used: 5070320<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 24<br>
pid: 37<br>
rank: -128<br>
used: 0<br>
free: 0<br>
real_used: 0<br>
total_size: 8388608<br>
total_frags: 0<br>
}<br>
{<br>
entry: 25<br>
pid: 38<br>
rank: -128<br>
used: 0<br>
free: 0<br>
real_used: 0<br>
total_size: 8388608<br>
total_frags: 0<br>
}<br>
{<br>
entry: 26<br>
pid: 39<br>
rank: -128<br>
used: 0<br>
free: 0<br>
real_used: 0<br>
total_size: 8388608<br>
total_frags: 0<br>
}<br>
{<br>
entry: 27<br>
pid: 40<br>
rank: -128<br>
used: 0<br>
free: 0<br>
real_used: 0<br>
total_size: 8388608<br>
total_frags: 0<br>
}<br>
{<br>
entry: 28<br>
pid: 41<br>
rank: -2<br>
used: 3657016<br>
free: 3304912<br>
real_used: 5083696<br>
total_size: 8388608<br>
total_frags: 23<br>
}<br>
{<br>
entry: 29<br>
pid: 42<br>
rank: -2<br>
used: 5810992<br>
free: 3816<br>
real_used: 8384792<br>
total_size: 8388608<br>
total_frags: 93<br>
}<br>
{<br>
entry: 30<br>
pid: 43<br>
rank: -2<br>
used: 3643744<br>
free: 3318288<br>
real_used: 5070320<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 31<br>
pid: 44<br>
rank: -2<br>
used: 3643744<br>
free: 3318288<br>
real_used: 5070320<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 32<br>
pid: 45<br>
rank: -1<br>
used: 3643600<br>
free: 3318536<br>
real_used: 5070072<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 33<br>
pid: 46<br>
rank: -1<br>
used: 3643600<br>
free: 3318536<br>
real_used: 5070072<br>
total_size: 8388608<br>
total_frags: 24<br>
}<br>
{<br>
entry: 34<br>
pid: 47<br>
rank: 17<br>
used: 3725080<br>
free: 3234768<br>
real_used: 5153840<br>
total_size: 8388608<br>
total_frags: 39<br>
}<br>
{<br>
entry: 35<br>
pid: 48<br>
rank: 18<br>
used: 3725064<br>
free: 3234992<br>
real_used: 5153616<br>
total_size: 8388608<br>
total_frags: 38<br>
}<br>
{<br>
entry: 36<br>
pid: 49<br>
rank: 19<br>
used: 3725048<br>
free: 3235112<br>
real_used: 5153496<br>
total_size: 8388608<br>
total_frags: 37<br>
}<br>
{<br>
entry: 37<br>
pid: 50<br>
rank: 20<br>
used: 3725048<br>
free: 3234800<br>
real_used: 5153808<br>
total_size: 8388608<br>
total_frags: 40<br>
}<br>
{<br>
entry: 38<br>
pid: 51<br>
rank: -4<br>
used: 3723728<br>
free: 3238824<br>
real_used: 5149784<br>
total_size: 8388608<br>
total_frags: 37<br>
}<br>
</div>
</div>
<div name="messageSignatureSection" style="font-size: 14px;
font-family: -apple-system, BlinkMacSystemFont, sans-serif;"><br>
Regards, Igor</div>
<div name="messageReplySection" style="font-size: 14px;
font-family: -apple-system, BlinkMacSystemFont, sans-serif;">On
Oct 2, 2018, 10:37 PM +0300, Daniel-Constantin Mierla
<a class="moz-txt-link-rfc2396E" href="mailto:miconda@gmail.com"><miconda@gmail.com></a>, wrote:<br>
<blockquote type="cite" style="margin: 5px 5px; padding-left:
10px; border-left: thin solid #1abc9c;">
<p>Hello,</p>
<p>pkg is out of free size, not shm. See the output for:</p>
<p>kamctl rpc pkg.stats</p>
<p>You may need to increase the pkg allocated pool with -M
command line parameter.<br>
</p>
<p>Cheers,<br>
Daniel<br>
</p>
<br>
<div class="moz-cite-prefix">On 02.10.18 20:02, Igor Olhovskiy
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8a4dd1bf-b781-4b25-a243-40d91f041e0f@Spark"
style="margin: 5px 5px; padding-left: 10px; border-left:
thin solid #e67e22;">
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<div name="messageBodySection" style="font-size: 14px;
font-family: -apple-system, BlinkMacSystemFont,
sans-serif;">Hi!
<div><br>
</div>
<div>What is the best way to debug messages like...</div>
<div><br>
</div>
<div>29(42) ERROR: [core/mem/q_malloc.c:290]:
qm_find_free(): qm_find_free(0x7fad301cf010, 56); Free
fragment not found!<br>
29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc():
qm_malloc(0x7fad301cf010, 56) called from core:
core/data_lump.c: dup_lump_list_r(537), module: core;
Free fragment not found!<br>
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r():
out of mem<br>
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r():
out of mem<br>
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r():
out of mem<br>
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r():
out of mem<br>
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r():
out of mem<br>
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r():
out of mem<br>
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r():
out of mem<br>
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r():
out of mem<br>
29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack():
failure to add branches<br>
</div>
<div><br>
</div>
<div>Cause core.shmmem shows</div>
<div> total: 536870912<br>
free: 522859104<br>
used: 12224144<br>
real_used: 14011808<br>
max_used: 14479048<br>
fragments: 10560<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Also </div>
<div><br>
</div>
<div>mod.stats all pkg<br>
Module: core<br>
{<br>
init_io_wait(475): 6960<br>
init_io_wait(524): 6312<br>
new_db_id(272): 512<br>
dupl_string_name(72): 32<br>
db_do_init2(298): 488<br>
mk_switch_cond_table(43): 144<br>
sr_wtimer_add(330): 48<br>
as_asciiz(161): 24<br>
init_modules(1026): 16<br>
rpc_hash_add(146): 2048<br>
db_allocate_columns(150): 32<br>
db_new_result(114): 56<br>
get_hdr_field(116): 1160<br>
db_allocate_columns(160): 16<br>
cnt_hash_add(383): 1024<br>
register_select_table(458): 48<br>
add_callback(59): 320<br>
sr_wtimer_init(308): 136<br>
cfg_new_group(79): 704<br>
cfg_declare(50): 3360<br>
fix_sock_str(420): 480<br>
init_dst_set(83): 32208<br>
grp_hash_add(234): 808<br>
fix_hostname(1301): 152<br>
add_alias(91): 464<br>
add_alias(93): 256<br>
dupl_string(48): 176<br>
parse_headers(325): 432<br>
subst_parser(291): 144<br>
subst_parser(274): 64<br>
exp_optimize_right(411): 24<br>
fix_expr(541): 600<br>
mk_case_stm(3775): 400<br>
fixup_regexp_null(213): 72<br>
mk_elem(90): 560<br>
route_new_list(200): 1344<br>
fix_param(1196): 1336<br>
parse_select(204): 1488<br>
fix_param(1257): 536<br>
fix_match_rve(3028): 320<br>
mk_rval_expr1(2616): 142656<br>
tr_new(1646): 1464<br>
fixup_pvar_all(279): 1624<br>
fix_param(1162): 16144<br>
mk_action(118): 350168<br>
mk_rval_expr2(2674): 489552<br>
mk_rval_expr_v(2537): 1475600<br>
yyparse(2757): 8792<br>
pv_parse_format(1150): 89064<br>
pv_cache_add(347): 31456<br>
parse_params2(585): 1400<br>
set_mod_param_regex(110): 376<br>
set_mod_param_regex(121): 832<br>
tr_table_add(1875): 400<br>
sr_cmd_exports_convert(235): 36288<br>
register_module(280): 6624<br>
load_module(496): 2640<br>
new_sock_info(235): 224<br>
fix_socket_list(1509): 48<br>
new_sock_info(230): 3920<br>
add_alias(93): 56<br>
add_alias(91): 96<br>
yyparse(1774): 64<br>
yyparse(667): 152<br>
yyparse(2306): 128<br>
yyparse(648): 144<br>
sr_push_yy_state(1666): 32<br>
sr_push_yy_state(1656): 104<br>
sr_push_yy_state(1596): 184<br>
addstr(1413): 415752<br>
rpc_hash_add(100): 25656<br>
str_hash_alloc(59): 512<br>
pv_table_add(236): 17864<br>
pv_init_buffer(2055): 327680<br>
pv_init_buffer(2045): 320<br>
init_nonsip_hooks(43): 24<br>
init_rlist(146): 8<br>
mk_rval_expr_v(2548): 1264<br>
route_add(124): 5872<br>
str_hash_alloc(59): 768<br>
rval_get_str(1256): 56632<br>
pp_define(1788): 2248<br>
init_counters(122): 128<br>
cnt_hash_add(332): 20608<br>
str_hash_alloc(59): 1280<br>
Total: 3602152<br>
}<br>
<br>
Module: rtpengine<br>
{<br>
child_init(1812): 256<br>
Total: 256<br>
}<br>
<br>
Module: db_mysql<br>
{<br>
db_mysql_new_connection(65): 96<br>
db_mysql_new_connection(75): 3040<br>
Total: 3136<br>
}<br>
<br>
Module: siptrace<br>
{<br>
mod_init(344): 384<br>
Total: 384<br>
}<br>
<br>
Module: sl<br>
{<br>
sl_register_callback(460): 88<br>
Total: 88<br>
}<br>
<br>
Module: acc<br>
{<br>
parse_acc_extra(116): 1512<br>
Total: 1512<br>
}<br>
<br>
Module: dialog<br>
{<br>
dlg_bridge_init_hdrs(66): 80<br>
Total: 80<br>
}<br>
<br>
Module: rr<br>
{<br>
register_rrcb(61): 80<br>
Total: 80<br>
}<br>
<br>
Module: db_text<br>
{<br>
dbt_get_columns(67): 64<br>
Total: 64<br>
}<br>
<br>
Module: permissions<br>
{<br>
get_pathname(242): 64<br>
Total: 64<br>
}<br>
<br>
Module: auth<br>
{<br>
generate_random_secret(239): 32<br>
generate_random_secret(238): 32<br>
Total: 64<br>
}<br>
<br>
Module: xlog<br>
{<br>
mod_init(214): 4104<br>
xdbg_fixup_helper(500): 6032<br>
xlog_fixup_helper(535): 26992<br>
Total: 37128<br>
}<br>
<br>
Module: dmq<br>
{<br>
make_socket_str_from_uri(149): 24<br>
Total: 24<br>
}<br>
<br>
Module: sqlops<br>
{<br>
sqlops_tr_buffer_init(46): 2048<br>
sql_init_con(83): 240<br>
Total: 2288<br>
}<br>
<br>
Module: textops<br>
{<br>
tr_txt_parse_re(212): 32<br>
hname_fixup(3217): 768<br>
fixup_method(3287): 1384<br>
Total: 2184<br>
}<br>
<br>
Module: pv<br>
{<br>
tr_parse_string(2343): 120<br>
tr_parse_string(2332): 32<br>
tr_parse_string(2378): 96<br>
tr_parse_string(2368): 184<br>
tr_parse_string(2544): 576<br>
add_var(58): 3616<br>
add_var(65): 1112<br>
Total: 5736<br>
}<br>
<br>
Module: textopsx<br>
{<br>
fixup_hname_param(598): 352<br>
Total: 352<br>
}<br>
<br>
Module: avpops<br>
{<br>
avpops_parse_pvar(57): 416<br>
Total: 416<br>
}<br>
<br>
Module: userblacklist<br>
{<br>
check_blacklist_fixup(522): 16<br>
Total: 16<br>
}<br>
<br>
Module: htable<br>
{<br>
fixup_ht_key(348): 176<br>
pv_parse_ht_name(158): 520<br>
Total: 696<br>
}<br>
<br>
Module: dialplan<br>
{<br>
dp_trans_fixup(396): 296<br>
Total: 296<br>
}<br>
<br>
Module: dispatcher<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: tmx<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: tm<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: kex<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: usrloc<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: topos<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: utils<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: app_python<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: cfgutils<br>
{<br>
Total: 0<br>
}<br>
<br>
Module: nathelper<br>
{<br>
Total: 0<br>
}<br>
</div>
<div><br>
</div>
<div>Where is the place to look at? Means what module can
cause this?</div>
</div>
<div name="messageSignatureSection" style="font-size: 14px;
font-family: -apple-system, BlinkMacSystemFont,
sans-serif;"><br>
Regards, Igor</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com" moz-do-not-send="true">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com" moz-do-not-send="true">www.asipto.com</a></pre>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a></pre>
</body>
</html>