[SR-Users] out of memory Error
Daniel-Constantin Mierla
miconda at gmail.com
Tue Sep 29 10:51:02 CEST 2015
Hello,
so these logs are only from main kamailio process, which doesn't handle
sip, so it looks like it has still a lot of pkg.
Is there another group of logs for "fm_status: summarizing all alloc'ed.
fragments" but for another PID than 11969?
If not, then do the test again. When you get out memory logs, identify
the PID of the process that wrote the log message, then run:
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
Replace <pid> with actual value for PID.
Then send some traffic so the process with PID is doing some work. It
should print the summary when it receives the first packet.
Cheers,
Daniel
On 29/09/15 10:34, ycaner wrote:
> Hello Daniel;
> i got summary on "kamailio -E -m 256 -M 8". as you said i did
> children 2. after 3000 calls i get this errors.
>
> in additon , i saw some logs about loops some packets.
> Thanks
>
> 0(11969) NOTICE: <core> [main.c:555]: cleanup(): Memory still-in-use
> summary (pkg):
> 0(11969) NOTICE: fm_status: summarizing all alloc'ed. fragments:
> 0(11969) NOTICE: fm_status: count= 1 size= 56 bytes from
> textops: textops.c: hname_fixup(2208)
> 0(11969) NOTICE: fm_status: count= 1 size= 128 bytes from
> <core>: mod_fix.c: fixup_pvar_all(270)
> 0(11969) NOTICE: fm_status: count= 1 size= 128 bytes from
> <core>: switch.c: mk_switch_cond_table(43)
> 0(11969) NOTICE: fm_status: count= 1 size= 232 bytes from
> <core>: parser/msg_parser.c: get_hdr_field(116)
> 0(11969) NOTICE: fm_status: count= 1 size= 64 bytes from
> <core>: parser/msg_parser.c: parse_headers(326)
> 0(11969) NOTICE: fm_status: count= 1 size= 56 bytes from
> acc: acc_mod.c: acc_register_engine(844)
> 0(11969) NOTICE: fm_status: count= 19 size= 2128 bytes from
> acc: acc_extra.c: parse_acc_extra(127)
> 0(11969) NOTICE: fm_status: count= 1 size= 112 bytes from
> misc_radius: extra.c: parse_extra_str(83)
> 0(11969) NOTICE: fm_status: count= 2 size= 32 bytes from
> <core>: select.c: register_select_table(448)
> 0(11969) NOTICE: fm_status: count= 1 size= 24 bytes from
> kex: mi_core.c: init_mi_uptime(82)
> 0(11969) NOTICE: fm_status: count= 1 size= 80 bytes from
> <core>: socket_info.c: fix_sock_str(419)
> 0(11969) NOTICE: fm_status: count= 1 size= 16 bytes from
> <core>: name_alias.h: add_alias(93)
> 0(11969) NOTICE: fm_status: count= 1 size= 32 bytes from
> <core>: name_alias.h: add_alias(91)
> 0(11969) NOTICE: fm_status: count= 1 size= 32208 bytes from
> <core>: dset.c: init_dst_set(83)
> 0(11969) NOTICE: fm_status: count= 1 size= 40 bytes from
> <core>: socket_info.c: fix_hostname(1357)
> 0(11969) NOTICE: fm_status: count= 1 size= 64 bytes from
> <core>: route.c: fix_expr(541)
> 0(11969) NOTICE: fm_status: count= 10 size= 656 bytes from
> <core>: cfg.y: mk_case_stm(3718)
> 0(11969) NOTICE: fm_status: count= 1 size= 256 bytes from
> <core>: switch.c: mk_match_cond_table(94)
> 0(11969) NOTICE: fm_status: count= 1 size= 168 bytes from
> <core>: mod_fix.c: fixup_regexp_null(213)
> 0(11969) NOTICE: fm_status: count= 5 size= 560 bytes from
> <core>: sr_module.c: fix_param(1211)
> 0(11969) NOTICE: fm_status: count= 3 size= 232 bytes from
> <core>: route_struct.c: mk_elem(90)
> 0(11969) NOTICE: fm_status: count= 1 size= 32 bytes from
> <core>: socket_info.c: fix_socket_list(1559)
> 0(11969) NOTICE: fm_status: count= 24 size= 632 bytes from
> textops: textops.c: fixup_method(2278)
> 0(11969) NOTICE: fm_status: count= 1 size= 72 bytes from
> htable: ht_var.c: pv_parse_ht_name(135)
> 0(11969) NOTICE: fm_status: count= 2 size= 64 bytes from
> pv: pv_trans.c: tr_parse_string(2120)
> 0(11969) NOTICE: fm_status: count= 3 size= 152 bytes from
> pv: pv_trans.c: tr_parse_string(2109)
> 0(11969) NOTICE: fm_status: count= 2 size= 16 bytes from
> <core>: re.c: subst_parser(300)
> 0(11969) NOTICE: fm_status: count= 3 size= 432 bytes from
> <core>: re.c: subst_parser(290)
> 0(11969) NOTICE: fm_status: count= 3 size= 192 bytes from
> <core>: re.c: subst_parser(273)
> 0(11969) NOTICE: fm_status: count= 3 size= 96 bytes from
> textops: txt_var.c: tr_txt_parse_re(212)
> 0(11969) NOTICE: fm_status: count= 9 size= 432 bytes from
> pv: pv_svar.c: add_var(58)
> 0(11969) NOTICE: fm_status: count= 1 size= 32 bytes from
> pv: pv_trans.c: tr_parse_string(2245)
> 0(11969) NOTICE: fm_status: count= 8 size= 384 bytes from
> <core>: pvapi.c: tr_new(1542)
> 0(11969) NOTICE: fm_status: count= 1 size= 128 bytes from
> <core>: rvalue.c: fix_match_rve(3018)
> 0(11969) NOTICE: fm_status: count= 9 size= 112 bytes from
> pv: pv_svar.c: add_var(65)
> 0(11969) NOTICE: fm_status: count= 48 size= 83328 bytes from
> <core>: rvalue.c: mk_rval_expr1(2611)
> 0(11969) NOTICE: fm_status: count= 211 size= 7624 bytes from
> <core>: pvapi.c: pv_parse_format(1057)
> 0(11969) NOTICE: fm_status: count= 147 size= 14440 bytes from
> <core>: sr_module.c: fix_param(1116)
> 0(11969) NOTICE: fm_status: count= 262 size= 5464 bytes from
> <core>: rvalue.c: rval_get_str(1252)
> 0(11969) NOTICE: fm_status: count= 52 size= 90272 bytes from
> <core>: rvalue.c: mk_rval_expr2(2669)
> 0(11969) NOTICE: fm_status: count= 54 size= 6496 bytes from
> xlog: xlog.c: xlog_fixup_helper(449)
> 0(11969) NOTICE: fm_status: count= 544 size= 136224 bytes from
> <core>: route_struct.c: mk_action(118)
> 0(11969) NOTICE: fm_status: count= 252 size= 437472 bytes from
> <core>: rvalue.c: mk_rval_expr_v(2532)
> 0(11969) NOTICE: fm_status: count= 87 size= 3704 bytes from
> <core>: cfg.y: yyparse(2700)
> 0(11969) NOTICE: fm_status: count= 71 size= 10040 bytes from
> <core>: pvapi.c: pv_cache_add(272)
> 0(11969) NOTICE: fm_status: count= 3 size= 168 bytes from
> <core>: parser/parse_param.c: parse_params2(584)
> 0(11969) NOTICE: fm_status: count= 18 size= 296 bytes from
> <core>: rvalue.c: mk_rval_expr_v(2543)
> 0(11969) NOTICE: fm_status: count= 17 size= 488 bytes from
> <core>: modparam.c: set_mod_param_regex(119)
> 0(11969) NOTICE: fm_status: count= 15 size= 1296 bytes from
> <core>: modparam.c: set_mod_param_regex(108)
> 0(11969) NOTICE: fm_status: count= 31 size= 25704 bytes from
> <core>: sr_module.c: sr_cmd_exports_convert(224)
> 0(11969) NOTICE: fm_status: count= 34 size= 1672 bytes from
> <core>: sr_module.c: load_module(488)
> 0(11969) NOTICE: fm_status: count= 2 size= 16 bytes from
> rtpproxy: rtpproxy.c: child_init(1013)
> 0(11969) NOTICE: fm_status: count= 1 size= 16 bytes from
> <core>: socket_info.c: new_sock_info(234)
> 0(11969) NOTICE: fm_status: count= 1 size= 264 bytes from
> <core>: socket_info.c: new_sock_info(229)
> 0(11969) NOTICE: fm_status: count= 1 size= 16 bytes from
> <core>: cfg.y: yyparse(703)
> 0(11969) NOTICE: fm_status: count= 1 size= 24 bytes from
> <core>: cfg.y: yyparse(1717)
> 0(11969) NOTICE: fm_status: count= 1005 size= 132640 bytes from
> <core>: cfg.lex: addstr(1410)
> 0(11969) NOTICE: fm_status: count= 58 size= 904 bytes from
> <core>: cfg.lex: pp_define(1771)
> 0(11969) NOTICE: fm_status: count= 1 size= 128 bytes from
> <core>: str_hash.h: str_hash_alloc(59)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> <core>: route.c: init_rlist(146)
> 0(11969) NOTICE: fm_status: count= 3 size= 48 bytes from
> <core>: cfg.y: yyparse(1767)
> 0(11969) NOTICE: fm_status: count= 35 size= 1832 bytes from
> <core>: route.c: route_add(124)
> 0(11969) NOTICE: fm_status: count= 57 size= 1184 bytes from
> xlog: xlog.c: xdbg_fixup_helper(415)
> 0(11969) NOTICE: fm_status: -----------------------------
> 0(11969) NOTICE: <core> [main.c:569]: cleanup(): Memory still-in-use
> summary (shm):
> 0(11969) NOTICE: fm_status: summarizing all alloc'ed. fragments:
> 0(11969) NOTICE: fm_status: count= 1 size= 128 bytes from
> tmx: tmx_pretran.c: tmx_init_pretran_table(90)
> 0(11969) NOTICE: fm_status: count= 3 size= 64 bytes from
> usrloc: udomain.c: build_stat_name(51)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> <core>: pt.c: init_pt(110)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> <core>: pt.c: init_pt(105)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> dialog: dlg_hash.c: init_dlg_table(269)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> dialog: dlg_hash.c: init_dlg_table(262)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> dialog: dlg_hash.c: init_dlg_table(257)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> domain: domain_mod.c: mod_init(219)
> 0(11969) NOTICE: fm_status: count= 2 size= 144 bytes from
> <core>: timer.c: register_timer(1011)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> <core>: usr_avp.c: init_avps(90)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> <core>: usr_avp.c: init_avps(89)
> 0(11969) NOTICE: fm_status: count= 1 size= 8 bytes from
> <core>: mem/shm_mem.c: shm_mem_init_mallocs(179)
> 0(11969) NOTICE: fm_status: -----------------------------
>
>
> 29.09.2015 09:40 tarihinde Daniel-Constantin Mierla-6 [via SIP Router]
> yazdı:
>> Hello,
>>
>> there are no logs.
>>
>> You should have followed my suggestions, because mem_summary=15 is
>> printing a lot of other unnecessary log messages. It is better to use
>> mem_summary=8 to get a smaller log. Also, the children is quite high,
>> try with children=2 -- the out of memory should appear sooner.
>>
>> Also, use -M 8 in order to reduce the time to using all memory.
>>
>> Cheers,
>> Daniel
>>
>> On 29/09/15 08:30, ycaner wrote:
>>> Hello Daniel;
>>>
>>> First of all , Thanks for reply.This project deserve a big award
>>> because of you. you try to give response any question as you can. i
>>> hope with my test results help you to make it bigger this project.
>>>
>>> Before you reply i set memlog=0 , debug=0 and mem_summary=15 ,
>>> children 64 ,"SHM_MEMORY=256" , "PKG_MEMORY=32" .i added to results.
>>> i dont know how to analyse results but i think the problem is "acc:
>>> acc_extra.c: extra2strar(265)". it increases bytes.
>>>
>>>
>>> Thanks.
>>>
>>>
>>> 29.09.2015 00:31 tarihinde Daniel-Constantin Mierla-6 [via SIP
>>> Router] yazdı:
>>>> The free size for private memory/pkg is getting low.
>>>>
>>>> Set in kamailio.cfg:
>>>>
>>>> memlog=1
>>>> mem_summary=8
>>>>
>>>> Be sure that debug parameter is at least 1.
>>>>
>>>> Restart kamailio and run the test again. When you get out of
>>>> memory, stop kamailio (e.g., using: killall kamailio).
>>>>
>>>> Look inside syslog file for the messages printed at the end. For
>>>> each kamailio process, there should be a pkg summary -- messages
>>>> can be mixed, so you can send all of them
>>>>
>>>> To have lower number of process (which will probably result in
>>>> faster consuming of memory), you can set children=2.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 28/09/15 17:31, ycaner wrote:
>>>>> {
>>>>> entry: 1
>>>>> pid: 12039
>>>>> rank: 1
>>>>> used: 13425600
>>>>> free: 148336
>>>>> real_used: 33406096
>>>>> total_size: 33554432
>>>>> total_frags: 20
>>>>> }
>>>>
>>>> --
>>>> Daniel-Constantin Mierla
>>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>> list
>>>> [hidden email] </user/SendEmail.jtp?type=node&node=141889&i=0>
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>> If you reply to this email, your message will be added to the
>>>> discussion below:
>>>> http://sip-router.1086192.n5.nabble.com/out-of-memory-Error-tp141866p141889.html
>>>>
>>>> To unsubscribe from out of memory Error, click here.
>>>> NAML
>>>> <http://sip-router.1086192.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>
>>> --
>>> <http://www.netgsm.com.tr>
>>>
>>> *kamailio_mem_dump_1.txt* (1M) Download Attachment
>>> <http://sip-router.1086192.n5.nabble.com/attachment/141893/1/kamailio_mem_dump_1.txt>
>>>
>>> ------------------------------------------------------------------------
>>> View this message in context: Re: out of memory Error
>>> <http://sip-router.1086192.n5.nabble.com/out-of-memory-Error-tp141866p141893.html>
>>> Sent from the Users mailing list archive
>>> <http://sip-router.1086192.n5.nabble.com/Users-f3.html> at Nabble.com.
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> [hidden email] </user/SendEmail.jtp?type=node&node=141895&i=0>
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Daniel-Constantin Mierla
>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> [hidden email] </user/SendEmail.jtp?type=node&node=141895&i=1>
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>> ------------------------------------------------------------------------
>> If you reply to this email, your message will be added to the
>> discussion below:
>> http://sip-router.1086192.n5.nabble.com/out-of-memory-Error-tp141866p141895.html
>>
>> To unsubscribe from out of memory Error, click here.
>> NAML
>> <http://sip-router.1086192.n5.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
> --
> <http://www.netgsm.com.tr>
>
> ------------------------------------------------------------------------
> View this message in context: Re: out of memory Error
> <http://sip-router.1086192.n5.nabble.com/out-of-memory-Error-tp141866p141898.html>
> Sent from the Users mailing list archive
> <http://sip-router.1086192.n5.nabble.com/Users-f3.html> at Nabble.com.
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20150929/58f2c316/attachment.html>
More information about the sr-users
mailing list