[SR-Users] Memory issue
Daniel-Constantin Mierla
miconda at gmail.com
Tue May 13 15:20:00 CEST 2014
Hello,
oh, 4.0 is not compiled with memory debugging by default, so it is not
printing the details about allocation place. Can you recompile with
MEMDBG=1? You can change that in Makefile.defs or give it to the command
line when compiling with make.
Cheers,
Daniel
On 13/05/14 14:44, Igor Potjevlesch wrote:
>
> Hello Daniel,
>
>
> Thank you for back porting the patches. I will schedule an update.
> Maybe I will go in the latest 4.1.x.
>
> Indeed, I have still the coredump. I tried the script you mentioned.
>
> Is this result make sense for you?:
>
> $1 = {size = 1728, u = {nxt_free = 0x0, is_free = 0}}
>
> $2 = {size = 32, u = {nxt_free = 0x0, is_free = 0}}
>
> $3 = {size = 128, u = {nxt_free = 0x0, is_free = 0}}
>
> $4 = {size = 224, u = {nxt_free = 0x0, is_free = 0}}
>
> $5 = {size = 128, u = {nxt_free = 0x0, is_free = 0}}
>
> $6 = {size = 256, u = {nxt_free = 0x0, is_free = 0}}
>
> $7 = {size = 224, u = {nxt_free = 0x0, is_free = 0}}
>
> $8 = {size = 224, u = {nxt_free = 0x0, is_free = 0}}
>
> […]
>
> $1289 = {size = 48, u = {nxt_free = 0x0, is_free = 0}}
>
> $1290 = {size = 1728, u = {nxt_free = 0x0, is_free = 0}}
>
> $1291 = {size = 1728, u = {nxt_free = 0x0, is_free = 0}}
>
> $1292 = {size = 1728, u = {nxt_free = 0x0, is_free = 0}}
>
> $1293 = {size = 128, u = {nxt_free = 0x0, is_free = 0}}
>
> $1294 = {size = 848, u = {nxt_free = 0x0, is_free = 0}}
>
> There are thousand and thousand lines.
>
> Regards,
>
> Igor.
>
> *De :*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Envoyé :* lundi 12 mai 2014 20:26
> *À :* Igor Potjevlesch; 'Kamailio (SER) - Users Mailing List'
> *Objet :* Re: [SR-Users] Memory issue
>
> Hello,
>
> On 12/05/14 17:29, Igor Potjevlesch wrote:
>
> Hello Daniel,
>
> Thank you for this feedback.
>
> We are running 4.0.4 without custom module.
>
> I do a sqlops request in a very small table (40 rows) but on each
> REGISTER and INVITE.
>
> The bigger table is “aliases” with 154k rows.
>
> Can you highlight me the particular bug concerned and corrected?
>
> In the meantime, for the memory allocation, I have to grow the
> shared and/or private memory? What would be the next increment :
> -M 128?
>
> it doesn't really matter the increment. I think the private memory is
> already too big, so if you don't load big amount of data, there could
> be a leak.
>
> I backported related patches to branch 4.0 -- you should upgrade to
> latest git branch 4.0. In this way you are sure you have all the
> fixes, because 4.0.4 is old in the 4.0 branch.
>
> You can use 'kamcmd pkg.stats' to monitor available pkg memory. When
> you see the available memory keep decreasing, you can issue:
>
> kam cmd cfg.set_now_int core mem_dump_pkg _PID_
>
> replace PID with an appropriate value you take from pkg.stats. You can
> send the output from syslog here for investigation, to see what can be
> the potential leak.
>
> If you still have the core file, you can look inside it to see
> allocated memory chunks -- see the gdb script at:
>
> -
> http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory#walking_through_pkg_with_gdb
>
> Cheers,
> Daniel
>
>
>
> Regards,
>
> Igor.
>
> *De :*sr-users-bounces at lists.sip-router.org
> <mailto:sr-users-bounces at lists.sip-router.org>
> [mailto:sr-users-bounces at lists.sip-router.org] *De la part de*
> Daniel-Constantin Mierla
> *Envoyé :* lundi 12 mai 2014 16:43
> *À :* Kamailio (SER) - Users Mailing List
> *Objet :* Re: [SR-Users] Memory issue
>
> Hello,
>
> what version are you using? Are you doing sql queries that return
> large number of records (e.g., via sqlops)? Are you using any
> custom module you developed?
>
> The crash was fixed in master branch, but somehow forgotten to be
> backported -- it is now in branch 4.1.
>
> Cheers,
> Daniel
>
> On 12/05/14 15:58, Igor Potjevlesch wrote:
>
> Hello,
>
> I had a memory issue this morning.
>
> First, I could see the following logs in /var/log/message :
>
> May 12 11:08:21 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: <core> [rvalue.c:2370]: rval_expr_eval(): rv eval int
> expression: out of memory
>
> May 12 11:08:25 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: <core> [msg_translator.c:2164]:
> build_res_buf_from_sip_req(): ERROR:
> build_res_buf_from_sip_req: out of memory ; needs 482
>
> May 12 11:08:26 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: <core> [msg_translator.c:2012]:
> build_res_buf_from_sip_res(): ERROR:
> build_res_buf_from_sip_res: out of mem
>
> May 12 11:08:27 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: <core> [msg_translator.c:1910]:
> build_req_buf_from_sip_req(): ERROR:
> build_req_buf_from_sip_req: out of memory
>
> May 12 11:08:28 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: <core> [rvalue.c:2370]: rval_expr_eval(): rv eval int
> expression: out of memory
>
> May 12 11:08:31 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: <core> [msg_translator.c:1910]:
> build_req_buf_from_sip_req(): ERROR:
> build_req_buf_from_sip_req: out of memory
>
> May 12 11:08:32 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: <core> [db_res.c:181]: db_allocate_rows(): no private
> memory left
>
> May 12 11:08:32 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: db_mysql [km_res.c:175]: db_mysql_convert_rows(): could
> not allocate rows
>
> May 12 11:08:32 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: db_mysql [km_res.c:217]: db_mysql_convert_result():
> error while converting rows
>
> May 12 11:08:32 localhost /usr/local/sbin/kamailio[30268]:
> ERROR: db_mysql [km_dbase.c:191]: db_mysql_store_result():
> error while converting result
>
> May 12 11:08:33 localhost kernel: kamailio[30268]: segfault at
> 30 ip 00007ff44dd93f93 sp 00007fffcb00a200 error 4 in
> db_mysql.so[7ff44dd8f000+19000]
>
> May 12 11:08:37 localhost /usr/local/sbin/kamailio[30235]:
> ALERT: <core> [main.c:788]: handle_sigs(): child process 30268
> exited by a signal 11
>
> May 12 11:08:37 localhost /usr/local/sbin/kamailio[30235]:
> ALERT: <core> [main.c:791]: handle_sigs(): core was generated
>
> May 12 11:08:37 localhost /usr/local/sbin/kamailio[30235]:
> INFO: <core> [main.c:803]: handle_sigs(): INFO: terminating
> due to SIGCHLD
>
> This problem have already occurred and I have set –M 64 to
> avoid it. It looks that it’s not enough anymore.
>
> What is the next increment recommended after 64M?
>
> On the contrary of the first occurrence, Kamailio crash with
> coredump just after these errors. I’m not sure that the memory
> message errors are linked to the crash. I have tried a “bt full”:
>
> (gdb) bt full
>
> #0 db_mysql_store_result (_h=0x7ff44e0e70e0,
> _r=0x7fffcb00a408) at km_dbase.c:198
>
> code = <value optimized out>
>
> __FUNCTION__ = "db_mysql_store_result"
>
> #1 0x00007ff44d971e72 in db_do_query_internal
> (_h=0x7ff44e0e70e0, _k=0x7fffcb00a3e0, _op=0x0,
> _v=0x7fffcb00a3a0, _c=<value optimized out>,
>
> _n=<value optimized out>, _nc=2, _o=0x0,
> _r=0x7fffcb00a408, val2str=0x7ff44dd97fa0 <db_mysql_val2str>,
>
> submit_query=0x7ff44dd92ed0 <db_mysql_submit_query>,
> store_result=0x7ff44dd93d80 <db_mysql_store_result>, _l=0) at
> db_query.c:137
>
> tmp = <value optimized out>
>
> off = <value optimized out>
>
> ret = <value optimized out>
>
> __FUNCTION__ = "db_do_query_internal"
>
> #2 0x00007ff44d97269a in db_do_query (_h=<value optimized
> out>, _k=<value optimized out>, _op=<value optimized out>,
>
> _v=<value optimized out>, _c=<value optimized out>,
> _n=<value optimized out>, _nc=2, _o=0x0, _r=0x7fffcb00a408,
>
> val2str=0x7ff44dd97fa0 <db_mysql_val2str>,
> submit_query=0x7ff44dd92ed0 <db_mysql_submit_query>,
>
> store_result=0x7ff44dd93d80 <db_mysql_store_result>) at
> db_query.c:156
>
> Thanks in advance for your help.
>
> Regards,
>
> Igor.
>
>
>
>
>
> _______________________________________________
>
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>
> sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>
>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
>
> --
>
> Daniel-Constantin Mierla -http://www.asipto.com
>
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>
>
>
> --
> Daniel-Constantin Mierla -http://www.asipto.com
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140513/aea8624b/attachment.html>
More information about the sr-users
mailing list