Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4
active workers (I can see all of them have grown roughly the same amount in that time).
Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance
shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message
is now being touched by it to mask the origin?
Thanks!
Andrew
On 3 Aug 2019, at 10:07 am, Andrew White
<andrew(a)uconnected.com.au> wrote:
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances
of PV.sets throughout the entire script. Additionally, this issue appears to have only
cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t
have any memory related crashes before that point), which leads me to believe it’s related
to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part
of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’])
For now, I’ve taken the output of a mem dump of one of the production workers, ps output,
etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory
increase, and send through the logs if that’s ok?
Thanks!
Andrew
> On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
>
> Hello,
>
> are you defining a lot of kamailio variables in your ruby script? In other words, are
you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is
passed from/computed in ruby script and is changing depending of the sip message?
>
> Cheers,
> Daniel
>
> On 01.08.19 15:34, Andrew White wrote:
>> Thanks Daniel, you’re fantastic!
>>
>> I have 4 children/workers configured with -m 128 -M 32. The machine in question
has 512MB of memory, 1 core and 1GB swap on an SSD.
>>
>> I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in.
I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I
suspect to be the primary children/workers. Immediately post restart:
>>
>> root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>>
>> After about 90 calls:
>>
>> root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>>
>> None of the other 9 Kamailio processes are increasing at all.
>>
>> I ran corex.pkg_summary against one of them and got the following dump:
>>
>>
https://paste.ubuntu.com/p/SqTF3K5knK/
<https://paste.ubuntu.com/p/SqTF3K5knK/>
>>
>> I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting
PVs that need to be unset?
>>
>> Here’s another after another 60 calls:
>>
>>
https://paste.ubuntu.com/p/9WQXqZtfT2/
<https://paste.ubuntu.com/p/9WQXqZtfT2/>
>>
>> root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>> root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00
/usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.cfg -m 128 -M 32
>>
>> The only changes I’ve made on this config over the last couple of weeks (since I
saw this issue) is removing the dispatcher module and adding in a small function in
app_ruby (which I already use) to query redis (which I also already use from app_ruby and
make a heap of queries per call) for some values and write $du manually. I also added in
the topoh module.
>>
>> It also makes a lot of sense to me to monitor the individual processes rather
than the aggregate. Is there a way to identify simply from bash what processes are workers
programmatically? I’d like to monitor just those individually in my monitoring.
>>
>> Thanks!
>>
>> Andrew
>>
>>
>>> On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> if it is pkg, then you have to see which process is increasing the use of
memory, because it is private memory, specific for each process. The sum is an indicator,
but the debugging has to be done for a specific process/pid.
>>>
>>> Once you indentify a process that is leaking pkg, execute the rpc command:
>>>
>>> -
https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pk…
<https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary>
>>> When that process is doing some runtime work (e.g., handling of a sip
message), the syslog will get a summary with used pkg chunks. Send those log messages here
for analysis. You have to set memlog core parameter to a value smaller than debug.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>>
>>> On 01.08.19 03:43, Andrew White wrote:
>>>> Hi all,
>>>>
>>>> I had a Kamailio crash the other day, and some debugging showed I ran out
of PKG memory.
>>>>
>>>> Since then I’ve run a simple bash script to compile the amount of memory
used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used
summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going
on, mostly increasing during our busier daytime hours.
>>>>
>>>>
https://i.imgur.com/UTzx2k1.png <https://i.imgur.com/UTzx2k1.png>
>>>>
>>>> Based on this, I suspect either a module loaded or something within my
app_ruby conf is leaking memory.
>>>>
>>>> I’ve been reading through
https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
<https://www.kamailio.org/wiki/tutorials/troubleshooting/memory>, but I’m a bit
nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can
attach to Kamailio, but is that going to use significant resources to run or impact the
running process? Is there a newer/better/alternative way to do this, and to help me break
this down?
>>>>
>>>> Thanks!
>>>>
>>>> Andrew
>>>>
>>>>
>>>> _______________________________________________
>>>> Kamailio (SER) - Users Mailing List
>>>> sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
>>>>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
<https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
>>> --
>>> Daniel-Constantin Mierla --
www.asipto.com <http://www.asipto.com/>
>>>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
> --
> Daniel-Constantin Mierla --
www.asipto.com <http://www.asipto.com/>
>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>