[SR-Users] Slowly increasing PKG memory usage

Daniel-Constantin Mierla miconda at gmail.com
Tue Aug 20 12:15:11 CEST 2019 

To close this discussion - patches for fixing this issue are now in
master and 5.2 branches.

Cheers,
Daniel

On 09.08.19 09:12, Andrew White wrote:
> If anyone is following this in the future and looking for further
> information, here is the GitHub issue:
>
> https://github.com/kamailio/kamailio/issues/2027
>
> Thanks!
>
> Andrew
>
>> On 6 Aug 2019, at 5:23 pm, Henning Westerholt <hw at skalatan.de
>> <mailto:hw at skalatan.de>> wrote:
>>
>> Hello Andrew,
>>
>> indeed, the difference is only 1.4k or so memory. This is probably
>> just an effect of the different time when you did the dump.
>>
>> Yes, an Github issue would be great. Add this dump as well
>> https://paste.ubuntu.com/p/85KYg7gD2z/
>>
>> Would be perfect if you could reproduce it with current git master,
>> though (if the app_ruby problem is also solved).
>>
>> Cheers,
>>
>> Henning
>>
>> Am 06.08.19 um 09:17 schrieb Andrew White:
>>> Very interesting results!
>>>
>>> 11pm last night:
>>> https://paste.ubuntu.com/p/qMz2tN7bcR/
>>>
>>> 2pm today:
>>> https://paste.ubuntu.com/p/QDmTwpV83X/
>>>
>>> The growth is pretty small. The only variance is disabling the topoh
>>> module.
>>>
>>> I’m not big on C, but I think this indicates a leak in the topoh
>>> module, does this sound right?
>>>
>>> I’m happy to put in an issue on GitHub if that’s necessary here.
>>>
>>> Cheers,
>>>
>>> Andrew
>>>
>>>> On 5 Aug 2019, at 11:30 pm, Andrew White <andrew at uconnected.com.au
>>>> <mailto:andrew at uconnected.com.au>> wrote:
>>>>
>>>> Hey Henning,
>>>>
>>>> Unfortunately I’ve had some troubles with app_ruby in dev-7.
>>>> Interestingly, dev-7 shows as “dirty” too:
>>>>
>>>> version: kamailio 5.3.0-dev7 (x86_64/linux) bd477f-dirty
>>>>
>>>> Here’s the ansible playbook I wrote to pull the specific version,
>>>> there’s no fancy patching or anything going on:
>>>>
>>>> - name: Download kamailio
>>>> git:
>>>> repo: https://github.com/kamailio/kamailio.git
>>>> version: "{{ kamailio_version }}"
>>>> dest: /usr/src/kamailio
>>>> - name: Roll back to specific commit
>>>> command: "git reset --hard {{ kamailio_commit }}"
>>>> args:
>>>> chdir: /usr/src/kamailio
>>>> when: kamailio_commit is defined
>>>> - name: Build kamailio from source
>>>> command: "{{ item }} chdir=/usr/src/kamailio"
>>>> with_items:
>>>> - make include_modules="app_ruby db_redis" cfg
>>>> - make all
>>>> - make install
>>>> args:
>>>> creates: /usr/local/sbin/kamailio
>>>>
>>>> Either way, I’ve removed the topoh module from my production
>>>> (dev-4) machine, and re-taken memory information. I’ll give it
>>>> about 12 hours and see where memory is sitting. I figure this
>>>> should isolate if it’s a topoh issue or not.
>>>>
>>>> Thanks!
>>>>
>>>> Andrew
>>>>
>>>>
>>>>> On 5 Aug 2019, at 10:46 am, Andrew White <andrew at uconnected.com.au
>>>>> <mailto:andrew at uconnected.com.au>> wrote:
>>>>>
>>>>> Hi Henning,
>>>>>
>>>>> That is interesting, I don’t know what the dirty version is. My
>>>>> deploy script I use for Kamailio
>>>>> pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as
>>>>> this had a fix in place Daniel did for app_ruby a while back. I
>>>>> don’t recall ever putting on any custom patches or modules. I
>>>>> compile from source on build from that git commit using:
>>>>>
>>>>> makeinclude_modules="app_rubydb_redis"cfg
>>>>>
>>>>> I’ll deploy dev7 tonight and report back!
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Andrew
>>>>>
>>>>>> On 4 Aug 2019, at 8:28 pm, Henning Westerholt <hw at skalatan.de
>>>>>> <mailto:hw at skalatan.de>> wrote:
>>>>>>
>>>>>> Hi Andrew,
>>>>>>
>>>>>> dev4 version - I see. I quickly looked into the topos module
>>>>>> commits, but there have been not that much changes there. But can
>>>>>> you try nevertheless to update to the latest git master (e.g.
>>>>>> dev7) version on your test setup?
>>>>>>
>>>>>> What about the "dirty" version flag, do you have some own patches
>>>>>> or modules that you use?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Henning
>>>>>>
>>>>>> Am 04.08.19 um 01:42 schrieb Andrew White:
>>>>>>> Hey Henning,
>>>>>>>
>>>>>>> It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux)
>>>>>>> 6d43ea-dirty. 
>>>>>>>
>>>>>>> I don’t think there’s any extra messages. The actual dispatching
>>>>>>> is simple, here’s the code:
>>>>>>>
>>>>>>>  1. if destination then
>>>>>>>  2. KSR::PV.sets("$du", destination['sip_uri'])
>>>>>>>  3. pbx = destination['display_name']
>>>>>>>  4. KSR.info <http://ksr.info/>("Chose #{pbx} as inbound PBX, as
>>>>>>>     it's received
>>>>>>>     #{destination['count'].to_i}/#{destination['total_count'].to_i}
>>>>>>>     (#{destination['calculated_percentage'].to_i}%) of calls in
>>>>>>>     this ruleset, where its max is
>>>>>>>     #{destination['percentage'].to_i}%")
>>>>>>>  5. end
>>>>>>>
>>>>>>>
>>>>>>> The code before that just pulls data from redis and returns a
>>>>>>> ruby hash with where we should send the call, so no kamailio
>>>>>>> variables set at all.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>>> On 3 Aug 2019, at 8:03 pm, Henning Westerholt <hw at skalatan.de
>>>>>>>> <mailto:hw at skalatan.de>> wrote:
>>>>>>>>
>>>>>>>> Hi Andrew,
>>>>>>>>
>>>>>>>> interesting insight from the logs indeed. According to your
>>>>>>>> changes this looks indeed like related to the topoh module or
>>>>>>>> custom dispatcher code. Which version of Kamailio do you use?
>>>>>>>>
>>>>>>>> Are you sending out some messages /replies as well from your
>>>>>>>> custom dispatcher app_ruby logic?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Henning
>>>>>>>>
>>>>>>>> Am 03.08.19 um 09:32 schrieb Andrew White:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I’ve done two outputs about 8 hours/3200 calls in between:
>>>>>>>>>
>>>>>>>>> First one (about 5 hours since last restart):
>>>>>>>>>
>>>>>>>>> https://paste.ubuntu.com/p/Y6Wvd9hsV2/
>>>>>>>>>
>>>>>>>>> Second one (about 8 hours after the first, 3200 odd cals later):
>>>>>>>>>
>>>>>>>>> https://paste.ubuntu.com/p/85KYg7gD2z/
>>>>>>>>>
>>>>>>>>> It looks like roughly 7MB of extra memory is being used on
>>>>>>>>> that process, which is 1/4 active workers (I can see all of
>>>>>>>>> them have grown roughly the same amount in that time). Looking
>>>>>>>>> at the memory status, there appear to be about 14,000 more
>>>>>>>>> lines. A cursory glance shows about 6k of those lines
>>>>>>>>> with msg_translator.c (compared to 1k in the first).
>>>>>>>>>
>>>>>>>>> I wonder if maybe this could be related to my use of the topoh
>>>>>>>>> module, given every message is now being touched by it to mask
>>>>>>>>> the origin?
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Andrew
>>>>>>>>>
>>>>>>>>>> On 3 Aug 2019, at 10:07 am, Andrew White
>>>>>>>>>> <andrew at uconnected.com.au <mailto:andrew at uconnected.com.au>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Daniel/Henning,
>>>>>>>>>>
>>>>>>>>>> Thanks so much for your quick responses, they’re very much
>>>>>>>>>> appreciated!
>>>>>>>>>>
>>>>>>>>>> Daniel: I don’t believe so. I’ve just checked over the code,
>>>>>>>>>> and I only have 9 instances of PV.sets throughout the entire
>>>>>>>>>> script. Additionally, this issue appears to have only cropped
>>>>>>>>>> up in the last week or two (I didn’t monitor PKG memory at
>>>>>>>>>> the time, but we didn’t have any memory related crashes
>>>>>>>>>> before that point), which leads me to believe it’s related to
>>>>>>>>>> my config code, the adding of the topoh module or the removal
>>>>>>>>>> of the dispatcher module. 
>>>>>>>>>>
>>>>>>>>>> Within that config code, there’s only one additional Kamailio
>>>>>>>>>> variable set, which is part of my replacement/custom
>>>>>>>>>> dispatcher (app_ruby):
>>>>>>>>>>
>>>>>>>>>> KSR::PV.sets("$du", destination['sip_uri’])
>>>>>>>>>> For now, I’ve taken the output of a mem dump of one of the
>>>>>>>>>> production workers, ps output, etc. I’ll leave that few a few
>>>>>>>>>> hours, monitor to ensure there is a measurable memory
>>>>>>>>>> increase, and send through the logs if that’s ok?
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Andrew
>>>>>>>>>>
>>>>>>>>>>> On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla
>>>>>>>>>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> are you defining a lot of kamailio variables in your ruby
>>>>>>>>>>> script? In other words, are you using variables with
>>>>>>>>>>> different name, like $var(xyz) or $sht(a=>xyz), where xyz is
>>>>>>>>>>> passed from/computed in ruby script and is changing
>>>>>>>>>>> depending of the sip message?
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Daniel
>>>>>>>>>>>
>>>>>>>>>>> On 01.08.19 15:34, Andrew White wrote:
>>>>>>>>>>>> Thanks Daniel, you’re fantastic!
>>>>>>>>>>>>
>>>>>>>>>>>> I have 4 children/workers configured with -m 128 -M 32. The
>>>>>>>>>>>> machine in question has 512MB of memory, 1 core and 1GB
>>>>>>>>>>>> swap on an SSD.
>>>>>>>>>>>>
>>>>>>>>>>>> I restarted Kamailio with memlog=1 and I’ve been sending
>>>>>>>>>>>> batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio
>>>>>>>>>>>> processes going up in memory after each batch, which I
>>>>>>>>>>>> suspect to be the primary children/workers. Immediately
>>>>>>>>>>>> post restart:
>>>>>>>>>>>>
>>>>>>>>>>>> root     28531  0.7  5.5 329368 27196 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28532  0.6  4.9 329368 24528 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28533  0.6  5.5 329368 27244 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28534  0.7  5.4 329368 26788 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>>
>>>>>>>>>>>> After about 90 calls:
>>>>>>>>>>>>
>>>>>>>>>>>> root     28531  0.0  6.7 330688 32948 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28532  0.0  6.5 330560 32264 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28533  0.0  6.5 330556 32272 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28534  0.0  6.6 330564 32592 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>>
>>>>>>>>>>>> None of the other 9 Kamailio processes are increasing at all.
>>>>>>>>>>>>
>>>>>>>>>>>> I ran corex.pkg_summary against one of them and got the
>>>>>>>>>>>> following dump:
>>>>>>>>>>>>
>>>>>>>>>>>> https://paste.ubuntu.com/p/SqTF3K5knK/
>>>>>>>>>>>>
>>>>>>>>>>>> I can see a lot of allocation to pvapi.c, does this
>>>>>>>>>>>> indicate maybe I’m setting PVs that need to be unset?
>>>>>>>>>>>>
>>>>>>>>>>>> Here’s another after another 60 calls:
>>>>>>>>>>>>
>>>>>>>>>>>> https://paste.ubuntu.com/p/9WQXqZtfT2/
>>>>>>>>>>>>
>>>>>>>>>>>> root     28531  0.0  6.9 330820 33928 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28532  0.0  6.7 330692 33352 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28533  0.0  6.7 330688 33280 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>> root     28534  0.0  6.7 330696 33192 ?        Sl   22:48  
>>>>>>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P
>>>>>>>>>>>> /var/run/kamailio/kamailio.pid -f
>>>>>>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32
>>>>>>>>>>>>
>>>>>>>>>>>> The only changes I’ve made on this config over the last
>>>>>>>>>>>> couple of weeks (since I saw this issue) is removing the
>>>>>>>>>>>> dispatcher module and adding in a small function in
>>>>>>>>>>>> app_ruby (which I already use) to query redis (which I also
>>>>>>>>>>>> already use from app_ruby and make a heap of queries per
>>>>>>>>>>>> call) for some values and write $du manually. I also added
>>>>>>>>>>>> in the topoh module.
>>>>>>>>>>>>
>>>>>>>>>>>> It also makes a lot of sense to me to monitor the
>>>>>>>>>>>> individual processes rather than the aggregate. Is there a
>>>>>>>>>>>> way to identify simply from bash what processes are workers
>>>>>>>>>>>> programmatically? I’d like to monitor just those
>>>>>>>>>>>> individually in my monitoring.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> Andrew
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla
>>>>>>>>>>>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> if it is pkg, then you have to see which process is
>>>>>>>>>>>>> increasing the use of memory, because it is private
>>>>>>>>>>>>> memory, specific for each process. The sum is an
>>>>>>>>>>>>> indicator, but the debugging has to be done for a specific
>>>>>>>>>>>>> process/pid.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Once you indentify a process that is leaking pkg, execute
>>>>>>>>>>>>> the rpc command:
>>>>>>>>>>>>>
>>>>>>>>>>>>>   -
>>>>>>>>>>>>> https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary
>>>>>>>>>>>>>
>>>>>>>>>>>>> When that process is doing some runtime work (e.g.,
>>>>>>>>>>>>> handling of a sip message), the syslog will get a summary
>>>>>>>>>>>>> with used pkg chunks. Send those log messages here for
>>>>>>>>>>>>> analysis. You have to set memlog core parameter to a value
>>>>>>>>>>>>> smaller than debug.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Daniel
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 01.08.19 03:43, Andrew White wrote:
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I had a Kamailio crash the other day, and some debugging
>>>>>>>>>>>>>> showed I ran out of PKG memory.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since then I’ve run a simple bash script to compile the
>>>>>>>>>>>>>> amount of memory used by all child processes,
>>>>>>>>>>>>>> effective /usr/local/sbin/kamcmd pkg.stats | grep
>>>>>>>>>>>>>> real_used summed together. I’ve graphed out the data, and
>>>>>>>>>>>>>> there’s a clear growth of PKG memory going on, mostly
>>>>>>>>>>>>>> increasing during our busier daytime hours.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://i.imgur.com/UTzx2k1.png
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Based on this, I suspect either a module loaded or
>>>>>>>>>>>>>> something within my app_ruby conf is leaking memory.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I’ve been reading
>>>>>>>>>>>>>> through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory,
>>>>>>>>>>>>>> but I’m a bit nervous, as I’m not really a C/deep memory
>>>>>>>>>>>>>> type of guy. I can see a GDB script I can attach to
>>>>>>>>>>>>>> Kamailio, but is that going to use significant resources
>>>>>>>>>>>>>> to run or impact the running process? Is there a
>>>>>>>>>>>>>> newer/better/alternative way to do this, and to help me
>>>>>>>>>>>>>> break this down?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Andrew
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Kamailio (SER) - Users Mailing List
>>>>>>>>>>>>>> sr-users at lists.kamailio.org
>>>>>>>>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>>>>> -- 
>>>>>>>>>>>>> Daniel-Constantin Mierla -- www.asipto.com
>>>>>>>>>>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>>>>>>>>>>
>>>>>>>>>>> -- 
>>>>>>>>>>> Daniel-Constantin Mierla -- www.asipto.com
>>>>>>>>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Kamailio (SER) - Users Mailing List
>>>>>>>>> sr-users at lists.kamailio.org
>>>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>> -- 
>>>>>>>> Henning Westerholt - https://skalatan.de/blog/
>>>>>>>> Kamailio services - https://skalatan.de/services
>>>>>>>
>>>>>> -- 
>>>>>> Henning Westerholt - https://skalatan.de/blog/
>>>>>> Kamailio services - https://skalatan.de/services
>>>>>
>>>>
>>>
>> -- 
>> Henning Westerholt - https://skalatan.de/blog/
>> Kamailio services - https://skalatan.de/services
>
-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190820/2c7fc32c/attachment.html>

More information about the sr-users mailing list