Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg...
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote:
Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote:
Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Hi Andrew,
answer inline below
Am 01.08.19 um 15:34 schrieb Andrew White: Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/
There is not that much difference between the two runs (diff):
-journal: Suppressed 19796 messages from /system.slice/kamailio.service +journal: Suppressed 1772 messages from /system.slice/kamailio.service ALERT: <core> [core/pt.c:548]: mem_dump_pkg_cb(): Memory status (pkg) of process 28531: ALERT: qm_status: (0x7f0fc3a2a010): ALERT: qm_status: heap size= 33554432 -ALERT: qm_status: used= 538712, used+overhead=883168, free=32671264 -ALERT: qm_status: max used (+overhead)= 893040 +ALERT: qm_status: used= 590552, used+overhead=939064, free=32615368 +ALERT: qm_status: max used (+overhead)= 948112 ALERT: qm_status: dumping all alloc'ed. fragments: ALERT: qm_status: 0. N address=0x7f0fc3a636d8 frag=0x7f0fc3a636a0 size=1024 used=1 ALERT: qm_status: alloc'd from core: core/str_hash.h: str_hash_alloc(59)
The second run shows about 50k more memory usage, but from the dumped allocations they are identical.
Two suggestions:
- your logging system suppresses actually a lot of debugging output from Kamailio (have a look to the first lines each). Check if its necessary to increase the respective configuration variable in your logging conf to get all log messages.
- If you are able to execute this test on a test plattform, configure Kamailio to just use one worker child each, to spot an error easier. Then execute 100 calls, and count the memory increase, another 100 calls and count. Then you can estimate how much bytes you actually leak per call. Try another memory dump and see if you spot something. Share your results again on the list.
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Have a look to the output of corex.ps or core.psx - the worker childs are named "receiver".
Cheers,
Henning
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg...
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote: Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
Hi Henning!
Thanks so much!
I spun up a dev machine with the same specs, Kamailio version and config. On each one, I’ve done a ps of the single receiver/worker process, as well as a full dump.
0 calls:
https://paste.ubuntu.com/p/Vdrqkfy9Cm/
100 calls:
https://paste.ubuntu.com/p/d765GRqKmx/
200 calls:
https://paste.ubuntu.com/p/rB2rzgZN7S/
300 calls:
https://paste.ubuntu.com/p/mK4gjJ4tcP/
Interestingly there doesn’t appear to be a major difference from 100-300, which seems odd to me. I feel my dev server isn’t giving me results mirrored with production maybe.
Please let me know what you think if you can!
Thanks,
Andrew
On 2 Aug 2019, at 6:19 am, Henning Westerholt hw@skalatan.de wrote:
Hi Andrew,
answer inline below
Am 01.08.19 um 15:34 schrieb Andrew White:
Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/ https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/ https://paste.ubuntu.com/p/9WQXqZtfT2/
There is not that much difference between the two runs (diff):
-journal: Suppressed 19796 messages from /system.slice/kamailio.service +journal: Suppressed 1772 messages from /system.slice/kamailio.service ALERT: <core> [core/pt.c:548]: mem_dump_pkg_cb(): Memory status (pkg) of process 28531: ALERT: qm_status: (0x7f0fc3a2a010): ALERT: qm_status: heap size= 33554432 -ALERT: qm_status: used= 538712, used+overhead=883168, free=32671264 -ALERT: qm_status: max used (+overhead)= 893040 +ALERT: qm_status: used= 590552, used+overhead=939064, free=32615368 +ALERT: qm_status: max used (+overhead)= 948112 ALERT: qm_status: dumping all alloc'ed. fragments: ALERT: qm_status: 0. N address=0x7f0fc3a636d8 frag=0x7f0fc3a636a0 size=1024 used=1 ALERT: qm_status: alloc'd from core: core/str_hash.h: str_hash_alloc(59)
The second run shows about 50k more memory usage, but from the dumped allocations they are identical.
Two suggestions:
your logging system suppresses actually a lot of debugging output from Kamailio (have a look to the first lines each). Check if its necessary to increase the respective configuration variable in your logging conf to get all log messages.
If you are able to execute this test on a test plattform, configure Kamailio to just use one worker child each, to spot an error easier. Then execute 100 calls, and count the memory increase, another 100 calls and count. Then you can estimate how much bytes you actually leak per call. Try another memory dump and see if you spot something. Share your results again on the list.
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Have a look to the output of corex.ps or core.psx - the worker childs are named "receiver".
Cheers,
Henning
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote:
Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
Hello Andrew,
I compared the 0 calls and 300 calls dump, indeed there is not that much of a difference. 688 bytes consumed more, they were allocated in pv_cache_add. Can you try another 300 calls and see if there is the same increase?
Cheers,
Henning
-root 9302 0.0 5.4 329264 26848 ? Sl 21:18 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 +root 9302 0.1 6.9 330460 33904 ? Sl 21:18 0:01 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
INFO: <core> [core/cfg/cfg_ctx.c:595]: cfg_set_now(): core.mem_dump_pkg has been changed to 9302 ALERT: <core> [core/pt.c:548]: mem_dump_pkg_cb(): Memory status (pkg) of process 9302: ALERT: qm_status: (0x7ff63029f010): ALERT: qm_status: heap size= 33554432 -ALERT: qm_status: used= 485216, used+overhead=815112, free=32739320 -ALERT: qm_status: max used (+overhead)= 821904 +ALERT: qm_status: used= 485904, used+overhead=826096, free=32728336 +ALERT: qm_status: max used (+overhead)= 838568 ALERT: qm_status: dumping all alloc'ed. fragments: ALERT: qm_status: 0. N address=0x7ff6302d86d8 frag=0x7ff6302d86a0 size=1024 used=1 ALERT: qm_status: alloc'd from core: core/str_hash.h: str_hash_alloc(59) @@ -2599,23 +2599,49 @@ ALERT: qm_status: 872. N address=0x7ff630365280 frag=0x7ff630365248 size=32 used=1 ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_parse_format(1152) ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed -ALERT: qm_status: 886. N address=0x7ff6303662c8 frag=0x7ff630366290 size=128 used=1 +ALERT: qm_status: 890. N address=0x7ff630365ff8 frag=0x7ff630365fc0 size=144 used=1 +ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347) +ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed +ALERT: qm_status: 894. N address=0x7ff6303662c8 frag=0x7ff630366290 size=128 used=1 +ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347) +ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed +ALERT: qm_status: 919. N address=0x7ff630367550 frag=0x7ff630367518 size=136 used=1 +ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347) +ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed +ALERT: qm_status: 922. N address=0x7ff630367e40 frag=0x7ff630367e08 size=136 used=1 +ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347) +ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed +ALERT: qm_status: 932. N address=0x7ff6303684c8 frag=0x7ff630368490 size=128 used=1 +ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347) +ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed +ALERT: qm_status: 975. N address=0x7ff63036a0a0 frag=0x7ff63036a068 size=144 used=1 ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347) ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Am 02.08.19 um 14:16 schrieb Andrew White: Hi Henning!
Thanks so much!
I spun up a dev machine with the same specs, Kamailio version and config. On each one, I’ve done a ps of the single receiver/worker process, as well as a full dump.
0 calls:
https://paste.ubuntu.com/p/Vdrqkfy9Cm/
100 calls:
https://paste.ubuntu.com/p/d765GRqKmx/
200 calls:
https://paste.ubuntu.com/p/rB2rzgZN7S/
300 calls:
https://paste.ubuntu.com/p/mK4gjJ4tcP/
Interestingly there doesn’t appear to be a major difference from 100-300, which seems odd to me. I feel my dev server isn’t giving me results mirrored with production maybe.
Please let me know what you think if you can!
Thanks,
Andrew
On 2 Aug 2019, at 6:19 am, Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote:
Hi Andrew,
answer inline below
Am 01.08.19 um 15:34 schrieb Andrew White: Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/
There is not that much difference between the two runs (diff):
-journal: Suppressed 19796 messages from /system.slice/kamailio.service +journal: Suppressed 1772 messages from /system.slice/kamailio.service ALERT: <core> [core/pt.c:548]: mem_dump_pkg_cb(): Memory status (pkg) of process 28531: ALERT: qm_status: (0x7f0fc3a2a010): ALERT: qm_status: heap size= 33554432 -ALERT: qm_status: used= 538712, used+overhead=883168, free=32671264 -ALERT: qm_status: max used (+overhead)= 893040 +ALERT: qm_status: used= 590552, used+overhead=939064, free=32615368 +ALERT: qm_status: max used (+overhead)= 948112 ALERT: qm_status: dumping all alloc'ed. fragments: ALERT: qm_status: 0. N address=0x7f0fc3a636d8 frag=0x7f0fc3a636a0 size=1024 used=1 ALERT: qm_status: alloc'd from core: core/str_hash.h: str_hash_alloc(59)
The second run shows about 50k more memory usage, but from the dumped allocations they are identical.
Two suggestions:
- your logging system suppresses actually a lot of debugging output from Kamailio (have a look to the first lines each). Check if its necessary to increase the respective configuration variable in your logging conf to get all log messages.
- If you are able to execute this test on a test plattform, configure Kamailio to just use one worker child each, to spot an error easier. Then execute 100 calls, and count the memory increase, another 100 calls and count. Then you can estimate how much bytes you actually leak per call. Try another memory dump and see if you spot something. Share your results again on the list.
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Have a look to the output of corex.ps or core.psx - the worker childs are named "receiver".
Cheers,
Henning
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg...
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote: Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
Hello,
are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message?
Cheers, Daniel
On 01.08.19 15:34, Andrew White wrote:
Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg...
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote:
Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’])
For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?
Thanks!
Andrew
On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message?
Cheers, Daniel
On 01.08.19 15:34, Andrew White wrote:
Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/ https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/ https://paste.ubuntu.com/p/9WQXqZtfT2/
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote:
Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
https://paste.ubuntu.com/p/Y6Wvd9hsV2/
Second one (about 8 hours after the first, 3200 odd cals later):
https://paste.ubuntu.com/p/85KYg7gD2z/
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?
Thanks!
Andrew
On 3 Aug 2019, at 10:07 am, Andrew White andrew@uconnected.com.au wrote:
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’])
For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?
Thanks!
Andrew
On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello,
are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message?
Cheers, Daniel
On 01.08.19 15:34, Andrew White wrote:
Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/ https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/ https://paste.ubuntu.com/p/9WQXqZtfT2/
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote:
Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Hi Andrew,
interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use?
Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic?
Cheers,
Henning
Am 03.08.19 um 09:32 schrieb Andrew White: Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
https://paste.ubuntu.com/p/Y6Wvd9hsV2/
Second one (about 8 hours after the first, 3200 odd cals later):
https://paste.ubuntu.com/p/85KYg7gD2z/
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?
Thanks!
Andrew
On 3 Aug 2019, at 10:07 am, Andrew White <andrew@uconnected.com.aumailto:andrew@uconnected.com.au> wrote:
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’]) For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?
Thanks!
Andrew
On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message?
Cheers, Daniel
On 01.08.19 15:34, Andrew White wrote: Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg...
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote: Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
Hey Henning,
It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty.
I don’t think there’s any extra messages. The actual dispatching is simple, here’s the code:
if destination then KSR::PV.sets("$du", destination['sip_uri']) pbx = destination['display_name'] KSR.info("Chose #{pbx} as inbound PBX, as it's received #{destination['count'].to_i}/#{destination['total_count'].to_i} (#{destination['calculated_percentage'].to_i}%) of calls in this ruleset, where its max is #{destination['percentage'].to_i}%") end
The code before that just pulls data from redis and returns a ruby hash with where we should send the call, so no kamailio variables set at all.
Thanks!
Andrew
On 3 Aug 2019, at 8:03 pm, Henning Westerholt hw@skalatan.de wrote:
Hi Andrew,
interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use?
Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic?
Cheers,
Henning
Am 03.08.19 um 09:32 schrieb Andrew White:
Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
https://paste.ubuntu.com/p/Y6Wvd9hsV2/ https://paste.ubuntu.com/p/Y6Wvd9hsV2/
Second one (about 8 hours after the first, 3200 odd cals later):
https://paste.ubuntu.com/p/85KYg7gD2z/ https://paste.ubuntu.com/p/85KYg7gD2z/
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?
Thanks!
Andrew
On 3 Aug 2019, at 10:07 am, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote:
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’]) For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?
Thanks!
Andrew
On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello,
are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message?
Cheers, Daniel
On 01.08.19 15:34, Andrew White wrote:
Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/ https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/ https://paste.ubuntu.com/p/9WQXqZtfT2/
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote: > Hi all, > > I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory. > > Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours. > > https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png > > Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory. > > I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down? > > Thanks! > > Andrew > > > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
Hi Andrew,
dev4 version - I see. I quickly looked into the topos module commits, but there have been not that much changes there. But can you try nevertheless to update to the latest git master (e.g. dev7) version on your test setup?
What about the "dirty" version flag, do you have some own patches or modules that you use?
Cheers,
Henning
Am 04.08.19 um 01:42 schrieb Andrew White: Hey Henning,
It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty.
I don’t think there’s any extra messages. The actual dispatching is simple, here’s the code:
1. if destination then 2. KSR::PV.sets("$du", destination['sip_uri']) 3. pbx = destination['display_name'] 4. KSR.infohttp://KSR.info("Chose #{pbx} as inbound PBX, as it's received #{destination['count'].to_i}/#{destination['total_count'].to_i} (#{destination['calculated_percentage'].to_i}%) of calls in this ruleset, where its max is #{destination['percentage'].to_i}%") 5. end
The code before that just pulls data from redis and returns a ruby hash with where we should send the call, so no kamailio variables set at all.
Thanks!
Andrew
On 3 Aug 2019, at 8:03 pm, Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote:
Hi Andrew,
interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use?
Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic?
Cheers,
Henning
Am 03.08.19 um 09:32 schrieb Andrew White: Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
https://paste.ubuntu.com/p/Y6Wvd9hsV2/
Second one (about 8 hours after the first, 3200 odd cals later):
https://paste.ubuntu.com/p/85KYg7gD2z/
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?
Thanks!
Andrew
On 3 Aug 2019, at 10:07 am, Andrew White <andrew@uconnected.com.aumailto:andrew@uconnected.com.au> wrote:
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’]) For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?
Thanks!
Andrew
On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message?
Cheers, Daniel
On 01.08.19 15:34, Andrew White wrote: Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg...
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote: Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
Hi Henning,
That is interesting, I don’t know what the dirty version is. My deploy script I use for Kamailio pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as this had a fix in place Daniel did for app_ruby a while back. I don’t recall ever putting on any custom patches or modules. I compile from source on build from that git commit using:
make include_modules="app_ruby db_redis" cfg
I’ll deploy dev7 tonight and report back!
Thanks,
Andrew
On 4 Aug 2019, at 8:28 pm, Henning Westerholt hw@skalatan.de wrote:
Hi Andrew,
dev4 version - I see. I quickly looked into the topos module commits, but there have been not that much changes there. But can you try nevertheless to update to the latest git master (e.g. dev7) version on your test setup?
What about the "dirty" version flag, do you have some own patches or modules that you use?
Cheers,
Henning
Am 04.08.19 um 01:42 schrieb Andrew White:
Hey Henning,
It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty.
I don’t think there’s any extra messages. The actual dispatching is simple, here’s the code:
if destination
then
KSR::PV.sets("$du", destination['sip_uri'])
pbx = destination['display_name']
KSR.info http://ksr.info/("Chose
#{pbx} as inbound PBX, as it's received #{destination['count'].to_i}/#{destination['total_count'].to_i} (#{destination['calculated_percentage'].to_i}%) of calls in this ruleset, where its max is #{destination['percentage'].to_i}%")
end
The code before that just pulls data from redis and returns a ruby hash with where we should send the call, so no kamailio variables set at all.
Thanks!
Andrew
On 3 Aug 2019, at 8:03 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hi Andrew,
interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use?
Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic?
Cheers,
Henning
Am 03.08.19 um 09:32 schrieb Andrew White:
Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
https://paste.ubuntu.com/p/Y6Wvd9hsV2/ https://paste.ubuntu.com/p/Y6Wvd9hsV2/
Second one (about 8 hours after the first, 3200 odd cals later):
https://paste.ubuntu.com/p/85KYg7gD2z/ https://paste.ubuntu.com/p/85KYg7gD2z/
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?
Thanks!
Andrew
On 3 Aug 2019, at 10:07 am, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote:
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’]) For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?
Thanks!
Andrew
On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello,
are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message?
Cheers, Daniel
On 01.08.19 15:34, Andrew White wrote: > Thanks Daniel, you’re fantastic! > > I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD. > > I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart: > > root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > > After about 90 calls: > > root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > > None of the other 9 Kamailio processes are increasing at all. > > I ran corex.pkg_summary against one of them and got the following dump: > > https://paste.ubuntu.com/p/SqTF3K5knK/ https://paste.ubuntu.com/p/SqTF3K5knK/ > > I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset? > > Here’s another after another 60 calls: > > https://paste.ubuntu.com/p/9WQXqZtfT2/ https://paste.ubuntu.com/p/9WQXqZtfT2/ > > root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 > > The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module. > > It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring. > > Thanks! > > Andrew > > >> On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote: >> >> Hello, >> >> if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid. >> >> Once you indentify a process that is leaking pkg, execute the rpc command: >> >> - https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary >> When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug. >> >> Cheers, >> Daniel >> >> >> >> On 01.08.19 03:43, Andrew White wrote: >>> Hi all, >>> >>> I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory. >>> >>> Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours. >>> >>> https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png >>> >>> Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory. >>> >>> I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down? >>> >>> Thanks! >>> >>> Andrew >>> >>> >>> _______________________________________________ >>> Kamailio (SER) - Users Mailing List >>> sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> -- >> Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/
>> www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
Hey Henning,
Unfortunately I’ve had some troubles with app_ruby in dev-7. Interestingly, dev-7 shows as “dirty” too:
version: kamailio 5.3.0-dev7 (x86_64/linux) bd477f-dirty
Here’s the ansible playbook I wrote to pull the specific version, there’s no fancy patching or anything going on:
- name: Download kamailio git: repo: https://github.com/kamailio/kamailio.git version: "{{ kamailio_version }}" dest: /usr/src/kamailio
- name: Roll back to specific commit command: "git reset --hard {{ kamailio_commit }}" args: chdir: /usr/src/kamailio when: kamailio_commit is defined
- name: Build kamailio from source command: "{{ item }} chdir=/usr/src/kamailio" with_items: - make include_modules="app_ruby db_redis" cfg - make all - make install args: creates: /usr/local/sbin/kamailio
Either way, I’ve removed the topoh module from my production (dev-4) machine, and re-taken memory information. I’ll give it about 12 hours and see where memory is sitting. I figure this should isolate if it’s a topoh issue or not.
Thanks!
Andrew
On 5 Aug 2019, at 10:46 am, Andrew White andrew@uconnected.com.au wrote:
Hi Henning,
That is interesting, I don’t know what the dirty version is. My deploy script I use for Kamailio pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as this had a fix in place Daniel did for app_ruby a while back. I don’t recall ever putting on any custom patches or modules. I compile from source on build from that git commit using:
make include_modules="app_ruby db_redis" cfg
I’ll deploy dev7 tonight and report back!
Thanks,
Andrew
On 4 Aug 2019, at 8:28 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hi Andrew,
dev4 version - I see. I quickly looked into the topos module commits, but there have been not that much changes there. But can you try nevertheless to update to the latest git master (e.g. dev7) version on your test setup?
What about the "dirty" version flag, do you have some own patches or modules that you use?
Cheers,
Henning
Am 04.08.19 um 01:42 schrieb Andrew White:
Hey Henning,
It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty.
I don’t think there’s any extra messages. The actual dispatching is simple, here’s the code:
if destination
then
KSR::PV.sets("$du", destination['sip_uri'])
pbx = destination['display_name']
KSR.info http://ksr.info/("Chose
#{pbx} as inbound PBX, as it's received #{destination['count'].to_i}/#{destination['total_count'].to_i} (#{destination['calculated_percentage'].to_i}%) of calls in this ruleset, where its max is #{destination['percentage'].to_i}%")
end
The code before that just pulls data from redis and returns a ruby hash with where we should send the call, so no kamailio variables set at all.
Thanks!
Andrew
On 3 Aug 2019, at 8:03 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hi Andrew,
interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use?
Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic?
Cheers,
Henning
Am 03.08.19 um 09:32 schrieb Andrew White:
Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
https://paste.ubuntu.com/p/Y6Wvd9hsV2/ https://paste.ubuntu.com/p/Y6Wvd9hsV2/
Second one (about 8 hours after the first, 3200 odd cals later):
https://paste.ubuntu.com/p/85KYg7gD2z/ https://paste.ubuntu.com/p/85KYg7gD2z/
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?
Thanks!
Andrew
On 3 Aug 2019, at 10:07 am, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote:
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’]) For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?
Thanks!
Andrew
> On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote: > > Hello, > > are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message? > > Cheers, > Daniel > > On 01.08.19 15:34, Andrew White wrote: >> Thanks Daniel, you’re fantastic! >> >> I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD. >> >> I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart: >> >> root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> >> After about 90 calls: >> >> root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> >> None of the other 9 Kamailio processes are increasing at all. >> >> I ran corex.pkg_summary against one of them and got the following dump: >> >> https://paste.ubuntu.com/p/SqTF3K5knK/ https://paste.ubuntu.com/p/SqTF3K5knK/ >> >> I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset? >> >> Here’s another after another 60 calls: >> >> https://paste.ubuntu.com/p/9WQXqZtfT2/ https://paste.ubuntu.com/p/9WQXqZtfT2/ >> >> root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >> >> The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module. >> >> It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring. >> >> Thanks! >> >> Andrew >> >> >>> On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote: >>> >>> Hello, >>> >>> if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid. >>> >>> Once you indentify a process that is leaking pkg, execute the rpc command: >>> >>> - https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary >>> When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug. >>> >>> Cheers, >>> Daniel >>> >>> >>> >>> On 01.08.19 03:43, Andrew White wrote: >>>> Hi all, >>>> >>>> I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory. >>>> >>>> Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours. >>>> >>>> https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png >>>> >>>> Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory. >>>> >>>> I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down? >>>> >>>> Thanks! >>>> >>>> Andrew >>>> >>>> >>>> _______________________________________________ >>>> Kamailio (SER) - Users Mailing List >>>> sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org >>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>> -- >>> Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ >>> www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda > -- > Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ > www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
Very interesting results!
11pm last night: https://paste.ubuntu.com/p/qMz2tN7bcR/
2pm today: https://paste.ubuntu.com/p/QDmTwpV83X/
The growth is pretty small. The only variance is disabling the topoh module.
I’m not big on C, but I think this indicates a leak in the topoh module, does this sound right?
I’m happy to put in an issue on GitHub if that’s necessary here.
Cheers,
Andrew
On 5 Aug 2019, at 11:30 pm, Andrew White andrew@uconnected.com.au wrote:
Hey Henning,
Unfortunately I’ve had some troubles with app_ruby in dev-7. Interestingly, dev-7 shows as “dirty” too:
version: kamailio 5.3.0-dev7 (x86_64/linux) bd477f-dirty
Here’s the ansible playbook I wrote to pull the specific version, there’s no fancy patching or anything going on:
name: Download kamailio git: repo: https://github.com/kamailio/kamailio.git https://github.com/kamailio/kamailio.git version: "{{ kamailio_version }}" dest: /usr/src/kamailio
name: Roll back to specific commit command: "git reset --hard {{ kamailio_commit }}" args: chdir: /usr/src/kamailio when: kamailio_commit is defined
name: Build kamailio from source command: "{{ item }} chdir=/usr/src/kamailio" with_items:
- make include_modules="app_ruby db_redis" cfg
- make all
- make install
args: creates: /usr/local/sbin/kamailio
Either way, I’ve removed the topoh module from my production (dev-4) machine, and re-taken memory information. I’ll give it about 12 hours and see where memory is sitting. I figure this should isolate if it’s a topoh issue or not.
Thanks!
Andrew
On 5 Aug 2019, at 10:46 am, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote:
Hi Henning,
That is interesting, I don’t know what the dirty version is. My deploy script I use for Kamailio pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as this had a fix in place Daniel did for app_ruby a while back. I don’t recall ever putting on any custom patches or modules. I compile from source on build from that git commit using:
make include_modules="app_ruby db_redis" cfg
I’ll deploy dev7 tonight and report back!
Thanks,
Andrew
On 4 Aug 2019, at 8:28 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hi Andrew,
dev4 version - I see. I quickly looked into the topos module commits, but there have been not that much changes there. But can you try nevertheless to update to the latest git master (e.g. dev7) version on your test setup?
What about the "dirty" version flag, do you have some own patches or modules that you use?
Cheers,
Henning
Am 04.08.19 um 01:42 schrieb Andrew White:
Hey Henning,
It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty.
I don’t think there’s any extra messages. The actual dispatching is simple, here’s the code:
if destination
then
KSR::PV.sets("$du", destination['sip_uri'])
pbx = destination['display_name']
KSR.info http://ksr.info/("Chose
#{pbx} as inbound PBX, as it's received #{destination['count'].to_i}/#{destination['total_count'].to_i} (#{destination['calculated_percentage'].to_i}%) of calls in this ruleset, where its max is #{destination['percentage'].to_i}%")
end
The code before that just pulls data from redis and returns a ruby hash with where we should send the call, so no kamailio variables set at all.
Thanks!
Andrew
On 3 Aug 2019, at 8:03 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hi Andrew,
interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use?
Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic?
Cheers,
Henning
Am 03.08.19 um 09:32 schrieb Andrew White:
Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
https://paste.ubuntu.com/p/Y6Wvd9hsV2/ https://paste.ubuntu.com/p/Y6Wvd9hsV2/
Second one (about 8 hours after the first, 3200 odd cals later):
https://paste.ubuntu.com/p/85KYg7gD2z/ https://paste.ubuntu.com/p/85KYg7gD2z/
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?
Thanks!
Andrew
> On 3 Aug 2019, at 10:07 am, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote: > > Hi Daniel/Henning, > > Thanks so much for your quick responses, they’re very much appreciated! > > Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module. > > Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby): > > KSR::PV.sets("$du", > destination['sip_uri’]) > For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok? > > Thanks! > > Andrew > >> On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote: >> >> Hello, >> >> are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message? >> >> Cheers, >> Daniel >> >> On 01.08.19 15:34, Andrew White wrote: >>> Thanks Daniel, you’re fantastic! >>> >>> I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD. >>> >>> I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart: >>> >>> root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> >>> After about 90 calls: >>> >>> root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> >>> None of the other 9 Kamailio processes are increasing at all. >>> >>> I ran corex.pkg_summary against one of them and got the following dump: >>> >>> https://paste.ubuntu.com/p/SqTF3K5knK/ https://paste.ubuntu.com/p/SqTF3K5knK/ >>> >>> I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset? >>> >>> Here’s another after another 60 calls: >>> >>> https://paste.ubuntu.com/p/9WQXqZtfT2/ https://paste.ubuntu.com/p/9WQXqZtfT2/ >>> >>> root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>> >>> The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module. >>> >>> It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring. >>> >>> Thanks! >>> >>> Andrew >>> >>> >>>> On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote: >>>> >>>> Hello, >>>> >>>> if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid. >>>> >>>> Once you indentify a process that is leaking pkg, execute the rpc command: >>>> >>>> - https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary >>>> When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> >>>> >>>> On 01.08.19 03:43, Andrew White wrote: >>>>> Hi all, >>>>> >>>>> I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory. >>>>> >>>>> Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours. >>>>> >>>>> https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png >>>>> >>>>> Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory. >>>>> >>>>> I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down? >>>>> >>>>> Thanks! >>>>> >>>>> Andrew >>>>> >>>>> >>>>> _______________________________________________ >>>>> Kamailio (SER) - Users Mailing List >>>>> sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> -- >>>> Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ >>>> www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda >> -- >> Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ >> www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
Hello Andrew,
indeed, the difference is only 1.4k or so memory. This is probably just an effect of the different time when you did the dump.
Yes, an Github issue would be great. Add this dump as well https://paste.ubuntu.com/p/85KYg7gD2z/
Would be perfect if you could reproduce it with current git master, though (if the app_ruby problem is also solved).
Cheers,
Henning
Am 06.08.19 um 09:17 schrieb Andrew White: Very interesting results!
11pm last night: https://paste.ubuntu.com/p/qMz2tN7bcR/
2pm today: https://paste.ubuntu.com/p/QDmTwpV83X/
The growth is pretty small. The only variance is disabling the topoh module.
I’m not big on C, but I think this indicates a leak in the topoh module, does this sound right?
I’m happy to put in an issue on GitHub if that’s necessary here.
Cheers,
Andrew
On 5 Aug 2019, at 11:30 pm, Andrew White <andrew@uconnected.com.aumailto:andrew@uconnected.com.au> wrote:
Hey Henning,
Unfortunately I’ve had some troubles with app_ruby in dev-7. Interestingly, dev-7 shows as “dirty” too:
version: kamailio 5.3.0-dev7 (x86_64/linux) bd477f-dirty
Here’s the ansible playbook I wrote to pull the specific version, there’s no fancy patching or anything going on:
- name: Download kamailio git: repo: https://github.com/kamailio/kamailio.git version: "{{ kamailio_version }}" dest: /usr/src/kamailio - name: Roll back to specific commit command: "git reset --hard {{ kamailio_commit }}" args: chdir: /usr/src/kamailio when: kamailio_commit is defined - name: Build kamailio from source command: "{{ item }} chdir=/usr/src/kamailio" with_items: - make include_modules="app_ruby db_redis" cfg - make all - make install args: creates: /usr/local/sbin/kamailio
Either way, I’ve removed the topoh module from my production (dev-4) machine, and re-taken memory information. I’ll give it about 12 hours and see where memory is sitting. I figure this should isolate if it’s a topoh issue or not.
Thanks!
Andrew
On 5 Aug 2019, at 10:46 am, Andrew White <andrew@uconnected.com.aumailto:andrew@uconnected.com.au> wrote:
Hi Henning,
That is interesting, I don’t know what the dirty version is. My deploy script I use for Kamailio pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as this had a fix in place Daniel did for app_ruby a while back. I don’t recall ever putting on any custom patches or modules. I compile from source on build from that git commit using:
make include_modules="app_ruby db_redis" cfg
I’ll deploy dev7 tonight and report back!
Thanks,
Andrew
On 4 Aug 2019, at 8:28 pm, Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote:
Hi Andrew,
dev4 version - I see. I quickly looked into the topos module commits, but there have been not that much changes there. But can you try nevertheless to update to the latest git master (e.g. dev7) version on your test setup?
What about the "dirty" version flag, do you have some own patches or modules that you use?
Cheers,
Henning
Am 04.08.19 um 01:42 schrieb Andrew White: Hey Henning,
It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty.
I don’t think there’s any extra messages. The actual dispatching is simple, here’s the code:
1. if destination then 2. KSR::PV.sets("$du", destination['sip_uri']) 3. pbx = destination['display_name'] 4. KSR.infohttp://ksr.info/("Chose #{pbx} as inbound PBX, as it's received #{destination['count'].to_i}/#{destination['total_count'].to_i} (#{destination['calculated_percentage'].to_i}%) of calls in this ruleset, where its max is #{destination['percentage'].to_i}%") 5. end
The code before that just pulls data from redis and returns a ruby hash with where we should send the call, so no kamailio variables set at all.
Thanks!
Andrew
On 3 Aug 2019, at 8:03 pm, Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote:
Hi Andrew,
interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use?
Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic?
Cheers,
Henning
Am 03.08.19 um 09:32 schrieb Andrew White: Hi all,
I’ve done two outputs about 8 hours/3200 calls in between:
First one (about 5 hours since last restart):
https://paste.ubuntu.com/p/Y6Wvd9hsV2/
Second one (about 8 hours after the first, 3200 odd cals later):
https://paste.ubuntu.com/p/85KYg7gD2z/
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?
Thanks!
Andrew
On 3 Aug 2019, at 10:07 am, Andrew White <andrew@uconnected.com.aumailto:andrew@uconnected.com.au> wrote:
Hi Daniel/Henning,
Thanks so much for your quick responses, they’re very much appreciated!
Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module.
Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):
KSR::PV.sets("$du", destination['sip_uri’]) For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?
Thanks!
Andrew
On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message?
Cheers, Daniel
On 01.08.19 15:34, Andrew White wrote: Thanks Daniel, you’re fantastic!
I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.
I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:
root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
After about 90 calls:
root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
None of the other 9 Kamailio processes are increasing at all.
I ran corex.pkg_summary against one of them and got the following dump:
https://paste.ubuntu.com/p/SqTF3K5knK/
I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?
Here’s another after another 60 calls:
https://paste.ubuntu.com/p/9WQXqZtfT2/
root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32
The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.
It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.
Thanks!
Andrew
On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.
Once you indentify a process that is leaking pkg, execute the rpc command:
- https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg...
When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.
Cheers, Daniel
On 01.08.19 03:43, Andrew White wrote: Hi all,
I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.
Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours.
https://i.imgur.com/UTzx2k1.png
Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.
I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?
Thanks!
Andrew
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
If anyone is following this in the future and looking for further information, here is the GitHub issue:
https://github.com/kamailio/kamailio/issues/2027
Thanks!
Andrew
On 6 Aug 2019, at 5:23 pm, Henning Westerholt hw@skalatan.de wrote:
Hello Andrew,
indeed, the difference is only 1.4k or so memory. This is probably just an effect of the different time when you did the dump.
Yes, an Github issue would be great. Add this dump as well https://paste.ubuntu.com/p/85KYg7gD2z/ https://paste.ubuntu.com/p/85KYg7gD2z/ Would be perfect if you could reproduce it with current git master, though (if the app_ruby problem is also solved).
Cheers,
Henning
Am 06.08.19 um 09:17 schrieb Andrew White:
Very interesting results!
11pm last night: https://paste.ubuntu.com/p/qMz2tN7bcR/ https://paste.ubuntu.com/p/qMz2tN7bcR/
2pm today: https://paste.ubuntu.com/p/QDmTwpV83X/ https://paste.ubuntu.com/p/QDmTwpV83X/
The growth is pretty small. The only variance is disabling the topoh module.
I’m not big on C, but I think this indicates a leak in the topoh module, does this sound right?
I’m happy to put in an issue on GitHub if that’s necessary here.
Cheers,
Andrew
On 5 Aug 2019, at 11:30 pm, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote:
Hey Henning,
Unfortunately I’ve had some troubles with app_ruby in dev-7. Interestingly, dev-7 shows as “dirty” too:
version: kamailio 5.3.0-dev7 (x86_64/linux) bd477f-dirty
Here’s the ansible playbook I wrote to pull the specific version, there’s no fancy patching or anything going on:
- name:
Download kamailio
git:
repo: https://github.com/kamailio/kamailio.git https://github.com/kamailio/kamailio.git version: "{{ kamailio_version }}"
dest: /usr/src/kamailio
- name:
Roll back to specific commit
command: "git reset --hard {{ kamailio_commit }}"
args:
chdir: /usr/src/kamailio
when: kamailio_commit is defined
- name:
Build kamailio from source
command: "{{ item }} chdir=/usr/src/kamailio"
with_items:
make include_modules="app_ruby db_redis" cfg
make all
make install
args:
creates: /usr/local/sbin/kamailio
Either way, I’ve removed the topoh module from my production (dev-4) machine, and re-taken memory information. I’ll give it about 12 hours and see where memory is sitting. I figure this should isolate if it’s a topoh issue or not.
Thanks!
Andrew
On 5 Aug 2019, at 10:46 am, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote:
Hi Henning,
That is interesting, I don’t know what the dirty version is. My deploy script I use for Kamailio pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as this had a fix in place Daniel did for app_ruby a while back. I don’t recall ever putting on any custom patches or modules. I compile from source on build from that git commit using:
make include_modules="app_ruby db_redis" cfg
I’ll deploy dev7 tonight and report back!
Thanks,
Andrew
On 4 Aug 2019, at 8:28 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hi Andrew,
dev4 version - I see. I quickly looked into the topos module commits, but there have been not that much changes there. But can you try nevertheless to update to the latest git master (e.g. dev7) version on your test setup?
What about the "dirty" version flag, do you have some own patches or modules that you use?
Cheers,
Henning
Am 04.08.19 um 01:42 schrieb Andrew White:
Hey Henning,
It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty.
I don’t think there’s any extra messages. The actual dispatching is simple, here’s the code:
if destination
then
KSR::PV.sets("$du", destination['sip_uri'])
pbx = destination['display_name']
KSR.info http://ksr.info/("Chose
#{pbx} as inbound PBX, as it's received #{destination['count'].to_i}/#{destination['total_count'].to_i} (#{destination['calculated_percentage'].to_i}%) of calls in this ruleset, where its max is #{destination['percentage'].to_i}%")
end
The code before that just pulls data from redis and returns a ruby hash with where we should send the call, so no kamailio variables set at all.
Thanks!
Andrew
> On 3 Aug 2019, at 8:03 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: > > Hi Andrew, > > interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use? > > Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic? > > Cheers, > > Henning > > Am 03.08.19 um 09:32 schrieb Andrew White: >> Hi all, >> >> I’ve done two outputs about 8 hours/3200 calls in between: >> >> First one (about 5 hours since last restart): >> >> https://paste.ubuntu.com/p/Y6Wvd9hsV2/ https://paste.ubuntu.com/p/Y6Wvd9hsV2/ >> >> Second one (about 8 hours after the first, 3200 odd cals later): >> >> https://paste.ubuntu.com/p/85KYg7gD2z/ https://paste.ubuntu.com/p/85KYg7gD2z/ >> >> It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first). >> >> I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin? >> >> Thanks! >> >> Andrew >> >>> On 3 Aug 2019, at 10:07 am, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote: >>> >>> Hi Daniel/Henning, >>> >>> Thanks so much for your quick responses, they’re very much appreciated! >>> >>> Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module. >>> >>> Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby): >>> >>> KSR::PV.sets("$du", >>> destination['sip_uri’]) >>> For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok? >>> >>> Thanks! >>> >>> Andrew >>> >>>> On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote: >>>> >>>> Hello, >>>> >>>> are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip message? >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 01.08.19 15:34, Andrew White wrote: >>>>> Thanks Daniel, you’re fantastic! >>>>> >>>>> I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD. >>>>> >>>>> I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart: >>>>> >>>>> root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> >>>>> After about 90 calls: >>>>> >>>>> root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> >>>>> None of the other 9 Kamailio processes are increasing at all. >>>>> >>>>> I ran corex.pkg_summary against one of them and got the following dump: >>>>> >>>>> https://paste.ubuntu.com/p/SqTF3K5knK/ https://paste.ubuntu.com/p/SqTF3K5knK/ >>>>> >>>>> I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset? >>>>> >>>>> Here’s another after another 60 calls: >>>>> >>>>> https://paste.ubuntu.com/p/9WQXqZtfT2/ https://paste.ubuntu.com/p/9WQXqZtfT2/ >>>>> >>>>> root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>> >>>>> The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module. >>>>> >>>>> It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring. >>>>> >>>>> Thanks! >>>>> >>>>> Andrew >>>>> >>>>> >>>>>> On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid. >>>>>> >>>>>> Once you indentify a process that is leaking pkg, execute the rpc command: >>>>>> >>>>>> - https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary >>>>>> When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug. >>>>>> >>>>>> Cheers, >>>>>> Daniel >>>>>> >>>>>> >>>>>> >>>>>> On 01.08.19 03:43, Andrew White wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory. >>>>>>> >>>>>>> Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory going on, mostly increasing during our busier daytime hours. >>>>>>> >>>>>>> https://i.imgur.com/UTzx2k1.png https://i.imgur.com/UTzx2k1.png >>>>>>> >>>>>>> Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory. >>>>>>> >>>>>>> I’ve been reading through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, but I’m a bit nervous, as I’m not really a C/deep memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down? >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Kamailio (SER) - Users Mailing List >>>>>>> sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org >>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>>> -- >>>>>> Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ >>>>>> www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda >>>> -- >>>> Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ >>>> www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda >> >> >> >> _______________________________________________ >> Kamailio (SER) - Users Mailing List >> sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > -- > Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ > Kamailio services - https://skalatan.de/services https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services https://skalatan.de/services
To close this discussion - patches for fixing this issue are now in master and 5.2 branches.
Cheers, Daniel
On 09.08.19 09:12, Andrew White wrote:
If anyone is following this in the future and looking for further information, here is the GitHub issue:
https://github.com/kamailio/kamailio/issues/2027
Thanks!
Andrew
On 6 Aug 2019, at 5:23 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hello Andrew,
indeed, the difference is only 1.4k or so memory. This is probably just an effect of the different time when you did the dump.
Yes, an Github issue would be great. Add this dump as well https://paste.ubuntu.com/p/85KYg7gD2z/
Would be perfect if you could reproduce it with current git master, though (if the app_ruby problem is also solved).
Cheers,
Henning
Am 06.08.19 um 09:17 schrieb Andrew White:
Very interesting results!
11pm last night: https://paste.ubuntu.com/p/qMz2tN7bcR/
2pm today: https://paste.ubuntu.com/p/QDmTwpV83X/
The growth is pretty small. The only variance is disabling the topoh module.
I’m not big on C, but I think this indicates a leak in the topoh module, does this sound right?
I’m happy to put in an issue on GitHub if that’s necessary here.
Cheers,
Andrew
On 5 Aug 2019, at 11:30 pm, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote:
Hey Henning,
Unfortunately I’ve had some troubles with app_ruby in dev-7. Interestingly, dev-7 shows as “dirty” too:
version: kamailio 5.3.0-dev7 (x86_64/linux) bd477f-dirty
Here’s the ansible playbook I wrote to pull the specific version, there’s no fancy patching or anything going on:
- name: Download kamailio
git: repo: https://github.com/kamailio/kamailio.git version: "{{ kamailio_version }}" dest: /usr/src/kamailio
- name: Roll back to specific commit
command: "git reset --hard {{ kamailio_commit }}" args: chdir: /usr/src/kamailio when: kamailio_commit is defined
- name: Build kamailio from source
command: "{{ item }} chdir=/usr/src/kamailio" with_items:
- make include_modules="app_ruby db_redis" cfg
- make all
- make install
args: creates: /usr/local/sbin/kamailio
Either way, I’ve removed the topoh module from my production (dev-4) machine, and re-taken memory information. I’ll give it about 12 hours and see where memory is sitting. I figure this should isolate if it’s a topoh issue or not.
Thanks!
Andrew
On 5 Aug 2019, at 10:46 am, Andrew White <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> wrote:
Hi Henning,
That is interesting, I don’t know what the dirty version is. My deploy script I use for Kamailio pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as this had a fix in place Daniel did for app_ruby a while back. I don’t recall ever putting on any custom patches or modules. I compile from source on build from that git commit using:
makeinclude_modules="app_rubydb_redis"cfg
I’ll deploy dev7 tonight and report back!
Thanks,
Andrew
On 4 Aug 2019, at 8:28 pm, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hi Andrew,
dev4 version - I see. I quickly looked into the topos module commits, but there have been not that much changes there. But can you try nevertheless to update to the latest git master (e.g. dev7) version on your test setup?
What about the "dirty" version flag, do you have some own patches or modules that you use?
Cheers,
Henning
Am 04.08.19 um 01:42 schrieb Andrew White: > Hey Henning, > > It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) > 6d43ea-dirty. > > I don’t think there’s any extra messages. The actual dispatching > is simple, here’s the code: > > 1. if destination then > 2. KSR::PV.sets("$du", destination['sip_uri']) > 3. pbx = destination['display_name'] > 4. KSR.info http://ksr.info/("Chose #{pbx} as inbound PBX, as > it's received > #{destination['count'].to_i}/#{destination['total_count'].to_i} > (#{destination['calculated_percentage'].to_i}%) of calls in > this ruleset, where its max is > #{destination['percentage'].to_i}%") > 5. end > > > The code before that just pulls data from redis and returns a > ruby hash with where we should send the call, so no kamailio > variables set at all. > > Thanks! > > Andrew > >> On 3 Aug 2019, at 8:03 pm, Henning Westerholt <hw@skalatan.de >> mailto:hw@skalatan.de> wrote: >> >> Hi Andrew, >> >> interesting insight from the logs indeed. According to your >> changes this looks indeed like related to the topoh module or >> custom dispatcher code. Which version of Kamailio do you use? >> >> Are you sending out some messages /replies as well from your >> custom dispatcher app_ruby logic? >> >> Cheers, >> >> Henning >> >> Am 03.08.19 um 09:32 schrieb Andrew White: >>> Hi all, >>> >>> I’ve done two outputs about 8 hours/3200 calls in between: >>> >>> First one (about 5 hours since last restart): >>> >>> https://paste.ubuntu.com/p/Y6Wvd9hsV2/ >>> >>> Second one (about 8 hours after the first, 3200 odd cals later): >>> >>> https://paste.ubuntu.com/p/85KYg7gD2z/ >>> >>> It looks like roughly 7MB of extra memory is being used on >>> that process, which is 1/4 active workers (I can see all of >>> them have grown roughly the same amount in that time). Looking >>> at the memory status, there appear to be about 14,000 more >>> lines. A cursory glance shows about 6k of those lines >>> with msg_translator.c (compared to 1k in the first). >>> >>> I wonder if maybe this could be related to my use of the topoh >>> module, given every message is now being touched by it to mask >>> the origin? >>> >>> Thanks! >>> >>> Andrew >>> >>>> On 3 Aug 2019, at 10:07 am, Andrew White >>>> <andrew@uconnected.com.au mailto:andrew@uconnected.com.au> >>>> wrote: >>>> >>>> Hi Daniel/Henning, >>>> >>>> Thanks so much for your quick responses, they’re very much >>>> appreciated! >>>> >>>> Daniel: I don’t believe so. I’ve just checked over the code, >>>> and I only have 9 instances of PV.sets throughout the entire >>>> script. Additionally, this issue appears to have only cropped >>>> up in the last week or two (I didn’t monitor PKG memory at >>>> the time, but we didn’t have any memory related crashes >>>> before that point), which leads me to believe it’s related to >>>> my config code, the adding of the topoh module or the removal >>>> of the dispatcher module. >>>> >>>> Within that config code, there’s only one additional Kamailio >>>> variable set, which is part of my replacement/custom >>>> dispatcher (app_ruby): >>>> >>>> KSR::PV.sets("$du", destination['sip_uri’]) >>>> For now, I’ve taken the output of a mem dump of one of the >>>> production workers, ps output, etc. I’ll leave that few a few >>>> hours, monitor to ensure there is a measurable memory >>>> increase, and send through the logs if that’s ok? >>>> >>>> Thanks! >>>> >>>> Andrew >>>> >>>>> On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla >>>>> <miconda@gmail.com mailto:miconda@gmail.com> wrote: >>>>> >>>>> Hello, >>>>> >>>>> are you defining a lot of kamailio variables in your ruby >>>>> script? In other words, are you using variables with >>>>> different name, like $var(xyz) or $sht(a=>xyz), where xyz is >>>>> passed from/computed in ruby script and is changing >>>>> depending of the sip message? >>>>> >>>>> Cheers, >>>>> Daniel >>>>> >>>>> On 01.08.19 15:34, Andrew White wrote: >>>>>> Thanks Daniel, you’re fantastic! >>>>>> >>>>>> I have 4 children/workers configured with -m 128 -M 32. The >>>>>> machine in question has 512MB of memory, 1 core and 1GB >>>>>> swap on an SSD. >>>>>> >>>>>> I restarted Kamailio with memlog=1 and I’ve been sending >>>>>> batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio >>>>>> processes going up in memory after each batch, which I >>>>>> suspect to be the primary children/workers. Immediately >>>>>> post restart: >>>>>> >>>>>> root 28531 0.7 5.5 329368 27196 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28532 0.6 4.9 329368 24528 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28533 0.6 5.5 329368 27244 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28534 0.7 5.4 329368 26788 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> >>>>>> After about 90 calls: >>>>>> >>>>>> root 28531 0.0 6.7 330688 32948 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28532 0.0 6.5 330560 32264 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28533 0.0 6.5 330556 32272 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28534 0.0 6.6 330564 32592 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> >>>>>> None of the other 9 Kamailio processes are increasing at all. >>>>>> >>>>>> I ran corex.pkg_summary against one of them and got the >>>>>> following dump: >>>>>> >>>>>> https://paste.ubuntu.com/p/SqTF3K5knK/ >>>>>> >>>>>> I can see a lot of allocation to pvapi.c, does this >>>>>> indicate maybe I’m setting PVs that need to be unset? >>>>>> >>>>>> Here’s another after another 60 calls: >>>>>> >>>>>> https://paste.ubuntu.com/p/9WQXqZtfT2/ >>>>>> >>>>>> root 28531 0.0 6.9 330820 33928 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28532 0.0 6.7 330692 33352 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28533 0.0 6.7 330688 33280 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> root 28534 0.0 6.7 330696 33192 ? Sl 22:48 >>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>> /var/run/kamailio/kamailio.pid -f >>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>> >>>>>> The only changes I’ve made on this config over the last >>>>>> couple of weeks (since I saw this issue) is removing the >>>>>> dispatcher module and adding in a small function in >>>>>> app_ruby (which I already use) to query redis (which I also >>>>>> already use from app_ruby and make a heap of queries per >>>>>> call) for some values and write $du manually. I also added >>>>>> in the topoh module. >>>>>> >>>>>> It also makes a lot of sense to me to monitor the >>>>>> individual processes rather than the aggregate. Is there a >>>>>> way to identify simply from bash what processes are workers >>>>>> programmatically? I’d like to monitor just those >>>>>> individually in my monitoring. >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Andrew >>>>>> >>>>>> >>>>>>> On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla >>>>>>> <miconda@gmail.com mailto:miconda@gmail.com> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> if it is pkg, then you have to see which process is >>>>>>> increasing the use of memory, because it is private >>>>>>> memory, specific for each process. The sum is an >>>>>>> indicator, but the debugging has to be done for a specific >>>>>>> process/pid. >>>>>>> >>>>>>> Once you indentify a process that is leaking pkg, execute >>>>>>> the rpc command: >>>>>>> >>>>>>> - >>>>>>> https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg... >>>>>>> >>>>>>> When that process is doing some runtime work (e.g., >>>>>>> handling of a sip message), the syslog will get a summary >>>>>>> with used pkg chunks. Send those log messages here for >>>>>>> analysis. You have to set memlog core parameter to a value >>>>>>> smaller than debug. >>>>>>> >>>>>>> Cheers, >>>>>>> Daniel >>>>>>> >>>>>>> >>>>>>> On 01.08.19 03:43, Andrew White wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I had a Kamailio crash the other day, and some debugging >>>>>>>> showed I ran out of PKG memory. >>>>>>>> >>>>>>>> Since then I’ve run a simple bash script to compile the >>>>>>>> amount of memory used by all child processes, >>>>>>>> effective /usr/local/sbin/kamcmd pkg.stats | grep >>>>>>>> real_used summed together. I’ve graphed out the data, and >>>>>>>> there’s a clear growth of PKG memory going on, mostly >>>>>>>> increasing during our busier daytime hours. >>>>>>>> >>>>>>>> https://i.imgur.com/UTzx2k1.png >>>>>>>> >>>>>>>> Based on this, I suspect either a module loaded or >>>>>>>> something within my app_ruby conf is leaking memory. >>>>>>>> >>>>>>>> I’ve been reading >>>>>>>> through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, >>>>>>>> but I’m a bit nervous, as I’m not really a C/deep memory >>>>>>>> type of guy. I can see a GDB script I can attach to >>>>>>>> Kamailio, but is that going to use significant resources >>>>>>>> to run or impact the running process? Is there a >>>>>>>> newer/better/alternative way to do this, and to help me >>>>>>>> break this down? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Kamailio (SER) - Users Mailing List >>>>>>>> sr-users@lists.kamailio.org >>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>>>> -- >>>>>>> Daniel-Constantin Mierla -- www.asipto.com >>>>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda >>>>>> >>>>> -- >>>>> Daniel-Constantin Mierla -- www.asipto.com >>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda >>>> >>> >>> >>> _______________________________________________ >>> Kamailio (SER) - Users Mailing List >>> sr-users@lists.kamailio.org >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> -- >> Henning Westerholt - https://skalatan.de/blog/ >> Kamailio services - https://skalatan.de/services
>
Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://skalatan.de/services
-
Andrew White -
Daniel-Constantin Mierla -
Henning Westerholt