[kamailio/kamailio] Mem leak in smsops which leads to the udp receiver process leak memory continuously (Issue #3833)
Hi, I am using Kamailio 5.8.1 ( ubuntu 22.04 Jammy 64 Bit),. Below are the observations.
I am running a load of 500 3GPP IMS SMS which kamailio decodes and sends to another application over Rabbit MQ (Kazoo Module). **-m 512 -M 256** , using 512 MB Shared memory and 256MB Package Memory.
When the load for 20 mins the available Memory drops from **80% to 25%**
And I see below error logs
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/pvapi.c:1136]: pv_parse_format(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/sr_module.c:1318]: fix_param(): bad PVE format: "PRD-IR92/11 term-Motorola/XT2205-1-T2STS33.105-51-11-1-2-2 device-type/smart-phone" Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:539 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:540 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:541 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:542 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:543 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:555 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:556 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1171]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} <core> [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:559
I am attaching the configuration for reference.
Help on this is much appreciated [configfiles.tar.gz](https://github.com/kamailio/kamailio/files/15148792/configfiles.tar.gz) , I am looking into this issue for quite a long time.
Thank you! Regards Paresh
Hello,
it looks like that you are observing a memory leak. You can look into it e.g. with memory debugging and try to debug it: https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
Generally, I would suggest decreasing the PGK memory, 256 MB is way too much, e.g, to 16-32 MB and increase the shared memory e.g. to 1GB. If there is a large memory leak, the memory will be used anyway so it makes no sense to increase it too much.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: paresh-panda via sr-dev sr-dev@lists.kamailio.org Sent: Montag, 29. April 2024 14:06 To: Kamailio Devel List sr-dev@lists.kamailio.org Cc: Subscribed subscribed@noreply.github.com; paresh-panda notifications@github.com Subject: [sr-dev] [kamailio/kamailio] Mem leak in smsops which leads to the udp receiver process leak memory continuously (Issue #3833)
Hi, I am using Kamailio 5.8.1 ( ubuntu 22.04 Jammy 64 Bit),. Below are the observations.
I am running a load of 500 3GPP IMS SMS which kamailio decodes and sends to another application over Rabbit MQ (Kazoo Module). -m 512 -M 256 , using 512 MB Shared memory and 256MB Package Memory.
When the load for 20 mins the available Memory drops from 80% to 25%
And I see below error logs
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/pvapi.c:1136]: pv_parse_format(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1318]: fix_param(): bad PVE format: "PRD-IR92/11 term-Motorola/XT2205-1-T2STS33.105-51-11-1-2-2 device-type/smart-phone" Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:539 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:540 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:541 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:542 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:543 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:555 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:556 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1171]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:559
I am attaching the configuration for reference.
Help on this is much appreciated configfiles.tar.gzhttps://github.com/kamailio/kamailio/files/15148792/configfiles.tar.gz , I am looking into this issue for quite a long time.
Thank you! Regards Paresh
— Reply to this email directly, view it on GitHubhttps://github.com/kamailio/kamailio/issues/3833, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABO7UZJMTNNEACXNITASO2TY7YZTTAVCNFSM6AAAAABG6GCT4CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DQOBWGU2TQMA. You are receiving this because you are subscribed to this thread.Message ID: kamailio/kamailio/issues/3833@github.com
Hi Henning,
Thank you for guidance and suggestion. I did change the PKG size to 32MB and increased SHM to 1GB. But the same issue observed with a lesser time duration.
While looking at the below error, found that something changes in the smsops module which was causing the leak
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes!
Below is my analysis: The initial code was
if (!rp_data) { rp_data = (sms_rp_data_t*)pkg_malloc(sizeof(struct _sms_rp_data)); if (!rp_data) { LM_ERR("Error allocating %lu bytes!\n", sizeof(struct _sms_rp_data)); return -1; } } else { freeRP_DATA(rp_data); } But it got changed in "smsops: Fix conversion from UCS-2 to UTF-8 and viceversa (#3546)" tag 8 months ago
if(rp_data) { freeRP_DATA(rp_data); } rp_data = (sms_rp_data_t *)pkg_malloc(sizeof(struct _sms_rp_data)); if(!rp_data) { LM_ERR("Error allocating %lu bytes!\n", (unsigned long)sizeof(struct _sms_rp_data)); return -1; } Now in the recent code it is not reusing the rp_data pointer, rather it allocates every time a new message reaches.
I will try to clone the tag and fix the code and keep you posted
if(_smsops_rp_data) { freeRP_DATA(_smsops_rp_data); } else { //Added by Paresh Panda
_smsops_rp_data = (sms_rp_data_t *)pkg_malloc(sizeof(struct _sms_rp_data)); } Is there any plan for new tag ? , I am planning to take the latest kamailio tag to production.
I will update you after testing the fix as above.
Thank you! Regards Paresh
Hello, thanks for the feedback. Your description sounds indeed like a memory leak. The code logic got changed, so its freeing the rp_data when its allocated now. Its a bit longer function, so it should be properly analyzed that its freed in all error conditions (which does not look like that right now).
Hi Henning, Thank you for the quick response!
I cloned the code and just changed back the block (which was problematic). Now I see error again as below :
May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5885]: INFO: <script>: {1 1 MESSAGE 1939882-1516@192.168.136.152} Message Outbound MESSAGE sip:+16475800172@192.168.136.152:5060 (sip:+15197963701@192.168.136.152:5061 (192.168.136.152:5062) to sip:+16475800172@192.168.136.151:5060, 1939882-1516@192.168.136.152) May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: INFO: {2 1 MESSAGE 1939882-1516@192.168.136.152} <script>: reply route May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: ERROR: {2 1 MESSAGE 1939882-1516@192.168.136.152} <core> [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7f3af4736010, 384); Free fragment not found! May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: ERROR: {2 1 MESSAGE 1939882-1516@192.168.136.152} <core> [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7f3af4736010, 384) called from core: core/msg_translator.c: generate_res_buf_from_sip_res(2435), module: core; Free fragment not found! May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: ERROR: {2 1 MESSAGE 1939882-1516@192.168.136.152} <core> [core/msg_translator.c:2438]: generate_res_buf_from_sip_res(): could not allocate private memory from pkg pool May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: ERROR: {2 1 MESSAGE 1939882-1516@192.168.136.152} tm [t_reply.c:2067]: relay_reply(): no mem for outbound reply buffer May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: ERROR: {2 1 MESSAGE 1939882-1516@192.168.136.152} <core> [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7f3af4736010, 416); Free fragment not found! May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: ERROR: {2 1 MESSAGE 1939882-1516@192.168.136.152} <core> [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7f3af4736010, 416) called from core: core/msg_translator.c: build_res_buf_from_sip_req(2608), module: core; Free fragment not found! May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: ERROR: {2 1 MESSAGE 1939882-1516@192.168.136.152} <core> [core/msg_translator.c:2610]: build_res_buf_from_sip_req(): could not allocate private memory from pkg pool May 1 16:54:43 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5875]: ERROR: {2 1 MESSAGE 1939882-1516@192.168.136.152} tm [t_reply.c:465]: _reply_light(): response building failed
Does it also look like memory leak this error log appeared and kamailio stopped processing traffic after 35 minutes or so.
When check the pkg memory for the pid as in log
{ entry: 6 pid: 5875 rank: 6 used: 5701888 free: 5728 real_used: 33548704 total_size: 33554432 total_frags: 28 desc: udp receiver child=5 sock=192.168.136.151:5060 }
It shows the udp receiver process PKG Mem is full , I have 8 childrens configured and all started showing the same issue.
Config : PKG 32 MB , SHM 1GB load 1000 msg/sec
Can you please quickly through some lights on this please? Shall I look at it as mem leak or config issue ( may be increase in PKG mem ?
Thank you!
@paresh-panda can you give this commit (https://github.com/kamailio/kamailio/commit/eccff1738ef074aa9fae08e4b10c851e...) a try and see if you still face the memory issue?
This issue is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks.
Closed #3833 as not planned.
-
github-actions[bot] -
Henning Westerholt -
Henning Westerholt
-
paresh-panda
-
Supreeth Herle