Hello,
it looks like that you are observing a memory leak. You can look into it e.g. with memory debugging and try to debug it: https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
Generally, I would suggest decreasing the PGK memory, 256 MB is way too much, e.g, to 16-32 MB and increase the shared memory e.g. to 1GB. If there is a large memory leak, the memory will be used anyway so it makes no sense to increase it too much.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: paresh-panda via sr-dev sr-dev@lists.kamailio.org Sent: Montag, 29. April 2024 14:06 To: Kamailio Devel List sr-dev@lists.kamailio.org Cc: Subscribed subscribed@noreply.github.com; paresh-panda notifications@github.com Subject: [sr-dev] [kamailio/kamailio] Mem leak in smsops which leads to the udp receiver process leak memory continuously (Issue #3833)
Hi, I am using Kamailio 5.8.1 ( ubuntu 22.04 Jammy 64 Bit),. Below are the observations.
I am running a load of 500 3GPP IMS SMS which kamailio decodes and sends to another application over Rabbit MQ (Kazoo Module). -m 512 -M 256 , using 512 MB Shared memory and 256MB Package Memory.
When the load for 20 mins the available Memory drops from 80% to 25%
And I see below error logs
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/pvapi.c:1136]: pv_parse_format(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1318]: fix_param(): bad PVE format: "PRD-IR92/11 term-Motorola/XT2205-1-T2STS33.105-51-11-1-2-2 device-type/smart-phone" Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:539 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:540 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:541 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:542 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:543 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:555 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:556 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1171]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:559
I am attaching the configuration for reference.
Help on this is much appreciated configfiles.tar.gzhttps://github.com/kamailio/kamailio/files/15148792/configfiles.tar.gz , I am looking into this issue for quite a long time.
Thank you! Regards Paresh
— Reply to this email directly, view it on GitHubhttps://github.com/kamailio/kamailio/issues/3833, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABO7UZJMTNNEACXNITASO2TY7YZTTAVCNFSM6AAAAABG6GCT4CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DQOBWGU2TQMA. You are receiving this because you are subscribed to this thread.Message ID: kamailio/kamailio/issues/3833@github.com
