[SR-Users] Re: Kamailio - could not allocate private memory from pkg pool

-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com > -----Original Message----- > From: Muhammad Danish Moosa danishmoosa@gmail.com > Sent: Dienstag, 6. Juni 2023 14:26 > To: Henning Westerholt hw@gilawa.com > Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org > Subject: Re: [SR-Users] Re: Kamailio - could not allocate private memory from pkg > pool > > Hi Henning, > > All good,I will re-examine the changes. There were positive changes after that. > Before that , I had to restart kamailio around every week, > > Quick question, is there any cleaner procedure to upgrade from 5.5.3 to 5.5.6. I > have compiled from sources. Please understand these are production systems > and should be done with 0 or minimal outage. > > Danish > > > > On Tue, Jun 6, 2023 at 10:14 PM Henning Westerholt hw@gilawa.com wrote: > > > > Hello Muhammad, > > > > maybe its related to the changes you've done, maybe its some load increase or > similar, but hard to say from the outside. > > > > As said, maybe try the minor update first, and then if its persists, lets go the > memory debugging route. > > > > Cheers, > > > > Henning > > > > -- > > Henning Westerholt – https://skalatan.de/blog/ Kamailio services – > > https://gilawa.com > > > > > -----Original Message----- > > > From: Muhammad Danish Moosa danishmoosa@gmail.com > > > Sent: Dienstag, 6. Juni 2023 13:58 > > > To: Henning Westerholt hw@gilawa.com > > > Cc: Kamailio (SER) - Users Mailing List > > > sr-users@lists.kamailio.org > > > Subject: Re: [SR-Users] Re: Kamailio - could not allocate private > > > memory from pkg pool > > > > > > Thanks for the response. I really appreciate it. > > > > > > This system was tested thoroughly in the lab but problems only > > > appeared in production. Actually it was working smoothly for 3 > > > months after I fixed some issues like external system calls etc and this > appeared all of sudden. > > > > > > > > > > > > > > > > > > On Tue, Jun 6, 2023 at 9:40 PM Henning Westerholt hw@gilawa.com wrote: > > > > > > > > Hello, > > > > > > > > first step should be probably to update to the latest version of > > > > the 5.5.x branch, > > > e.g., 5.5.6. There have been several fixes, some also related to > > > memory leaks included if I remember correctly. > > > > > > > > If the problem persists, it might be a new memory leak. This can > > > > be found by > > > enabling memory debugging, preferable on a test system or lab system. > > > > https://www.kamailio.org/wiki/tutorials/troubleshooting/memory > > > > > > > > Cheers, > > > > > > > > Henning > > > > > > > > -- > > > > Henning Westerholt – https://skalatan.de/blog/ Kamailio services – > > > > https://gilawa.com > > > > > > > > > -----Original Message----- > > > > > From: Muhammad Danish Moosa danishmoosa@gmail.com > > > > > Sent: Dienstag, 6. Juni 2023 10:46 > > > > > To: Kamailio (SER) - Users Mailing List > > > > > sr-users@lists.kamailio.org > > > > > Subject: [SR-Users] Re: Kamailio - could not allocate private > > > > > memory from pkg pool > > > > > > > > > > Mainly if I am looking at the private memory usage , it keeps > > > > > increasing very slowly but does not go back even after the calls > > > > > are literally zero. I suspect some UDP listener process are not > > > > > releasing the > > > memory. > > > > > > > > > > If I consider that my kamailio cfg and lua code has problem with > > > > > variables or something , I want to understand how it works when > > > > > calls are released. My understanding is usage should be minimum > > > > > when calls are released. Looking for feedback please? > > > > > > > > > > [roottmp]# kamcmd pkg.stats |grep real_used > > > > > real_used: 836144 > > > > > real_used: 1017328 > > > > > real_used: 1028656 > > > > > real_used: 1020864 > > > > > real_used: 1020632 > > > > > real_used: 1006120 > > > > > real_used: 1014536 > > > > > real_used: 1023984 > > > > > real_used: 1017584 > > > > > real_used: 835784 > > > > > real_used: 835784 > > > > > real_used: 835784 > > > > > real_used: 844344 > > > > > real_used: 835784 > > > > > real_used: 901344 > > > > > real_used: 835784 > > > > > real_used: 1259728 > > > > > real_used: 1196688 > > > > > real_used: 1181320 > > > > > real_used: 1166624 > > > > > real_used: 1216032 > > > > > real_used: 1173464 > > > > > real_used: 1168552 > > > > > real_used: 1235528 > > > > > real_used: 917904 > > > > > > > > > > > > > > > > > > > > For short term I increase the pvt memory allocation or keep > > > > > restarting kamailio after every week or fortnight but I am > > > > > looking for more stable solution. I am using kamailio 5.5.3 > > > > > > > > > > On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa > > > > > danishmoosa@gmail.com wrote: > > > > > > > > > > > > Any recommendations? > > > > > > I have good settings for pkg and shm but I can increase more > > > > > > > > > > > > [root1 ~]# ps -ef |grep kam > > > > > > root 1816 1 0 09:48 ? 00:00:00 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1817 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1818 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1819 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1820 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1821 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1822 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1823 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1824 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1825 1816 0 09:48 ? 00:00:00 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1826 1816 0 09:48 ? 00:00:06 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1827 1816 0 09:48 ? 00:00:00 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1828 1816 0 09:48 ? 00:00:00 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1829 1816 0 09:48 ? 00:00:00 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1830 1816 0 09:48 ? 00:00:00 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1831 1816 0 09:48 ? 00:00:00 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1832 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1833 1816 0 09:48 ? 00:00:02 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1834 1816 0 09:48 ? 00:00:02 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1835 1816 0 09:48 ? 00:00:01 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1836 1816 0 09:48 ? 00:00:02 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1837 1816 0 09:48 ? 00:00:02 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1838 1816 0 09:48 ? 00:00:02 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1839 1816 0 09:48 ? 00:00:03 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 1840 1816 0 09:48 ? 00:00:00 > > > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > > > root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam > > > > > > [root@msteams-sip-01 ~]# free -m > > > > > > total used free shared buff/cache available > > > > > > Mem: 7822 221 4055 386 3545 6900 > > > > > > Swap: 5119 0 5119 > > > > > > > > > > > > On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa > > > > > > danishmoosa@gmail.com wrote: > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I have been kamailio services working for few months and > > > > > > > suddenly I got this error. > > > > > > > > > > > > > > ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable > > > > > > > to set pv [$rp] > > > > > > > ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not > > > > > > > allocate private memory from pkg pool > > > > > > > ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot > > > > > > > get pv spec for > > > > > > > [$dlg_var(BW0930526150606231188928742@10.10.16.100)] > > > > > > > ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): > > > > > > > could not allocate private memory from pkg pool > > > > > > > ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): > > > > > > > out of mem > > > > > > > ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to > > > > > > > add branches > > > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > > > build_res_buf_from_sip_req(): could not allocate private > > > > > > > memory from pkg pool > > > > > > > ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless > > > > > > > error reply > > > > > > > used: I'm terribly sorry, server error occurred (1/SL) > > > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > > > build_res_buf_from_sip_req(): could not allocate private > > > > > > > memory from pkg pool > > > > > > > ERROR: tm [t_reply.c:475]: _reply_light(): response > > > > > > > building failed > > > > > > > WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a > > > > > > > delayed stateful reply failed > > > > > > > > > > > > > > Issue was fixed after the restart of kamailio. > > > > > > > > > > > > > > -- > > > > > > > Muhammad Danish Moosa > > > > > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > > > Christopher McCandless > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Muhammad Danish Moosa > > > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > > Christopher McCandless > > > > > > > > > > > > > > > > > > > > -- > > > > > Muhammad Danish Moosa > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > Christopher McCandless > > > > > __________________________________________________________ > > > > > Kamailio - Users Mailing List - Non Commercial Discussions To > > > > > unsubscribe send an email to sr-users-leave@lists.kamailio.org > > > > > Important: keep the mailing list in the recipients, do not reply > > > > > only to the > > > sender! > > > > > Edit mailing list options or unsubscribe: > > > > > > > > > > > > -- > > > Muhammad Danish Moosa > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > Christopher McCandless > > > > -- > Muhammad Danish Moosa > > " The core of mans' spirit comes from new experiences. "___ Christopher > McCandless