Kamailio - could not allocate private memory from pkg pool - sr-users

List overview All Threads
Download

newer

Kamailio - could not allocate private memory from pkg pool

older

Dialog DMQ state after restart

Kamailio World

Muhammad Danish Moosa

5 Jun 2023 5 Jun '23

11:19 p.m.

Hi, I have been kamailio services working for few months and suddenly I got this error. ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable to set pv [$rp] ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not allocate private memory from pkg pool ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot get pv spec for [$dlg_var(BW0930526150606231188928742(a)10.10.16.100)] ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): could not allocate private memory from pkg pool ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): out of mem ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to add branches ERROR: <core> [core/msg_translator.c:2501]: build_res_buf_from_sip_req(): could not allocate private memory from pkg pool ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) ERROR: <core> [core/msg_translator.c:2501]: build_res_buf_from_sip_req(): could not allocate private memory from pkg pool ERROR: tm [t_reply.c:475]: _reply_light(): response building failed WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a delayed stateful reply failed Issue was fixed after the restart of kamailio. -- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Show replies by date

Muhammad Danish Moosa

5 Jun 5 Jun

11:21 p.m.

Any recommendations? I have good settings for pkg and shm but I can increase more [root1 ~]# ps -ef |grep kam root 1816 1 0 09:48 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1817 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1818 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1819 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1820 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1821 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1822 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1823 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1824 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1825 1816 0 09:48 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1826 1816 0 09:48 ? 00:00:06 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1827 1816 0 09:48 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1828 1816 0 09:48 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1829 1816 0 09:48 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1830 1816 0 09:48 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1831 1816 0 09:48 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1832 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1833 1816 0 09:48 ? 00:00:02 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1834 1816 0 09:48 ? 00:00:02 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1835 1816 0 09:48 ? 00:00:01 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1836 1816 0 09:48 ? 00:00:02 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1837 1816 0 09:48 ? 00:00:02 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1838 1816 0 09:48 ? 00:00:02 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1839 1816 0 09:48 ? 00:00:03 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 1840 1816 0 09:48 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam [root@msteams-sip-01 ~]# free -m total used free shared buff/cache available Mem: 7822 221 4055 386 3545 6900 Swap: 5119 0 5119 On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa <danishmoosa(a)gmail.com> wrote:

...

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Muhammad Danish Moosa

6 Jun 6 Jun

4:45 a.m.

Mainly if I am looking at the private memory usage , it keeps increasing very slowly but does not go back even after the calls are literally zero. I suspect some UDP listener process are not releasing the memory. If I consider that my kamailio cfg and lua code has problem with variables or something , I want to understand how it works when calls are released. My understanding is usage should be minimum when calls are released. Looking for feedback please? [roottmp]# kamcmd pkg.stats |grep real_used real_used: 836144 real_used: 1017328 real_used: 1028656 real_used: 1020864 real_used: 1020632 real_used: 1006120 real_used: 1014536 real_used: 1023984 real_used: 1017584 real_used: 835784 real_used: 835784 real_used: 835784 real_used: 844344 real_used: 835784 real_used: 901344 real_used: 835784 real_used: 1259728 real_used: 1196688 real_used: 1181320 real_used: 1166624 real_used: 1216032 real_used: 1173464 real_used: 1168552 real_used: 1235528 real_used: 917904 For short term I increase the pvt memory allocation or keep restarting kamailio after every week or fortnight but I am looking for more stable solution. I am using kamailio 5.5.3 On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa <danishmoosa(a)gmail.com> wrote:

...

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Henning Westerholt

7:40 a.m.

...

-----Original Message----- From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> Sent: Dienstag, 6. Juni 2023 10:46 To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org> Subject: [SR-Users] Re: Kamailio - could not allocate private memory from pkg pool Mainly if I am looking at the private memory usage , it keeps increasing very slowly but does not go back even after the calls are literally zero. I suspect some UDP listener process are not releasing the memory. If I consider that my kamailio cfg and lua code has problem with variables or something , I want to understand how it works when calls are released. My understanding is usage should be minimum when calls are released. Looking for feedback please? [roottmp]# kamcmd pkg.stats |grep real_used real_used: 836144 real_used: 1017328 real_used: 1028656 real_used: 1020864 real_used: 1020632 real_used: 1006120 real_used: 1014536 real_used: 1023984 real_used: 1017584 real_used: 835784 real_used: 835784 real_used: 835784 real_used: 844344 real_used: 835784 real_used: 901344 real_used: 835784 real_used: 1259728 real_used: 1196688 real_used: 1181320 real_used: 1166624 real_used: 1216032 real_used: 1173464 real_used: 1168552 real_used: 1235528 real_used: 917904 For short term I increase the pvt memory allocation or keep restarting kamailio after every week or fortnight but I am looking for more stable solution. I am using kamailio 5.5.3 On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa <danishmoosa(a)gmail.com> wrote:

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Muhammad Danish Moosa

7:57 a.m.

Thanks for the response. I really appreciate it. This system was tested thoroughly in the lab but problems only appeared in production. Actually it was working smoothly for 3 months after I fixed some issues like external system calls etc and this appeared all of sudden. On Tue, Jun 6, 2023 at 9:40 PM Henning Westerholt <hw(a)gilawa.com> wrote:

...

Hello, first step should be probably to update to the latest version of the 5.5.x branch, e.g., 5.5.6. There have been several fixes, some also related to memory leaks included if I remember correctly. If the problem persists, it might be a new memory leak. This can be found by enabling memory debugging, preferable on a test system or lab system. https://www.kamailio.org/wiki/tutorials/troubleshooting/memory Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com > -----Original Message----- > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > Sent: Dienstag, 6. Juni 2023 10:46 > To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org> > Subject: [SR-Users] Re: Kamailio - could not allocate private memory from pkg > pool > > Mainly if I am looking at the private memory usage , it keeps increasing very > slowly but does not go back even after the calls are literally zero. I suspect some > UDP listener process are not releasing the memory. > > If I consider that my kamailio cfg and lua code has problem with variables or > something , I want to understand how it works when calls are released. My > understanding is usage should be minimum when calls are released. Looking for > feedback please? > > [roottmp]# kamcmd pkg.stats |grep real_used > real_used: 836144 > real_used: 1017328 > real_used: 1028656 > real_used: 1020864 > real_used: 1020632 > real_used: 1006120 > real_used: 1014536 > real_used: 1023984 > real_used: 1017584 > real_used: 835784 > real_used: 835784 > real_used: 835784 > real_used: 844344 > real_used: 835784 > real_used: 901344 > real_used: 835784 > real_used: 1259728 > real_used: 1196688 > real_used: 1181320 > real_used: 1166624 > real_used: 1216032 > real_used: 1173464 > real_used: 1168552 > real_used: 1235528 > real_used: 917904 > > > > For short term I increase the pvt memory allocation or keep restarting kamailio > after every week or fortnight but I am looking for more stable solution. I am using > kamailio 5.5.3 > > On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa > <danishmoosa(a)gmail.com> wrote: > > > > Any recommendations? > > I have good settings for pkg and shm but I can increase more > > > > [root1 ~]# ps -ef |grep kam > > root 1816 1 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1817 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1818 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1819 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1820 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1821 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1822 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1823 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1824 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1825 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1826 1816 0 09:48 ? 00:00:06 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1827 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1828 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1829 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1830 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1831 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1832 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1833 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1834 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1835 1816 0 09:48 ? 00:00:01 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1836 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1837 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1838 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1839 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1840 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam > > [root@msteams-sip-01 ~]# free -m > > total used free shared buff/cache available > > Mem: 7822 221 4055 386 3545 6900 > > Swap: 5119 0 5119 > > > > On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa > > <danishmoosa(a)gmail.com> wrote: > > > > > > Hi, > > > > > > I have been kamailio services working for few months and suddenly I > > > got this error. > > > > > > ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable to set > > > pv [$rp] > > > ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not > > > allocate private memory from pkg pool > > > ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot get pv > > > spec for [$dlg_var(BW0930526150606231188928742(a)10.10.16.100)] > > > ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): could not > > > allocate private memory from pkg pool > > > ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): out of mem > > > ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to add > > > branches > > > ERROR: <core> [core/msg_translator.c:2501]: > > > build_res_buf_from_sip_req(): could not allocate private memory from > > > pkg pool > > > ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless error reply > > > used: I'm terribly sorry, server error occurred (1/SL) > > > ERROR: <core> [core/msg_translator.c:2501]: > > > build_res_buf_from_sip_req(): could not allocate private memory from > > > pkg pool > > > ERROR: tm [t_reply.c:475]: _reply_light(): response building failed > > > WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a delayed > > > stateful reply failed > > > > > > Issue was fixed after the restart of kamailio. > > > > > > -- > > > Muhammad Danish Moosa > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > Christopher McCandless > > > > > > > > -- > > Muhammad Danish Moosa > > > > " The core of mans' spirit comes from new experiences. "___ > > Christopher McCandless > > > > -- > Muhammad Danish Moosa > > " The core of mans' spirit comes from new experiences. "___ Christopher > McCandless > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send > an email to sr-users-leave(a)lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the sender! > Edit mailing list options or unsubscribe:

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Henning Westerholt

8:14 a.m.

...

-----Original Message----- From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> Sent: Dienstag, 6. Juni 2023 13:58 To: Henning Westerholt <hw(a)gilawa.com> Cc: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org> Subject: Re: [SR-Users] Re: Kamailio - could not allocate private memory from pkg pool Thanks for the response. I really appreciate it. This system was tested thoroughly in the lab but problems only appeared in production. Actually it was working smoothly for 3 months after I fixed some issues like external system calls etc and this appeared all of sudden. On Tue, Jun 6, 2023 at 9:40 PM Henning Westerholt <hw(a)gilawa.com> wrote:

Hello, first step should be probably to update to the latest version of the 5.5.x branch,

e.g., 5.5.6. There have been several fixes, some also related to memory leaks included if I remember correctly.

If the problem persists, it might be a new memory leak. This can be found by

enabling memory debugging, preferable on a test system or lab system.

https://www.kamailio.org/wiki/tutorials/troubleshooting/memory Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com > -----Original Message----- > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > Sent: Dienstag, 6. Juni 2023 10:46 > To: Kamailio (SER) - Users Mailing List > <sr-users(a)lists.kamailio.org> > Subject: [SR-Users] Re: Kamailio - could not allocate private memory > from pkg pool > > Mainly if I am looking at the private memory usage , it keeps > increasing very slowly but does not go back even after the calls are > literally zero. I suspect some UDP listener process are not releasing the

memory.

> > If I consider that my kamailio cfg and lua code has problem with > variables or something , I want to understand how it works when > calls are released. My understanding is usage should be minimum when > calls are released. Looking for feedback please? > > [roottmp]# kamcmd pkg.stats |grep real_used > real_used: 836144 > real_used: 1017328 > real_used: 1028656 > real_used: 1020864 > real_used: 1020632 > real_used: 1006120 > real_used: 1014536 > real_used: 1023984 > real_used: 1017584 > real_used: 835784 > real_used: 835784 > real_used: 835784 > real_used: 844344 > real_used: 835784 > real_used: 901344 > real_used: 835784 > real_used: 1259728 > real_used: 1196688 > real_used: 1181320 > real_used: 1166624 > real_used: 1216032 > real_used: 1173464 > real_used: 1168552 > real_used: 1235528 > real_used: 917904 > > > > For short term I increase the pvt memory allocation or keep > restarting kamailio after every week or fortnight but I am looking > for more stable solution. I am using kamailio 5.5.3 > > On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa > <danishmoosa(a)gmail.com> wrote: > > > > Any recommendations? > > I have good settings for pkg and shm but I can increase more > > > > [root1 ~]# ps -ef |grep kam > > root 1816 1 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1817 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1818 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1819 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1820 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1821 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1822 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1823 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1824 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1825 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1826 1816 0 09:48 ? 00:00:06 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1827 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1828 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1829 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1830 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1831 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1832 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1833 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1834 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1835 1816 0 09:48 ? 00:00:01 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1836 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1837 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1838 1816 0 09:48 ? 00:00:02 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1839 1816 0 09:48 ? 00:00:03 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 1840 1816 0 09:48 ? 00:00:00 > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam > > [root@msteams-sip-01 ~]# free -m > > total used free shared buff/cache available > > Mem: 7822 221 4055 386 3545 6900 > > Swap: 5119 0 5119 > > > > On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa > > <danishmoosa(a)gmail.com> wrote: > > > > > > Hi, > > > > > > I have been kamailio services working for few months and > > > suddenly I got this error. > > > > > > ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable to > > > set pv [$rp] > > > ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not > > > allocate private memory from pkg pool > > > ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot get > > > pv spec for [$dlg_var(BW0930526150606231188928742(a)10.10.16.100)] > > > ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): could > > > not allocate private memory from pkg pool > > > ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): out of > > > mem > > > ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to add > > > branches > > > ERROR: <core> [core/msg_translator.c:2501]: > > > build_res_buf_from_sip_req(): could not allocate private memory > > > from pkg pool > > > ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless error > > > reply > > > used: I'm terribly sorry, server error occurred (1/SL) > > > ERROR: <core> [core/msg_translator.c:2501]: > > > build_res_buf_from_sip_req(): could not allocate private memory > > > from pkg pool > > > ERROR: tm [t_reply.c:475]: _reply_light(): response building > > > failed > > > WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a > > > delayed stateful reply failed > > > > > > Issue was fixed after the restart of kamailio. > > > > > > -- > > > Muhammad Danish Moosa > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > Christopher McCandless > > > > > > > > -- > > Muhammad Danish Moosa > > > > " The core of mans' spirit comes from new experiences. "___ > > Christopher McCandless > > > > -- > Muhammad Danish Moosa > > " The core of mans' spirit comes from new experiences. "___ > Christopher McCandless > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions To > unsubscribe send an email to sr-users-leave(a)lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the

sender!

> Edit mailing list options or unsubscribe:

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Muhammad Danish Moosa

8:25 a.m.

Hi Henning, All good,I will re-examine the changes. There were positive changes after that. Before that , I had to restart kamailio around every week, Quick question, is there any cleaner procedure to upgrade from 5.5.3 to 5.5.6. I have compiled from sources. Please understand these are production systems and should be done with 0 or minimal outage. Danish On Tue, Jun 6, 2023 at 10:14 PM Henning Westerholt <hw(a)gilawa.com> wrote:

...

Hello Muhammad, maybe its related to the changes you've done, maybe its some load increase or similar, but hard to say from the outside. As said, maybe try the minor update first, and then if its persists, lets go the memory debugging route. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com > -----Original Message----- > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > Sent: Dienstag, 6. Juni 2023 13:58 > To: Henning Westerholt <hw(a)gilawa.com> > Cc: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org> > Subject: Re: [SR-Users] Re: Kamailio - could not allocate private memory from pkg > pool > > Thanks for the response. I really appreciate it. > > This system was tested thoroughly in the lab but problems only appeared in > production. Actually it was working smoothly for 3 months after I fixed some > issues like external system calls etc and this appeared all of sudden. > > > > > > On Tue, Jun 6, 2023 at 9:40 PM Henning Westerholt <hw(a)gilawa.com> wrote: > > > > Hello, > > > > first step should be probably to update to the latest version of the 5.5.x branch, > e.g., 5.5.6. There have been several fixes, some also related to memory leaks > included if I remember correctly. > > > > If the problem persists, it might be a new memory leak. This can be found by > enabling memory debugging, preferable on a test system or lab system. > > https://www.kamailio.org/wiki/tutorials/troubleshooting/memory > > > > Cheers, > > > > Henning > > > > -- > > Henning Westerholt – https://skalatan.de/blog/ Kamailio services – > > https://gilawa.com > > > > > -----Original Message----- > > > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > > > Sent: Dienstag, 6. Juni 2023 10:46 > > > To: Kamailio (SER) - Users Mailing List > > > <sr-users(a)lists.kamailio.org> > > > Subject: [SR-Users] Re: Kamailio - could not allocate private memory > > > from pkg pool > > > > > > Mainly if I am looking at the private memory usage , it keeps > > > increasing very slowly but does not go back even after the calls are > > > literally zero. I suspect some UDP listener process are not releasing the > memory. > > > > > > If I consider that my kamailio cfg and lua code has problem with > > > variables or something , I want to understand how it works when > > > calls are released. My understanding is usage should be minimum when > > > calls are released. Looking for feedback please? > > > > > > [roottmp]# kamcmd pkg.stats |grep real_used > > > real_used: 836144 > > > real_used: 1017328 > > > real_used: 1028656 > > > real_used: 1020864 > > > real_used: 1020632 > > > real_used: 1006120 > > > real_used: 1014536 > > > real_used: 1023984 > > > real_used: 1017584 > > > real_used: 835784 > > > real_used: 835784 > > > real_used: 835784 > > > real_used: 844344 > > > real_used: 835784 > > > real_used: 901344 > > > real_used: 835784 > > > real_used: 1259728 > > > real_used: 1196688 > > > real_used: 1181320 > > > real_used: 1166624 > > > real_used: 1216032 > > > real_used: 1173464 > > > real_used: 1168552 > > > real_used: 1235528 > > > real_used: 917904 > > > > > > > > > > > > For short term I increase the pvt memory allocation or keep > > > restarting kamailio after every week or fortnight but I am looking > > > for more stable solution. I am using kamailio 5.5.3 > > > > > > On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa > > > <danishmoosa(a)gmail.com> wrote: > > > > > > > > Any recommendations? > > > > I have good settings for pkg and shm but I can increase more > > > > > > > > [root1 ~]# ps -ef |grep kam > > > > root 1816 1 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1817 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1818 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1819 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1820 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1821 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1822 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1823 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1824 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1825 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1826 1816 0 09:48 ? 00:00:06 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1827 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1828 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1829 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1830 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1831 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1832 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1833 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1834 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1835 1816 0 09:48 ? 00:00:01 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1836 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1837 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1838 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1839 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1840 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam > > > > [root@msteams-sip-01 ~]# free -m > > > > total used free shared buff/cache available > > > > Mem: 7822 221 4055 386 3545 6900 > > > > Swap: 5119 0 5119 > > > > > > > > On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa > > > > <danishmoosa(a)gmail.com> wrote: > > > > > > > > > > Hi, > > > > > > > > > > I have been kamailio services working for few months and > > > > > suddenly I got this error. > > > > > > > > > > ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable to > > > > > set pv [$rp] > > > > > ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not > > > > > allocate private memory from pkg pool > > > > > ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot get > > > > > pv spec for [$dlg_var(BW0930526150606231188928742(a)10.10.16.100)] > > > > > ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): could > > > > > not allocate private memory from pkg pool > > > > > ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): out of > > > > > mem > > > > > ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to add > > > > > branches > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > build_res_buf_from_sip_req(): could not allocate private memory > > > > > from pkg pool > > > > > ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless error > > > > > reply > > > > > used: I'm terribly sorry, server error occurred (1/SL) > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > build_res_buf_from_sip_req(): could not allocate private memory > > > > > from pkg pool > > > > > ERROR: tm [t_reply.c:475]: _reply_light(): response building > > > > > failed > > > > > WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a > > > > > delayed stateful reply failed > > > > > > > > > > Issue was fixed after the restart of kamailio. > > > > > > > > > > -- > > > > > Muhammad Danish Moosa > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > Christopher McCandless > > > > > > > > > > > > > > > > -- > > > > Muhammad Danish Moosa > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > Christopher McCandless > > > > > > > > > > > > -- > > > Muhammad Danish Moosa > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > Christopher McCandless > > > __________________________________________________________ > > > Kamailio - Users Mailing List - Non Commercial Discussions To > > > unsubscribe send an email to sr-users-leave(a)lists.kamailio.org > > > Important: keep the mailing list in the recipients, do not reply only to the > sender! > > > Edit mailing list options or unsubscribe: > > > > -- > Muhammad Danish Moosa > > " The core of mans' spirit comes from new experiences. "___ Christopher > McCandless

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Henning Westerholt

8:31 a.m.

Hello, you can use RPM and Debian packages, provided from the project, sure. - deb.kamailio.org - rpm.kamailio.org Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com

...

-----Original Message----- From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> Sent: Dienstag, 6. Juni 2023 14:26 To: Henning Westerholt <hw(a)gilawa.com> Cc: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org> Subject: Re: [SR-Users] Re: Kamailio - could not allocate private memory from pkg pool Hi Henning, All good,I will re-examine the changes. There were positive changes after that. Before that , I had to restart kamailio around every week, Quick question, is there any cleaner procedure to upgrade from 5.5.3 to 5.5.6. I have compiled from sources. Please understand these are production systems and should be done with 0 or minimal outage. Danish On Tue, Jun 6, 2023 at 10:14 PM Henning Westerholt <hw(a)gilawa.com> wrote:

Hello Muhammad, maybe its related to the changes you've done, maybe its some load increase or

similar, but hard to say from the outside.

As said, maybe try the minor update first, and then if its persists, lets go the

memory debugging route.

Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com > -----Original Message----- > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > Sent: Dienstag, 6. Juni 2023 13:58 > To: Henning Westerholt <hw(a)gilawa.com> > Cc: Kamailio (SER) - Users Mailing List > <sr-users(a)lists.kamailio.org> > Subject: Re: [SR-Users] Re: Kamailio - could not allocate private > memory from pkg pool > > Thanks for the response. I really appreciate it. > > This system was tested thoroughly in the lab but problems only > appeared in production. Actually it was working smoothly for 3 > months after I fixed some issues like external system calls etc and this

appeared all of sudden.

> > > > > > On Tue, Jun 6, 2023 at 9:40 PM Henning Westerholt <hw(a)gilawa.com> wrote: > > > > Hello, > > > > first step should be probably to update to the latest version of > > the 5.5.x branch, > e.g., 5.5.6. There have been several fixes, some also related to > memory leaks included if I remember correctly. > > > > If the problem persists, it might be a new memory leak. This can > > be found by > enabling memory debugging, preferable on a test system or lab system. > > https://www.kamailio.org/wiki/tutorials/troubleshooting/memory > > > > Cheers, > > > > Henning > > > > -- > > Henning Westerholt – https://skalatan.de/blog/ Kamailio services – > > https://gilawa.com > > > > > -----Original Message----- > > > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > > > Sent: Dienstag, 6. Juni 2023 10:46 > > > To: Kamailio (SER) - Users Mailing List > > > <sr-users(a)lists.kamailio.org> > > > Subject: [SR-Users] Re: Kamailio - could not allocate private > > > memory from pkg pool > > > > > > Mainly if I am looking at the private memory usage , it keeps > > > increasing very slowly but does not go back even after the calls > > > are literally zero. I suspect some UDP listener process are not > > > releasing the > memory. > > > > > > If I consider that my kamailio cfg and lua code has problem with > > > variables or something , I want to understand how it works when > > > calls are released. My understanding is usage should be minimum > > > when calls are released. Looking for feedback please? > > > > > > [roottmp]# kamcmd pkg.stats |grep real_used > > > real_used: 836144 > > > real_used: 1017328 > > > real_used: 1028656 > > > real_used: 1020864 > > > real_used: 1020632 > > > real_used: 1006120 > > > real_used: 1014536 > > > real_used: 1023984 > > > real_used: 1017584 > > > real_used: 835784 > > > real_used: 835784 > > > real_used: 835784 > > > real_used: 844344 > > > real_used: 835784 > > > real_used: 901344 > > > real_used: 835784 > > > real_used: 1259728 > > > real_used: 1196688 > > > real_used: 1181320 > > > real_used: 1166624 > > > real_used: 1216032 > > > real_used: 1173464 > > > real_used: 1168552 > > > real_used: 1235528 > > > real_used: 917904 > > > > > > > > > > > > For short term I increase the pvt memory allocation or keep > > > restarting kamailio after every week or fortnight but I am > > > looking for more stable solution. I am using kamailio 5.5.3 > > > > > > On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa > > > <danishmoosa(a)gmail.com> wrote: > > > > > > > > Any recommendations? > > > > I have good settings for pkg and shm but I can increase more > > > > > > > > [root1 ~]# ps -ef |grep kam > > > > root 1816 1 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1817 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1818 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1819 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1820 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1821 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1822 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1823 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1824 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1825 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1826 1816 0 09:48 ? 00:00:06 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1827 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1828 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1829 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1830 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1831 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1832 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1833 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1834 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1835 1816 0 09:48 ? 00:00:01 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1836 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1837 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1838 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1839 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1840 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam > > > > [root@msteams-sip-01 ~]# free -m > > > > total used free shared buff/cache available > > > > Mem: 7822 221 4055 386 3545 6900 > > > > Swap: 5119 0 5119 > > > > > > > > On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa > > > > <danishmoosa(a)gmail.com> wrote: > > > > > > > > > > Hi, > > > > > > > > > > I have been kamailio services working for few months and > > > > > suddenly I got this error. > > > > > > > > > > ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable > > > > > to set pv [$rp] > > > > > ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not > > > > > allocate private memory from pkg pool > > > > > ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot > > > > > get pv spec for > > > > > [$dlg_var(BW0930526150606231188928742(a)10.10.16.100)] > > > > > ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): > > > > > could not allocate private memory from pkg pool > > > > > ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): > > > > > out of mem > > > > > ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to > > > > > add branches > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > build_res_buf_from_sip_req(): could not allocate private > > > > > memory from pkg pool > > > > > ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless > > > > > error reply > > > > > used: I'm terribly sorry, server error occurred (1/SL) > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > build_res_buf_from_sip_req(): could not allocate private > > > > > memory from pkg pool > > > > > ERROR: tm [t_reply.c:475]: _reply_light(): response > > > > > building failed > > > > > WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a > > > > > delayed stateful reply failed > > > > > > > > > > Issue was fixed after the restart of kamailio. > > > > > > > > > > -- > > > > > Muhammad Danish Moosa > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > Christopher McCandless > > > > > > > > > > > > > > > > -- > > > > Muhammad Danish Moosa > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > Christopher McCandless > > > > > > > > > > > > -- > > > Muhammad Danish Moosa > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > Christopher McCandless > > > __________________________________________________________ > > > Kamailio - Users Mailing List - Non Commercial Discussions To > > > unsubscribe send an email to sr-users-leave(a)lists.kamailio.org > > > Important: keep the mailing list in the recipients, do not reply > > > only to the > sender! > > > Edit mailing list options or unsubscribe: > > > > -- > Muhammad Danish Moosa > > " The core of mans' spirit comes from new experiences. "___ > Christopher McCandless

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Kaufman

9:56 a.m.

While it's not "official" my recollection is that `make uninstall` will provide a list of the files to be removed to uninstall kamailio. I've done this and validated what I want to keep/delete to make a simple 'uninstall' shell script (i.e. I'd want to keep my config files in /etc/kamailio). You should be able to git pull the desired branch/tag into your working directory, run `make distclean`, rerun `./configure` with whatever parameters you want, and `make` all without any impact to the running software at all. The bulk of the compilation time is running `make`, so there's no reason this can't be done ahead of time. I have, at various times, run the uninstall script and `make install` before restarting kamailio and had success, but I would guess the "correct" way to do this is to stop kamailio, run the uninstall script, run `make install`, then start Kamailio. You might also consider building packages from source depending if there's supporting code for your distribution. While this is slightly more work, it does leave you with a simpler installation pattern. - Kaufman -----Original Message----- From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> Sent: Tuesday, June 6, 2023 7:26 AM To: Henning Westerholt <hw(a)gilawa.com> Cc: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org> Subject: [SR-Users] Re: Kamailio - could not allocate private memory from pkg pool CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Henning, All good,I will re-examine the changes. There were positive changes after that. Before that , I had to restart kamailio around every week, Quick question, is there any cleaner procedure to upgrade from 5.5.3 to 5.5.6. I have compiled from sources. Please understand these are production systems and should be done with 0 or minimal outage. Danish On Tue, Jun 6, 2023 at 10:14 PM Henning Westerholt <hw(a)gilawa.com> wrote:

...

Hello Muhammad, maybe its related to the changes you've done, maybe its some load increase or similar, but hard to say from the outside. As said, maybe try the minor update first, and then if its persists, lets go the memory debugging route. Cheers, Henning -- Henning Westerholt – https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskal atan.de%2Fblog%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af403b8b14 266a53608db668a59c4%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C63821 6516674656903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu MzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=irgb9nc1zWh0X Ssq5JuKlApvUZBlT5nvwpzYKwjHFOg%3D&reserved=0 Kamailio services – https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgila wa.com%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af403b8b14266a5360 8db668a59c4%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C6382165166746 56903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KhlWLRiIZwEAb2JvgfUaS gX7kk5WKTWUAfJGiE%2BhLGo%3D&reserved=0 > -----Original Message----- > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > Sent: Dienstag, 6. Juni 2023 13:58 > To: Henning Westerholt <hw(a)gilawa.com> > Cc: Kamailio (SER) - Users Mailing List > <sr-users(a)lists.kamailio.org> > Subject: Re: [SR-Users] Re: Kamailio - could not allocate private > memory from pkg pool > > Thanks for the response. I really appreciate it. > > This system was tested thoroughly in the lab but problems only > appeared in production. Actually it was working smoothly for 3 > months after I fixed some issues like external system calls etc and this appeared all of sudden. > > > > > > On Tue, Jun 6, 2023 at 9:40 PM Henning Westerholt <hw(a)gilawa.com> wrote: > > > > Hello, > > > > first step should be probably to update to the latest version of > > the 5.5.x branch, > e.g., 5.5.6. There have been several fixes, some also related to > memory leaks included if I remember correctly. > > > > If the problem persists, it might be a new memory leak. This can > > be found by > enabling memory debugging, preferable on a test system or lab system. > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > www.kamailio.org%2Fwiki%2Ftutorials%2Ftroubleshooting%2Fmemory&dat > > a=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af403b8b14266a53608db668a59 > > c4%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C638216516674656903 > > %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT > > iI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3c61kNO7YPT6EYTlnl > > LgQARPKyQQ9KJjQkualoxUvJ0%3D&reserved=0 > > > > Cheers, > > > > Henning > > > > -- > > Henning Westerholt – > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > skalatan.de%2Fblog%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af > > 403b8b14266a53608db668a59c4%7Cafc1818e7b6848568913201b9396c4fc%7C1 > > %7C0%7C638216516674656903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw > > MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C > > &sdata=irgb9nc1zWh0XSsq5JuKlApvUZBlT5nvwpzYKwjHFOg%3D&reserved=0 > > Kamailio services – > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > gilawa.com%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af403b8b14 > > 266a53608db668a59c4%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C6 > > 38216516674656903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ > > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=K > > hlWLRiIZwEAb2JvgfUaSgX7kk5WKTWUAfJGiE%2BhLGo%3D&reserved=0 > > > > > -----Original Message----- > > > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > > > Sent: Dienstag, 6. Juni 2023 10:46 > > > To: Kamailio (SER) - Users Mailing List > > > <sr-users(a)lists.kamailio.org> > > > Subject: [SR-Users] Re: Kamailio - could not allocate private > > > memory from pkg pool > > > > > > Mainly if I am looking at the private memory usage , it keeps > > > increasing very slowly but does not go back even after the calls > > > are literally zero. I suspect some UDP listener process are not > > > releasing the > memory. > > > > > > If I consider that my kamailio cfg and lua code has problem with > > > variables or something , I want to understand how it works when > > > calls are released. My understanding is usage should be minimum > > > when calls are released. Looking for feedback please? > > > > > > [roottmp]# kamcmd pkg.stats |grep real_used > > > real_used: 836144 > > > real_used: 1017328 > > > real_used: 1028656 > > > real_used: 1020864 > > > real_used: 1020632 > > > real_used: 1006120 > > > real_used: 1014536 > > > real_used: 1023984 > > > real_used: 1017584 > > > real_used: 835784 > > > real_used: 835784 > > > real_used: 835784 > > > real_used: 844344 > > > real_used: 835784 > > > real_used: 901344 > > > real_used: 835784 > > > real_used: 1259728 > > > real_used: 1196688 > > > real_used: 1181320 > > > real_used: 1166624 > > > real_used: 1216032 > > > real_used: 1173464 > > > real_used: 1168552 > > > real_used: 1235528 > > > real_used: 917904 > > > > > > > > > > > > For short term I increase the pvt memory allocation or keep > > > restarting kamailio after every week or fortnight but I am > > > looking for more stable solution. I am using kamailio 5.5.3 > > > > > > On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa > > > <danishmoosa(a)gmail.com> wrote: > > > > > > > > Any recommendations? > > > > I have good settings for pkg and shm but I can increase more > > > > > > > > [root1 ~]# ps -ef |grep kam > > > > root 1816 1 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1817 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1818 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1819 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1820 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1821 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1822 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1823 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1824 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1825 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1826 1816 0 09:48 ? 00:00:06 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1827 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1828 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1829 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1830 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1831 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1832 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1833 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1834 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1835 1816 0 09:48 ? 00:00:01 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1836 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1837 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1838 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1839 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1840 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam > > > > [root@msteams-sip-01 ~]# free -m > > > > total used free shared buff/cache available > > > > Mem: 7822 221 4055 386 3545 6900 > > > > Swap: 5119 0 5119 > > > > > > > > On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa > > > > <danishmoosa(a)gmail.com> wrote: > > > > > > > > > > Hi, > > > > > > > > > > I have been kamailio services working for few months and > > > > > suddenly I got this error. > > > > > > > > > > ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable > > > > > to set pv [$rp] > > > > > ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not > > > > > allocate private memory from pkg pool > > > > > ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot > > > > > get pv spec for > > > > > [$dlg_var(BW0930526150606231188928742(a)10.10.16.100)] > > > > > ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): > > > > > could not allocate private memory from pkg pool > > > > > ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): > > > > > out of mem > > > > > ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to > > > > > add branches > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > build_res_buf_from_sip_req(): could not allocate private > > > > > memory from pkg pool > > > > > ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless > > > > > error reply > > > > > used: I'm terribly sorry, server error occurred (1/SL) > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > build_res_buf_from_sip_req(): could not allocate private > > > > > memory from pkg pool > > > > > ERROR: tm [t_reply.c:475]: _reply_light(): response > > > > > building failed > > > > > WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a > > > > > delayed stateful reply failed > > > > > > > > > > Issue was fixed after the restart of kamailio. > > > > > > > > > > -- > > > > > Muhammad Danish Moosa > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > Christopher McCandless > > > > > > > > > > > > > > > > -- > > > > Muhammad Danish Moosa > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > Christopher McCandless > > > > > > > > > > > > -- > > > Muhammad Danish Moosa > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > Christopher McCandless > > > __________________________________________________________ > > > Kamailio - Users Mailing List - Non Commercial Discussions To > > > unsubscribe send an email to sr-users-leave(a)lists.kamailio.org > > > Important: keep the mailing list in the recipients, do not reply > > > only to the > sender! > > > Edit mailing list options or unsubscribe: > > > > -- > Muhammad Danish Moosa > > " The core of mans' spirit comes from new experiences. "___ > Christopher McCandless

Muhammad Danish Moosa

7 Jun 7 Jun

9:21 a.m.

Just a quick question: How does kamailio acquire memory? I have started kamailio with 32mb Pkg and 1.5gb shared memory but when I look at free , it does not seem to match. Only 265 is used. Does it reside in cache buff? root 11195 11171 0 18:06 ? 00:00:00 /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamailio//kamailio.cfg -M 32 -m 1536 [root@ ~]# free -m total used free shared buff/cache available Mem: 7822 265 3881 377 3675 6862 Swap: 5119 0 5119 [root@msteams-sip-01 ~]# On Wed, Jun 7, 2023 at 12:32 AM Kaufman <bkaufman(a)bcmone.com> wrote:

...

Hello Muhammad, maybe its related to the changes you've done, maybe its some load increase or similar, but hard to say from the outside. As said, maybe try the minor update first, and then if its persists, lets go the memory debugging route. Cheers, Henning -- Henning Westerholt – https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskal atan.de%2Fblog%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af403b8b14 266a53608db668a59c4%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C63821 6516674656903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2lu MzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=irgb9nc1zWh0X Ssq5JuKlApvUZBlT5nvwpzYKwjHFOg%3D&reserved=0 Kamailio services – https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgila wa.com%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af403b8b14266a5360 8db668a59c4%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C6382165166746 56903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KhlWLRiIZwEAb2JvgfUaS gX7kk5WKTWUAfJGiE%2BhLGo%3D&reserved=0 > -----Original Message----- > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > Sent: Dienstag, 6. Juni 2023 13:58 > To: Henning Westerholt <hw(a)gilawa.com> > Cc: Kamailio (SER) - Users Mailing List > <sr-users(a)lists.kamailio.org> > Subject: Re: [SR-Users] Re: Kamailio - could not allocate private > memory from pkg pool > > Thanks for the response. I really appreciate it. > > This system was tested thoroughly in the lab but problems only > appeared in production. Actually it was working smoothly for 3 > months after I fixed some issues like external system calls etc and this appeared all of sudden. > > > > > > On Tue, Jun 6, 2023 at 9:40 PM Henning Westerholt <hw(a)gilawa.com> wrote: > > > > Hello, > > > > first step should be probably to update to the latest version of > > the 5.5.x branch, > e.g., 5.5.6. There have been several fixes, some also related to > memory leaks included if I remember correctly. > > > > If the problem persists, it might be a new memory leak. This can > > be found by > enabling memory debugging, preferable on a test system or lab system. > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > www.kamailio.org%2Fwiki%2Ftutorials%2Ftroubleshooting%2Fmemory&dat > > a=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af403b8b14266a53608db668a59 > > c4%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C638216516674656903 > > %7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBT > > iI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3c61kNO7YPT6EYTlnl > > LgQARPKyQQ9KJjQkualoxUvJ0%3D&reserved=0 > > > > Cheers, > > > > Henning > > > > -- > > Henning Westerholt – > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > skalatan.de%2Fblog%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af > > 403b8b14266a53608db668a59c4%7Cafc1818e7b6848568913201b9396c4fc%7C1 > > %7C0%7C638216516674656903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw > > MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C > > &sdata=irgb9nc1zWh0XSsq5JuKlApvUZBlT5nvwpzYKwjHFOg%3D&reserved=0 > > Kamailio services – > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > gilawa.com%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7C1c9af403b8b14 > > 266a53608db668a59c4%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C6 > > 38216516674656903%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ > > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=K > > hlWLRiIZwEAb2JvgfUaSgX7kk5WKTWUAfJGiE%2BhLGo%3D&reserved=0 > > > > > -----Original Message----- > > > From: Muhammad Danish Moosa <danishmoosa(a)gmail.com> > > > Sent: Dienstag, 6. Juni 2023 10:46 > > > To: Kamailio (SER) - Users Mailing List > > > <sr-users(a)lists.kamailio.org> > > > Subject: [SR-Users] Re: Kamailio - could not allocate private > > > memory from pkg pool > > > > > > Mainly if I am looking at the private memory usage , it keeps > > > increasing very slowly but does not go back even after the calls > > > are literally zero. I suspect some UDP listener process are not > > > releasing the > memory. > > > > > > If I consider that my kamailio cfg and lua code has problem with > > > variables or something , I want to understand how it works when > > > calls are released. My understanding is usage should be minimum > > > when calls are released. Looking for feedback please? > > > > > > [roottmp]# kamcmd pkg.stats |grep real_used > > > real_used: 836144 > > > real_used: 1017328 > > > real_used: 1028656 > > > real_used: 1020864 > > > real_used: 1020632 > > > real_used: 1006120 > > > real_used: 1014536 > > > real_used: 1023984 > > > real_used: 1017584 > > > real_used: 835784 > > > real_used: 835784 > > > real_used: 835784 > > > real_used: 844344 > > > real_used: 835784 > > > real_used: 901344 > > > real_used: 835784 > > > real_used: 1259728 > > > real_used: 1196688 > > > real_used: 1181320 > > > real_used: 1166624 > > > real_used: 1216032 > > > real_used: 1173464 > > > real_used: 1168552 > > > real_used: 1235528 > > > real_used: 917904 > > > > > > > > > > > > For short term I increase the pvt memory allocation or keep > > > restarting kamailio after every week or fortnight but I am > > > looking for more stable solution. I am using kamailio 5.5.3 > > > > > > On Tue, Jun 6, 2023 at 1:21 PM Muhammad Danish Moosa > > > <danishmoosa(a)gmail.com> wrote: > > > > > > > > Any recommendations? > > > > I have good settings for pkg and shm but I can increase more > > > > > > > > [root1 ~]# ps -ef |grep kam > > > > root 1816 1 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1817 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1818 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1819 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1820 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1821 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1822 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1823 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1824 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1825 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1826 1816 0 09:48 ? 00:00:06 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1827 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1828 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1829 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1830 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1831 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1832 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1833 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1834 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1835 1816 0 09:48 ? 00:00:01 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1836 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1837 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1838 1816 0 09:48 ? 00:00:02 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1839 1816 0 09:48 ? 00:00:03 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 1840 1816 0 09:48 ? 00:00:00 > > > > /usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f > > > > /usr/local/etc/kamailio//kamailio.cfg -M 16 -m 1024 > > > > root 3100 3062 0 13:13 pts/1 00:00:00 grep --color=auto kam > > > > [root@msteams-sip-01 ~]# free -m > > > > total used free shared buff/cache available > > > > Mem: 7822 221 4055 386 3545 6900 > > > > Swap: 5119 0 5119 > > > > > > > > On Tue, Jun 6, 2023 at 1:19 PM Muhammad Danish Moosa > > > > <danishmoosa(a)gmail.com> wrote: > > > > > > > > > > Hi, > > > > > > > > > > I have been kamailio services working for few months and > > > > > suddenly I got this error. > > > > > > > > > > ERROR: <core> [core/kemi.c:2923]: sr_kemi_pv_sets(): unable > > > > > to set pv [$rp] > > > > > ERROR: <core> [core/pvapi.c:352]: pv_cache_add(): could not > > > > > allocate private memory from pkg pool > > > > > ERROR: <core> [core/kemi.c:2914]: sr_kemi_pv_sets(): cannot > > > > > get pv spec for > > > > > [$dlg_var(BW0930526150606231188928742(a)10.10.16.100)] > > > > > ERROR: <core> [core/data_lump.c:544]: dup_lump_list_r(): > > > > > could not allocate private memory from pkg pool > > > > > ERROR: <core> [core/data_lump.c:582]: dup_lump_list_r(): > > > > > out of mem > > > > > ERROR: tm [t_fwd.c:1759]: t_forward_nonack(): failure to > > > > > add branches > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > build_res_buf_from_sip_req(): could not allocate private > > > > > memory from pkg pool > > > > > ERROR: sl [sl_funcs.c:414]: sl_reply_error(): stateless > > > > > error reply > > > > > used: I'm terribly sorry, server error occurred (1/SL) > > > > > ERROR: <core> [core/msg_translator.c:2501]: > > > > > build_res_buf_from_sip_req(): could not allocate private > > > > > memory from pkg pool > > > > > ERROR: tm [t_reply.c:475]: _reply_light(): response > > > > > building failed > > > > > WARNING: tm [t_lookup.c:1499]: t_unref(): generation of a > > > > > delayed stateful reply failed > > > > > > > > > > Issue was fixed after the restart of kamailio. > > > > > > > > > > -- > > > > > Muhammad Danish Moosa > > > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > > Christopher McCandless > > > > > > > > > > > > > > > > -- > > > > Muhammad Danish Moosa > > > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > > Christopher McCandless > > > > > > > > > > > > -- > > > Muhammad Danish Moosa > > > > > > " The core of mans' spirit comes from new experiences. "___ > > > Christopher McCandless > > > __________________________________________________________ > > > Kamailio - Users Mailing List - Non Commercial Discussions To > > > unsubscribe send an email to sr-users-leave(a)lists.kamailio.org > > > Important: keep the mailing list in the recipients, do not reply > > > only to the > sender! > > > Edit mailing list options or unsubscribe: > > > > -- > Muhammad Danish Moosa > > " The core of mans' spirit comes from new experiences. "___ > Christopher McCandless

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:

-- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless

Daniel-Constantin Mierla

9 Jun 9 Jun

2:13 a.m.

Hello, as I can see you use KEMI and the out of pkg could be because you define too many pseudo-variables. Does your kemi script set/get pseudo-variables with different name each time processing a sip message? Like $avp(x) or $var(z) where x or z are different for each sip message? Cheers, Daniel On 06.06.23 05:19, Muhammad Danish Moosa wrote:

...

-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - www.kamailioworld.com

566

days inactive

569

days old

sr-users@lists.kamailio.org

Manage subscription

10 comments

4 participants

tags (0)

participants (4)

Daniel-Constantin Mierla
Henning Westerholt
Kaufman
Muhammad Danish Moosa