Hello,
We had been experiencing memory leaks issues with our Kamailio deployment. Some of the process started to use all their assigned memory and arises allocation memory errors. Using the "kill -SIGUSR1" as explained in the Kamailio wiki and found the problem:
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: <core> [main.c:788]: INFO: signal 15 received
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: <core> [main.c:799]: Memory status (pkg):
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: qm_status: (0x885a60):
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: qm_status: heap size= 67108864
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: qm_status: used= 60641440, used+overhead=67096424, free=12440
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: qm_status: max used (+overhead)= 67102896
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: qm_status: dumping all alloc'ed. fragments:
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: qm_status: 0. N address=0x8baf78 frag=0x8baf48 size=16 used=1
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: qm_status: alloc'd from <core>: rvalue.c: mk_rval_expr_v(2452)
Sep 8 11:34:04 localhost /sbin/kamailio[17275]: INFO: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
...
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: 1618. N address=0x95e920 frag=0x95e8f0 size=936 used=1
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: alloc'd from <core>: rvalue.c: rval_new_empty(233)
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: 1633. N address=0x95f528 frag=0x95f4f8 size=936 used=1
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: alloc'd from <core>: rvalue.c: rval_new_empty(233)
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: 1634. N address=0x95f930 frag=0x95f900 size=936 used=1
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: alloc'd from <core>: rvalue.c: rval_new_empty(233)
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: 1696. N address=0x962298 frag=0x962268 size=944 used=1
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: alloc'd from <core>: rvalue.c: rval_new_empty(233)
Sep 8 11:34:06 localhost /sbin/kamailio[17275]: INFO: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
.
. (the rvalue.c repeats all the time)
.
Sep 8 11:34:23 localhost /sbin/kamailio[17275]: INFO: qm_status: 64963. N address=0x4885118 frag=0x48850e8 size=944 used=1
Sep 8 11:34:23 localhost /sbin/kamailio[17275]: INFO: qm_status: alloc'd from <core>: rvalue.c: rval_new_empty(233)
Sep 8 11:34:23 localhost /sbin/kamailio[17275]: INFO: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Sep 8 11:34:23 localhost /sbin/kamailio[17275]: INFO: qm_status: 64964. N address=0x4885528 frag=0x48854f8 size=1288 used=1
Sep 8 11:34:23 localhost /sbin/kamailio[17275]: INFO: qm_status: alloc'd from <core>: rvalue.c: rval_new_empty(233)
Sep 8 11:34:23 localhost /sbin/kamailio[17275]: INFO: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
We saw there was several core memory leaks resolved in the 3.0.3 version (we had the 3.0.2), so we decided to upgrade to that version. Now I was trying to check the memory status of the process (in order to see if the problem is arising again) with "kill -SIGUSR1" but now only show the memory status of the shm and the memory of the attendant process (the one who launch the other ones). I had tried to send the signal to each of the son process but I see nothing in the log.
Is this a new bug or is that there is other new way to see the memory status of the processes?
Thanks in advance,
http://www.alhambra-eidos.es/CO/1.gifhttp://www.alhambra-eidos.es/CO/1.gif
César Pinto. ICM Team +34 91 787 23 00
http://www.alhambra-eidos.es/CO/3.gif http://www.alhambra-eidos.es/
On Friday 24 September 2010, César Pinto Magán wrote:
We had been experiencing memory leaks issues with our Kamailio deployment. Some of the process started to use all their assigned memory and arises allocation memory errors. Using the “kill –SIGUSR1” as explained in the Kamailio wiki and found the problem: [..] We saw there was several core memory leaks resolved in the 3.0.3 version (we had the 3.0.2), so we decided to upgrade to that version. Now I was trying to check the memory status of the process (in order to see if the problem is arising again) with “kill –SIGUSR1” but now only show the memory status of the shm and the memory of the attendant process (the one who launch the other ones). I had tried to send the signal to each of the son process but I see nothing in the log. Is this a new bug or is that there is other new way to see the memory status of the processes?
Hi César,
normally it should work here as in the old version. Do you compiled the new version also with memory debug enabled and enabled also the appropriate logging in the configuration?
build: make cfg mode=debug MEMDBG=1 make all
cfg: memlog=1 memdbg=1
Cheers,
Henning
Hello Henning.
Yes, I made both things. The only thing is that I have the next configuration:
debug=5 log_stderror=no memdbg=3 memlog=3
But it worked in the previous version, so it should work now also (or that I suppose). Really I think that's not the problem, because the debug for the shared memory and the package memory of the principal thread appears, so it should be in other place (or I'm wrong in my supposes and I should configure the Kamailio in another way).
Regards,
César Pinto (2439) +34 91 787 23 00 alhambra-eidos.es
-----Mensaje original----- De: Henning Westerholt [mailto:henning.westerholt@1und1.de] Enviado el: martes, 28 de septiembre de 2010 14:54 Para: sr-users@lists.sip-router.org CC: César Pinto Magán; users@lists.kamailio.org Asunto: Re: [SR-Users] Problem with debug memory: kill -SIGUSR1 doesn't work fine
On Friday 24 September 2010, César Pinto Magán wrote:
We had been experiencing memory leaks issues with our Kamailio deployment. Some of the process started to use all their assigned memory and arises allocation memory errors. Using the “kill –SIGUSR1” as explained in the Kamailio wiki and found the problem: [..] We saw there was several core memory leaks resolved in the 3.0.3 version (we had the 3.0.2), so we decided to upgrade to that version. Now I was trying to check the memory status of the process (in order to see if the problem is arising again) with “kill –SIGUSR1” but now only show the memory status of the shm and the memory of the attendant process (the one who launch the other ones). I had tried to send the signal to each of the son process but I see nothing in the log. Is this a new bug or is that there is other new way to see the memory status of the processes?
Hi César,
normally it should work here as in the old version. Do you compiled the new version also with memory debug enabled and enabled also the appropriate logging in the configuration?
build: make cfg mode=debug MEMDBG=1 make all
cfg: memlog=1 memdbg=1
Cheers,
Henning
-
César Pinto Magán -
Henning Westerholt