Hello,
there is no indication yet where is the leak, so you have to upgrade core
and all modules you are using.
If you configured with 4GB of memory, then there is a leak. You say you
have your own customized modules, you should investigate your changes first
to see if there are leaks.
There are some notes on how to compile with memory debugger and
troubleshoot leaks at:
-
Cheers,
Daniel
On 06/05/15 10:40, Bui Minh Yen wrote:
Hello Daniel,
Thank you so much for your answer
I want to ask you something before upgrade kamailio, currently we are
config shared memory size is 4GB, and normal kamailio only use 12-14 MB. We
are changed some module for our purpose like using cnxcc (using for
prepaid) and alias_db module for call forwarding feature. Like you said we
should upgrade to kamailio-4.1.8, so we have to upgrade all modules and
core or only some modules which related to this error? If only some
modules, could you tell me what modules?
Thanks so much for your help!
On Wed, May 6, 2015 at 2:34 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
upgrade to kamailio 4.1.8, because there are fixes since 4.1.5. Same
config and database structure are used in all 4.1.x releases, therefore you
just need to deploy the new version and restart.
Also, be sure you have allocated sufficient shared memory (-m command
line parameter), by default is 32MB, once that is filled, kamailio throws
out of memory errors.
Cheers,
Daniel
On 06/05/15 04:27, Bui Minh Yen wrote:
Hi everyone,
I am using Kamailio-4.1.5. Sometime kamailio was crash because memory
leak.
Log when out of mem
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm
[t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm
[t_lookup.c:1332]: new_t(): ERROR: new_t: out of mem:
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm
[t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc
[ucontact.c:103]: new_ucontact(): no more shm memory
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc
[urecord.c:155]: mem_insert_ucontact(): failed to create new contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc
[urecord.c:547]: insert_ucontact(): failed to insert contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
registrar [save.c:708]: update_contacts(): failed to insert contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm
[t_lookup.c:1332]: new_t(): ERROR: new_t: out of mem:
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm
[t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc
[ucontact.c:103]: new_ucontact(): no more shm memory
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc
[urecord.c:155]: mem_insert_ucontact(): failed to create new contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc
[urecord.c:547]: insert_ucontact(): failed to insert contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
registrar [save.c:521]: insert_contacts(): failed to insert contact
Log when kamailio was crash
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: ALERT: <core>
[main.c:775]: handle_sigs(): child process 30821 exited by a signal 11
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: ALERT: <core>
[main.c:778]: handle_sigs(): core was not generated
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: INFO: <core>
[main.c:790]: handle_sigs(): INFO: terminating due to SIGCHLD
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: INFO: <core>
[main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: <core>
[main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: heap size= 749731840
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: used= 1110960, used+overhead=1663384, free=748068456
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: max used (+overhead)= 1825240
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958 size=256
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: INFO: <core>
[main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 2. N address=0x7f6af599dae8 frag=0x7f6af599dab8 size=176
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: counters.c: cnt_hash_add(339)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 3. N address=0x7f6af599dbf8 frag=0x7f6af599dbc8 size=240
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: counters.c: cnt_hash_add(339)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 4. N address=0x7f6af599dd48 frag=0x7f6af599dd18 size=128
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: counters.c: init_counters(129)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 5. N address=0x7f6af599de28 frag=0x7f6af599ddf8 size=16
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: modparam.c:
set_mod_param_regex(124)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 6. N address=0x7f6af599de98 frag=0x7f6af599de68 size=128
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 7. N address=0x7f6af599df78 frag=0x7f6af599df48 size=48
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 8. N address=0x7f6af599e008 frag=0x7f6af599dfd8 size=16
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(181)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 9. N address=0x7f6af599e078 frag=0x7f6af599e048 size=128
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 10. N address=0x7f6af599e158 frag=0x7f6af599e128 size=48
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 11. N address=0x7f6af599e1e8 frag=0x7f6af599e1b8 size=16
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(181)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 12. N address=0x7f6af599e258 frag=0x7f6af599e228 size=128
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 13. N address=0x7f6af599e338 frag=0x7f6af599e308 size=48
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 14. N address=0x7f6af599e3c8 frag=0x7f6af599e398 size=16
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(181)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 15. N address=0x7f6af599e438 frag=0x7f6af599e408 size=128
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: INFO: <core>
[main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: <core>
[main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: heap size= 749731840
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: used= 1110768, used+overhead=1663192, free=748068648
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: max used (+overhead)= 1825240
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958 size=256
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30827]: INFO: <core>
[main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: INFO: <core>
[main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: NOTICE: <core>
[main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: <core>
[main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: heap size= 749731840
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: used= 1110768, used+overhead=1663192, free=748068648
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: max used (+overhead)= 1825240
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: INFO: <core>
[main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: <core>
[main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: heap size= 749731840
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: used= 1110768, used+overhead=1663192, free=748068648
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: max used (+overhead)= 1825240
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958 size=256
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: INFO: <core>
[main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: 2. N address=0x7f6af599dae8 frag=0x7f6af599dab8 size=176
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: alloc'd from <core>: counters.c: cnt_hash_add(339)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958 size=256
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: NOTICE: <core>
[main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 16. N address=0x7f6af599e518 frag=0x7f6af599e4e8 size=48
used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30822]: INFO: <core>
[main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30822]: NOTICE: <core>
[main.c:852]: sig_usr(): Memory status (pkg):
Could you help me to fix this problem!
Thanks so much!
--
---------------------------
*Bui Thi Minh Yen*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin
Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany -
http://www.kamailioworld.com