Hello,
there is no indication yet where is the leak, so you have to upgrade
core and all modules you are using.
If you configured with 4GB of memory, then there is a leak. You say you
have your own customized modules, you should investigate your changes
first to see if there are leaks.
There are some notes on how to compile with memory debugger and
troubleshoot leaks at:
-
Hello Daniel,
Thank you so much for your answer
I want to ask you something before upgrade kamailio, currently we are
config shared memory size is 4GB, and normal kamailio only use 12-14
MB. We are changed some module for our purpose like using cnxcc (using
for prepaid) and alias_db module for call forwarding feature. Like you
said we should upgrade to kamailio-4.1.8, so we have to upgrade all
modules and core or only some modules which related to this error? If
only some modules, could you tell me what modules?
Thanks so much for your help!
On Wed, May 6, 2015 at 2:34 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
upgrade to kamailio 4.1.8, because there are fixes since 4.1.5.
Same config and database structure are used in all 4.1.x releases,
therefore you just need to deploy the new version and restart.
Also, be sure you have allocated sufficient shared memory (-m
command line parameter), by default is 32MB, once that is filled,
kamailio throws out of memory errors.
Cheers,
Daniel
On 06/05/15 04:27, Bui Minh Yen wrote:
Hi everyone,
I am using Kamailio-4.1.5. Sometime kamailio was crash because
memory leak.
Log when out of mem
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
tm [t_lookup.c:1332]: new_t(): ERROR: new_t: out of mem:
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
usrloc [ucontact.c:103]: new_ucontact(): no more shm memory
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
usrloc [urecord.c:155]: mem_insert_ucontact(): failed to create
new contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
usrloc [urecord.c:547]: insert_ucontact(): failed to insert contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
registrar [save.c:708]: update_contacts(): failed to insert contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
tm [t_lookup.c:1332]: new_t(): ERROR: new_t: out of mem:
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failed
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
usrloc [ucontact.c:103]: new_ucontact(): no more shm memory
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
usrloc [urecord.c:155]: mem_insert_ucontact(): failed to create
new contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
usrloc [urecord.c:547]: insert_ucontact(): failed to insert contact
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR:
registrar [save.c:521]: insert_contacts(): failed to insert contact
Log when kamailio was crash
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: ALERT:
<core> [main.c:775]: handle_sigs(): child process 30821 exited by
a signal 11
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: ALERT:
<core> [main.c:778]: handle_sigs(): core was not generated
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: INFO:
<core> [main.c:790]: handle_sigs(): INFO: terminating due to SIGCHLD
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: INFO:
<core> [main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
<core> [main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: heap size= 749731840
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: used= 1110960, used+overhead=1663384, free=748068456
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: max used (+overhead)= 1825240
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8
size=1024 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958
size=256 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: INFO:
<core> [main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 2. N address=0x7f6af599dae8 frag=0x7f6af599dab8
size=176 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: counters.c:
cnt_hash_add(339)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 3. N address=0x7f6af599dbf8 frag=0x7f6af599dbc8
size=240 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: counters.c:
cnt_hash_add(339)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 4. N address=0x7f6af599dd48 frag=0x7f6af599dd18
size=128 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: counters.c:
init_counters(129)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 5. N address=0x7f6af599de28 frag=0x7f6af599ddf8
size=16 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: modparam.c:
set_mod_param_regex(124)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 6. N address=0x7f6af599de98 frag=0x7f6af599de68
size=128 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 7. N address=0x7f6af599df78 frag=0x7f6af599df48
size=48 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 8. N address=0x7f6af599e008 frag=0x7f6af599dfd8
size=16 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(181)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 9. N address=0x7f6af599e078 frag=0x7f6af599e048
size=128 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 10. N address=0x7f6af599e158 frag=0x7f6af599e128
size=48 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 11. N address=0x7f6af599e1e8 frag=0x7f6af599e1b8
size=16 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(181)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 12. N address=0x7f6af599e258 frag=0x7f6af599e228
size=128 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 13. N address=0x7f6af599e338 frag=0x7f6af599e308
size=48 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 14. N address=0x7f6af599e3c8 frag=0x7f6af599e398
size=16 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(181)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 15. N address=0x7f6af599e438 frag=0x7f6af599e408
size=128 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: INFO:
<core> [main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
<core> [main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: heap size= 749731840
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: used= 1110768, used+overhead=1663192, free=748068648
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: max used (+overhead)= 1825240
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8
size=1024 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958
size=256 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30827]: INFO:
<core> [main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: INFO:
<core> [main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: NOTICE:
<core> [main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
<core> [main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: heap size= 749731840
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: used= 1110768, used+overhead=1663192, free=748068648
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: max used (+overhead)= 1825240
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8
size=1024 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: INFO:
<core> [main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
<core> [main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: heap size= 749731840
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: used= 1110768, used+overhead=1663192, free=748068648
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: max used (+overhead)= 1825240
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8
size=1024 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958
size=256 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: INFO:
<core> [main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: 2. N address=0x7f6af599dae8 frag=0x7f6af599dab8
size=176 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE:
qm_status: alloc'd from <core>: counters.c:
cnt_hash_add(339)
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE:
qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958
size=256 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: NOTICE:
<core> [main.c:852]: sig_usr(): Memory status (pkg):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE:
qm_status: 16. N address=0x7f6af599e518 frag=0x7f6af599e4e8
size=48 used=1
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: NOTICE:
qm_status: (0x7f6af5968010):
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30822]: INFO:
<core> [main.c:841]: sig_usr(): INFO: signal 15 received
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30822]: NOTICE:
<core> [main.c:852]: sig_usr(): Memory status (pkg):
Could you help me to fix this problem!
Thanks so much!
--
---------------------------
*Bui Thi Minh Yen*
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany -
http://www.kamailioworld.com
--
---------------------------
*Bui Thi Minh Yen*
SSDC, VNPT Technology
124 Hoang Quoc Viet | Mobile: (+84) 973110551 | Skype: minhyenbt