Hello,
there is no indication yet where is the leak, so you have to upgrade core and all modules you are using.
If you configured with 4GB of memory, then there is a leak. You say you have your own customized modules, you should investigate your changes first to see if there are leaks.
There are some notes on how to compile with memory debugger and troubleshoot leaks at:
- http://kamailio.org/wiki/tutorials/troubleshooting/memory
Cheers,
Daniel
On 06/05/15 10:40, Bui Minh Yen wrote:
Hello Daniel,
Thank you so much for your answerI want to ask you something before upgrade kamailio, currently we are config shared memory size is 4GB, and normal kamailio only use 12-14 MB. We are changed some module for our purpose like using cnxcc (using for prepaid) and alias_db module for call forwarding feature. Like you said we should upgrade to kamailio-4.1.8, so we have to upgrade all modules and core or only some modules which related to this error? If only some modules, could you tell me what modules?
Thanks so much for your help!
On Wed, May 6, 2015 at 2:34 PM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,
upgrade to kamailio 4.1.8, because there are fixes since 4.1.5. Same config and database structure are used in all 4.1.x releases, therefore you just need to deploy the new version and restart.
Also, be sure you have allocated sufficient shared memory (-m command line parameter), by default is 32MB, once that is filled, kamailio throws out of memory errors.
Cheers,
Daniel
On 06/05/15 04:27, Bui Minh Yen wrote:
Thanks so much!Hi everyone,I am using Kamailio-4.1.5. Sometime kamailio was crash because memory leak.
Log when out of mem
May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failedMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1332]: new_t(): ERROR: new_t: out of mem:May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failedMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [ucontact.c:103]: new_ucontact(): no more shm memoryMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [urecord.c:155]: mem_insert_ucontact(): failed to create new contactMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [urecord.c:547]: insert_ucontact(): failed to insert contactMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: registrar [save.c:708]: update_contacts(): failed to insert contactMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1332]: new_t(): ERROR: new_t: out of mem:May 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: tm [t_lookup.c:1472]: t_newtran(): ERROR: t_newtran: new_t failedMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [ucontact.c:103]: new_ucontact(): no more shm memoryMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [urecord.c:155]: mem_insert_ucontact(): failed to create new contactMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: usrloc [urecord.c:547]: insert_ucontact(): failed to insert contactMay 2 09:49:00 vnpsip01 /usr/local/sbin/kamailio[30821]: ERROR: registrar [save.c:521]: insert_contacts(): failed to insert contact
Log when kamailio was crash
May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: ALERT: <core> [main.c:775]: handle_sigs(): child process 30821 exited by a signal 11May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: ALERT: <core> [main.c:778]: handle_sigs(): core was not generatedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30818]: INFO: <core> [main.c:790]: handle_sigs(): INFO: terminating due to SIGCHLDMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 receivedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: (0x7f6af5968010):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: heap size= 749731840May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: used= 1110960, used+overhead=1663384, free=748068456May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: max used (+overhead)= 1825240May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: dumping all alloc'ed. fragments:May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958 size=256 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 receivedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 2. N address=0x7f6af599dae8 frag=0x7f6af599dab8 size=176 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: counters.c: cnt_hash_add(339)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 3. N address=0x7f6af599dbf8 frag=0x7f6af599dbc8 size=240 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: counters.c: cnt_hash_add(339)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 4. N address=0x7f6af599dd48 frag=0x7f6af599dd18 size=128 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: counters.c: init_counters(129)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 5. N address=0x7f6af599de28 frag=0x7f6af599ddf8 size=16 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: modparam.c: set_mod_param_regex(124)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 6. N address=0x7f6af599de98 frag=0x7f6af599de68 size=128 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 7. N address=0x7f6af599df78 frag=0x7f6af599df48 size=48 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 8. N address=0x7f6af599e008 frag=0x7f6af599dfd8 size=16 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(181)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 9. N address=0x7f6af599e078 frag=0x7f6af599e048 size=128 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 10. N address=0x7f6af599e158 frag=0x7f6af599e128 size=48 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 11. N address=0x7f6af599e1e8 frag=0x7f6af599e1b8 size=16 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(181)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 12. N address=0x7f6af599e258 frag=0x7f6af599e228 size=128 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 13. N address=0x7f6af599e338 frag=0x7f6af599e308 size=48 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 14. N address=0x7f6af599e3c8 frag=0x7f6af599e398 size=16 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(181)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 15. N address=0x7f6af599e438 frag=0x7f6af599e408 size=128 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 receivedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: (0x7f6af5968010):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: heap size= 749731840May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: used= 1110768, used+overhead=1663192, free=748068648May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: max used (+overhead)= 1825240May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: dumping all alloc'ed. fragments:May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958 size=256 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30833]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30827]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 receivedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 receivedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: (0x7f6af5968010):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: heap size= 749731840May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: used= 1110768, used+overhead=1663192, free=748068648May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: max used (+overhead)= 1825240May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: dumping all alloc'ed. fragments:May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30828]: NOTICE: qm_status: (0x7f6af5968010):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 receivedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: (0x7f6af5968010):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: heap size= 749731840May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: used= 1110768, used+overhead=1663192, free=748068648May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: max used (+overhead)= 1825240May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: dumping all alloc'ed. fragments:May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: 0. N address=0x7f6af599d528 frag=0x7f6af599d4f8 size=1024 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958 size=256 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 receivedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: 2. N address=0x7f6af599dae8 frag=0x7f6af599dab8 size=176 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30830]: NOTICE: qm_status: alloc'd from <core>: counters.c: cnt_hash_add(339)May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30832]: NOTICE: qm_status: 1. N address=0x7f6af599d988 frag=0x7f6af599d958 size=256 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30823]: NOTICE: qm_status: 16. N address=0x7f6af599e518 frag=0x7f6af599e4e8 size=48 used=1May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30834]: NOTICE: qm_status: (0x7f6af5968010):May 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30822]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 receivedMay 2 10:50:41 vnpsip01 /usr/local/sbin/kamailio[30822]: NOTICE: <core> [main.c:852]: sig_usr(): Memory status (pkg):Could you help me to fix this problem!
--
---------------------------
Bui Thi Minh Yen
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com
---------------------------
Bui Thi Minh Yen
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com
---------------------------
Bui Thi Minh Yen
SSDC, VNPT Technology
124 Hoang Quoc Viet | Mobile: (+84) 973110551 | Skype: minhyenbt