[SR-Users] "new_t: out_of_mem" errors, yet core.shmmem reports >60% shmmem free

[elactrum at jamailca.com]

Hi,

I have a Kamailio 3.3.3 server that currently reports the following 
errors upon execution of t_relay() for certain initial INVITEs.

daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> 
[sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory
daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: 
ERROR: new_t: out of mem:
daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: 
ERROR: t_newtran: new_t failed

I understand this error to be related to shared memory, based on [1]. 
However, sercmd reports normal shared memory utilization:

# sercmd core.shmmem
{
	total: 67108864
	free: 43189728
	used: 20505784
	real_used: 23919136
	max_used: 65948568
	fragments: 36887
}

These figures are quite similar to another server with the same 
configuration and Kamailio version, which is not reporting out-of-memory 
errors.

# sercmd core.shmmem
{
	total: 67108864
	free: 40219536
	used: 23241952
	real_used: 26889328
	max_used: 66306288
	fragments: 37832
}


I recently saw similar symptoms on a Kamailio 4.0.4 server, but 
Kamailio unfortunately crashed before I could get further details. In 
this case, I can consistently duplicate the problem with certain 
INVITEs. It seems that the failure is somehow related to the size of the 
SIP headers. For example, I have an INVITE from a Polycom phone which 
t_relay() routes successfully with 741 bytes of headers (total SIP 
length 1039 bytes). However, it seems that if I add one character to any 
of its SIP headers (742 bytes of headers; total SIP length 1039 bytes) 
the t_relay() will consistently fail.

This binary doesn't have DBG_QM_MALLOC configured (I am working on 
getting it re-compiled). However, since I don't know what caused the 
problem, I would still like to get as much information as I can out of 
the running system. I have attached the output of mi_statistics, as well 
as mem_dump_pkg for the external UDP receivers and the ctl handler. Are 
there any other diagnostics that could be useful?


-Andrew


[1] 
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_errorsip_msg_clonercannot_allocate_memory_and_errornew_tout_of_memmean_and_how_do_i_fix_them
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mem_dump_pkg
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131129/c517bd6a/attachment-0002.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mi_statistics
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131129/c517bd6a/attachment-0003.ksh>