[SR-Users] "new_t: out_of_mem" errors, yet core.shmmem reports >60% shmmem free

[elactrum at jamailca.com]

On 2013-11-29 14:37, Will Ferrer wrote:
> Hi Andrew
> 
> A very simple place to start -- have you tried upping the memory
> allocations in /etc/default/kamailio to see if it fixes the issues? We
> had some issues with shared memory that went away when we changed
> around these settings.
> 
> All the best.
> 
> Will Ferrer
> 

Will,

Thanks for the suggestion. What sort of issues did you have with shared 
memory?

On my other servers, I am already increasing the shared memory 
allocation from 64M to 512M in case that might reduce the risk of this. 
However, is it normal to have out-of-memory errors when core.shmmem does 
not report the memory being full? If not, what could cause this?


-Andrew


> On Fri, Nov 29, 2013 at 11:30 AM, <elactrum at jamailca.com> wrote:
> 
>> Hi,
>> 
>> I have a Kamailio 3.3.3 server that currently reports the following 
>> errors upon execution of t_relay() for certain initial INVITEs.
>> 
>> daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> 
>> [sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory
>> daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: 
>> ERROR: new_t: out of mem:
>> daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: 
>> ERROR: t_newtran: new_t failed
>> 
>> I understand this error to be related to shared memory, based on [1]. 
>> However, sercmd reports normal shared memory utilization:
>> 
>> # sercmd core.shmmem
>> {
>>         total: 67108864
>>         free: 43189728
>>         used: 20505784
>>         real_used: 23919136
>>         max_used: 65948568
>>         fragments: 36887
>> }
>> 
>> These figures are quite similar to another server with the same 
>> configuration and Kamailio version, which is not reporting 
>> out-of-memory errors.
>> 
>> # sercmd core.shmmem
>> {
>>         total: 67108864
>>         free: 40219536
>>         used: 23241952
>>         real_used: 26889328
>>         max_used: 66306288
>>         fragments: 37832
>> }
>> 
>> I recently saw similar symptoms on a Kamailio 4.0.4 server, but 
>> Kamailio unfortunately crashed before I could get further details. In 
>> this case, I can consistently duplicate the problem with certain 
>> INVITEs. It seems that the failure is somehow related to the size of 
>> the SIP headers. For example, I have an INVITE from a Polycom phone 
>> which t_relay() routes successfully with 741 bytes of headers (total 
>> SIP length 1039 bytes). However, it seems that if I add one character 
>> to any of its SIP headers (742 bytes of headers; total SIP length 1039 
>> bytes) the t_relay() will consistently fail.
>> 
>> This binary doesn't have DBG_QM_MALLOC configured (I am working on 
>> getting it re-compiled). However, since I don't know what caused the 
>> problem, I would still like to get as much information as I can out of 
>> the running system. I have attached the output of mi_statistics, as 
>> well as mem_dump_pkg for the external UDP receivers and the ctl 
>> handler. Are there any other diagnostics that could be useful?
>> 
>> -Andrew
>> 
>> [1] 
>> http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_errorsip_msg_clonercannot_allocate_memory_and_errornew_tout_of_memmean_and_how_do_i_fix_them 
>> [1]
>> 
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing 
>> list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users [2]
> 
> 
> 
> Links:
> ------
> [1]
> http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_errorsip_msg_clonercannot_allocate_memory_and_errornew_tout_of_memmean_and_how_do_i_fix_them
> [2] http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
> 
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing 
> list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users