[SR-Users] "new_t: out_of_mem" errors, yet core.shmmem reports >60% shmmem free

[Daniel-Constantin Mierla]

Hello,

do you have mem_join set in the config file?

I saw there is quite a lot of fragments, which means that there could 
demand for large size of memory, but available packets have lower size. 
mem_join should help in this case.

Cheers,
Daniel

On 12/3/13 5:10 PM, elactrum at jamailca.com wrote:
> On 2013-11-29 14:37, Will Ferrer wrote:
>> Hi Andrew
>>
>> A very simple place to start -- have you tried upping the memory
>> allocations in /etc/default/kamailio to see if it fixes the issues? We
>> had some issues with shared memory that went away when we changed
>> around these settings.
>>
>> All the best.
>>
>> Will Ferrer
>>
>
> Will,
>
> Thanks for the suggestion. What sort of issues did you have with 
> shared memory?
>
> On my other servers, I am already increasing the shared memory 
> allocation from 64M to 512M in case that might reduce the risk of 
> this. However, is it normal to have out-of-memory errors when 
> core.shmmem does not report the memory being full? If not, what could 
> cause this?
>
>
> -Andrew
>
>
>> On Fri, Nov 29, 2013 at 11:30 AM, <elactrum at jamailca.com> wrote:
>>
>>> Hi,
>>>
>>> I have a Kamailio 3.3.3 server that currently reports the following 
>>> errors upon execution of t_relay() for certain initial INVITEs.
>>>
>>> daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> 
>>> [sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory
>>> daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: 
>>> ERROR: new_t: out of mem:
>>> daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: 
>>> ERROR: t_newtran: new_t failed
>>>
>>> I understand this error to be related to shared memory, based on 
>>> [1]. However, sercmd reports normal shared memory utilization:
>>>
>>> # sercmd core.shmmem
>>> {
>>>         total: 67108864
>>>         free: 43189728
>>>         used: 20505784
>>>         real_used: 23919136
>>>         max_used: 65948568
>>>         fragments: 36887
>>> }
>>>
>>> These figures are quite similar to another server with the same 
>>> configuration and Kamailio version, which is not reporting 
>>> out-of-memory errors.
>>>
>>> # sercmd core.shmmem
>>> {
>>>         total: 67108864
>>>         free: 40219536
>>>         used: 23241952
>>>         real_used: 26889328
>>>         max_used: 66306288
>>>         fragments: 37832
>>> }
>>>
>>> I recently saw similar symptoms on a Kamailio 4.0.4 server, but 
>>> Kamailio unfortunately crashed before I could get further details. 
>>> In this case, I can consistently duplicate the problem with certain 
>>> INVITEs. It seems that the failure is somehow related to the size of 
>>> the SIP headers. For example, I have an INVITE from a Polycom phone 
>>> which t_relay() routes successfully with 741 bytes of headers (total 
>>> SIP length 1039 bytes). However, it seems that if I add one 
>>> character to any of its SIP headers (742 bytes of headers; total SIP 
>>> length 1039 bytes) the t_relay() will consistently fail.
>>>
>>> This binary doesn't have DBG_QM_MALLOC configured (I am working on 
>>> getting it re-compiled). However, since I don't know what caused the 
>>> problem, I would still like to get as much information as I can out 
>>> of the running system. I have attached the output of mi_statistics, 
>>> as well as mem_dump_pkg for the external UDP receivers and the ctl 
>>> handler. Are there any other diagnostics that could be useful?
>>>
>>> -Andrew
>>>
>>> [1] 
>>> http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_errorsip_msg_clonercannot_allocate_memory_and_errornew_tout_of_memmean_and_how_do_i_fix_them 
>>> [1]
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users at lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users [2]
>>
>>
>>
>> Links:
>> ------
>> [1]
>> http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_errorsip_msg_clonercannot_allocate_memory_and_errornew_tout_of_memmean_and_how_do_i_fix_them 
>>
>> [2] http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda