[SR-Users] "new_t: out_of_mem" errors, yet core.shmmem reports >60% shmmem free

[Will Ferrer]

Hi Andrew

A very simple place to start -- have you tried upping the memory
allocations in /etc/default/kamailio to see if it fixes the issues? We had
some issues with shared memory that went away when we changed around these
settings.

All the best.

Will Ferrer




On Fri, Nov 29, 2013 at 11:30 AM, <elactrum at jamailca.com> wrote:

> Hi,
>
> I have a Kamailio 3.3.3 server that currently reports the following errors
> upon execution of t_relay() for certain initial INVITEs.
>
> daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> [sip_msg_clone.c:505]:
> ERROR: sip_msg_cloner: cannot allocate memory
> daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: ERROR:
> new_t: out of mem:
> daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: ERROR:
> t_newtran: new_t failed
>
> I understand this error to be related to shared memory, based on [1].
> However, sercmd reports normal shared memory utilization:
>
> # sercmd core.shmmem
> {
>         total: 67108864
>         free: 43189728
>         used: 20505784
>         real_used: 23919136
>         max_used: 65948568
>         fragments: 36887
> }
>
> These figures are quite similar to another server with the same
> configuration and Kamailio version, which is not reporting out-of-memory
> errors.
>
> # sercmd core.shmmem
> {
>         total: 67108864
>         free: 40219536
>         used: 23241952
>         real_used: 26889328
>         max_used: 66306288
>         fragments: 37832
> }
>
>
> I recently saw similar symptoms on a Kamailio 4.0.4 server, but Kamailio
> unfortunately crashed before I could get further details. In this case, I
> can consistently duplicate the problem with certain INVITEs. It seems that
> the failure is somehow related to the size of the SIP headers. For example,
> I have an INVITE from a Polycom phone which t_relay() routes successfully
> with 741 bytes of headers (total SIP length 1039 bytes). However, it seems
> that if I add one character to any of its SIP headers (742 bytes of
> headers; total SIP length 1039 bytes) the t_relay() will consistently fail.
>
> This binary doesn't have DBG_QM_MALLOC configured (I am working on getting
> it re-compiled). However, since I don't know what caused the problem, I
> would still like to get as much information as I can out of the running
> system. I have attached the output of mi_statistics, as well as
> mem_dump_pkg for the external UDP receivers and the ctl handler. Are there
> any other diagnostics that could be useful?
>
>
> -Andrew
>
>
> [1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_
> does_errorsip_msg_clonercannot_allocate_memory_
> and_errornew_tout_of_memmean_and_how_do_i_fix_them
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131129/9e24fa68/attachment.html>