Hi,
I have a Kamailio 3.3.3 server that currently reports the following errors upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> [sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
I understand this error to be related to shared memory, based on [1]. However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem { total: 67108864 free: 43189728 used: 20505784 real_used: 23919136 max_used: 65948568 fragments: 36887 }
These figures are quite similar to another server with the same configuration and Kamailio version, which is not reporting out-of-memory errors.
# sercmd core.shmmem { total: 67108864 free: 40219536 used: 23241952 real_used: 26889328 max_used: 66306288 fragments: 37832 }
I recently saw similar symptoms on a Kamailio 4.0.4 server, but Kamailio unfortunately crashed before I could get further details. In this case, I can consistently duplicate the problem with certain INVITEs. It seems that the failure is somehow related to the size of the SIP headers. For example, I have an INVITE from a Polycom phone which t_relay() routes successfully with 741 bytes of headers (total SIP length 1039 bytes). However, it seems that if I add one character to any of its SIP headers (742 bytes of headers; total SIP length 1039 bytes) the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on getting it re-compiled). However, since I don't know what caused the problem, I would still like to get as much information as I can out of the running system. I have attached the output of mi_statistics, as well as mem_dump_pkg for the external UDP receivers and the ctl handler. Are there any other diagnostics that could be useful?
-Andrew
[1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_err...