Hi,
I have a Kamailio 3.3.3 server that currently reports the following errors upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> [sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
I understand this error to be related to shared memory, based on [1]. However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem { total: 67108864 free: 43189728 used: 20505784 real_used: 23919136 max_used: 65948568 fragments: 36887 }
These figures are quite similar to another server with the same configuration and Kamailio version, which is not reporting out-of-memory errors.
# sercmd core.shmmem { total: 67108864 free: 40219536 used: 23241952 real_used: 26889328 max_used: 66306288 fragments: 37832 }
I recently saw similar symptoms on a Kamailio 4.0.4 server, but Kamailio unfortunately crashed before I could get further details. In this case, I can consistently duplicate the problem with certain INVITEs. It seems that the failure is somehow related to the size of the SIP headers. For example, I have an INVITE from a Polycom phone which t_relay() routes successfully with 741 bytes of headers (total SIP length 1039 bytes). However, it seems that if I add one character to any of its SIP headers (742 bytes of headers; total SIP length 1039 bytes) the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on getting it re-compiled). However, since I don't know what caused the problem, I would still like to get as much information as I can out of the running system. I have attached the output of mi_statistics, as well as mem_dump_pkg for the external UDP receivers and the ctl handler. Are there any other diagnostics that could be useful?
-Andrew
[1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_err...
Hi Andrew
A very simple place to start -- have you tried upping the memory allocations in /etc/default/kamailio to see if it fixes the issues? We had some issues with shared memory that went away when we changed around these settings.
All the best.
Will Ferrer
On Fri, Nov 29, 2013 at 11:30 AM, elactrum@jamailca.com wrote:
Hi,
I have a Kamailio 3.3.3 server that currently reports the following errors upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> [sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
I understand this error to be related to shared memory, based on [1]. However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem { total: 67108864 free: 43189728 used: 20505784 real_used: 23919136 max_used: 65948568 fragments: 36887 }
These figures are quite similar to another server with the same configuration and Kamailio version, which is not reporting out-of-memory errors.
# sercmd core.shmmem { total: 67108864 free: 40219536 used: 23241952 real_used: 26889328 max_used: 66306288 fragments: 37832 }
I recently saw similar symptoms on a Kamailio 4.0.4 server, but Kamailio unfortunately crashed before I could get further details. In this case, I can consistently duplicate the problem with certain INVITEs. It seems that the failure is somehow related to the size of the SIP headers. For example, I have an INVITE from a Polycom phone which t_relay() routes successfully with 741 bytes of headers (total SIP length 1039 bytes). However, it seems that if I add one character to any of its SIP headers (742 bytes of headers; total SIP length 1039 bytes) the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on getting it re-compiled). However, since I don't know what caused the problem, I would still like to get as much information as I can out of the running system. I have attached the output of mi_statistics, as well as mem_dump_pkg for the external UDP receivers and the ctl handler. Are there any other diagnostics that could be useful?
-Andrew
[1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_ does_errorsip_msg_clonercannot_allocate_memory_ and_errornew_tout_of_memmean_and_how_do_i_fix_them
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 2013-11-29 14:37, Will Ferrer wrote:
Hi Andrew
A very simple place to start -- have you tried upping the memory allocations in /etc/default/kamailio to see if it fixes the issues? We had some issues with shared memory that went away when we changed around these settings.
All the best.
Will Ferrer
Will,
Thanks for the suggestion. What sort of issues did you have with shared memory?
On my other servers, I am already increasing the shared memory allocation from 64M to 512M in case that might reduce the risk of this. However, is it normal to have out-of-memory errors when core.shmmem does not report the memory being full? If not, what could cause this?
-Andrew
On Fri, Nov 29, 2013 at 11:30 AM, elactrum@jamailca.com wrote:
Hi,
I have a Kamailio 3.3.3 server that currently reports the following errors upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> [sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
I understand this error to be related to shared memory, based on [1]. However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem { total: 67108864 free: 43189728 used: 20505784 real_used: 23919136 max_used: 65948568 fragments: 36887 }
These figures are quite similar to another server with the same configuration and Kamailio version, which is not reporting out-of-memory errors.
# sercmd core.shmmem { total: 67108864 free: 40219536 used: 23241952 real_used: 26889328 max_used: 66306288 fragments: 37832 }
I recently saw similar symptoms on a Kamailio 4.0.4 server, but Kamailio unfortunately crashed before I could get further details. In this case, I can consistently duplicate the problem with certain INVITEs. It seems that the failure is somehow related to the size of the SIP headers. For example, I have an INVITE from a Polycom phone which t_relay() routes successfully with 741 bytes of headers (total SIP length 1039 bytes). However, it seems that if I add one character to any of its SIP headers (742 bytes of headers; total SIP length 1039 bytes) the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on getting it re-compiled). However, since I don't know what caused the problem, I would still like to get as much information as I can out of the running system. I have attached the output of mi_statistics, as well as mem_dump_pkg for the external UDP receivers and the ctl handler. Are there any other diagnostics that could be useful?
-Andrew
[1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_err... [1]
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users [2]
Links:
[1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_err... [2] http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
do you have mem_join set in the config file?
I saw there is quite a lot of fragments, which means that there could demand for large size of memory, but available packets have lower size. mem_join should help in this case.
Cheers, Daniel
On 12/3/13 5:10 PM, elactrum@jamailca.com wrote:
On 2013-11-29 14:37, Will Ferrer wrote:
Hi Andrew
A very simple place to start -- have you tried upping the memory allocations in /etc/default/kamailio to see if it fixes the issues? We had some issues with shared memory that went away when we changed around these settings.
All the best.
Will Ferrer
Will,
Thanks for the suggestion. What sort of issues did you have with shared memory?
On my other servers, I am already increasing the shared memory allocation from 64M to 512M in case that might reduce the risk of this. However, is it normal to have out-of-memory errors when core.shmmem does not report the memory being full? If not, what could cause this?
-Andrew
On Fri, Nov 29, 2013 at 11:30 AM, elactrum@jamailca.com wrote:
Hi,
I have a Kamailio 3.3.3 server that currently reports the following errors upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> [sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: ERROR: new_t: out of mem: daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed
I understand this error to be related to shared memory, based on [1]. However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem { total: 67108864 free: 43189728 used: 20505784 real_used: 23919136 max_used: 65948568 fragments: 36887 }
These figures are quite similar to another server with the same configuration and Kamailio version, which is not reporting out-of-memory errors.
# sercmd core.shmmem { total: 67108864 free: 40219536 used: 23241952 real_used: 26889328 max_used: 66306288 fragments: 37832 }
I recently saw similar symptoms on a Kamailio 4.0.4 server, but Kamailio unfortunately crashed before I could get further details. In this case, I can consistently duplicate the problem with certain INVITEs. It seems that the failure is somehow related to the size of the SIP headers. For example, I have an INVITE from a Polycom phone which t_relay() routes successfully with 741 bytes of headers (total SIP length 1039 bytes). However, it seems that if I add one character to any of its SIP headers (742 bytes of headers; total SIP length 1039 bytes) the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on getting it re-compiled). However, since I don't know what caused the problem, I would still like to get as much information as I can out of the running system. I have attached the output of mi_statistics, as well as mem_dump_pkg for the external UDP receivers and the ctl handler. Are there any other diagnostics that could be useful?
-Andrew
[1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_err... [1]
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users [2]
Links:
[1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_err...
[2] http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Daniel-Constantin Mierla -
elactrum＠jamailca.com -
Will Ferrer