Hi everyone,

I have a problem with Kamailio consuming memory. I started testing with Kamailio 3.2.1 upgraded to 3.2.2 and also tested 3.1.5, everywhere the same problem. Whereas Kamailio 1.5.0 is working fine. Kamailio -V shows the following:

kamailio@test:~$ kamailio -V version: kamailio 3.2.2 (i386/linux) 98ba92 flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 98ba92 compiled on 17:51:30 Feb 29 2012 with gcc 4.2.4 kamailio@test:~$

My test scenario consists of two SIPp servers (a UAC and a UAS) and the Kamailio in the middle proxying the SIP traffic. The UAC sends 300 calls per second (INVITE, 100, 180, 200, ACK) and releases the calls (BYE, 200) after 20 sec call duration. UAS is simply sending back the right responds. Protocol is UDP.

After some minutes the Kamailio floods the syslog file with error messages indicating that he is running out of memory. The SIPp shows retransmissions and also timeouts. When I stop the test and do a single call I get some lines of error messages. I found three different sets of error messages for a single call so far:

Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [parser/msg_parser.c:172]: get_hdr_field: out of memory Mar 2 09:35:44 test P-CSCFtest[21251]: INFO:<core> [parser/msg_parser.c:353]: ERROR: bad header field [To:<sip:00498111111] Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR: tm [t_lookup.c:1118]: ERROR: reply cannot be parsed Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [msg_translator.c:1884]: ERROR: build_res_buf_from_sip_res: out of mem Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [forward.c:794]: ERROR: forward_reply: building failed Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR:<core> [msg_translator.c:1884]: ERROR: build_res_buf_from_sip_res: out of mem Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR: tm [t_reply.c:1786]: ERROR: relay_reply: no mem for outbound reply buffer Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR:<core> [msg_translator.c:2036]: ERROR: build_res_buf_from_sip_req: out of memory ; needs 311

Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR:<core> [msg_translator.c:1884]: ERROR: build_res_buf_from_sip_res: out of mem Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR: tm [t_reply.c:1786]: ERROR: relay_reply: no mem for outbound reply buffer Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR:<core> [msg_translator.c:2036]: ERROR: build_res_buf_from_sip_req: out of memory ; needs 311

Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:852]: ERROR: do_action: memory allocation failure Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:1568]: run action error at: /usr/local/etc/kamailio//kamailio.cfg:28 Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:852]: ERROR: do_action: memory allocation failure Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:1568]: run action error at: /usr/local/etc/kamailio//kamailio.cfg:32 Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [parser/parse_from.c:71]: ERROR:parse_from_header: out of pkg_memory Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR: tm [t_lookup.c:1326]: ERROR: new_t: no valid From in INVITE Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran: new_t failed Mar 2 10:20:16 test P-CSCFtest[21328]: WARNING:<core> [receive.c:209]: WARNING: receive_msg: error while trying script

I repeated the test various times, it always generates a memory problem. The "top" command shows that the free memory is decreasing constantly during the test. Stopping the test and waiting for some time (1 hour) doesn't increase the free memory showed by "top". Only a restart of the Kamailio frees the memory again. I executed the test with 30 calls per second too and got the same behaviour (after some more minutes).

I finally simplified the Kamailio config as far as possible and came to this minimum:

#!KAMAILIO listen=10.10.10.10:5060 children=30 user="kamailio" group="kamailio" debug=2 disable_core_dump=yes log_facility=LOG_LOCAL0 log_name="P-CSCFtest" server_signature=no sip_warning=0 syn_branch=0 disable_tcp=yes

####### Modules Section ####### mpath="/usr/local/lib/kamailio/modules_k/:/usr/local/lib/kamailio/modules/" loadmodule "tm.so" ####### Main Routing Logic ######## route{ if (src_ip==1.2.3.4) { rewritehost("5.6.7.8"); } else { rewritehost("1.2.3.4"); }; rewriteport("5061"); t_relay(); }

This config still generates the memory problem. After replacing the "t_relay()" with the stateless "forward()" the problem is gone. I played around with the number of child processes too, but no difference.

Is this a memory leak of the TM module or is there a different explanation?

Regards Fred