Hi, I've experimented PKG out of memory in a production server. Of course I'm already increasing such value but I would like to know if the current settings and traffic could run into PKG memory issues:
- Dual core INTEL XEON 3.00 GHz server. - Kamailio PKG memory = 4 MB. - Kamailio Shared memory = 64 MB. - Kamailio just listens in a single UDP port (8 childrens). - Just INVITE method is handled (no registration, no subscription).
The script does the following for each request:
- 'permissions' module to match source IP (just ~20 entries in 'address' table). - 2 custom SQL queries (returning a simple value). - 'dialog' memory (just in memory). - 'uac' module to change From header. - There are 10 AVP's set per transaction. - 'lcr' module for routing to two gateways (just 2 entries in 'lcr' table). - 'rtpproxy' is forced for every call.
The server has been working properly for months but these days the traffic has been duplicated, having ~400 simultaneous calls in peak hours. Also note that such calls come from callcenters so they are ver "fast".
With this environment, is it normal to get into PKG memory issues (4 MB)? I understand that it makes sense, but I would like to hear some opinions. Thanks a lot.
Hello,
On 4/7/10 10:37 PM, Iñaki Baz Castillo wrote:
Hi, I've experimented PKG out of memory in a production server. Of course I'm already increasing such value but I would like to know if the current settings and traffic could run into PKG memory issues:
- Dual core INTEL XEON 3.00 GHz server.
- Kamailio PKG memory = 4 MB.
- Kamailio Shared memory = 64 MB.
- Kamailio just listens in a single UDP port (8 childrens).
- Just INVITE method is handled (no registration, no subscription).
The script does the following for each request:
- 'permissions' module to match source IP (just ~20 entries in 'address' table).
- 2 custom SQL queries (returning a simple value).
- 'dialog' memory (just in memory).
- 'uac' module to change From header.
- There are 10 AVP's set per transaction.
- 'lcr' module for routing to two gateways (just 2 entries in 'lcr' table).
- 'rtpproxy' is forced for every call.
The server has been working properly for months but these days the traffic has been duplicated, having ~400 simultaneous calls in peak hours. Also note that such calls come from callcenters so they are ver "fast".
With this environment, is it normal to get into PKG memory issues (4 MB)? I understand that it makes sense, but I would like to hear some opinions. Thanks a lot.
does not look like a very pkg intensive processing. What version are you running? Do you get other frequent error messages (e.g., bad sip message)?
Cheers, Daniel
2010/4/7 Daniel-Constantin Mierla miconda@gmail.com:
does not look like a very pkg intensive processing. What version are you running? Do you get other frequent error messages (e.g., bad sip message)?
No, no errors at all, I check the logs very often (however I set it to INFO level, but wrong messages should be displayed at this level).
2010/4/7 Iñaki Baz Castillo ibc@aliax.net:
2010/4/7 Daniel-Constantin Mierla miconda@gmail.com:
does not look like a very pkg intensive processing. What version are you running? Do you get other frequent error messages (e.g., bad sip message)?
No, no errors at all, I check the logs very often (however I set it to INFO level, but wrong messages should be displayed at this level).
Sorry, kamailio 1.5 rev 5990
On 4/7/10 10:47 PM, Iñaki Baz Castillo wrote:
2010/4/7 Daniel-Constantin Mierlamiconda@gmail.com:
does not look like a very pkg intensive processing. What version are you running? Do you get other frequent error messages (e.g., bad sip message)?
No, no errors at all, I check the logs very often (however I set it to INFO level, but wrong messages should be displayed at this level).
yes, info level displays everything but 'debug'.
You haven't mentioned the version yet.
Cheers, Daniel
2010/4/7 Daniel-Constantin Mierla miconda@gmail.com:
You haven't mentioned the version yet.
Sorry, my fault, I'm forking myself now with different tasks :)
It's Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased PKG memory to 16.
Thanks.
On 4/7/10 10:56 PM, Iñaki Baz Castillo wrote:
2010/4/7 Daniel-Constantin Mierlamiconda@gmail.com:
You haven't mentioned the version yet.
Sorry, my fault, I'm forking myself now with different tasks :)
It's Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased PKG memory to 16.
if I spot it right on the svn commit log, there was a fix for a leak related to dst_uri when changed from failure route, nothing else important.
Would be good if you can compile with memory debugging, let it run for a while, then send a sigusr2 to a sip worker process to dump the pkg status -- that can reveal if it is a leak somewhere.
Cheers, Daniel
2010/4/7 Daniel-Constantin Mierla miconda@gmail.com:
It's Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased PKG memory to 16.
if I spot it right on the svn commit log, there was a fix for a leak related to dst_uri when changed from failure route, nothing else important.
Would it affect when use LCR module 'next_gw()' funtion in failure_route? It creates a new branch internally
Would be good if you can compile with memory debugging, let it run for a while, then send a sigusr2 to a sip worker process to dump the pkg status -- that can reveal if it is a leak somewhere.
Thanks. I cannot do it in the production server as it's very critical now :) But I'll do it in a mirror server generating traffic with SIPp.
Just a question: I usually compile kamailio with "make deb", is it compiled with memory debugging? if not, would it be enabled by editing Makefile.vars (MEMDBG=1) and running "make deb"?
Thanks a lot.
2010/4/7 Iñaki Baz Castillo ibc@aliax.net:
Just a question: I usually compile kamailio with "make deb", is it compiled with memory debugging?
Autoreply: Yes, DEB packages are compiled with memory debugging built in. I've checked it by setting "memlog=1" and sending a SIGUSR1 to any worker so I get the process PKG status.
NOTE: It's SIGUSR1 rather than SIGUSR2 :)
On 4/8/10 12:07 AM, Iñaki Baz Castillo wrote:
2010/4/7 Iñaki Baz Castilloibc@aliax.net:
Just a question: I usually compile kamailio with "make deb", is it compiled with memory debugging?
Autoreply: Yes, DEB packages are compiled with memory debugging built in. I've checked it by setting "memlog=1" and sending a SIGUSR1 to any worker so I get the process PKG status.
NOTE: It's SIGUSR1 rather than SIGUSR2 :)
ok, I mixed them, it was too late to check the source code.
Cheers, Daniel
Iñaki Baz Castillo writes:
Would it affect when use LCR module 'next_gw()' funtion in failure_route? It creates a new branch internally
inaki,
next_gw() sets dst uri if you have given a hostname for the gateway in hostname column of gw table. otherwise, it is not setting dst uri.
-- juha
2010/4/8 Juha Heinanen jh@tutpro.com:
Iñaki Baz Castillo writes:
> Would it affect when use LCR module 'next_gw()' funtion in > failure_route? It creates a new branch internally
inaki,
next_gw() sets dst uri if you have given a hostname for the gateway in hostname column of gw table. otherwise, it is not setting dst uri.
'hostname' column is NULL in my case so LCR is setting the RURI. Thanks.
On 4/7/10 11:37 PM, Iñaki Baz Castillo wrote:
2010/4/7 Daniel-Constantin Mierlamiconda@gmail.com:
It's Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased PKG memory to 16.
if I spot it right on the svn commit log, there was a fix for a leak related to dst_uri when changed from failure route, nothing else important.
Would it affect when use LCR module 'next_gw()' funtion in failure_route? It creates a new branch internally
Would be good if you can compile with memory debugging, let it run for a while, then send a sigusr2 to a sip worker process to dump the pkg status -- that can reveal if it is a leak somewhere.
Thanks. I cannot do it in the production server as it's very critical now :) But I'll do it in a mirror server generating traffic with SIPp.
Just a question: I usually compile kamailio with "make deb", is it compiled with memory debugging? if not, would it be enabled by editing Makefile.vars (MEMDBG=1) and running "make deb"?
yes, setting MEMDBG=1 will compile with memory debugging.
Cheers, Daniel
2010/4/7 Daniel-Constantin Mierla miconda@gmail.com:
It's Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased PKG memory to 16.
if I spot it right on the svn commit log, there was a fix for a leak related to dst_uri when changed from failure route, nothing else important.
Hi, after upgrading to kamailio 1.5.4 (before I used 1.5.1) I haven't experimented the memory leak anymore (even if the traffic is being increased each day). PKG MEM remains constant for all the processes (I check it every 5 minutes):
kamailio[11770]: used= 190936, used+overhead=250712, free=16526504 kamailio[11770]: max used (+overhead)= 258520 kamailio[11758]: used= 191176, used+overhead=250648, free=16526568 kamailio[11758]: max used (+overhead)= 257864
Yes, setting 16 MB per process is a terrible ammount of memory, I just set it in order to hold up a possible memory leak.
Note however that I don't use "hostname" column in LCR gw table, so next_gw() doesn't set dst_uri in failure_route (the fixed memory leak). Then it's unclear the reason of the PKG memory congestion I issued past week using 1.5.1. I'll continue checking it.
Regards.
2010/4/7 Iñaki Baz Castillo ibc@aliax.net:
With this environment, is it normal to get into PKG memory issues (4 MB)? I understand that it makes sense, but I would like to hear some opinions. Thanks a lot.
The PMG memory error looks like this:
kamailio[8147]: ERROR:core:build_res_buf_from_sip_res: out of pkg mem kamailio[8147]: ERROR:tm:relay_reply: no mem for outbound reply buffer kamailio[8147]: ERROR:core:parse_headers: pkg memory allocation failed kamailio[8147]: ERROR:tm:build_local: failed to reparse the request buffer kamailio[8147]: ERROR:tm:cancel_branch: attempt to build a CANCEL failed
-
Daniel-Constantin Mierla -
Iñaki Baz Castillo -
jh＠tutpro.com