7 Apr
2010
7 Apr
'10
4:37 p.m.
Hi, I've experimented PKG out of memory in a production server. Of
course I'm already increasing such value but I would like to know if
the current settings and traffic could run into PKG memory issues:
- Dual core INTEL XEON 3.00 GHz server.
- Kamailio PKG memory = 4 MB.
- Kamailio Shared memory = 64 MB.
- Kamailio just listens in a single UDP port (8 childrens).
- Just INVITE method is handled (no registration, no subscription).
The script does the following for each request:
- 'permissions' module to match source IP (just ~20 entries in 'address'
table).
- 2 custom SQL queries (returning a simple value).
- 'dialog' memory (just in memory).
- 'uac' module to change From header.
- There are 10 AVP's set per transaction.
- 'lcr' module for routing to two gateways (just 2 entries in 'lcr'
table).
- 'rtpproxy' is forced for every call.
The server has been working properly for months but these days the
traffic has been duplicated, having ~400 simultaneous calls in peak
hours. Also note that such calls come from callcenters so they are ver
"fast".
With this environment, is it normal to get into PKG memory issues (4
MB)? I understand that it makes sense, but I would like to hear some
opinions. Thanks a lot.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
7 Apr
7 Apr
4:44 p.m.
Hello,
On 4/7/10 10:37 PM, Iñaki Baz Castillo wrote:
Hi, I've experimented PKG out of memory in a
production server. Of
course I'm already increasing such value but I would like to know if
the current settings and traffic could run into PKG memory issues:
- Dual core INTEL XEON 3.00 GHz server.
- Kamailio PKG memory = 4 MB.
- Kamailio Shared memory = 64 MB.
- Kamailio just listens in a single UDP port (8 childrens).
- Just INVITE method is handled (no registration, no subscription).
The script does the following for each request:
- 'permissions' module to match source IP (just ~20 entries in 'address'
table).
- 2 custom SQL queries (returning a simple value).
- 'dialog' memory (just in memory).
- 'uac' module to change From header.
- There are 10 AVP's set per transaction.
- 'lcr' module for routing to two gateways (just 2 entries in 'lcr'
table).
- 'rtpproxy' is forced for every call.
The server has been working properly for months but these days the
traffic has been duplicated, having ~400 simultaneous calls in peak
hours. Also note that such calls come from callcenters so they are ver
"fast".
With this environment, is it normal to get into PKG memory issues (4
MB)? I understand that it makes sense, but I would like to hear some
opinions. Thanks a lot.
does not look like a very pkg intensive processing. What version are you
running? Do you get other frequent error messages (e.g., bad sip message)?
Cheers,
Daniel
--
Daniel-Constantin Mierla * http://www.asipto.com/ *
http://twitter.com/miconda *
http://www.linkedin.com/in/danielconstantinmierla
4:47 p.m.
2010/4/7 Daniel-Constantin Mierla <miconda(a)gmail.com>om>:
does not look like a very pkg intensive processing.
What version are you
running? Do you get other frequent error messages (e.g., bad sip message)?
No, no errors at all, I check the logs very often (however I set it to
INFO level, but wrong messages should be displayed at this level).
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
4:49 p.m.
2010/4/7 Iñaki Baz Castillo <ibc(a)aliax.net>et>:
2010/4/7 Daniel-Constantin Mierla
<miconda(a)gmail.com>om>:
Sorry, kamailio 1.5 rev 5990
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
does not look like a very pkg intensive
processing. What version are you
running? Do you get other frequent error messages (e.g., bad sip message)?
No, no errors at all, I check the logs very often (however I set it to
INFO level, but wrong messages should be displayed at this level).
4:51 p.m.
On 4/7/10 10:47 PM, Iñaki Baz Castillo wrote:
2010/4/7 Daniel-Constantin
Mierla<miconda(a)gmail.com>om>:
yes, info level displays everything but 'debug'.
You haven't mentioned the version yet.
Cheers,
Daniel
--
Daniel-Constantin Mierla * http://www.asipto.com/ *
http://twitter.com/miconda *
http://www.linkedin.com/in/danielconstantinmierla
does not look like a very pkg intensive
processing. What version are you
running? Do you get other frequent error messages (e.g., bad sip message)?
No, no errors at all, I check the logs very often (however I set it to
INFO level, but wrong messages should be displayed at this level).
4:56 p.m.
2010/4/7 Daniel-Constantin Mierla <miconda(a)gmail.com>om>:
You haven't mentioned the version yet.
Sorry, my fault, I'm forking myself now with different tasks :)
It's Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased
PKG memory to 16.
Thanks.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
5:16 p.m.
On 4/7/10 10:56 PM, Iñaki Baz Castillo wrote:
2010/4/7 Daniel-Constantin
Mierla<miconda(a)gmail.com>om>:
if I spot it right on the svn commit log, there was a fix for a leak
related to dst_uri when changed from failure route, nothing else important.
Would be good if you can compile with memory debugging, let it run for a
while, then send a sigusr2 to a sip worker process to dump the pkg
status -- that can reveal if it is a leak somewhere.
Cheers,
Daniel
--
Daniel-Constantin Mierla * http://www.asipto.com/ *
http://twitter.com/miconda *
http://www.linkedin.com/in/danielconstantinmierla
You haven't mentioned the version yet.
Sorry, my fault, I'm forking myself now with different tasks :)
It's Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased
PKG memory to 16.
5:37 p.m.
2010/4/7 Daniel-Constantin Mierla <miconda(a)gmail.com>om>:
Would it affect when use LCR module 'next_gw()' funtion in
failure_route? It creates a new branch internally
It's
Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased
PKG memory to 16.
if I spot it right on the svn commit log, there was a fix for a leak related
to dst_uri when changed from failure route, nothing else important. Would be good if you can compile with memory
debugging, let it run for a
while, then send a sigusr2 to a sip worker process to dump the pkg status --
that can reveal if it is a leak somewhere.
Thanks. I cannot do it in the production server as it's very critical now :)
But I'll do it in a mirror server generating traffic with SIPp.
Just a question: I usually compile kamailio with "make deb", is it
compiled with memory debugging? if not, would it be enabled by editing
Makefile.vars (MEMDBG=1) and running "make deb"?
Thanks a lot.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
6:07 p.m.
2010/4/7 Iñaki Baz Castillo <ibc(a)aliax.net>et>:
Just a question: I usually compile kamailio with
"make deb", is it
compiled with memory debugging?
Autoreply: Yes, DEB packages are compiled with memory debugging built
in. I've checked it by setting "memlog=1" and sending a SIGUSR1 to any
worker so I get the process PKG status.
NOTE: It's SIGUSR1 rather than SIGUSR2 :)
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
8 Apr
8 Apr
3:41 a.m.
On 4/8/10 12:07 AM, Iñaki Baz Castillo wrote:
2010/4/7 Iñaki Baz Castillo<ibc(a)aliax.net>et>:
ok, I mixed them, it was too late to check the source code.
Cheers,
Daniel
--
Daniel-Constantin Mierla * http://www.asipto.com/ *
http://twitter.com/miconda *
http://www.linkedin.com/in/danielconstantinmierla
Just a question: I usually compile kamailio with
"make deb", is it
compiled with memory debugging?
Autoreply: Yes, DEB packages are compiled with memory debugging built
in. I've checked it by setting "memlog=1" and sending a SIGUSR1 to any
worker so I get the process PKG status.
NOTE: It's SIGUSR1 rather than SIGUSR2 :)
1 a.m.
Iñaki Baz Castillo writes:
Would it affect when use LCR module
'next_gw()' funtion in
failure_route? It creates a new branch internally
inaki,
next_gw() sets dst uri if you have given a hostname for the gateway in
hostname column of gw table. otherwise, it is not setting dst uri.
-- juha
4:33 a.m.
2010/4/8 Juha Heinanen <jh(a)tutpro.com>om>:
Iñaki Baz Castillo writes:
'hostname' column is NULL in my case so LCR is setting the RURI.
Thanks.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
Would it affect when use LCR module
'next_gw()' funtion in
failure_route? It creates a new branch internally
inaki,
next_gw() sets dst uri if you have given a hostname for the gateway in
hostname column of gw table. otherwise, it is not setting dst uri.
3:40 a.m.
On 4/7/10 11:37 PM, Iñaki Baz Castillo wrote:
2010/4/7 Daniel-Constantin
Mierla<miconda(a)gmail.com>om>:
Would it affect when use LCR module 'next_gw()' funtion in
failure_route? It creates a new branch internally
yes, setting MEMDBG=1 will compile with memory debugging.
Cheers,
Daniel
--
Daniel-Constantin Mierla * http://www.asipto.com/ *
http://twitter.com/miconda *
http://www.linkedin.com/in/danielconstantinmierla
It's
Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased
PKG memory to 16.
if I spot it right on the svn commit log, there was a fix for a leak
related
to dst_uri when changed from failure route, nothing else important.
Would be good if you can compile with memory
debugging, let it run for a
while, then send a sigusr2 to a sip worker process to dump the pkg status --
that can reveal if it is a leak somewhere.
Thanks. I cannot do it in the production server as it's very critical
now :)
But I'll do it in a mirror server generating traffic with SIPp.
Just a question: I usually compile kamailio with "make deb", is it
compiled with memory debugging? if not, would it be enabled by editing
Makefile.vars (MEMDBG=1) and running "make deb"?
15 Apr
15 Apr
11:56 a.m.
2010/4/7 Daniel-Constantin Mierla <miconda(a)gmail.com>om>:
Hi, after upgrading to kamailio 1.5.4 (before I used 1.5.1) I haven't
experimented the memory leak anymore (even if the traffic is being
increased each day). PKG MEM remains constant for all the processes (I
check it every 5 minutes):
kamailio[11770]: used= 190936, used+overhead=250712, free=16526504
kamailio[11770]: max used (+overhead)= 258520
kamailio[11758]: used= 191176, used+overhead=250648, free=16526568
kamailio[11758]: max used (+overhead)= 257864
Yes, setting 16 MB per process is a terrible ammount of memory, I just
set it in order to hold up a possible memory leak.
Note however that I don't use "hostname" column in LCR gw table, so
next_gw() doesn't set dst_uri in failure_route (the fixed memory
leak). Then it's unclear the reason of the PKG memory congestion I
issued past week using 1.5.1. I'll continue checking it.
Regards.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
It's
Kamailio 1.5.1-notls. Now I've compiled 1.5.4-notls and increased
PKG memory to 16.
if I spot it right on the svn commit log, there was a fix for a leak related
to dst_uri when changed from failure route, nothing else important.
7 Apr
7 Apr
4:44 p.m.
2010/4/7 Iñaki Baz Castillo <ibc(a)aliax.net>et>:
With this environment, is it normal to get into PKG
memory issues (4
MB)? I understand that it makes sense, but I would like to hear some
opinions. Thanks a lot.
The PMG memory error looks like this:
kamailio[8147]: ERROR:core:build_res_buf_from_sip_res: out of pkg mem
kamailio[8147]: ERROR:tm:relay_reply: no mem for outbound reply buffer
kamailio[8147]: ERROR:core:parse_headers: pkg memory allocation failed
kamailio[8147]: ERROR:tm:build_local: failed to reparse the request buffer
kamailio[8147]: ERROR:tm:cancel_branch: attempt to build a CANCEL failed
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
5373
days inactive
5381
days old
14 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Iñaki Baz Castillo -
jh＠tutpro.com