16 Jul
2010
16 Jul
'10
5:51 p.m.
Hi All,
I loaded up the PDT database with about 35K records and when I issue
the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR: <core> [tree.c:139]: no more pkg mem
3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which
suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/
#define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try:
#define PKG_MEM_POOL_SIZE 4*2048*2048
or
#define PKG_MEM_POOL_SIZE 8*1024*1024
Which one would be a logical adjustment? Also, is there a correlation
between pkg mem and database record size as related to pdt_list?
The idea is to have a few 100K records loaded in kamailio and be able
to perform "kamctl fifo pdt_list | grep 512345" to show this prefix
route. But without enough memory, doesn't work.
Thanks.
JR
--
JR Richardson
Engineering for the Masses
16 Jul
16 Jul
6:19 p.m.
On Fri, Jul 16, 2010 at 4:51 PM, JR Richardson <jmr.richardson(a)gmail.com> wrote:
Hi All,
I loaded up the PDT database with about 35K records and when I issue
the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR: <core> [tree.c:139]: no more pkg mem
3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which
suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/
#define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try:
#define PKG_MEM_POOL_SIZE 4*2048*2048
or
#define PKG_MEM_POOL_SIZE 8*1024*1024
I tried #define PKG_MEM_POOL_SIZE 8*1024*1024 and recompiled with good
results, was able to run pdt_list just fine. So what do I look for in
the memory statistics to show how much memory to configure when using
large database record sets? And if I need to go to 16* or 32*, would
that have any adverse affect on any other kamailio operations?
Which one would be a logical adjustment? Also, is there a correlation
between pkg mem and database record size as related to pdt_list?
The idea is to have a few 100K records loaded in kamailio and be able
to perform "kamctl fifo pdt_list | grep 512345" to show this prefix
route. But without enough memory, doesn't work.
Thanks.
JR
--
JR Richardson
Engineering for the Masses
Thanks.
JR
--
JR Richardson
Engineering for the Masses
19 Jul
19 Jul
3:41 a.m.
JR Richardson wrote:
On Fri, Jul 16, 2010 at 4:51 PM, JR Richardson
<jmr.richardson(a)gmail.com> wrote:
Hello JR,
Compile kamailio with memory debug support. Modify
Makefile.vars and set MEMDBG to 1 . In the config file(kamailio.cfg),
set memlog to a lower level than the general debug level. You can do a
dump of pkg (private) memory by
sending a SIGUSR1 to one of the worker process ('kamctl ps' to find the
pid).
Unless you are working with a huge number of calls, I can't see why you
need more than 8 MB or 10 maximum.
Carefull if you have a 64 bit system you might need just a little more.
Hi All,
I loaded up the PDT database with about 35K records and when I issue
the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR: <core> [tree.c:139]: no more pkg mem
3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which
suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/
#define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try:
#define PKG_MEM_POOL_SIZE 4*2048*2048
or
#define PKG_MEM_POOL_SIZE 8*1024*1024
I tried #define PKG_MEM_POOL_SIZE 8*1024*1024 and recompiled with good
results, was able to run pdt_list just fine. So what do I look for in
the memory statistics to show how much memory to configure when using
large database record sets? And if I need to go to 16* or 32*, would
that have any adverse affect on any other kamailio operations?
> Which one would be a logical adjustment? Also, is
there a correlation
> between pkg mem and database record size as related to pdt_list?
>
> The idea is to have a few 100K records loaded in kamailio and be able
> to perform "kamctl fifo pdt_list | grep 512345" to show this prefix
> route. But without enough memory, doesn't work.
>
>
AFAIK, these mi commands will construct a complete result tree, thus
meaning it should have enough memory for 100K records.
If it works with 8 MB it is ok . Keep in mind that the memory dump is
not that usefull to track these OOM condition, it is usefull to track
memory leaks, as after the OOM condition happens the already malloc'ed
memory is free'd. It would be usefull to test that is those OOM
conditions, the free is done corectly and no leaks are induced
Cheers
Marius
Thanks.
JR
--
JR Richardson
Engineering for the Masses
Thanks.
JR
4:12 p.m.
On Mon, Jul 19, 2010 at 2:41 AM, marius zbihlei <marius.zbihlei(a)1and1.ro> wrote:
JR Richardson wrote:
So I started seeing this issue with only 35K records in pdt database,
I increased
#define PKG_MEM_POOL_SIZE 8*1024*1024
Then I was able to load 35K records with good results. I increased
record count to 60K records and hit the limit again, pdt_list would
not work with same "no more pkg mem" error.
I increased again:
#define PKG_MEM_POOL_SIZE 16*1024*1024
This allowed me to execute pdt_list with 60K to 120K records loaded.
When I added 180K records in the database, I got the "no more pkg mem"
error again.
I increased again:
#define PKG_MEM_POOL_SIZE 32*1024*1024
This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg
mem" error again.
So I don't think it is prudent to just keep increasing
PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo
pkg_mem, shouldn't this be a dynamic allocation if it's not within
shmem and has no affect on core sip-router function? While the
pdt_reload and pdt_list is going on (takes a few seconds to load and
list), I don't see any problems with the sip-router executions. I
guess I can just use the old fashion database query to look up routes
instead of fifo pdt_list.
Thanks.
JR
--
JR Richardson
Engineering for the Masses
On Fri, Jul 16, 2010 at 4:51 PM, JR Richardson <jmr.richardson(a)gmail.com>
wrote:
Hello JR,
Compile kamailio with memory debug support. Modify
Makefile.vars and set MEMDBG to 1 . In the config file(kamailio.cfg),
set memlog to a lower level than the general debug level. You can do a dump
of pkg (private) memory by
sending a SIGUSR1 to one of the worker process ('kamctl ps' to find the
pid).
Unless you are working with a huge number of calls, I can't see why you need
more than 8 MB or 10 maximum.
Carefull if you have a 64 bit system you might need just a little more.
Hi All,
I loaded up the PDT database with about 35K records and when I issue
the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR: <core> [tree.c:139]: no more pkg mem
3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing
failed
Searching around I found the
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which
suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/
#define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try:
#define PKG_MEM_POOL_SIZE 4*2048*2048
or
#define PKG_MEM_POOL_SIZE 8*1024*1024
I tried #define PKG_MEM_POOL_SIZE 8*1024*1024 and recompiled with good
results, was able to run pdt_list just fine. So what do I look for in
the memory statistics to show how much memory to configure when using
large database record sets? And if I need to go to 16* or 32*, would
that have any adverse affect on any other kamailio operations?
> Which one would be a logical adjustment?
Also, is there a correlation
> between pkg mem and database record size as related to pdt_list?
>
> The idea is to have a few 100K records loaded in kamailio and be able
> to perform "kamctl fifo pdt_list | grep 512345" to show this prefix
> route. But without enough memory, doesn't work.
>
>
AFAIK, these mi commands will construct a complete result tree, thus meaning
it should have enough memory for 100K records.
If it works with 8 MB it is ok . Keep in mind that the memory dump is not
that usefull to track these OOM condition, it is usefull to track
memory leaks, as after the OOM condition happens the already malloc'ed
memory is free'd. It would be usefull to test that is those OOM conditions,
the free is done corectly and no leaks are induced
Cheers
Marius
6:16 p.m.
So I started seeing this issue with only 35K records
in pdt database,
I increased
#define PKG_MEM_POOL_SIZE 8*1024*1024
Then I was able to load 35K records with good results. I increased
record count to 60K records and hit the limit again, pdt_list would
not work with same "no more pkg mem" error.
I increased again:
#define PKG_MEM_POOL_SIZE 16*1024*1024
This allowed me to execute pdt_list with 60K to 120K records loaded.
When I added 180K records in the database, I got the "no more pkg mem"
error again.
I increased again:
#define PKG_MEM_POOL_SIZE 32*1024*1024
This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg
mem" error again.
So I don't think it is prudent to just keep increasing
PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo
pkg_mem, shouldn't this be a dynamic allocation if it's not within
shmem and has no affect on core sip-router function? While the
pdt_reload and pdt_list is going on (takes a few seconds to load and
list), I don't see any problems with the sip-router executions. I
guess I can just use the old fashion database query to look up routes
instead of fifo pdt_list.
Thanks.
JR
So now I'm running out of shmem as well. After I loaded more than 320K
records in the pdt database table, I started getting these errors:
0(17599) ERROR: pdt [pdtree.c:283]: bad parameters
0(17599) INFO: pdt [pdt.c:490]: no prefix found in [7000011234]
shmem:real_used_size = 1247512
This parameter does not get above 1247512 without having a problem,
even though I have plenty of shmem available.
shmem:total_size = 33554432
shmem:used_size = 1229472
shmem:real_used_size = 1247512
shmem:max_used_size = 1247512
shmem:free_size = 32306920
I increased these parameters in config.h but that did not help:
/*used only if PKG_MALLOC is defined*/
#define PKG_MEM_POOL_SIZE 64*1024*1024
/*used if SH_MEM is defined*/
#define SHM_MEM_SIZE 64
So now it looks like there is some hard limit on the route size you
can deploy for the pdt module? Is this the case or is there somewhere
I can increase the available shmem for pdt to use?
Thanks.
JR
--
JR Richardson
Engineering for the Masses
20 Jul
20 Jul
5:27 a.m.
On Tuesday 20 July 2010, JR Richardson wrote:
[..]
So now I'm running out of shmem as well. After I loaded more than 320K
records in the pdt database table, I started getting these errors:
0(17599) ERROR: pdt [pdtree.c:283]: bad parameters
0(17599) INFO: pdt [pdt.c:490]: no prefix found in [7000011234]
shmem:real_used_size = 1247512
This parameter does not get above 1247512 without having a problem,
even though I have plenty of shmem available.
shmem:total_size = 33554432
shmem:used_size = 1229472
shmem:real_used_size = 1247512
shmem:max_used_size = 1247512
shmem:free_size = 32306920
I increased these parameters in config.h but that did not help:
/*used only if PKG_MALLOC is defined*/
#define PKG_MEM_POOL_SIZE 64*1024*1024
/*used if SH_MEM is defined*/
#define SHM_MEM_SIZE 64
Hi Jr,
its not necessary to re-compile the server to just increase the SHM mem pool.
You can give this as a config parameter or a server daemon parameter during
startup. Maybe there is also an bug present in the PDT, which somehow causes
this problems if you load that much records. I did a short look into the code,
there is a variable
#define PDT_MAX_DEPTH 32
which somehow seems to restrict the maximum size of the internal (tree)
datastructure for queries. But i did not used this module so far - all i can
say is that during my test with cr we loaded more records without any problems
coming from the internal SHM MM, so i guess this is a limitation from the
module. Maybe you should also try to increase the pool to 512 MB (the mem is
just free also on a system level if its not used).
Regards,
Henning
6:25 a.m.
Hi,
On 07/20/2010 11:27 AM, Henning Westerholt wrote:
On Tuesday 20 July 2010, JR Richardson wrote:
this defines the max length for prefixes (DIDs) that can be stored in
the internal trees.
[..]
So now I'm running out of shmem as well. After I loaded more than 320K
records in the pdt database table, I started getting these errors:
0(17599) ERROR: pdt [pdtree.c:283]: bad parameters
0(17599) INFO: pdt [pdt.c:490]: no prefix found in [7000011234]
shmem:real_used_size = 1247512
This parameter does not get above 1247512 without having a problem,
even though I have plenty of shmem available.
shmem:total_size = 33554432
shmem:used_size = 1229472
shmem:real_used_size = 1247512
shmem:max_used_size = 1247512
shmem:free_size = 32306920
I increased these parameters in config.h but that did not help:
/*used only if PKG_MALLOC is defined*/
#define PKG_MEM_POOL_SIZE 64*1024*1024
/*used if SH_MEM is defined*/
#define SHM_MEM_SIZE 64
Hi Jr,
its not necessary to re-compile the server to just increase the SHM mem pool.
You can give this as a config parameter or a server daemon parameter during
startup. Maybe there is also an bug present in the PDT, which somehow causes
this problems if you load that much records. I did a short look into the code,
there is a variable
#define PDT_MAX_DEPTH 32
which somehow seems to restrict the maximum size of
the internal (tree)
datastructure for queries. But i did not used this module so far - all i can
say is that during my test with cr we loaded more records without any problems
coming from the internal SHM MM, so i guess this is a limitation from the
module. Maybe you should also try to increase the pool to 512 MB (the mem is
just free also on a system level if its not used).
If there are many records then the shared memory size must be increased
(command line parameter -m, e.g., -m 512). I used the module with over
1million records without problem.
Regards,
Ramona
6:28 a.m.
On Tuesday 20 July 2010, Elena-Ramona Modroiu wrote:
Hi Elena-Ramona,
ah, now i understand better - so my comment was misleading. Thanks for
clarification!
Cheers,
Henning
[..]
#define PDT_MAX_DEPTH 32
this defines the max length for prefixes (DIDs) that can be stored in
the internal trees.
5:21 a.m.
On Monday 19 July 2010, JR Richardson wrote:
[..]
When I added 180K records in the database, I got the "no more pkg mem"
error again.
I increased again:
#define PKG_MEM_POOL_SIZE 32*1024*1024
This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg
mem" error again.
So I don't think it is prudent to just keep increasing
PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo
pkg_mem, shouldn't this be a dynamic allocation if it's not within
shmem and has no affect on core sip-router function? While the
pdt_reload and pdt_list is going on (takes a few seconds to load and
list), I don't see any problems with the sip-router executions. I
guess I can just use the old fashion database query to look up routes
instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio 1.5
will fallback to the system malloc, which don't have this size restrictions.
For 3.x/ sr there is an option to build the MI interface with system malloc as
well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
Author: Daniel-Constantin Mierla <miconda(a)gmail.com>
Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc
- MI lib can be built independently of core with system malloc for pkg
memory (see Makefile)
Both workarounds should help in your case.
But you're right, its probably more meaningfull to just execute a DB query
instead of trying to pipe all this data through your (productive) sip server.
Regards,
Henning
6:21 a.m.
Hi,
On 07/20/2010 11:21 AM, Henning Westerholt wrote:
On Monday 19 July 2010, JR Richardson wrote:
this is the option added to overcome the MI design limitations - while
most of dumped structures are in shared memory, MI builds the output in
PKG memory, resulting in insufficient space since shm is much bigger
than pkg - also this sometime ended as well in lot of mem fragmentation
for mi process in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile
and reinstall.
Regards,
Ramona
[..]
When I added 180K records in the database, I got the "no more pkg mem"
error again.
I increased again:
#define PKG_MEM_POOL_SIZE 32*1024*1024
This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg
mem" error again.
So I don't think it is prudent to just keep increasing
PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo
pkg_mem, shouldn't this be a dynamic allocation if it's not within
shmem and has no affect on core sip-router function? While the
pdt_reload and pdt_list is going on (takes a few seconds to load and
list), I don't see any problems with the sip-router executions. I
guess I can just use the old fashion database query to look up routes
instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio 1.5
will fallback to the system malloc, which don't have this size restrictions.
For 3.x/ sr there is an option to build the MI interface with system malloc as
well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
Author: Daniel-Constantin Mierla<miconda(a)gmail.com>
Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc
- MI lib can be built independently of core with system malloc for pkg
memory (see Makefile)
Both workarounds should help in your case.
But you're right, its probably more meaningfull to just execute a DB query
instead of trying to pipe all this data through your (productive) sip server.
Regards,
Henning
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
6:29 a.m.
2010/7/20 Elena-Ramona Modroiu <ramona(a)asipto.com>om>:
this is the option added to overcome the MI design
limitations - while most
of dumped structures are in shared memory, MI builds the output in PKG
memory, resulting in insufficient space since shm is much bigger than pkg -
also this sometime ended as well in lot of mem fragmentation for mi process
in the past.
Hi Elena. This is also I'm a bit afraid of when using MI command to
manage large ammount of records (LCR, locations, 'address'
entries...).
Wouldn't make sense MI to use shared memory instead of private memory?
Regards.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
6:37 a.m.
On Tuesday 20 July 2010, Iñaki Baz Castillo wrote:
Hi Iñaki,
this would be possible, but its IMHO a bit wrong from a conceptional POV. This
FIFO command execution belongs to the MI process, and it should not affect the
working of other process which are busy routing SIP requests - they don't need
to see this as all. One workaround is as discussed is to just use system
malloc, another would be to partition the command execution in small steps
like we do for data loading in several modules.
But as i said earlier, its maybe not the best idea in general to pipe the
output of several houndred thousands rules through the MI interface,
especially if the information is available more or less similar in the
database.
Regards,
Henning
this is the
option added to overcome the MI design limitations - while
most of dumped structures are in shared memory, MI builds the output in
PKG memory, resulting in insufficient space since shm is much bigger than
pkg - also this sometime ended as well in lot of mem fragmentation for mi
process in the past.
Hi Elena. This is also I'm a bit afraid of when using MI command to
manage large ammount of records (LCR, locations, 'address'
entries...).
Wouldn't make sense MI to use shared memory instead of private memory?
6:50 a.m.
Hi,
On 07/20/2010 12:37 PM, Henning Westerholt wrote:
On Tuesday 20 July 2010, Iñaki Baz Castillo wrote:
Hi Iñaki,
this would be possible, but its IMHO a bit wrong from a conceptional POV. This
FIFO command execution belongs to the MI process, and it should not affect the
working of other process which are busy routing SIP requests - they don't need
to see this as all. One workaround is as discussed is to just use system
malloc, another would be to partition the command execution in small steps
like we do for data loading in several modules.
agree, using shm does not bring benefits, but some drawbacks since shm
involves locking for access, thus it affects the other processes (e.g.,
sip workers). It is better with system malloc, maybe this should be made
default for MI.
Regards,
Ramona
this is
the option added to overcome the MI design limitations - while
most of dumped structures are in shared memory, MI builds the output in
PKG memory, resulting in insufficient space since shm is much bigger than
pkg - also this sometime ended as well in lot of mem fragmentation for mi
process in the past.
Hi Elena. This is also I'm a bit afraid of when using MI command to
manage large ammount of records (LCR, locations, 'address'
entries...).
Wouldn't make sense MI to use shared memory instead of private memory?
But as i said earlier, its maybe not the best idea in
general to pipe the
output of several houndred thousands rules through the MI interface,
especially if the information is available more or less similar in the
database.
Regards,
Henning
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
7:09 a.m.
2010/7/20 Henning Westerholt <henning.westerholt(a)1und1.de>de>:
Sure, you are right.
Hi Elena. This
is also I'm a bit afraid of when using MI command to
manage large ammount of records (LCR, locations, 'address'
entries...).
Wouldn't make sense MI to use shared memory instead of private memory?
Hi Iñaki,
this would be possible, but its IMHO a bit wrong from a conceptional POV. This
FIFO command execution belongs to the MI process, and it should not affect the
working of other process which are busy routing SIP requests - they don't need
to see this as all. One workaround is as discussed is to just use system
malloc,
It seems a good solution.
another would be to partition the command execution in
small steps
like we do for data loading in several modules.
Yes, but this involves MI process to be designed for that, and it's
not the case, right? This is, in case the retrieved information
exceeds some size it should be automatically partitioned in various
steps which are sent to the MI process one by one.
But as i said earlier, its maybe not the best idea in
general to pipe the
output of several houndred thousands rules through the MI interface,
especially if the information is available more or less similar in the
database.
Sure, but what about when using just memory (as in db_mode=0 for
usrloc)? Also there are other modules (as LCR) which just use the
database to load initial values and later those values or their
attributes are modified in memory (i.e: LCR gateways status).
Regards.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
12:30 p.m.
On Tuesday 20 July 2010, Iñaki Baz Castillo wrote:
Hey Inaki,
yes, this is not the case at the moment.
This are indeed valid use cases, did not thought about it. Also the dialog
module outputs some bigger list, but normally not several 100k. ;-)
Regards,
Henning
another would
be to partition the command execution in small steps
like we do for data loading in several modules.
Yes, but this involves MI process to be designed for that, and it's
not the case, right? This is, in case the retrieved information
exceeds some size it should be automatically partitioned in various
steps which are sent to the MI process one by one. But as i said
earlier, its maybe not the best idea in general to pipe the
output of several houndred thousands rules through the MI interface,
especially if the information is available more or less similar in the
database.
Sure, but what about when using just memory (as in db_mode=0 for
usrloc)? Also there are other modules (as LCR) which just use the
database to load initial values and later those values or their
attributes are modified in memory (i.e: LCR gateways status).
11:06 a.m.
[..]
When I added 180K records in the database, I got the "no more pkg mem"
error again.
I increased again:
#define PKG_MEM_POOL_SIZE 32*1024*1024
This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg
mem" error again.
So I don't think it is prudent to just keep increasing
PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo
pkg_mem, shouldn't this be a dynamic allocation if it's not within
shmem and has no affect on core sip-router function? While the
pdt_reload and pdt_list is going on (takes a few seconds to load and
list), I don't see any problems with the sip-router executions. I
guess I can just use the old fashion database query to look up routes
instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio
1.5
will fallback to the system malloc, which don't have this size
restrictions.
For 3.x/ sr there is an option to build the MI interface with system
malloc as
well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
Author: Daniel-Constantin Mierla<miconda(a)gmail.com>
Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc
- MI lib can be built independently of core with system malloc for pkg
memory (see Makefile)
1:04 p.m.
On 07/20/2010 05:06 PM, JR Richardson wrote:
this is the option added to overcome the MI design limitations - while
most
of dumped structures are in shared memory, MI builds the output in PKG
memory, resulting in insufficient space since shm is much bigger than pkg -
also this sometime ended as well in lot of mem fragmentation for mi process
in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile and
reinstall.
Regards,
Ramona
Hi Ramona,
Thanks for your suggestions. I started over with default mem
parameters in config.h and compiled with:
## uncomment next line for using system malloc with MI
DEFS+= -DMI_SYSTEM_MALLOC
I loaded the pdt database table with 360K records and also started
kamailio with '-m 512' but the table still has a max record
limitation, will not load propperly and I get these errors:
0(22456) ERROR: pdt [pdtree.c:283]: bad parameters
0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
Of course wihout a propper table load, pdt_list does not produce any
results (but i don;t get the pkg_mem error any more).
sip-router2:~# kamctl fifo get_statistics all
shmem:total_size = 536870912
shmem:used_size = 1229472
shmem:real_used_size = 1247560
shmem:max_used_size = 1255376
shmem:free_size = 535623352
There seems to still be a bottleneck with shmem, even through I have
512M+ available, I only seem to be using ~1.2M. Is there an internal
shmem hard limit that can be adjusted?
Hi JR,
There is no limit hard-coded, I just loaded 1mio records in pdt
(sdomain=*, prefix=1mio..2mio, domain=test.org) and statistics show:
shmem:total_size = 268435456
shmem:used_size = 26594256
shmem:real_used_size = 80148548
shmem:max_used_size = 80148548
shmem:free_size = 188286908
So the records are reflected in shared memory. What is the output of
"kamailio -V"? What are the parameters for pdt?
Regards,
Ramona
[..]
When I added 180K records in the database, I got the "no more pkg mem"
error again.
I increased again:
#define PKG_MEM_POOL_SIZE 32*1024*1024
This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg
mem" error again.
So I don't think it is prudent to just keep increasing
PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo
pkg_mem, shouldn't this be a dynamic allocation if it's not within
shmem and has no affect on core sip-router function? While the
pdt_reload and pdt_list is going on (takes a few seconds to load and
list), I don't see any problems with the sip-router executions. I
guess I can just use the old fashion database query to look up routes
instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio
1.5
will fallback to the system malloc, which don't have this size
restrictions.
For 3.x/ sr there is an option to build the MI interface with system
malloc as
well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
Author: Daniel-Constantin Mierla<miconda(a)gmail.com>
Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc
- MI lib can be built independently of core with system malloc for pkg
memory (see Makefile)
1:45 p.m.
On Tue, Jul 20, 2010 at 12:04 PM, Elena-Ramona Modroiu
<ramona(a)asipto.com> wrote:
On 07/20/2010 05:06 PM, JR Richardson wrote:
Hi Ramona,
Thanks for your suggestions. I started over with default mem
parameters in config.h and compiled with:
## uncomment next line for using system malloc with MI
DEFS+= -DMI_SYSTEM_MALLOC
I loaded the pdt database table with 360K records and also started
kamailio with '-m 512' but the table still has a max record
limitation, will not load propperly and I get these errors:
0(22456) ERROR: pdt [pdtree.c:283]: bad parameters
0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
Of course wihout a propper table load, pdt_list does not produce any
results (but i don;t get the pkg_mem error any more).
sip-router2:~# kamctl fifo get_statistics all
shmem:total_size = 536870912
shmem:used_size = 1229472
shmem:real_used_size = 1247560
shmem:max_used_size = 1255376
shmem:free_size = 535623352
There seems to still be a bottleneck with shmem, even through I have
512M+ available, I only seem to be using ~1.2M. Is there an internal
shmem hard limit that can be adjusted?
Hi JR,
There is no limit hard-coded, I just loaded 1mio records in pdt (sdomain=*,
prefix=1mio..2mio, domain=test.org) and statistics show:
shmem:total_size = 268435456
shmem:used_size = 26594256
shmem:real_used_size = 80148548
shmem:max_used_size = 80148548
shmem:free_size = 188286908
So the records are reflected in shared memory. What is the output of
"kamailio -V"? What are the parameters for pdt?
Regards,
Ramona
sip-router2:~# kamailio -V
version: kamailio 3.0.2 (i386/linux) a6141a
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
@(#) $Id$
main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2
sip-router2:~#
# ----- pdt params -----
modparam("pdt", "db_url",
"mysql://openserro:openserro@localhost/openser")
modparam("pdt", "char_list", "0123456789")
modparam("pdt", "check_domain", 0)
modparam("pdt", "fetch_rows", 5000)
Here is an interesting observation:
When I have 300K+ records in the pdt table, it seems like it will not
load properly and the shmem is very low, as expected without a large
memory pool needed to hold the records. Statistics show:
shmem:total_size = 33554432
shmem:used_size = 1229472
shmem:real_used_size = 1247512
shmem:max_used_size = 1247512
shmem:free_size = 32306920
shmem:fragments = 11
When I decrease the table recrods to 240K, the complete table loads ok
and shmem reflect properly:
shmem:total_size = 33554432
shmem:used_size = 6349784
shmem:real_used_size = 8501144
shmem:max_used_size = 8501144
shmem:free_size = 25053288
shmem:fragments = 1
So maybe I'm not having an issue with shmem at all but a loading
problem with the pdt tabe after a certain record count.
I changed the "modparam("pdt", "fetch_rows", 5000)" to see
if that
would help. When the table loads properly is is quicker than with the
default 1000 rows, but that is the only difference I experience. When
the pdt table has more than 300K records, it just does not load into
memory.
Could there be a pdt module timeout that is being met when loading?
For instance, if it takes more than N seconds to load, consider a
fault and not load?
sip-router2:/etc/kamailio# mysql -V
mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2
sip-router2:/etc/kamailio#
Thanks.
JR
--
JR Richardson
Engineering for the Masses
>
> [..]
> When I added 180K records in the database, I got the "no more pkg mem"
> error again.
> I increased again:
> #define PKG_MEM_POOL_SIZE 32*1024*1024
> This allowed me to execute pdt_list with 180K records loaded.
>
> I increased database record count to 240K and got the "no more pkg
> mem" error again.
>
> So I don't think it is prudent to just keep increasing
> PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo
> pkg_mem, shouldn't this be a dynamic allocation if it's not within
> shmem and has no affect on core sip-router function? While the
> pdt_reload and pdt_list is going on (takes a few seconds to load and
> list), I don't see any problems with the sip-router executions. I
> guess I can just use the old fashion database query to look up routes
> instead of fifo pdt_list.
>
>
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio
1.5
will fallback to the system malloc, which don't have this size
restrictions.
For 3.x/ sr there is an option to build the MI interface with system
malloc as
well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
Author: Daniel-Constantin Mierla<miconda(a)gmail.com>
Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc
- MI lib can be built independently of core with system malloc for
pkg
memory (see Makefile)
this is the option added to overcome the MI design limitations - while
most
of dumped structures are in shared memory, MI builds the output in PKG
memory, resulting in insufficient space since shm is much bigger than pkg
-
also this sometime ended as well in lot of mem fragmentation for mi
process
in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile and
reinstall.
Regards,
Ramona
10:12 p.m.
New subject: Variables for UNIX timestamps and acc/callid?
Hi there,
Hopefully there are two simple answers for these. Running Kamailio 1.4.2.
1. Is there a variable that holds the current UNIX timestamp? Either when
called in a route() or method.
2. What is the variable that is equivalent to the "callid" in the ACC
module? Or better yet, what is the best way to retrieve that ACC field in
the config? I need a unique ID for the complete call ie. From INVITE to
BYE...
I am calling out to a new external CDR "gatherer of sorts" and would like
this information to be present...
Thanks,
-graham
10:15 p.m.
New subject: Variables for UNIX timestamps and acc/callid?
On 07/20/2010 10:12 PM, Graham Wooden wrote:
1. Is there a variable that holds the current UNIX
timestamp? Either when
called in a route() or method.
Yep:
http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:1.4.x#unix_time_s…
2. What is the variable that is equivalent to the
"callid" in the ACC
module? Or better yet, what is the best way to retrieve that ACC field in
the config? I need a unique ID for the complete call ie. From INVITE to
BYE...
$ci --
http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:1.4.x#call-id
--
Alex Balashov - Principal
Evariste Systems LLC
1170 Peachtree Street
12th Floor, Suite 1200
Atlanta, GA 30309
Tel: +1-678-954-0670
Fax: +1-404-961-1892
Web: http://www.evaristesys.com/
10:32 p.m.
New subject: Variables for UNIX timestamps and acc/callid?
Ah - perfect. Thank you Alex.
-graham
On 7/20/10 9:15 PM, "Alex Balashov" <abalashov(a)evaristesys.com> wrote:
On 07/20/2010 10:12 PM, Graham Wooden wrote:
http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:1.4.x#unix_time_s…
p
1. Is there a variable that holds the current
UNIX timestamp? Either when
called in a route() or method.
Yep:
2. What is the variable that is equivalent to the
"callid" in the ACC
module? Or better yet, what is the best way to retrieve that ACC field in
the config? I need a unique ID for the complete call ie. From INVITE to
BYE...
$ci --
http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:1.4.x#call-id
7 Aug
7 Aug
9:58 p.m.
New subject: Simple Call Accounting
Hi there,
I wanted to share with the folks on this list a simple way to get some
simple call accounting.
My goal was to have a simple, one line in a DB entry, depicting when the
call started (epoch and date/time), when the call ended (epoch and
date/time), the call-to number and the calling-from, and finally the
direction of the call.
I found that ACC was too buggy with this - meaning, that about 1/2 of the
time the acc_extra and acc_to would actually be correct & it had a bunch of
extra stuff that I really didn't need or care about. Maybe the buggy-ness
is because I am running kamailio 1.4.2 - ?
I used CDRTool and Radius a while back, on my first production machine, but
after 2 years of service I had to fail over to my secondary and could never
get that combination going again. I didn't keep good notes when I struggled
to get it working the first time.
So, with that - for the folks out there that want a simple, no thrills call
accounting, from which you can add and tailor to your environment - here is
what I did:
First, create a table in your openser database. For this example, I called
it "rawlines". Here is the mySQL schema:
+------------+-------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| timenow | datetime | NO | | NULL | |
| epoch | varchar(10) | NO | | NULL | |
| endtimenow | datetime | NO | | NULL | |
| endepoch | varchar(10) | NO | | NULL | |
| callid | varchar(64) | NO | MUL | NULL | |
| thisside | varchar(28) | NO | | NULL | |
| otherside | varchar(28) | NO | | NULL | |
| direction | varchar(8) | NO | | NULL | |
+------------+-------------+------+-----+---------+----------------+
Then, the placement the SQL calls. Of course, everyone's cfg file is
different, as well as you may have a better place inside ... Here is what I
am doing:
route{
.
.
if (is_method("BYE")) {
.
.
avp_db_query("UPDATE rawlines set endtimenow=NOW(), endepoch='$Ts' WHERE
callid='$ci'");
.
.
}
.
.
if (!lookup("location")) {
.
.
avp_db_query("INSERT INTO rawlines
(timenow,epoch,callid,thisside,otherside,direction)
VALUES(NOW(),'$Ts','$ci','$fU','$rU','outbound')");
.
.
} else {
avp_db_query("INSERT INTO rawlines
(timenow,epoch,callid,thisside,otherside,direction)
VALUES(NOW(),'$Ts','$ci','$tU','$Au','inbound')");
.
.
}
That's pretty much it. Your results will be a single, readable line for each
call. Now, is this perfect - heck no. For instance, for calls that don't
complete (like just ring/hangup with no BYE), you will get 0000-00-00 for
"endtimenow" which could be misleading thinking the call is still happening.
I like depicting both epoch and readable date/time. I use the epoch to
calculate billing time and use the readable for quick reference for users.
My main goal was to get something quick and friendly out there to start
expanding on. Fits my needs currently and maybe it will help benefit someone
else out there - or better yet, come with a better way to achieve this
approach.
-graham
8 Aug
8 Aug
9:54 p.m.
New subject: Follow up: Simple Call Accounting
That's pretty much it. Your results will be a
single, readable line for each
call. Now, is this perfect - heck no. For instance, for calls that don't
complete (like just ring/hangup with no BYE), you will get 0000-00-00 for
"endtimenow" which could be misleading thinking the call is still happening.
Just to follow up - I meant to add to this by saying that thus far, any
incomplete's that have occurred (ie. Call not being answered or not active
in the dialog table), I was able to reference all of them in the
missed_calls table. So with that, I have a couple worker perl scripts that
go out from a centralized box that picks up and collates all my data from
the proxies. Not once, yet, had a call become a strangler or have
incomplete/inaccurate data.
Your mileage may vary.
-graham
10 p.m.
New subject: Simple Call Accounting
For comparison and contrast: We handle this in PostgreSQL by having
Kamailio write to the 'acc' table, and adding triggers that operate on
'acc' events and populate or update another table, 'cdr', with actual
CDRs in the desirable one-row-per-call way.
The advantage to doing it that way versus in route script is that it's
transactional in nature; an error will result in all the cascading
queries being rolled back. It also makes it much easier to insinuate
business layer stuff into the cascade, or extend the trigger cascade
with additional logic, because PL/PgSQL provides certain features of a
general-purpose programming environment that Kamailio does not, given
its domain specificity.
It also eliminates unnecessary data interchange between the Kamailio
process and the RDBM, given that most of the work is done in the
database itself, causing additional latency and being a possible
source of data corruption.
In short, it lets us have a relatively clean Kamailio config and
deputise all the business layer stuff to the database backend in a
very hierarchical, maintainable way. That way, the Kamailio route
script can stick to doing what it does best: operating on SIP messages.
--
Alex Balashov - Principal
Evariste Systems LLC
1170 Peachtree Street
12th Floor, Suite 1200
Atlanta, GA 30309
Tel: +1-678-954-0670
Fax: +1-404-961-1892
Web: http://www.evaristesys.com/
9 Aug
9 Aug
9:10 a.m.
New subject: Simple Call Accounting
Just as an addendum to Alex's comment, I'm doing the exact same thing
with MySQL triggers.
Most strange situations can be worked around using some parameter in
ACC or DIALOG modules (if working together).
Our main cause is that we have many proxies and a central DB, so the
work is performed by only one process that can be measured and
optimized alone.
Uriel
On Sun, Aug 8, 2010 at 11:00 PM, Alex Balashov
<abalashov(a)evaristesys.com> wrote:
For comparison and contrast: We handle this in
PostgreSQL by having
Kamailio write to the 'acc' table, and adding triggers that operate on
'acc'
events and populate or update another table, 'cdr', with actual CDRs in the
desirable one-row-per-call way.
The advantage to doing it that way versus in route script is that it's
transactional in nature; an error will result in all the cascading queries
being rolled back. It also makes it much easier to insinuate business layer
stuff into the cascade, or extend the trigger cascade with additional logic,
because PL/PgSQL provides certain features of a general-purpose programming
environment that Kamailio does not, given its domain specificity.
It also eliminates unnecessary data interchange between the Kamailio process
and the RDBM, given that most of the work is done in the database itself,
causing additional latency and being a possible source of data corruption.
In short, it lets us have a relatively clean Kamailio config and deputise
all the business layer stuff to the database backend in a very hierarchical,
maintainable way. That way, the Kamailio route script can stick to doing
what it does best: operating on SIP messages.
--
Alex Balashov - Principal
Evariste Systems LLC
1170 Peachtree Street
12th Floor, Suite 1200
Atlanta, GA 30309
Tel: +1-678-954-0670
Fax: +1-404-961-1892
Web: http://www.evaristesys.com/
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
10:05 p.m.
New subject: Simple Call Accounting
I appreciate yours and Alex's' input and insight.
While I don't have a many to one (proxy to DB), but rather one to one, I
have a few worker scripts from a central reporting box that pulls those
records and anything in the missed_calls and dialog and wraps up any
collation. I don't use anything in ACC...
-graham
On 8/9/10 8:10 AM, "Uriel Rozenbaum" <uriel.rozenbaum(a)gmail.com> wrote:
Just as an addendum to Alex's comment, I'm
doing the exact same thing
with MySQL triggers.
Most strange situations can be worked around using some parameter in
ACC or DIALOG modules (if working together).
Our main cause is that we have many proxies and a central DB, so the
work is performed by only one process that can be measured and
optimized alone.
Uriel
On Sun, Aug 8, 2010 at 11:00 PM, Alex Balashov
<abalashov(a)evaristesys.com> wrote:
For comparison and contrast: We handle this in
PostgreSQL by having
Kamailio write to the 'acc' table, and adding triggers that operate on
'acc'
events and populate or update another table, 'cdr', with actual CDRs in the
desirable one-row-per-call way.
The advantage to doing it that way versus in route script is that it's
transactional in nature; an error will result in all the cascading queries
being rolled back. It also makes it much easier to insinuate business layer
stuff into the cascade, or extend the trigger cascade with additional logic,
because PL/PgSQL provides certain features of a general-purpose programming
environment that Kamailio does not, given its domain specificity.
It also eliminates unnecessary data interchange between the Kamailio process
and the RDBM, given that most of the work is done in the database itself,
causing additional latency and being a possible source of data corruption.
In short, it lets us have a relatively clean Kamailio config and deputise
all the business layer stuff to the database backend in a very hierarchical,
maintainable way. That way, the Kamailio route script can stick to doing
what it does best: operating on SIP messages.
--
Alex Balashov - Principal
Evariste Systems LLC
1170 Peachtree Street
12th Floor, Suite 1200
Atlanta, GA 30309
Tel: +1-678-954-0670
Fax: +1-404-961-1892
Web: http://www.evaristesys.com/
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
21 Jul
21 Jul
4:56 a.m.
On 07/20/2010 07:45 PM, JR Richardson wrote:
On Tue, Jul 20, 2010 at 12:04 PM, Elena-Ramona
Modroiu
<ramona(a)asipto.com> wrote:
try to play with mysql module timeout parameter in case the records are
not loaded fast enough:
http://kamailio.org/docs/modules/stable/modules/db_mysql.html#id2928400
Regards,
Ramona
On 07/20/2010 05:06 PM, JR Richardson wrote:
Hi JR,
There is no limit hard-coded, I just loaded 1mio records in pdt (sdomain=*,
prefix=1mio..2mio, domain=test.org) and statistics show:
shmem:total_size = 268435456
shmem:used_size = 26594256
shmem:real_used_size = 80148548
shmem:max_used_size = 80148548
shmem:free_size = 188286908
So the records are reflected in shared memory. What is the output of
"kamailio -V"? What are the parameters for pdt?
Regards,
Ramona
sip-router2:~# kamailio -V
version: kamailio 3.0.2 (i386/linux) a6141a
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
@(#) $Id$
main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2
sip-router2:~#
# ----- pdt params -----
modparam("pdt", "db_url",
"mysql://openserro:openserro@localhost/openser")
modparam("pdt", "char_list", "0123456789")
modparam("pdt", "check_domain", 0)
modparam("pdt", "fetch_rows", 5000)
Here is an interesting observation:
When I have 300K+ records in the pdt table, it seems like it will not
load properly and the shmem is very low, as expected without a large
memory pool needed to hold the records. Statistics show:
shmem:total_size = 33554432
shmem:used_size = 1229472
shmem:real_used_size = 1247512
shmem:max_used_size = 1247512
shmem:free_size = 32306920
shmem:fragments = 11
When I decrease the table recrods to 240K, the complete table loads ok
and shmem reflect properly:
shmem:total_size = 33554432
shmem:used_size = 6349784
shmem:real_used_size = 8501144
shmem:max_used_size = 8501144
shmem:free_size = 25053288
shmem:fragments = 1
So maybe I'm not having an issue with shmem at all but a loading
problem with the pdt tabe after a certain record count.
I changed the "modparam("pdt", "fetch_rows", 5000)" to see
if that
would help. When the table loads properly is is quicker than with the
default 1000 rows, but that is the only difference I experience. When
the pdt table has more than 300K records, it just does not load into
memory.
Could there be a pdt module timeout that is being met when loading?
For instance, if it takes more than N seconds to load, consider a
fault and not load?
sip-router2:/etc/kamailio# mysql -V
mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2
sip-router2:/etc/kamailio#
>>
[..]
>> When I added 180K records in the database, I got the "no more pkg mem"
>> error again.
>> I increased again:
>> #define PKG_MEM_POOL_SIZE 32*1024*1024
>> This allowed me to execute pdt_list with 180K records loaded.
>>
>> I increased database record count to 240K and got the "no more pkg
>> mem" error again.
>>
>> So I don't think it is prudent to just keep increasing
>> PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo
>> pkg_mem, shouldn't this be a dynamic allocation if it's not within
>> shmem and has no affect on core sip-router function? While the
>> pdt_reload and pdt_list is going on (takes a few seconds to load and
>> list), I don't see any problems with the sip-router executions. I
>> guess I can just use the old fashion database query to look up routes
>> instead of fifo pdt_list.
>>
>>
>>
> Hi Jr,
>
> if you don't use the PKG_MALLOC MM in your system at least the kamailio
> 1.5
> will fallback to the system malloc, which don't have this size
> restrictions.
> For 3.x/ sr there is an option to build the MI interface with system
> malloc as
> well, it this commit:
>
> commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec
> Author: Daniel-Constantin Mierla<miconda(a)gmail.com>
> Date: Mon Jun 29 12:34:08 2009 +0200
>
> lib/kmi: option to build it with system malloc
>
> - MI lib can be built independently of core with system malloc for
> pkg
> memory (see Makefile)
>
>
>
this is the option added to overcome the MI design limitations - while
most
of dumped structures are in shared memory, MI builds the output in PKG
memory, resulting in insufficient space since shm is much bigger than pkg
-
also this sometime ended as well in lot of mem fragmentation for mi
process
in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile and
reinstall.
Regards,
Ramona
Hi Ramona,
Thanks for your suggestions. I started over with default mem
parameters in config.h and compiled with:
## uncomment next line for using system malloc with MI
DEFS+= -DMI_SYSTEM_MALLOC
I loaded the pdt database table with 360K records and also started
kamailio with '-m 512' but the table still has a max record
limitation, will not load propperly and I get these errors:
0(22456) ERROR: pdt [pdtree.c:283]: bad parameters
0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
Of course wihout a propper table load, pdt_list does not produce any
results (but i don;t get the pkg_mem error any more).
sip-router2:~# kamctl fifo get_statistics all
shmem:total_size = 536870912
shmem:used_size = 1229472
shmem:real_used_size = 1247560
shmem:max_used_size = 1255376
shmem:free_size = 535623352
There seems to still be a bottleneck with shmem, even through I have
512M+ available, I only seem to be using ~1.2M. Is there an internal
shmem hard limit that can be adjusted?
5:55 p.m.
sip-router2:~#
kamailio -V
version: kamailio 3.0.2 (i386/linux) a6141a
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
@(#) $Id$
main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2
sip-router2:~#
# ----- pdt params -----
modparam("pdt", "db_url",
"mysql://openserro:openserro@localhost/openser")
modparam("pdt", "char_list", "0123456789")
modparam("pdt", "check_domain", 0)
modparam("pdt", "fetch_rows", 5000)
Here is an interesting observation:
When I have 300K+ records in the pdt table, it seems like it will not
load properly and the shmem is very low, as expected without a large
memory pool needed to hold the records. Statistics show:
shmem:total_size = 33554432
shmem:used_size = 1229472
shmem:real_used_size = 1247512
shmem:max_used_size = 1247512
shmem:free_size = 32306920
shmem:fragments = 11
When I decrease the table recrods to 240K, the complete table loads ok
and shmem reflect properly:
shmem:total_size = 33554432
shmem:used_size = 6349784
shmem:real_used_size = 8501144
shmem:max_used_size = 8501144
shmem:free_size = 25053288
shmem:fragments = 1
So maybe I'm not having an issue with shmem at all but a loading
problem with the pdt tabe after a certain record count.
I changed the "modparam("pdt", "fetch_rows", 5000)" to see
if that
would help. When the table loads properly is is quicker than with the
default 1000 rows, but that is the only difference I experience. When
the pdt table has more than 300K records, it just does not load into
memory.
Could there be a pdt module timeout that is being met when loading?
For instance, if it takes more than N seconds to load, consider a
fault and not load?
sip-router2:/etc/kamailio# mysql -V
mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using
readline 5.2
sip-router2:/etc/kamailio#
try to play with mysql module timeout parameter in case the records are not
loaded fast enough:
http://kamailio.org/docs/modules/stable/modules/db_mysql.html#id2928400
Regards,
Ramona
5:34 a.m.
On Tuesday 20 July 2010, JR Richardson wrote:
[..]
Here is an interesting observation:
When I have 300K+ records in the pdt table, it seems like it will not
load properly and the shmem is very low, as expected without a large
memory pool needed to hold the records. Statistics show:
shmem:total_size = 33554432
[..]
When I decrease the table recrods to 240K, the complete table loads ok
and shmem reflect properly:
shmem:total_size = 33554432
Hi JR,
it seems you've a problem with your SHM pool size configuration.
33.554.432 ~ 34MB
It should normally output the proper size, like this:
test-proxy:~# grep "MEM" /etc/default/kamailio
MEMORY=1024
test-proxy:~# kamctl fifo get_statistics all | grep total_size
shmem:total_size = 1073741824
1073.741.824 = 1024 MB
Cheers,
Henning
20 Jul
20 Jul
6:33 a.m.
Hi,
On 07/16/2010 11:51 PM, JR Richardson wrote:
Hi All,
I loaded up the PDT database with about 35K records and when I issue
the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR:<core> [tree.c:139]: no more pkg mem
3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which
suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/
#define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try:
#define PKG_MEM_POOL_SIZE 4*2048*2048
or
#define PKG_MEM_POOL_SIZE 8*1024*1024
Which one would be a logical adjustment? Also, is there a correlation
between pkg mem and database record size as related to pdt_list?
The idea is to have a few 100K records loaded in kamailio and be able
to perform "kamctl fifo pdt_list | grep 512345" to show this prefix
route. But without enough memory, doesn't work.
this is a limitation of MI control interface (as written in other email,
there are workarounds).
Anyway, pdt_list can take the prefix as parameter, so you do not need to
list entire tree and do grep, just list desired prefixes:
http://kamailio.org/docs/modules/stable/modules_k/pdt.html#id2854918
Regards,
Ramona
5253
days inactive
5278
days old
29 comments
8 participants
participants (8)
-
Alex Balashov -
Elena-Ramona Modroiu -
Graham Wooden -
Henning Westerholt -
Iñaki Baz Castillo -
JR Richardson -
marius zbihlei -
Uriel Rozenbaum