Hi All,
I loaded up the PDT database with about 35K records and when I issue the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR: <core> [tree.c:139]: no more pkg mem 3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/ #define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try: #define PKG_MEM_POOL_SIZE 4*2048*2048 or #define PKG_MEM_POOL_SIZE 8*1024*1024
Which one would be a logical adjustment? Also, is there a correlation between pkg mem and database record size as related to pdt_list?
The idea is to have a few 100K records loaded in kamailio and be able to perform "kamctl fifo pdt_list | grep 512345" to show this prefix route. But without enough memory, doesn't work.
Thanks.
JR
On Fri, Jul 16, 2010 at 4:51 PM, JR Richardson jmr.richardson@gmail.com wrote:
Hi All,
I loaded up the PDT database with about 35K records and when I issue the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR: <core> [tree.c:139]: no more pkg mem 3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/ #define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try: #define PKG_MEM_POOL_SIZE 4*2048*2048 or #define PKG_MEM_POOL_SIZE 8*1024*1024
I tried #define PKG_MEM_POOL_SIZE 8*1024*1024 and recompiled with good results, was able to run pdt_list just fine. So what do I look for in the memory statistics to show how much memory to configure when using large database record sets? And if I need to go to 16* or 32*, would that have any adverse affect on any other kamailio operations?
Which one would be a logical adjustment? Also, is there a correlation between pkg mem and database record size as related to pdt_list?
The idea is to have a few 100K records loaded in kamailio and be able to perform "kamctl fifo pdt_list | grep 512345" to show this prefix route. But without enough memory, doesn't work.
Thanks.
JR
JR Richardson Engineering for the Masses
Thanks.
JR
JR Richardson wrote:
On Fri, Jul 16, 2010 at 4:51 PM, JR Richardson jmr.richardson@gmail.com wrote:
Hi All,
I loaded up the PDT database with about 35K records and when I issue the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR: <core> [tree.c:139]: no more pkg mem 3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/ #define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try: #define PKG_MEM_POOL_SIZE 4*2048*2048 or #define PKG_MEM_POOL_SIZE 8*1024*1024
I tried #define PKG_MEM_POOL_SIZE 8*1024*1024 and recompiled with good results, was able to run pdt_list just fine. So what do I look for in the memory statistics to show how much memory to configure when using large database record sets? And if I need to go to 16* or 32*, would that have any adverse affect on any other kamailio operations?
Hello JR,
Compile kamailio with memory debug support. Modify Makefile.vars and set MEMDBG to 1 . In the config file(kamailio.cfg), set memlog to a lower level than the general debug level. You can do a dump of pkg (private) memory by sending a SIGUSR1 to one of the worker process ('kamctl ps' to find the pid). Unless you are working with a huge number of calls, I can't see why you need more than 8 MB or 10 maximum. Carefull if you have a 64 bit system you might need just a little more.
Which one would be a logical adjustment? Also, is there a correlation between pkg mem and database record size as related to pdt_list?
The idea is to have a few 100K records loaded in kamailio and be able to perform "kamctl fifo pdt_list | grep 512345" to show this prefix route. But without enough memory, doesn't work.
AFAIK, these mi commands will construct a complete result tree, thus meaning it should have enough memory for 100K records. If it works with 8 MB it is ok . Keep in mind that the memory dump is not that usefull to track these OOM condition, it is usefull to track memory leaks, as after the OOM condition happens the already malloc'ed memory is free'd. It would be usefull to test that is those OOM conditions, the free is done corectly and no leaks are induced
Cheers Marius
Thanks.
JR
JR Richardson Engineering for the Masses
Thanks.
JR
On Mon, Jul 19, 2010 at 2:41 AM, marius zbihlei marius.zbihlei@1and1.ro wrote:
JR Richardson wrote:
On Fri, Jul 16, 2010 at 4:51 PM, JR Richardson jmr.richardson@gmail.com wrote:
Hi All,
I loaded up the PDT database with about 35K records and when I issue the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR: <core> [tree.c:139]: no more pkg mem 3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/ #define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try: #define PKG_MEM_POOL_SIZE 4*2048*2048 or #define PKG_MEM_POOL_SIZE 8*1024*1024
I tried #define PKG_MEM_POOL_SIZE 8*1024*1024 and recompiled with good results, was able to run pdt_list just fine. So what do I look for in the memory statistics to show how much memory to configure when using large database record sets? And if I need to go to 16* or 32*, would that have any adverse affect on any other kamailio operations?
Hello JR,
Compile kamailio with memory debug support. Modify Makefile.vars and set MEMDBG to 1 . In the config file(kamailio.cfg), set memlog to a lower level than the general debug level. You can do a dump of pkg (private) memory by sending a SIGUSR1 to one of the worker process ('kamctl ps' to find the pid). Unless you are working with a huge number of calls, I can't see why you need more than 8 MB or 10 maximum. Carefull if you have a 64 bit system you might need just a little more.
Which one would be a logical adjustment? Also, is there a correlation between pkg mem and database record size as related to pdt_list?
The idea is to have a few 100K records loaded in kamailio and be able to perform "kamctl fifo pdt_list | grep 512345" to show this prefix route. But without enough memory, doesn't work.
AFAIK, these mi commands will construct a complete result tree, thus meaning it should have enough memory for 100K records. If it works with 8 MB it is ok . Keep in mind that the memory dump is not that usefull to track these OOM condition, it is usefull to track memory leaks, as after the OOM condition happens the already malloc'ed memory is free'd. It would be usefull to test that is those OOM conditions, the free is done corectly and no leaks are induced
Cheers Marius
So I started seeing this issue with only 35K records in pdt database, I increased #define PKG_MEM_POOL_SIZE 8*1024*1024 Then I was able to load 35K records with good results. I increased record count to 60K records and hit the limit again, pdt_list would not work with same "no more pkg mem" error.
I increased again: #define PKG_MEM_POOL_SIZE 16*1024*1024 This allowed me to execute pdt_list with 60K to 120K records loaded.
When I added 180K records in the database, I got the "no more pkg mem" error again. I increased again: #define PKG_MEM_POOL_SIZE 32*1024*1024 This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg mem" error again.
So I don't think it is prudent to just keep increasing PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo pkg_mem, shouldn't this be a dynamic allocation if it's not within shmem and has no affect on core sip-router function? While the pdt_reload and pdt_list is going on (takes a few seconds to load and list), I don't see any problems with the sip-router executions. I guess I can just use the old fashion database query to look up routes instead of fifo pdt_list.
Thanks.
JR
So I started seeing this issue with only 35K records in pdt database, I increased #define PKG_MEM_POOL_SIZE 8*1024*1024 Then I was able to load 35K records with good results. I increased record count to 60K records and hit the limit again, pdt_list would not work with same "no more pkg mem" error.
I increased again: #define PKG_MEM_POOL_SIZE 16*1024*1024 This allowed me to execute pdt_list with 60K to 120K records loaded.
When I added 180K records in the database, I got the "no more pkg mem" error again. I increased again: #define PKG_MEM_POOL_SIZE 32*1024*1024 This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg mem" error again.
So I don't think it is prudent to just keep increasing PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo pkg_mem, shouldn't this be a dynamic allocation if it's not within shmem and has no affect on core sip-router function? While the pdt_reload and pdt_list is going on (takes a few seconds to load and list), I don't see any problems with the sip-router executions. I guess I can just use the old fashion database query to look up routes instead of fifo pdt_list.
Thanks.
JR
So now I'm running out of shmem as well. After I loaded more than 320K records in the pdt database table, I started getting these errors: 0(17599) ERROR: pdt [pdtree.c:283]: bad parameters 0(17599) INFO: pdt [pdt.c:490]: no prefix found in [7000011234]
shmem:real_used_size = 1247512 This parameter does not get above 1247512 without having a problem, even though I have plenty of shmem available.
shmem:total_size = 33554432 shmem:used_size = 1229472 shmem:real_used_size = 1247512 shmem:max_used_size = 1247512 shmem:free_size = 32306920
I increased these parameters in config.h but that did not help:
/*used only if PKG_MALLOC is defined*/ #define PKG_MEM_POOL_SIZE 64*1024*1024
/*used if SH_MEM is defined*/ #define SHM_MEM_SIZE 64
So now it looks like there is some hard limit on the route size you can deploy for the pdt module? Is this the case or is there somewhere I can increase the available shmem for pdt to use?
Thanks.
JR
On Tuesday 20 July 2010, JR Richardson wrote:
[..] So now I'm running out of shmem as well. After I loaded more than 320K records in the pdt database table, I started getting these errors: 0(17599) ERROR: pdt [pdtree.c:283]: bad parameters 0(17599) INFO: pdt [pdt.c:490]: no prefix found in [7000011234]
shmem:real_used_size = 1247512 This parameter does not get above 1247512 without having a problem, even though I have plenty of shmem available.
shmem:total_size = 33554432 shmem:used_size = 1229472 shmem:real_used_size = 1247512 shmem:max_used_size = 1247512 shmem:free_size = 32306920
I increased these parameters in config.h but that did not help:
/*used only if PKG_MALLOC is defined*/ #define PKG_MEM_POOL_SIZE 64*1024*1024
/*used if SH_MEM is defined*/ #define SHM_MEM_SIZE 64
Hi Jr,
its not necessary to re-compile the server to just increase the SHM mem pool. You can give this as a config parameter or a server daemon parameter during startup. Maybe there is also an bug present in the PDT, which somehow causes this problems if you load that much records. I did a short look into the code, there is a variable
#define PDT_MAX_DEPTH 32
which somehow seems to restrict the maximum size of the internal (tree) datastructure for queries. But i did not used this module so far - all i can say is that during my test with cr we loaded more records without any problems coming from the internal SHM MM, so i guess this is a limitation from the module. Maybe you should also try to increase the pool to 512 MB (the mem is just free also on a system level if its not used).
Regards,
Henning
Hi,
On 07/20/2010 11:27 AM, Henning Westerholt wrote:
On Tuesday 20 July 2010, JR Richardson wrote:
[..] So now I'm running out of shmem as well. After I loaded more than 320K records in the pdt database table, I started getting these errors: 0(17599) ERROR: pdt [pdtree.c:283]: bad parameters 0(17599) INFO: pdt [pdt.c:490]: no prefix found in [7000011234]
shmem:real_used_size = 1247512 This parameter does not get above 1247512 without having a problem, even though I have plenty of shmem available.
shmem:total_size = 33554432 shmem:used_size = 1229472 shmem:real_used_size = 1247512 shmem:max_used_size = 1247512 shmem:free_size = 32306920
I increased these parameters in config.h but that did not help:
/*used only if PKG_MALLOC is defined*/ #define PKG_MEM_POOL_SIZE 64*1024*1024
/*used if SH_MEM is defined*/ #define SHM_MEM_SIZE 64
Hi Jr,
its not necessary to re-compile the server to just increase the SHM mem pool. You can give this as a config parameter or a server daemon parameter during startup. Maybe there is also an bug present in the PDT, which somehow causes this problems if you load that much records. I did a short look into the code, there is a variable
#define PDT_MAX_DEPTH 32
this defines the max length for prefixes (DIDs) that can be stored in the internal trees.
which somehow seems to restrict the maximum size of the internal (tree) datastructure for queries. But i did not used this module so far - all i can say is that during my test with cr we loaded more records without any problems coming from the internal SHM MM, so i guess this is a limitation from the module. Maybe you should also try to increase the pool to 512 MB (the mem is just free also on a system level if its not used).
If there are many records then the shared memory size must be increased (command line parameter -m, e.g., -m 512). I used the module with over 1million records without problem.
Regards, Ramona
On Tuesday 20 July 2010, Elena-Ramona Modroiu wrote:
[..] #define PDT_MAX_DEPTH 32
this defines the max length for prefixes (DIDs) that can be stored in the internal trees.
Hi Elena-Ramona,
ah, now i understand better - so my comment was misleading. Thanks for clarification!
Cheers,
Henning
On Monday 19 July 2010, JR Richardson wrote:
[..] When I added 180K records in the database, I got the "no more pkg mem" error again. I increased again: #define PKG_MEM_POOL_SIZE 32*1024*1024 This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg mem" error again.
So I don't think it is prudent to just keep increasing PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo pkg_mem, shouldn't this be a dynamic allocation if it's not within shmem and has no affect on core sip-router function? While the pdt_reload and pdt_list is going on (takes a few seconds to load and list), I don't see any problems with the sip-router executions. I guess I can just use the old fashion database query to look up routes instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio 1.5 will fallback to the system malloc, which don't have this size restrictions. For 3.x/ sr there is an option to build the MI interface with system malloc as well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec Author: Daniel-Constantin Mierla miconda@gmail.com Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc
- MI lib can be built independently of core with system malloc for pkg memory (see Makefile)
Both workarounds should help in your case.
But you're right, its probably more meaningfull to just execute a DB query instead of trying to pipe all this data through your (productive) sip server.
Regards,
Henning
Hi,
On 07/20/2010 11:21 AM, Henning Westerholt wrote:
On Monday 19 July 2010, JR Richardson wrote:
[..] When I added 180K records in the database, I got the "no more pkg mem" error again. I increased again: #define PKG_MEM_POOL_SIZE 32*1024*1024 This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg mem" error again.
So I don't think it is prudent to just keep increasing PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo pkg_mem, shouldn't this be a dynamic allocation if it's not within shmem and has no affect on core sip-router function? While the pdt_reload and pdt_list is going on (takes a few seconds to load and list), I don't see any problems with the sip-router executions. I guess I can just use the old fashion database query to look up routes instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio 1.5 will fallback to the system malloc, which don't have this size restrictions. For 3.x/ sr there is an option to build the MI interface with system malloc as well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec Author: Daniel-Constantin Mierlamiconda@gmail.com Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc - MI lib can be built independently of core with system malloc for pkg memory (see Makefile)
this is the option added to overcome the MI design limitations - while most of dumped structures are in shared memory, MI builds the output in PKG memory, resulting in insufficient space since shm is much bigger than pkg - also this sometime ended as well in lot of mem fragmentation for mi process in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile and reinstall.
Regards, Ramona
Both workarounds should help in your case.
But you're right, its probably more meaningfull to just execute a DB query instead of trying to pipe all this data through your (productive) sip server.
Regards,
Henning
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
2010/7/20 Elena-Ramona Modroiu ramona@asipto.com:
this is the option added to overcome the MI design limitations - while most of dumped structures are in shared memory, MI builds the output in PKG memory, resulting in insufficient space since shm is much bigger than pkg - also this sometime ended as well in lot of mem fragmentation for mi process in the past.
Hi Elena. This is also I'm a bit afraid of when using MI command to manage large ammount of records (LCR, locations, 'address' entries...). Wouldn't make sense MI to use shared memory instead of private memory?
Regards.
On Tuesday 20 July 2010, Iñaki Baz Castillo wrote:
this is the option added to overcome the MI design limitations - while most of dumped structures are in shared memory, MI builds the output in PKG memory, resulting in insufficient space since shm is much bigger than pkg - also this sometime ended as well in lot of mem fragmentation for mi process in the past.
Hi Elena. This is also I'm a bit afraid of when using MI command to manage large ammount of records (LCR, locations, 'address' entries...). Wouldn't make sense MI to use shared memory instead of private memory?
Hi Iñaki,
this would be possible, but its IMHO a bit wrong from a conceptional POV. This FIFO command execution belongs to the MI process, and it should not affect the working of other process which are busy routing SIP requests - they don't need to see this as all. One workaround is as discussed is to just use system malloc, another would be to partition the command execution in small steps like we do for data loading in several modules.
But as i said earlier, its maybe not the best idea in general to pipe the output of several houndred thousands rules through the MI interface, especially if the information is available more or less similar in the database.
Regards,
Henning
Hi,
On 07/20/2010 12:37 PM, Henning Westerholt wrote:
On Tuesday 20 July 2010, Iñaki Baz Castillo wrote:
this is the option added to overcome the MI design limitations - while most of dumped structures are in shared memory, MI builds the output in PKG memory, resulting in insufficient space since shm is much bigger than pkg - also this sometime ended as well in lot of mem fragmentation for mi process in the past.
Hi Elena. This is also I'm a bit afraid of when using MI command to manage large ammount of records (LCR, locations, 'address' entries...). Wouldn't make sense MI to use shared memory instead of private memory?
Hi Iñaki,
this would be possible, but its IMHO a bit wrong from a conceptional POV. This FIFO command execution belongs to the MI process, and it should not affect the working of other process which are busy routing SIP requests - they don't need to see this as all. One workaround is as discussed is to just use system malloc, another would be to partition the command execution in small steps like we do for data loading in several modules.
agree, using shm does not bring benefits, but some drawbacks since shm involves locking for access, thus it affects the other processes (e.g., sip workers). It is better with system malloc, maybe this should be made default for MI.
Regards, Ramona
But as i said earlier, its maybe not the best idea in general to pipe the output of several houndred thousands rules through the MI interface, especially if the information is available more or less similar in the database.
Regards,
Henning
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
2010/7/20 Henning Westerholt henning.westerholt@1und1.de:
Hi Elena. This is also I'm a bit afraid of when using MI command to manage large ammount of records (LCR, locations, 'address' entries...). Wouldn't make sense MI to use shared memory instead of private memory?
Hi Iñaki,
this would be possible, but its IMHO a bit wrong from a conceptional POV. This FIFO command execution belongs to the MI process, and it should not affect the working of other process which are busy routing SIP requests - they don't need to see this as all.
Sure, you are right.
One workaround is as discussed is to just use system malloc,
It seems a good solution.
another would be to partition the command execution in small steps like we do for data loading in several modules.
Yes, but this involves MI process to be designed for that, and it's not the case, right? This is, in case the retrieved information exceeds some size it should be automatically partitioned in various steps which are sent to the MI process one by one.
But as i said earlier, its maybe not the best idea in general to pipe the output of several houndred thousands rules through the MI interface, especially if the information is available more or less similar in the database.
Sure, but what about when using just memory (as in db_mode=0 for usrloc)? Also there are other modules (as LCR) which just use the database to load initial values and later those values or their attributes are modified in memory (i.e: LCR gateways status).
Regards.
On Tuesday 20 July 2010, Iñaki Baz Castillo wrote:
another would be to partition the command execution in small steps like we do for data loading in several modules.
Yes, but this involves MI process to be designed for that, and it's not the case, right? This is, in case the retrieved information exceeds some size it should be automatically partitioned in various steps which are sent to the MI process one by one.
Hey Inaki,
yes, this is not the case at the moment.
But as i said earlier, its maybe not the best idea in general to pipe the output of several houndred thousands rules through the MI interface, especially if the information is available more or less similar in the database.
Sure, but what about when using just memory (as in db_mode=0 for usrloc)? Also there are other modules (as LCR) which just use the database to load initial values and later those values or their attributes are modified in memory (i.e: LCR gateways status).
This are indeed valid use cases, did not thought about it. Also the dialog module outputs some bigger list, but normally not several 100k. ;-)
Regards,
Henning
[..] When I added 180K records in the database, I got the "no more pkg mem" error again. I increased again: #define PKG_MEM_POOL_SIZE 32*1024*1024 This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg mem" error again.
So I don't think it is prudent to just keep increasing PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo pkg_mem, shouldn't this be a dynamic allocation if it's not within shmem and has no affect on core sip-router function? While the pdt_reload and pdt_list is going on (takes a few seconds to load and list), I don't see any problems with the sip-router executions. I guess I can just use the old fashion database query to look up routes instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio 1.5 will fallback to the system malloc, which don't have this size restrictions. For 3.x/ sr there is an option to build the MI interface with system malloc as well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec Author: Daniel-Constantin Mierlamiconda@gmail.com Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc
- MI lib can be built independently of core with system malloc for pkg memory (see Makefile)
this is the option added to overcome the MI design limitations - while most of dumped structures are in shared memory, MI builds the output in PKG memory, resulting in insufficient space since shm is much bigger than pkg - also this sometime ended as well in lot of mem fragmentation for mi process in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile and reinstall.
Regards, Ramona
Hi Ramona,
Thanks for your suggestions. I started over with default mem parameters in config.h and compiled with: ## uncomment next line for using system malloc with MI DEFS+= -DMI_SYSTEM_MALLOC
I loaded the pdt database table with 360K records and also started kamailio with '-m 512' but the table still has a max record limitation, will not load propperly and I get these errors: 0(22456) ERROR: pdt [pdtree.c:283]: bad parameters 0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
Of course wihout a propper table load, pdt_list does not produce any results (but i don;t get the pkg_mem error any more).
sip-router2:~# kamctl fifo get_statistics all shmem:total_size = 536870912 shmem:used_size = 1229472 shmem:real_used_size = 1247560 shmem:max_used_size = 1255376 shmem:free_size = 535623352
There seems to still be a bottleneck with shmem, even through I have 512M+ available, I only seem to be using ~1.2M. Is there an internal shmem hard limit that can be adjusted?
Thanks
JR
On 07/20/2010 05:06 PM, JR Richardson wrote:
[..] When I added 180K records in the database, I got the "no more pkg mem" error again. I increased again: #define PKG_MEM_POOL_SIZE 32*1024*1024 This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg mem" error again.
So I don't think it is prudent to just keep increasing PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo pkg_mem, shouldn't this be a dynamic allocation if it's not within shmem and has no affect on core sip-router function? While the pdt_reload and pdt_list is going on (takes a few seconds to load and list), I don't see any problems with the sip-router executions. I guess I can just use the old fashion database query to look up routes instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio 1.5 will fallback to the system malloc, which don't have this size restrictions. For 3.x/ sr there is an option to build the MI interface with system malloc as well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec Author: Daniel-Constantin Mierlamiconda@gmail.com Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc - MI lib can be built independently of core with system malloc for pkg memory (see Makefile)this is the option added to overcome the MI design limitations - while most of dumped structures are in shared memory, MI builds the output in PKG memory, resulting in insufficient space since shm is much bigger than pkg - also this sometime ended as well in lot of mem fragmentation for mi process in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile and reinstall.
Regards, Ramona
Hi Ramona,
Thanks for your suggestions. I started over with default mem parameters in config.h and compiled with: ## uncomment next line for using system malloc with MI DEFS+= -DMI_SYSTEM_MALLOC
I loaded the pdt database table with 360K records and also started kamailio with '-m 512' but the table still has a max record limitation, will not load propperly and I get these errors: 0(22456) ERROR: pdt [pdtree.c:283]: bad parameters 0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
Of course wihout a propper table load, pdt_list does not produce any results (but i don;t get the pkg_mem error any more).
sip-router2:~# kamctl fifo get_statistics all shmem:total_size = 536870912 shmem:used_size = 1229472 shmem:real_used_size = 1247560 shmem:max_used_size = 1255376 shmem:free_size = 535623352
There seems to still be a bottleneck with shmem, even through I have 512M+ available, I only seem to be using ~1.2M. Is there an internal shmem hard limit that can be adjusted?
Hi JR,
There is no limit hard-coded, I just loaded 1mio records in pdt (sdomain=*, prefix=1mio..2mio, domain=test.org) and statistics show:
shmem:total_size = 268435456 shmem:used_size = 26594256 shmem:real_used_size = 80148548 shmem:max_used_size = 80148548 shmem:free_size = 188286908
So the records are reflected in shared memory. What is the output of "kamailio -V"? What are the parameters for pdt?
Regards, Ramona
On Tue, Jul 20, 2010 at 12:04 PM, Elena-Ramona Modroiu ramona@asipto.com wrote:
On 07/20/2010 05:06 PM, JR Richardson wrote:
[..] When I added 180K records in the database, I got the "no more pkg mem" error again. I increased again: #define PKG_MEM_POOL_SIZE 32*1024*1024 This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg mem" error again.
So I don't think it is prudent to just keep increasing PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo pkg_mem, shouldn't this be a dynamic allocation if it's not within shmem and has no affect on core sip-router function? While the pdt_reload and pdt_list is going on (takes a few seconds to load and list), I don't see any problems with the sip-router executions. I guess I can just use the old fashion database query to look up routes instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio 1.5 will fallback to the system malloc, which don't have this size restrictions. For 3.x/ sr there is an option to build the MI interface with system malloc as well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec Author: Daniel-Constantin Mierlamiconda@gmail.com Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc
- MI lib can be built independently of core with system malloc for pkg memory (see Makefile)
this is the option added to overcome the MI design limitations - while most of dumped structures are in shared memory, MI builds the output in PKG memory, resulting in insufficient space since shm is much bigger than pkg
also this sometime ended as well in lot of mem fragmentation for mi process in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile and reinstall.
Regards, Ramona
Hi Ramona,
Thanks for your suggestions. I started over with default mem parameters in config.h and compiled with: ## uncomment next line for using system malloc with MI DEFS+= -DMI_SYSTEM_MALLOC
I loaded the pdt database table with 360K records and also started kamailio with '-m 512' but the table still has a max record limitation, will not load propperly and I get these errors: 0(22456) ERROR: pdt [pdtree.c:283]: bad parameters 0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
Of course wihout a propper table load, pdt_list does not produce any results (but i don;t get the pkg_mem error any more).
sip-router2:~# kamctl fifo get_statistics all shmem:total_size = 536870912 shmem:used_size = 1229472 shmem:real_used_size = 1247560 shmem:max_used_size = 1255376 shmem:free_size = 535623352
There seems to still be a bottleneck with shmem, even through I have 512M+ available, I only seem to be using ~1.2M. Is there an internal shmem hard limit that can be adjusted?
Hi JR,
There is no limit hard-coded, I just loaded 1mio records in pdt (sdomain=*, prefix=1mio..2mio, domain=test.org) and statistics show:
shmem:total_size = 268435456 shmem:used_size = 26594256 shmem:real_used_size = 80148548 shmem:max_used_size = 80148548 shmem:free_size = 188286908
So the records are reflected in shared memory. What is the output of "kamailio -V"? What are the parameters for pdt?
Regards, Ramona
sip-router2:~# kamailio -V version: kamailio 3.0.2 (i386/linux) a6141a flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. @(#) $Id$ main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2 sip-router2:~#
# ----- pdt params ----- modparam("pdt", "db_url", "mysql://openserro:openserro@localhost/openser") modparam("pdt", "char_list", "0123456789") modparam("pdt", "check_domain", 0) modparam("pdt", "fetch_rows", 5000)
Here is an interesting observation: When I have 300K+ records in the pdt table, it seems like it will not load properly and the shmem is very low, as expected without a large memory pool needed to hold the records. Statistics show: shmem:total_size = 33554432 shmem:used_size = 1229472 shmem:real_used_size = 1247512 shmem:max_used_size = 1247512 shmem:free_size = 32306920 shmem:fragments = 11
When I decrease the table recrods to 240K, the complete table loads ok and shmem reflect properly: shmem:total_size = 33554432 shmem:used_size = 6349784 shmem:real_used_size = 8501144 shmem:max_used_size = 8501144 shmem:free_size = 25053288 shmem:fragments = 1
So maybe I'm not having an issue with shmem at all but a loading problem with the pdt tabe after a certain record count.
I changed the "modparam("pdt", "fetch_rows", 5000)" to see if that would help. When the table loads properly is is quicker than with the default 1000 rows, but that is the only difference I experience. When the pdt table has more than 300K records, it just does not load into memory.
Could there be a pdt module timeout that is being met when loading? For instance, if it takes more than N seconds to load, consider a fault and not load?
sip-router2:/etc/kamailio# mysql -V mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2 sip-router2:/etc/kamailio#
Thanks.
JR
Hi there,
Hopefully there are two simple answers for these. Running Kamailio 1.4.2.
1. Is there a variable that holds the current UNIX timestamp? Either when called in a route() or method.
2. What is the variable that is equivalent to the "callid" in the ACC module? Or better yet, what is the best way to retrieve that ACC field in the config? I need a unique ID for the complete call ie. From INVITE to BYE...
I am calling out to a new external CDR "gatherer of sorts" and would like this information to be present...
Thanks,
-graham
On 07/20/2010 10:12 PM, Graham Wooden wrote:
- Is there a variable that holds the current UNIX timestamp? Either when
called in a route() or method.
Yep:
http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:1.4.x#unix_time_st...
- What is the variable that is equivalent to the "callid" in the ACC
module? Or better yet, what is the best way to retrieve that ACC field in the config? I need a unique ID for the complete call ie. From INVITE to BYE...
$ci --
http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:1.4.x#call-id
Ah - perfect. Thank you Alex.
-graham
On 7/20/10 9:15 PM, "Alex Balashov" abalashov@evaristesys.com wrote:
On 07/20/2010 10:12 PM, Graham Wooden wrote:
- Is there a variable that holds the current UNIX timestamp? Either when
called in a route() or method.
Yep:
http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:1.4.x#unix_time_st... p
- What is the variable that is equivalent to the "callid" in the ACC
module? Or better yet, what is the best way to retrieve that ACC field in the config? I need a unique ID for the complete call ie. From INVITE to BYE...
$ci --
http://www.kamailio.org/dokuwiki/doku.php/pseudovariables:1.4.x#call-id
Hi there,
I wanted to share with the folks on this list a simple way to get some simple call accounting.
My goal was to have a simple, one line in a DB entry, depicting when the call started (epoch and date/time), when the call ended (epoch and date/time), the call-to number and the calling-from, and finally the direction of the call.
I found that ACC was too buggy with this - meaning, that about 1/2 of the time the acc_extra and acc_to would actually be correct & it had a bunch of extra stuff that I really didn't need or care about. Maybe the buggy-ness is because I am running kamailio 1.4.2 - ?
I used CDRTool and Radius a while back, on my first production machine, but after 2 years of service I had to fail over to my secondary and could never get that combination going again. I didn't keep good notes when I struggled to get it working the first time.
So, with that - for the folks out there that want a simple, no thrills call accounting, from which you can add and tailor to your environment - here is what I did:
First, create a table in your openser database. For this example, I called it "rawlines". Here is the mySQL schema:
+------------+-------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +------------+-------------+------+-----+---------+----------------+ | id | int(11) | NO | PRI | NULL | auto_increment | | timenow | datetime | NO | | NULL | | | epoch | varchar(10) | NO | | NULL | | | endtimenow | datetime | NO | | NULL | | | endepoch | varchar(10) | NO | | NULL | | | callid | varchar(64) | NO | MUL | NULL | | | thisside | varchar(28) | NO | | NULL | | | otherside | varchar(28) | NO | | NULL | | | direction | varchar(8) | NO | | NULL | | +------------+-------------+------+-----+---------+----------------+
Then, the placement the SQL calls. Of course, everyone's cfg file is different, as well as you may have a better place inside ... Here is what I am doing:
route{ . . if (is_method("BYE")) { . . avp_db_query("UPDATE rawlines set endtimenow=NOW(), endepoch='$Ts' WHERE callid='$ci'"); . . } . . if (!lookup("location")) { . . avp_db_query("INSERT INTO rawlines (timenow,epoch,callid,thisside,otherside,direction) VALUES(NOW(),'$Ts','$ci','$fU','$rU','outbound')"); . . } else { avp_db_query("INSERT INTO rawlines (timenow,epoch,callid,thisside,otherside,direction) VALUES(NOW(),'$Ts','$ci','$tU','$Au','inbound')"); . . }
That's pretty much it. Your results will be a single, readable line for each call. Now, is this perfect - heck no. For instance, for calls that don't complete (like just ring/hangup with no BYE), you will get 0000-00-00 for "endtimenow" which could be misleading thinking the call is still happening.
I like depicting both epoch and readable date/time. I use the epoch to calculate billing time and use the readable for quick reference for users.
My main goal was to get something quick and friendly out there to start expanding on. Fits my needs currently and maybe it will help benefit someone else out there - or better yet, come with a better way to achieve this approach.
-graham
That's pretty much it. Your results will be a single, readable line for each call. Now, is this perfect - heck no. For instance, for calls that don't complete (like just ring/hangup with no BYE), you will get 0000-00-00 for "endtimenow" which could be misleading thinking the call is still happening.
Just to follow up - I meant to add to this by saying that thus far, any incomplete's that have occurred (ie. Call not being answered or not active in the dialog table), I was able to reference all of them in the missed_calls table. So with that, I have a couple worker perl scripts that go out from a centralized box that picks up and collates all my data from the proxies. Not once, yet, had a call become a strangler or have incomplete/inaccurate data.
Your mileage may vary.
-graham
For comparison and contrast: We handle this in PostgreSQL by having Kamailio write to the 'acc' table, and adding triggers that operate on 'acc' events and populate or update another table, 'cdr', with actual CDRs in the desirable one-row-per-call way.
The advantage to doing it that way versus in route script is that it's transactional in nature; an error will result in all the cascading queries being rolled back. It also makes it much easier to insinuate business layer stuff into the cascade, or extend the trigger cascade with additional logic, because PL/PgSQL provides certain features of a general-purpose programming environment that Kamailio does not, given its domain specificity.
It also eliminates unnecessary data interchange between the Kamailio process and the RDBM, given that most of the work is done in the database itself, causing additional latency and being a possible source of data corruption.
In short, it lets us have a relatively clean Kamailio config and deputise all the business layer stuff to the database backend in a very hierarchical, maintainable way. That way, the Kamailio route script can stick to doing what it does best: operating on SIP messages.
Just as an addendum to Alex's comment, I'm doing the exact same thing with MySQL triggers.
Most strange situations can be worked around using some parameter in ACC or DIALOG modules (if working together).
Our main cause is that we have many proxies and a central DB, so the work is performed by only one process that can be measured and optimized alone.
Uriel
On Sun, Aug 8, 2010 at 11:00 PM, Alex Balashov abalashov@evaristesys.com wrote:
For comparison and contrast: We handle this in PostgreSQL by having Kamailio write to the 'acc' table, and adding triggers that operate on 'acc' events and populate or update another table, 'cdr', with actual CDRs in the desirable one-row-per-call way.
The advantage to doing it that way versus in route script is that it's transactional in nature; an error will result in all the cascading queries being rolled back. It also makes it much easier to insinuate business layer stuff into the cascade, or extend the trigger cascade with additional logic, because PL/PgSQL provides certain features of a general-purpose programming environment that Kamailio does not, given its domain specificity.
It also eliminates unnecessary data interchange between the Kamailio process and the RDBM, given that most of the work is done in the database itself, causing additional latency and being a possible source of data corruption.
In short, it lets us have a relatively clean Kamailio config and deputise all the business layer stuff to the database backend in a very hierarchical, maintainable way. That way, the Kamailio route script can stick to doing what it does best: operating on SIP messages.
-- Alex Balashov - Principal Evariste Systems LLC 1170 Peachtree Street 12th Floor, Suite 1200 Atlanta, GA 30309 Tel: +1-678-954-0670 Fax: +1-404-961-1892 Web: http://www.evaristesys.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
I appreciate yours and Alex's' input and insight.
While I don't have a many to one (proxy to DB), but rather one to one, I have a few worker scripts from a central reporting box that pulls those records and anything in the missed_calls and dialog and wraps up any collation. I don't use anything in ACC...
-graham
On 8/9/10 8:10 AM, "Uriel Rozenbaum" uriel.rozenbaum@gmail.com wrote:
Just as an addendum to Alex's comment, I'm doing the exact same thing with MySQL triggers.
Most strange situations can be worked around using some parameter in ACC or DIALOG modules (if working together).
Our main cause is that we have many proxies and a central DB, so the work is performed by only one process that can be measured and optimized alone.
Uriel
On Sun, Aug 8, 2010 at 11:00 PM, Alex Balashov abalashov@evaristesys.com wrote:
For comparison and contrast: We handle this in PostgreSQL by having Kamailio write to the 'acc' table, and adding triggers that operate on 'acc' events and populate or update another table, 'cdr', with actual CDRs in the desirable one-row-per-call way.
The advantage to doing it that way versus in route script is that it's transactional in nature; an error will result in all the cascading queries being rolled back. It also makes it much easier to insinuate business layer stuff into the cascade, or extend the trigger cascade with additional logic, because PL/PgSQL provides certain features of a general-purpose programming environment that Kamailio does not, given its domain specificity.
It also eliminates unnecessary data interchange between the Kamailio process and the RDBM, given that most of the work is done in the database itself, causing additional latency and being a possible source of data corruption.
In short, it lets us have a relatively clean Kamailio config and deputise all the business layer stuff to the database backend in a very hierarchical, maintainable way. That way, the Kamailio route script can stick to doing what it does best: operating on SIP messages.
-- Alex Balashov - Principal Evariste Systems LLC 1170 Peachtree Street 12th Floor, Suite 1200 Atlanta, GA 30309 Tel: +1-678-954-0670 Fax: +1-404-961-1892 Web: http://www.evaristesys.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 07/20/2010 07:45 PM, JR Richardson wrote:
On Tue, Jul 20, 2010 at 12:04 PM, Elena-Ramona Modroiu ramona@asipto.com wrote:
On 07/20/2010 05:06 PM, JR Richardson wrote:
[..] When I added 180K records in the database, I got the "no more pkg mem" error again. I increased again: #define PKG_MEM_POOL_SIZE 32*1024*1024 This allowed me to execute pdt_list with 180K records loaded.
I increased database record count to 240K and got the "no more pkg mem" error again.
So I don't think it is prudent to just keep increasing PKG_MEM_POOL_SIZE. Is this an architectural limitation with fifo pkg_mem, shouldn't this be a dynamic allocation if it's not within shmem and has no affect on core sip-router function? While the pdt_reload and pdt_list is going on (takes a few seconds to load and list), I don't see any problems with the sip-router executions. I guess I can just use the old fashion database query to look up routes instead of fifo pdt_list.
Hi Jr,
if you don't use the PKG_MALLOC MM in your system at least the kamailio 1.5 will fallback to the system malloc, which don't have this size restrictions. For 3.x/ sr there is an option to build the MI interface with system malloc as well, it this commit:
commit 85f2302e8448ee20edb12bc3b4e2911ca3b111ec Author: Daniel-Constantin Mierlamiconda@gmail.com Date: Mon Jun 29 12:34:08 2009 +0200
lib/kmi: option to build it with system malloc - MI lib can be built independently of core with system malloc forpkg memory (see Makefile)
this is the option added to overcome the MI design limitations - while most of dumped structures are in shared memory, MI builds the output in PKG memory, resulting in insufficient space since shm is much bigger than pkg
also this sometime ended as well in lot of mem fragmentation for mi process in the past.
It needs to edit lib/kmi/Makefile and enable system malloc, recompile and reinstall.
Regards, Ramona
Hi Ramona,
Thanks for your suggestions. I started over with default mem parameters in config.h and compiled with: ## uncomment next line for using system malloc with MI DEFS+= -DMI_SYSTEM_MALLOC
I loaded the pdt database table with 360K records and also started kamailio with '-m 512' but the table still has a max record limitation, will not load propperly and I get these errors: 0(22456) ERROR: pdt [pdtree.c:283]: bad parameters 0(22456) INFO: pdt [pdt.c:490]: no prefix found in [2000171212]
Of course wihout a propper table load, pdt_list does not produce any results (but i don;t get the pkg_mem error any more).
sip-router2:~# kamctl fifo get_statistics all shmem:total_size = 536870912 shmem:used_size = 1229472 shmem:real_used_size = 1247560 shmem:max_used_size = 1255376 shmem:free_size = 535623352
There seems to still be a bottleneck with shmem, even through I have 512M+ available, I only seem to be using ~1.2M. Is there an internal shmem hard limit that can be adjusted?
Hi JR,
There is no limit hard-coded, I just loaded 1mio records in pdt (sdomain=*, prefix=1mio..2mio, domain=test.org) and statistics show:
shmem:total_size = 268435456 shmem:used_size = 26594256 shmem:real_used_size = 80148548 shmem:max_used_size = 80148548 shmem:free_size = 188286908
So the records are reflected in shared memory. What is the output of "kamailio -V"? What are the parameters for pdt?
Regards, Ramona
sip-router2:~# kamailio -V version: kamailio 3.0.2 (i386/linux) a6141a flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. @(#) $Id$ main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2 sip-router2:~#
# ----- pdt params ----- modparam("pdt", "db_url", "mysql://openserro:openserro@localhost/openser") modparam("pdt", "char_list", "0123456789") modparam("pdt", "check_domain", 0) modparam("pdt", "fetch_rows", 5000)
Here is an interesting observation: When I have 300K+ records in the pdt table, it seems like it will not load properly and the shmem is very low, as expected without a large memory pool needed to hold the records. Statistics show: shmem:total_size = 33554432 shmem:used_size = 1229472 shmem:real_used_size = 1247512 shmem:max_used_size = 1247512 shmem:free_size = 32306920 shmem:fragments = 11
When I decrease the table recrods to 240K, the complete table loads ok and shmem reflect properly: shmem:total_size = 33554432 shmem:used_size = 6349784 shmem:real_used_size = 8501144 shmem:max_used_size = 8501144 shmem:free_size = 25053288 shmem:fragments = 1
So maybe I'm not having an issue with shmem at all but a loading problem with the pdt tabe after a certain record count.
I changed the "modparam("pdt", "fetch_rows", 5000)" to see if that would help. When the table loads properly is is quicker than with the default 1000 rows, but that is the only difference I experience. When the pdt table has more than 300K records, it just does not load into memory.
Could there be a pdt module timeout that is being met when loading? For instance, if it takes more than N seconds to load, consider a fault and not load?
sip-router2:/etc/kamailio# mysql -V mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2 sip-router2:/etc/kamailio#
try to play with mysql module timeout parameter in case the records are not loaded fast enough:
http://kamailio.org/docs/modules/stable/modules/db_mysql.html#id2928400
Regards, Ramona
sip-router2:~# kamailio -V version: kamailio 3.0.2 (i386/linux) a6141a flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. @(#) $Id$ main.c compiled on 09:25:15 Jul 20 2010 with gcc 4.3.2 sip-router2:~#
# ----- pdt params ----- modparam("pdt", "db_url", "mysql://openserro:openserro@localhost/openser") modparam("pdt", "char_list", "0123456789") modparam("pdt", "check_domain", 0) modparam("pdt", "fetch_rows", 5000)
Here is an interesting observation: When I have 300K+ records in the pdt table, it seems like it will not load properly and the shmem is very low, as expected without a large memory pool needed to hold the records. Statistics show: shmem:total_size = 33554432 shmem:used_size = 1229472 shmem:real_used_size = 1247512 shmem:max_used_size = 1247512 shmem:free_size = 32306920 shmem:fragments = 11
When I decrease the table recrods to 240K, the complete table loads ok and shmem reflect properly: shmem:total_size = 33554432 shmem:used_size = 6349784 shmem:real_used_size = 8501144 shmem:max_used_size = 8501144 shmem:free_size = 25053288 shmem:fragments = 1
So maybe I'm not having an issue with shmem at all but a loading problem with the pdt tabe after a certain record count.
I changed the "modparam("pdt", "fetch_rows", 5000)" to see if that would help. When the table loads properly is is quicker than with the default 1000 rows, but that is the only difference I experience. When the pdt table has more than 300K records, it just does not load into memory.
Could there be a pdt module timeout that is being met when loading? For instance, if it takes more than N seconds to load, consider a fault and not load?
sip-router2:/etc/kamailio# mysql -V mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (i486) using readline 5.2 sip-router2:/etc/kamailio#
try to play with mysql module timeout parameter in case the records are not loaded fast enough:
http://kamailio.org/docs/modules/stable/modules/db_mysql.html#id2928400
Regards, Ramona
Ok, I figured out that sr was taking about 9-10 seconds to load 300K records so I increased: # ----- db_mysql params ----- modparam("db_mysql", "timeout_interval", 20)
And found that the table would load, so I started increasing the record count by 60K, then the fault condition occurred again at around 400K records, then went away by itself. I got all the way to 760K records without a problem then hit another wall at 800K records.
I think there is something flaky going on, I can not reproduce it reliable, sometimes the table loads just fine and other times it does not load at all.
I started monitoring mysqld. When I restart sr or issue pdt_reload and msqld has a load, the table seems to load fine, regardless of the table size or "timeout_interval" setting. When the problem is occurring, myslqd has zero usage. When the issue was occurring, I restarted the mysql process and the table loaded into sr. I'm now suspecting I have a buggy mysqld so I'm going to try to update that and see if I can reproduce the problem. So for now I can't isolate the issue, but I'm still working on it.
I do see a difference with the "timeout_interval" during sr restarts, the longer the time the better it does. Also there is a time difference when loading the table, sometimes it takes 30 seconds, sometimes 90 seconds. I think maybe there is a combination of issues going on here. I need to rebuild a fresh install and test a bit more methodically to find a root cause. Also I want to grab a newer version of mysql-server.
I'll update with my findings.
Thanks.
JR
On Tuesday 20 July 2010, JR Richardson wrote:
[..] Here is an interesting observation: When I have 300K+ records in the pdt table, it seems like it will not load properly and the shmem is very low, as expected without a large memory pool needed to hold the records. Statistics show: shmem:total_size = 33554432 [..] When I decrease the table recrods to 240K, the complete table loads ok and shmem reflect properly: shmem:total_size = 33554432
Hi JR,
it seems you've a problem with your SHM pool size configuration.
33.554.432 ~ 34MB
It should normally output the proper size, like this:
test-proxy:~# grep "MEM" /etc/default/kamailio MEMORY=1024
test-proxy:~# kamctl fifo get_statistics all | grep total_size shmem:total_size = 1073741824
1073.741.824 = 1024 MB
Cheers,
Henning
Hi,
On 07/16/2010 11:51 PM, JR Richardson wrote:
Hi All,
I loaded up the PDT database with about 35K records and when I issue the commad "kamctl fifo pdt_list" I get:
3(3018) ERROR:<core> [tree.c:139]: no more pkg mem 3(3018) ERROR: mi_fifo [fifo_fnc.c:509]: command (pdt_list) processing failed
Searching around I found the http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory which suggest I adjust the pkg mem size in config.h.
In config.h, I found:
/*used only if PKG_MALLOC is defined*/ #define PKG_MEM_POOL_SIZE 4*1024*1024
So is this what I am supposed to adjust? Maybe try: #define PKG_MEM_POOL_SIZE 4*2048*2048 or #define PKG_MEM_POOL_SIZE 8*1024*1024
Which one would be a logical adjustment? Also, is there a correlation between pkg mem and database record size as related to pdt_list?
The idea is to have a few 100K records loaded in kamailio and be able to perform "kamctl fifo pdt_list | grep 512345" to show this prefix route. But without enough memory, doesn't work.
this is a limitation of MI control interface (as written in other email, there are workarounds).
Anyway, pdt_list can take the prefix as parameter, so you do not need to list entire tree and do grep, just list desired prefixes: http://kamailio.org/docs/modules/stable/modules_k/pdt.html#id2854918
Regards, Ramona
-
Alex Balashov -
Elena-Ramona Modroiu -
Graham Wooden -
Henning Westerholt -
Iñaki Baz Castillo -
JR Richardson -
marius zbihlei -
Uriel Rozenbaum