7 Nov
2005
7 Nov
'05
4:23 p.m.
Hello,
I am having some problems configuring memory for SER..
Here is what I have in my test instance:
Server: Linux x64
Real Memory: 4GB
Database: MySQL/NDB
SER Shared memory: 256M
Children: 16
Subscribers: 40000
In test environment:
Registrations per sec: 120
Call per second: 120
When the SER process was started last week, it had only 4000 subscribers
provisioned. Then another 36000 subscribers were provisioned. The SER
was running fine. But when I decided to bounce SER, it would not
restart.
It seems from the log file that SER ran out of memory trying to load the
LOCATION database into memory upon startup, failing around row #3500.
LOCATION table had in excess of 10000 records at that point.
The related piece of the logfile is below..
I wonder whether it is a matter of configuration or is it a bug.. The
system definitely seem to have enough memory to store those few tens of
thousands of records.. Even at 1K each, it amounts to nothing..
Especially given that at runtime it handles that amount of location info
easily.. It's only a startup that seems to be a problem. I tried to play
with giving it more or less of shared memory.. It does not seem to be
relevant..
For now I just have to clean up the LOCATION table each time before
restarting SER. Obviously, it is not a nice workaround..
Thanks,
Sergei
Mon Nov 7 12:29:59.281894 200: 0
(0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so lookup
Mon Nov 7 12:29:59.281964 200: 0(0):DEBUG:get_connection(): Connection
not found in the pool
Mon Nov 7 12:29:59.282329 200: 0(0):DEBUG:preload_udomain(): Table is
empty
Mon Nov 7 12:29:59.282342 200: 0(0):DEBUG:release_connection():
Removing connection from the pool
Mon Nov 7 12:29:59.282360 200: 0
(0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so save
Mon Nov 7 12:29:59.282414 200: 0(0):DEBUG:get_connection(): Connection
not found in the pool
Mon Nov 7 12:29:59.357157 200: 0(0):WARNING:convert_row(): No memory
left
Mon Nov 7 12:29:59.357189 200: 0(0):WARNING:convert_rows(): Error while
converting row #3445
Mon Nov 7 12:29:59.357413 200: 0(0):WARNING:convert_result(): Error
while converting rows
Mon Nov 7 12:29:59.357424 200: 0(0):WARNING:store_result(): Error while
converting result
Mon Nov 7 12:29:59.357432 200: 0(0):WARNING:preload_udomain(): Error
while doing db_query
Mon Nov 7 12:29:59.357438 200: 0(0):WARNING:register_udomain(): Error
while preloading domain 'location'
Mon Nov 7 12:29:59.357447 200: 0(0):DEBUG:release_connection():
Removing connection from the pool
Mon Nov 7 12:29:59.357856 200: 0(0):WARNING:domain_fixup(): Error while
registering domain
ERROR: error -1 while trying to fix configuration
Mon Nov 7 12:29:59.357901 200: 0(0)::DEBUG: tm_shutdown : start
Mon Nov 7 12:29:59.357913 200: 0(0)::DEBUG: unlink_timer_lists :
emptying DELETE list
Mon Nov 7 12:29:59.357920 200: 0(0)::DEBUG: tm_shutdown : emptying hash
table
Mon Nov 7 12:29:59.359085 200: 0(0)::DEBUG: tm_shutdown : releasing
timers
Mon Nov 7 12:29:59.359108 200: 0(0)::DEBUG: tm_shutdown : removing
semaphores
Mon Nov 7 12:29:59.359114 200: 0(0)::DEBUG: tm_shutdown : destroying
tmcb lists
Mon Nov 7 12:29:59.359122 200: 0(0)::DEBUG: tm_shutdown : done
Mon Nov 7 12:29:59.359143 200: 0(11662):DEBUG:shm_mem_destroy
Mon Nov 7 12:29:59.359154 200: 0(11662):DEBUG:destroying the shared
memory lock
7 Nov
7 Nov
6:04 p.m.
Hello,
try to start SER with more shared memory next time (with that amount of memory
you use only a fraction of the memory). I'm not 100% sure but I think SER
uses shared memory when loading the location table. Thus it runs probably
pretty soon out of shared memory with such a big subscriber population.
Nils
On Monday 07 November 2005 22:23, yserge wrote:
Hello,
I am having some problems configuring memory for SER..
Here is what I have in my test instance:
Server: Linux x64
Real Memory: 4GB
Database: MySQL/NDB
SER Shared memory: 256M
Children: 16
Subscribers: 40000
In test environment:
Registrations per sec: 120
Call per second: 120
When the SER process was started last week, it had only 4000 subscribers
provisioned. Then another 36000 subscribers were provisioned. The SER
was running fine. But when I decided to bounce SER, it would not
restart.
It seems from the log file that SER ran out of memory trying to load the
LOCATION database into memory upon startup, failing around row #3500.
LOCATION table had in excess of 10000 records at that point.
The related piece of the logfile is below..
I wonder whether it is a matter of configuration or is it a bug.. The
system definitely seem to have enough memory to store those few tens of
thousands of records.. Even at 1K each, it amounts to nothing..
Especially given that at runtime it handles that amount of location info
easily.. It's only a startup that seems to be a problem. I tried to play
with giving it more or less of shared memory.. It does not seem to be
relevant..
For now I just have to clean up the LOCATION table each time before
restarting SER. Obviously, it is not a nice workaround..
Thanks,
Sergei
Mon Nov 7 12:29:59.281894 200: 0
(0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so lookup
Mon Nov 7 12:29:59.281964 200: 0(0):DEBUG:get_connection(): Connection
not found in the pool
Mon Nov 7 12:29:59.282329 200: 0(0):DEBUG:preload_udomain(): Table is
empty
Mon Nov 7 12:29:59.282342 200: 0(0):DEBUG:release_connection():
Removing connection from the pool
Mon Nov 7 12:29:59.282360 200: 0
(0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so save
Mon Nov 7 12:29:59.282414 200: 0(0):DEBUG:get_connection(): Connection
not found in the pool
Mon Nov 7 12:29:59.357157 200: 0(0):WARNING:convert_row(): No memory
left
Mon Nov 7 12:29:59.357189 200: 0(0):WARNING:convert_rows(): Error while
converting row #3445
Mon Nov 7 12:29:59.357413 200: 0(0):WARNING:convert_result(): Error
while converting rows
Mon Nov 7 12:29:59.357424 200: 0(0):WARNING:store_result(): Error while
converting result
Mon Nov 7 12:29:59.357432 200: 0(0):WARNING:preload_udomain(): Error
while doing db_query
Mon Nov 7 12:29:59.357438 200: 0(0):WARNING:register_udomain(): Error
while preloading domain 'location'
Mon Nov 7 12:29:59.357447 200: 0(0):DEBUG:release_connection():
Removing connection from the pool
Mon Nov 7 12:29:59.357856 200: 0(0):WARNING:domain_fixup(): Error while
registering domain
ERROR: error -1 while trying to fix configuration
Mon Nov 7 12:29:59.357901 200: 0(0)::DEBUG: tm_shutdown : start
Mon Nov 7 12:29:59.357913 200: 0(0)::DEBUG: unlink_timer_lists :
emptying DELETE list
Mon Nov 7 12:29:59.357920 200: 0(0)::DEBUG: tm_shutdown : emptying hash
table
Mon Nov 7 12:29:59.359085 200: 0(0)::DEBUG: tm_shutdown : releasing
timers
Mon Nov 7 12:29:59.359108 200: 0(0)::DEBUG: tm_shutdown : removing
semaphores
Mon Nov 7 12:29:59.359114 200: 0(0)::DEBUG: tm_shutdown : destroying
tmcb lists
Mon Nov 7 12:29:59.359122 200: 0(0)::DEBUG: tm_shutdown : done
Mon Nov 7 12:29:59.359143 200: 0(11662):DEBUG:shm_mem_destroy
Mon Nov 7 12:29:59.359154 200: 0(11662):DEBUG:destroying the shared
memory lock
_______________________________________________
Serusers mailing list
serusers(a)lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serusers
6:21 p.m.
That's what I first tried.. I tried to start with 512M.. No difference
whatsoever.
Then, checking the source code, it looks like it is the "PKG_MALLOC"
macro that fails, which , AFAIU, is not a shared memory.
Sergei
Hello,
try to start SER with more shared memory next time (with that amount of memory
you use only a fraction of the memory). I'm not 100% sure but I think SER
uses shared memory when loading the location table. Thus it runs probably
pretty soon out of shared memory with such a big subscriber population.
Nils
On Monday 07 November 2005 22:23, yserge wrote:
> Hello,
>
> I am having some problems configuring memory for SER..
>
> Here is what I have in my test instance:
>
> Server: Linux x64
> Real Memory: 4GB
> Database: MySQL/NDB
> SER Shared memory: 256M
> Children: 16
> Subscribers: 40000
>
> In test environment:
> Registrations per sec: 120
> Call per second: 120
>
> When the SER process was started last week, it had only 4000 subscribers
> provisioned. Then another 36000 subscribers were provisioned. The SER
> was running fine. But when I decided to bounce SER, it would not
> restart.
>
> It seems from the log file that SER ran out of memory trying to load the
> LOCATION database into memory upon startup, failing around row #3500.
> LOCATION table had in excess of 10000 records at that point.
>
> The related piece of the logfile is below..
>
>
> I wonder whether it is a matter of configuration or is it a bug.. The
> system definitely seem to have enough memory to store those few tens of
> thousands of records.. Even at 1K each, it amounts to nothing..
> Especially given that at runtime it handles that amount of location info
> easily.. It's only a startup that seems to be a problem. I tried to play
> with giving it more or less of shared memory.. It does not seem to be
> relevant..
>
> For now I just have to clean up the LOCATION table each time before
> restarting SER. Obviously, it is not a nice workaround..
>
>
> Thanks,
> Sergei
>
>
>
>
> Mon Nov 7 12:29:59.281894 200: 0
> (0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so lookup
> Mon Nov 7 12:29:59.281964 200: 0(0):DEBUG:get_connection(): Connection
> not found in the pool
> Mon Nov 7 12:29:59.282329 200: 0(0):DEBUG:preload_udomain(): Table is
> empty
> Mon Nov 7 12:29:59.282342 200: 0(0):DEBUG:release_connection():
> Removing connection from the pool
> Mon Nov 7 12:29:59.282360 200: 0
> (0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so save
> Mon Nov 7 12:29:59.282414 200: 0(0):DEBUG:get_connection(): Connection
> not found in the pool
> Mon Nov 7 12:29:59.357157 200: 0(0):WARNING:convert_row(): No memory
> left
> Mon Nov 7 12:29:59.357189 200: 0(0):WARNING:convert_rows(): Error while
> converting row #3445
> Mon Nov 7 12:29:59.357413 200: 0(0):WARNING:convert_result(): Error
> while converting rows
> Mon Nov 7 12:29:59.357424 200: 0(0):WARNING:store_result(): Error while
> converting result
> Mon Nov 7 12:29:59.357432 200: 0(0):WARNING:preload_udomain(): Error
> while doing db_query
> Mon Nov 7 12:29:59.357438 200: 0(0):WARNING:register_udomain(): Error
> while preloading domain 'location'
> Mon Nov 7 12:29:59.357447 200: 0(0):DEBUG:release_connection():
> Removing connection from the pool
> Mon Nov 7 12:29:59.357856 200: 0(0):WARNING:domain_fixup(): Error while
> registering domain
> ERROR: error -1 while trying to fix configuration
> Mon Nov 7 12:29:59.357901 200: 0(0)::DEBUG: tm_shutdown : start
> Mon Nov 7 12:29:59.357913 200: 0(0)::DEBUG: unlink_timer_lists :
> emptying DELETE list
> Mon Nov 7 12:29:59.357920 200: 0(0)::DEBUG: tm_shutdown : emptying hash
> table
> Mon Nov 7 12:29:59.359085 200: 0(0)::DEBUG: tm_shutdown : releasing
> timers
> Mon Nov 7 12:29:59.359108 200: 0(0)::DEBUG: tm_shutdown : removing
> semaphores
> Mon Nov 7 12:29:59.359114 200: 0(0)::DEBUG: tm_shutdown : destroying
> tmcb lists
> Mon Nov 7 12:29:59.359122 200: 0(0)::DEBUG: tm_shutdown : done
> Mon Nov 7 12:29:59.359143 200: 0(11662):DEBUG:shm_mem_destroy
> Mon Nov 7 12:29:59.359154 200: 0(11662):DEBUG:destroying the shared
> memory lock
>
>
>
> _______________________________________________
> Serusers mailing list
> serusers(a)lists.iptel.org
> http://lists.iptel.org/mailman/listinfo/serusers
7:32 p.m.
I think in config.h you should try changing the following line.
#define PKG_MEM_POOL_SIZE 1024*1024
to
#define PKG_MEM_POOL_SIZE 1024*1024*512
if you want to use 512 meg for PKG_MEM, which is where aliases would be
cached.
Let me know if this helps,
William
yserge wrote:
That's what I first tried.. I tried to start with
512M.. No difference
whatsoever.
Then, checking the source code, it looks like it is the "PKG_MALLOC"
macro that fails, which , AFAIU, is not a shared memory.
Sergei
Hello,
try to start SER with more shared memory next time (with that amount of memory
you use only a fraction of the memory). I'm not 100% sure but I think SER
uses shared memory when loading the location table. Thus it runs probably
pretty soon out of shared memory with such a big subscriber population.
Nils
On Monday 07 November 2005 22:23, yserge wrote:
>Hello,
>
>I am having some problems configuring memory for SER..
>
>Here is what I have in my test instance:
>
>Server: Linux x64
>Real Memory: 4GB
>Database: MySQL/NDB
>SER Shared memory: 256M
>Children: 16
>Subscribers: 40000
>
>In test environment:
>Registrations per sec: 120
>Call per second: 120
>
>When the SER process was started last week, it had only 4000 subscribers
>provisioned. Then another 36000 subscribers were provisioned. The SER
>was running fine. But when I decided to bounce SER, it would not
>restart.
>
>It seems from the log file that SER ran out of memory trying to load the
>LOCATION database into memory upon startup, failing around row #3500.
>LOCATION table had in excess of 10000 records at that point.
>
>The related piece of the logfile is below..
>
>
>I wonder whether it is a matter of configuration or is it a bug.. The
>system definitely seem to have enough memory to store those few tens of
>thousands of records.. Even at 1K each, it amounts to nothing..
>Especially given that at runtime it handles that amount of location info
>easily.. It's only a startup that seems to be a problem. I tried to play
>with giving it more or less of shared memory.. It does not seem to be
>relevant..
>
>For now I just have to clean up the LOCATION table each time before
>restarting SER. Obviously, it is not a nice workaround..
>
>
>Thanks,
>Sergei
>
>
>
>
>Mon Nov 7 12:29:59.281894 200: 0
>(0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so lookup
>Mon Nov 7 12:29:59.281964 200: 0(0):DEBUG:get_connection(): Connection
>not found in the pool
>Mon Nov 7 12:29:59.282329 200: 0(0):DEBUG:preload_udomain(): Table is
>empty
>Mon Nov 7 12:29:59.282342 200: 0(0):DEBUG:release_connection():
>Removing connection from the pool
>Mon Nov 7 12:29:59.282360 200: 0
>(0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so save
>Mon Nov 7 12:29:59.282414 200: 0(0):DEBUG:get_connection(): Connection
>not found in the pool
>Mon Nov 7 12:29:59.357157 200: 0(0):WARNING:convert_row(): No memory
>left
>Mon Nov 7 12:29:59.357189 200: 0(0):WARNING:convert_rows(): Error while
>converting row #3445
>Mon Nov 7 12:29:59.357413 200: 0(0):WARNING:convert_result(): Error
>while converting rows
>Mon Nov 7 12:29:59.357424 200: 0(0):WARNING:store_result(): Error while
>converting result
>Mon Nov 7 12:29:59.357432 200: 0(0):WARNING:preload_udomain(): Error
>while doing db_query
>Mon Nov 7 12:29:59.357438 200: 0(0):WARNING:register_udomain(): Error
>while preloading domain 'location'
>Mon Nov 7 12:29:59.357447 200: 0(0):DEBUG:release_connection():
>Removing connection from the pool
>Mon Nov 7 12:29:59.357856 200: 0(0):WARNING:domain_fixup(): Error while
>registering domain
>ERROR: error -1 while trying to fix configuration
>Mon Nov 7 12:29:59.357901 200: 0(0)::DEBUG: tm_shutdown : start
>Mon Nov 7 12:29:59.357913 200: 0(0)::DEBUG: unlink_timer_lists :
>emptying DELETE list
>Mon Nov 7 12:29:59.357920 200: 0(0)::DEBUG: tm_shutdown : emptying hash
>table
>Mon Nov 7 12:29:59.359085 200: 0(0)::DEBUG: tm_shutdown : releasing
>timers
>Mon Nov 7 12:29:59.359108 200: 0(0)::DEBUG: tm_shutdown : removing
>semaphores
>Mon Nov 7 12:29:59.359114 200: 0(0)::DEBUG: tm_shutdown : destroying
>tmcb lists
>Mon Nov 7 12:29:59.359122 200: 0(0)::DEBUG: tm_shutdown : done
>Mon Nov 7 12:29:59.359143 200: 0(11662):DEBUG:shm_mem_destroy
>Mon Nov 7 12:29:59.359154 200: 0(11662):DEBUG:destroying the shared
>memory lock
>
>
>
>_______________________________________________
>Serusers mailing list
>serusers(a)lists.iptel.org
>http://lists.iptel.org/mailman/listinfo/serusers
>
>
_______________________________________________
Serusers mailing list
serusers(a)lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serusers
8 Nov
8 Nov
12:21 p.m.
William,
that would help.. But the solution might be not pretty.. The memory pool
is allocated at startup and, if I understand it correctly, will be
replicated to every forked child, meaning that increasing memory pool
to, say, 16MB would result ( for 30+ children forked for all interface,
fifo, etc) in total memory waste of about half gigabyte. Memory that
would likely not be used at all.. That, if I understand it correctly..
Another option would probably be recompiling mysql module using regular
malloc/free calls instead of F_MALLOC. But that, I presume, would make
mysql module slower quite a bit..
And I don't see a good solution within existing design..
So, I am actually leaning to extending db_func_t interface locally with
an open_cursor/fetch/close_cursor type of functionality, implementing it
for mysql module and using them in the usrloc module instead of plain
"query" interface.
Sergei
I think in config.h you should try changing the
following line.
#define PKG_MEM_POOL_SIZE 1024*1024
to
#define PKG_MEM_POOL_SIZE 1024*1024*512
if you want to use 512 meg for PKG_MEM, which is where aliases would be
cached.
Let me know if this helps,
William
yserge wrote:
That's what I first tried.. I tried to start
with 512M.. No difference
whatsoever.
Then, checking the source code, it looks like it is the "PKG_MALLOC"
macro that fails, which , AFAIU, is not a shared memory.
Sergei
Hello,
try to start SER with more shared memory next time (with that amount of memory
you use only a fraction of the memory). I'm not 100% sure but I think SER
uses shared memory when loading the location table. Thus it runs probably
pretty soon out of shared memory with such a big subscriber population.
Nils
On Monday 07 November 2005 22:23, yserge wrote:
>Hello,
>
>I am having some problems configuring memory for SER..
>
>Here is what I have in my test instance:
>
>Server: Linux x64
>Real Memory: 4GB
>Database: MySQL/NDB
>SER Shared memory: 256M
>Children: 16
>Subscribers: 40000
>
>In test environment:
>Registrations per sec: 120
>Call per second: 120
>
>When the SER process was started last week, it had only 4000 subscribers
>provisioned. Then another 36000 subscribers were provisioned. The SER
>was running fine. But when I decided to bounce SER, it would not
>restart.
>
>It seems from the log file that SER ran out of memory trying to load the
>LOCATION database into memory upon startup, failing around row #3500.
>LOCATION table had in excess of 10000 records at that point.
>
>The related piece of the logfile is below..
>
>
>I wonder whether it is a matter of configuration or is it a bug.. The
>system definitely seem to have enough memory to store those few tens of
>thousands of records.. Even at 1K each, it amounts to nothing..
>Especially given that at runtime it handles that amount of location info
>easily.. It's only a startup that seems to be a problem. I tried to play
>with giving it more or less of shared memory.. It does not seem to be
>relevant..
>
>For now I just have to clean up the LOCATION table each time before
>restarting SER. Obviously, it is not a nice workaround..
>
>
>Thanks,
>Sergei
>
>
>
>
>Mon Nov 7 12:29:59.281894 200: 0
>(0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so lookup
>Mon Nov 7 12:29:59.281964 200: 0(0):DEBUG:get_connection(): Connection
>not found in the pool
>Mon Nov 7 12:29:59.282329 200: 0(0):DEBUG:preload_udomain(): Table is
>empty
>Mon Nov 7 12:29:59.282342 200: 0(0):DEBUG:release_connection():
>Removing connection from the pool
>Mon Nov 7 12:29:59.282360 200: 0
>(0):DEBUG:fixing /usr/local/ser/lib/ser/modules/registrar.so save
>Mon Nov 7 12:29:59.282414 200: 0(0):DEBUG:get_connection(): Connection
>not found in the pool
>Mon Nov 7 12:29:59.357157 200: 0(0):WARNING:convert_row(): No memory
>left
>Mon Nov 7 12:29:59.357189 200: 0(0):WARNING:convert_rows(): Error while
>converting row #3445
>Mon Nov 7 12:29:59.357413 200: 0(0):WARNING:convert_result(): Error
>while converting rows
>Mon Nov 7 12:29:59.357424 200: 0(0):WARNING:store_result(): Error while
>converting result
>Mon Nov 7 12:29:59.357432 200: 0(0):WARNING:preload_udomain(): Error
>while doing db_query
>Mon Nov 7 12:29:59.357438 200: 0(0):WARNING:register_udomain(): Error
>while preloading domain 'location'
>Mon Nov 7 12:29:59.357447 200: 0(0):DEBUG:release_connection():
>Removing connection from the pool
>Mon Nov 7 12:29:59.357856 200: 0(0):WARNING:domain_fixup(): Error while
>registering domain
>ERROR: error -1 while trying to fix configuration
>Mon Nov 7 12:29:59.357901 200: 0(0)::DEBUG: tm_shutdown : start
>Mon Nov 7 12:29:59.357913 200: 0(0)::DEBUG: unlink_timer_lists :
>emptying DELETE list
>Mon Nov 7 12:29:59.357920 200: 0(0)::DEBUG: tm_shutdown : emptying hash
>table
>Mon Nov 7 12:29:59.359085 200: 0(0)::DEBUG: tm_shutdown : releasing
>timers
>Mon Nov 7 12:29:59.359108 200: 0(0)::DEBUG: tm_shutdown : removing
>semaphores
>Mon Nov 7 12:29:59.359114 200: 0(0)::DEBUG: tm_shutdown : destroying
>tmcb lists
>Mon Nov 7 12:29:59.359122 200: 0(0)::DEBUG: tm_shutdown : done
>Mon Nov 7 12:29:59.359143 200: 0(11662):DEBUG:shm_mem_destroy
>Mon Nov 7 12:29:59.359154 200: 0(11662):DEBUG:destroying the shared
>memory lock
>
>
>
>_______________________________________________
>Serusers mailing list
>serusers(a)lists.iptel.org
>http://lists.iptel.org/mailman/listinfo/serusers
>
>
_______________________________________________
Serusers mailing list
serusers(a)lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serusers
6805
days inactive
6806
days old
4 comments
3 participants
participants (3)
-
Nils Ohlmeier -
William Towfiq -
yserge