Hi Marius,
since you did some updates to this module, I am opening for debate some needed enhancements I did during 3.0 testing phase and want to get opinions how to get in the code repo.
Practically is a new module I named for now ratelimit2 and my last idea is to get it named pipelimit in the trunk.
The reason for a new module are some major changes. The module uses the same algorithm but its core is overhaul. - definitions of pipes are loaded from database - there can be unlimited number of pipes - pipes are identified by string names - should be possible to reload pipes at runtime (iirc, not yet in) - new pipes can be added at runtime - functions accept variables to identify the pipe
Since I never used queues from this module and haven't spent time to understand the concept behind, this functionality is completely missing.
The old module might be good to keep in place, probably many people are using it in this form. So, proposals? What is the way to go on? Common code (algorithms) can be made lib at some point.
Cheers, Daniel
Daniel-Constantin Mierla wrote:
Hi Marius,
Hello,
I am in no way a heavy user of ratelimit, only done a couple of tests using the TAILDROP algorithm. My primary interest was in extending the cfg framework from ser to k modules. I have dug around the documentation and the code so I can say I am familiar with some of the insights.
since you did some updates to this module, I am opening for debate some needed enhancements I did during 3.0 testing phase and want to get opinions how to get in the code repo.
I want to clear some documentation issues first. The ratelimit docs says the queue/pipe use the ipfw nomenclature.
<quote> Packets belonging to the same flow are then passed to either of two different objects, which implement the traffic regulation:
_pipe_ A pipe emulates a link with given bandwidth, propagation delay, queue size and packet loss rate. Packets are queued in front of the pipe as they come out from the classifier, and then transferred to the pipe according to the pipe's parameters.
_queue_ A queue is an abstraction used to implement the WF2Q+ (Worst-case Fair Weighted Fair Queue-ing) Queueing) ing) policy, which is an efficient variant of the WFQ policy. The queue associates a _weight_ and a reference pipe to each flow, and then all backlogged (i.e., with packets queued) flows linked to the same pipe share the pipe's bandwidth pro-portionally proportionally portionally to their weights. Note that weights are not priorities; a flow with a lower weight is still guaranteed to get its fraction of the bandwidth even if a flow with a higher weight is permanently backlogged.
In practice, _pipes_ can be used to set hard limits to the bandwidth that a flow can use, whereas _queues_ can be used to determine how different flow share the available bandwidth.
The _pipe_ and _queue_ configuration commands are the following: </quote>
Practically is a new module I named for now ratelimit2 and my last idea is to get it named pipelimit in the trunk.
The reason for a new module are some major changes. The module uses the same algorithm but its core is overhaul.
- definitions of pipes are loaded from database
- there can be unlimited number of pipes
- pipes are identified by string names
- should be possible to reload pipes at runtime (iirc, not yet in)
- new pipes can be added at runtime
- functions accept variables to identify the pipe
I might add the following nice to have features(provided by implementing and extending queues):
- Matching of packets to a specific queue should be done by via regexp, not only method (ratelimit.c find_queue() method). If you remove queues, then matching to a specific pipe... - Add support for weighted queue eg. Specify the same pipe for invite and register but allow a 25%-75% priorities. I yet don't know if we can have several queues assigned per pipe. As a comment, ATM I guess queues are only used to match request to pipes(Say REGISTER and SUBSCRIBE requests go to pipe 1, INVITEs to pipe 2) - Also it would be nice to modify rates(bandwidth in ipfw terminology) at runtime. AFAIK one can only modify algorithm parameters, not the rate. I am looking into it.
Since I never used queues from this module and haven't spent time to understand the concept behind, this functionality is completely missing.
The old module might be good to keep in place, probably many people are using it in this form. So, proposals? What is the way to go on? Common code (algorithms) can be made lib at some point.
Hm, I would suggest keeping compatibility between the new module and the old and mark the old as obsolete. The alg section is just a few hundred lines, and the queue functionality isn't yet that advanced(no weights and such), and could be easily duplicated and extended into the new.
Any ideas?!
Cheers,
Marius
Cheers, Daniel
The way that the original module was written did not allow dynamic numbers of queues and pipes. I am using the ratelimit module to control incoming/outgoing traffic on specific trunks by invoking rate limitation based on pipes (by forcing a specific pipe). Choosing a specific pipe is up to the kamailio config designer (any type of operations ca be applied to the pvar before choosing a specific pipe, so IMHO it is pretty flexible as is right now - one can apply any regex operations to a pvar). Pipes and queues can be dynamically modified via mi commands. The major limitation of the module is the hard coded number of queues and pipes and the fact that dynamic changes to pipes and queues are not saved on restart.
Loading the queues and pipes from a db will require a redesign of how the pipes and queues are stored internally.
Regards, Ovidiu Sas
On Fri, Jan 22, 2010 at 6:52 AM, marius zbihlei marius.zbihlei@1and1.ro wrote:
Daniel-Constantin Mierla wrote:
Hi Marius,
Hello,
I am in no way a heavy user of ratelimit, only done a couple of tests using the TAILDROP algorithm. My primary interest was in extending the cfg framework from ser to k modules. I have dug around the documentation and the code so I can say I am familiar with some of the insights.
since you did some updates to this module, I am opening for debate some needed enhancements I did during 3.0 testing phase and want to get opinions how to get in the code repo.
I want to clear some documentation issues first. The ratelimit docs says the queue/pipe use the ipfw nomenclature.
<quote> Packets belonging to the same flow are then passed to either of two different objects, which implement the traffic regulation:
_pipe_ A pipe emulates a link with given bandwidth, propagation delay, queue size and packet loss rate. Packets are queued in front of the pipe as they come out from the classifier, and then transferred to the pipe according to the pipe's parameters.
_queue_ A queue is an abstraction used to implement the WF2Q+ (Worst-case Fair Weighted Fair Queue-ing) Queueing) ing) policy, which is an efficient variant of the WFQ policy. The queue associates a _weight_ and a reference pipe to each flow, and then all backlogged (i.e., with packets queued) flows linked to the same pipe share the pipe's bandwidth pro-portionally proportionally portionally to their weights. Note that weights are not priorities; a flow with a lower weight is still guaranteed to get its fraction of the bandwidth even if a flow with a higher weight is permanently backlogged.
In practice, _pipes_ can be used to set hard limits to the bandwidth that a flow can use, whereas _queues_ can be used to determine how different flow share the available bandwidth.
The _pipe_ and _queue_ configuration commands are the following:
</quote> > > Practically is a new module I named for now ratelimit2 and my last idea is > to get it named pipelimit in the trunk. > > The reason for a new module are some major changes. The module uses the > same algorithm but its core is overhaul. > - definitions of pipes are loaded from database > - there can be unlimited number of pipes > - pipes are identified by string names > - should be possible to reload pipes at runtime (iirc, not yet in) > - new pipes can be added at runtime > - functions accept variables to identify the pipe > I might add the following nice to have features(provided by implementing and extending queues):
- Matching of packets to a specific queue should be done by via regexp, not
only method (ratelimit.c find_queue() method). If you remove queues, then matching to a specific pipe...
- Add support for weighted queue eg. Specify the same pipe for invite and
register but allow a 25%-75% priorities. I yet don't know if we can have several queues assigned per pipe. As a comment, ATM I guess queues are only used to match request to pipes(Say REGISTER and SUBSCRIBE requests go to pipe 1, INVITEs to pipe 2)
- Also it would be nice to modify rates(bandwidth in ipfw terminology) at
runtime. AFAIK one can only modify algorithm parameters, not the rate. I am looking into it.
Since I never used queues from this module and haven't spent time to understand the concept behind, this functionality is completely missing.
The old module might be good to keep in place, probably many people are using it in this form. So, proposals? What is the way to go on? Common code (algorithms) can be made lib at some point.
Hm, I would suggest keeping compatibility between the new module and the old and mark the old as obsolete. The alg section is just a few hundred lines, and the queue functionality isn't yet that advanced(no weights and such), and could be easily duplicated and extended into the new.
Any ideas?!
Cheers,
Marius
Cheers, Daniel
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello,
On 1/22/10 3:31 PM, Ovidiu Sas wrote:
The way that the original module was written did not allow dynamic numbers of queues and pipes. I am using the ratelimit module to control incoming/outgoing traffic on specific trunks by invoking rate limitation based on pipes (by forcing a specific pipe). Choosing a specific pipe is up to the kamailio config designer (any type of operations ca be applied to the pvar before choosing a specific pipe, so IMHO it is pretty flexible as is right now - one can apply any regex operations to a pvar). Pipes and queues can be dynamically modified via mi commands. The major limitation of the module is the hard coded number of queues and pipes and the fact that dynamic changes to pipes and queues are not saved on restart.
Loading the queues and pipes from a db will require a redesign of how the pipes and queues are stored internally.
yes, it is a hash table. A benefit of dynamic/dynamic names is you can simply use ip address as pipe name. Also strings tend to look more meaningful when coming back to a config after some time :-) .
IIRC, one issue with the old design is using single lock for all pipes. The advantage was using static indexing, therefore faster access (still under one lock). The new one has a lock per slot, so there can be quite some parallelism of updates/checks, therefore overall could be same results, tending to be faster with old for low number of pipes and not so heavy traffic, better with lot of pipes and lot of traffic.
On the other hand, I find useful what Marius proposed as new features.
Cheers, Daniel
Regards, Ovidiu Sas
On Fri, Jan 22, 2010 at 6:52 AM, marius zbihleimarius.zbihlei@1and1.ro wrote:
Daniel-Constantin Mierla wrote:
Hi Marius,
Hello,
I am in no way a heavy user of ratelimit, only done a couple of tests using the TAILDROP algorithm. My primary interest was in extending the cfg framework from ser to k modules. I have dug around the documentation and the code so I can say I am familiar with some of the insights.
since you did some updates to this module, I am opening for debate some needed enhancements I did during 3.0 testing phase and want to get opinions how to get in the code repo.
I want to clear some documentation issues first. The ratelimit docs says the queue/pipe use the ipfw nomenclature.
<quote> Packets belonging to the same flow are then passed to either of two different objects, which implement the traffic regulation:
_pipe_ A pipe emulates a link with given bandwidth, propagationdelay, queue size and packet loss rate. Packets are queued in front of the pipe as they come out from the classifier, and then transferred to the pipe according to the pipe's parameters.
_queue_ A queue is an abstraction used to implement the WF2Q+(Worst-case Fair Weighted Fair Queue-ing) Queueing) ing) policy, which is an efficient variant of the WFQ policy. The queue associates a _weight_ and a reference pipe to each flow, and then all backlogged (i.e., with packets queued) flows linked to the same pipe share the pipe's bandwidth pro-portionally proportionally portionally to their weights. Note that weights are not priorities; a flow with a lower weight is still guaranteed to get its fraction of the bandwidth even if a flow with a higher weight is permanently backlogged.
In practice, _pipes_ can be used to set hard limits to the bandwidth that a flow can use, whereas _queues_ can be used to determine how different flow share the available bandwidth.
The _pipe_ and _queue_ configuration commands are the following:
</quote>
Practically is a new module I named for now ratelimit2 and my last idea is to get it named pipelimit in the trunk.
The reason for a new module are some major changes. The module uses the same algorithm but its core is overhaul.
- definitions of pipes are loaded from database
- there can be unlimited number of pipes
- pipes are identified by string names
- should be possible to reload pipes at runtime (iirc, not yet in)
- new pipes can be added at runtime
- functions accept variables to identify the pipe
I might add the following nice to have features(provided by implementing and extending queues):
- Matching of packets to a specific queue should be done by via regexp, not
only method (ratelimit.c find_queue() method). If you remove queues, then matching to a specific pipe...
- Add support for weighted queue eg. Specify the same pipe for invite and
register but allow a 25%-75% priorities. I yet don't know if we can have several queues assigned per pipe. As a comment, ATM I guess queues are only used to match request to pipes(Say REGISTER and SUBSCRIBE requests go to pipe 1, INVITEs to pipe 2)
- Also it would be nice to modify rates(bandwidth in ipfw terminology) at
runtime. AFAIK one can only modify algorithm parameters, not the rate. I am looking into it.
Since I never used queues from this module and haven't spent time to understand the concept behind, this functionality is completely missing.
The old module might be good to keep in place, probably many people are using it in this form. So, proposals? What is the way to go on? Common code (algorithms) can be made lib at some point.
Hm, I would suggest keeping compatibility between the new module and the old and mark the old as obsolete. The alg section is just a few hundred lines, and the queue functionality isn't yet that advanced(no weights and such), and could be easily duplicated and extended into the new.
Any ideas?!
Cheers,
Marius
Cheers, Daniel
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Daniel-Constantin Mierla wrote:
Hello,
On 1/22/10 3:31 PM, Ovidiu Sas wrote:
The way that the original module was written did not allow dynamic numbers of queues and pipes. I am using the ratelimit module to control incoming/outgoing traffic on specific trunks by invoking rate limitation based on pipes (by forcing a specific pipe). Choosing a specific pipe is up to the kamailio config designer (any type of operations ca be applied to the pvar before choosing a specific pipe, so IMHO it is pretty flexible as is right now - one can apply any regex operations to a pvar). Pipes and queues can be dynamically modified via mi commands. The major limitation of the module is the hard coded number of queues and pipes and the fact that dynamic changes to pipes and queues are not saved on restart.
Loading the queues and pipes from a db will require a redesign of how the pipes and queues are stored internally.
yes, it is a hash table. A benefit of dynamic/dynamic names is you can simply use ip address as pipe name. Also strings tend to look more meaningful when coming back to a config after some time :-) .
IIRC, one issue with the old design is using single lock for all pipes. The advantage was using static indexing, therefore faster access (still under one lock). The new one has a lock per slot, so there can be quite some parallelism of updates/checks, therefore overall could be same results, tending to be faster with old for low number of pipes and not so heavy traffic, better with lot of pipes and lot of traffic.
On the other hand, I find useful what Marius proposed as new features.
Cheers, Daniel
Hello,
I saw the pipelimit commit, and I am going to have a look to better understand the design. The first question is about the new pipelimit module... Does it require(as in it's a must) a working DB connection to get the pipe configuration ? Because this could be a problem for some systems, (like a light stateless proxy in front of other machines) because this will mean the requirement of loading a DB module. If there is indeed the need for a database, than I suggest that we still allow cfg configuration , will the limitation that runtime changes aren't permanent and are lost in case of a restart.
Regarding the static indexing versus dynamic allocation of pipes, I was thinking that most setups only need a few pipes and that number doesn't change very often. With this in mind I was thinking about combining the two approaches: start to a low number of pipes (16 for example which is basicaly an array on the stack or continuous allocation on the heap). In case of need of a 17th pipe double the capacity (allocate a 32 continuous array on the heap ) and deep copy the old 16 pipes (something that std::vector in c++ does when we alter its capacity). This will add the advantage on using static indexing but still provide with the new functionality of a variable number of pipes.
What do you think?
Greetings, Marius
Hello,
On 1/25/10 10:45 AM, marius zbihlei wrote:
Daniel-Constantin Mierla wrote:
Hello,
On 1/22/10 3:31 PM, Ovidiu Sas wrote:
The way that the original module was written did not allow dynamic numbers of queues and pipes. I am using the ratelimit module to control incoming/outgoing traffic on specific trunks by invoking rate limitation based on pipes (by forcing a specific pipe). Choosing a specific pipe is up to the kamailio config designer (any type of operations ca be applied to the pvar before choosing a specific pipe, so IMHO it is pretty flexible as is right now - one can apply any regex operations to a pvar). Pipes and queues can be dynamically modified via mi commands. The major limitation of the module is the hard coded number of queues and pipes and the fact that dynamic changes to pipes and queues are not saved on restart.
Loading the queues and pipes from a db will require a redesign of how the pipes and queues are stored internally.
yes, it is a hash table. A benefit of dynamic/dynamic names is you can simply use ip address as pipe name. Also strings tend to look more meaningful when coming back to a config after some time :-) .
IIRC, one issue with the old design is using single lock for all pipes. The advantage was using static indexing, therefore faster access (still under one lock). The new one has a lock per slot, so there can be quite some parallelism of updates/checks, therefore overall could be same results, tending to be faster with old for low number of pipes and not so heavy traffic, better with lot of pipes and lot of traffic.
On the other hand, I find useful what Marius proposed as new features.
Cheers, Daniel
Hello,
I saw the pipelimit commit, and I am going to have a look to better understand the design.
the changes are related to the storage engine (db and in memory) by allowing string names and dynamic add/remove pipes.
The first question is about the new pipelimit module... Does it require(as in it's a must) a working DB connection to get the pipe configuration ?
yes. That's why I said the old module should be kept and maybe export pipe algo in a lib ir from module.
I was thinking at some point more or less to your idea, to combine. But the complexity introduced to adapt from one to another does not pay the effort imo when comparing with having two modules clear targeting different sizes.
Most of the cases are as you say, but there are cases when the limit wants to be for each destination or source ip, and that can be controlled via other meanings, like dispatcher, lcr, etc... new ips can appear/disappear dynamically. Having static array with fixed indexing is not convenient.
I uploaded the module so people can discuss based on it. The sql to create the table is in one of the c files at the top.
Cheers, Daniel
Because this could be a problem for some systems, (like a light stateless proxy in front of other machines) because this will mean the requirement of loading a DB module. If there is indeed the need for a database, than I suggest that we still allow cfg configuration , will the limitation that runtime changes aren't permanent and are lost in case of a restart.
Regarding the static indexing versus dynamic allocation of pipes, I was thinking that most setups only need a few pipes and that number doesn't change very often. With this in mind I was thinking about combining the two approaches: start to a low number of pipes (16 for example which is basicaly an array on the stack or continuous allocation on the heap). In case of need of a 17th pipe double the capacity (allocate a 32 continuous array on the heap ) and deep copy the old 16 pipes (something that std::vector in c++ does when we alter its capacity). This will add the advantage on using static indexing but still provide with the new functionality of a variable number of pipes.
What do you think?
Greetings, Marius
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello,
I'm in favor of keeping this module as a db only module. Combining both init methods (param and db config) is messy due to the current module initialization (the shm mem is not available during the param parsing). For small/embedded systems db_text is always a viable alternative.
In the future, the ratelimit module should become obsolete and only pipelimit should be maintained. Also, I'm in favor of keeping the new module as simple as possible and perform any fancy operations in the config file (like regex to pv via textops).
Regards, Ovidiu Sas
On Mon, Jan 25, 2010 at 4:57 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
On 1/25/10 10:45 AM, marius zbihlei wrote:
Daniel-Constantin Mierla wrote:
Hello,
On 1/22/10 3:31 PM, Ovidiu Sas wrote:
The way that the original module was written did not allow dynamic numbers of queues and pipes. I am using the ratelimit module to control incoming/outgoing traffic on specific trunks by invoking rate limitation based on pipes (by forcing a specific pipe). Choosing a specific pipe is up to the kamailio config designer (any type of operations ca be applied to the pvar before choosing a specific pipe, so IMHO it is pretty flexible as is right now - one can apply any regex operations to a pvar). Pipes and queues can be dynamically modified via mi commands. The major limitation of the module is the hard coded number of queues and pipes and the fact that dynamic changes to pipes and queues are not saved on restart.
Loading the queues and pipes from a db will require a redesign of how the pipes and queues are stored internally.
yes, it is a hash table. A benefit of dynamic/dynamic names is you can simply use ip address as pipe name. Also strings tend to look more meaningful when coming back to a config after some time :-) .
IIRC, one issue with the old design is using single lock for all pipes. The advantage was using static indexing, therefore faster access (still under one lock). The new one has a lock per slot, so there can be quite some parallelism of updates/checks, therefore overall could be same results, tending to be faster with old for low number of pipes and not so heavy traffic, better with lot of pipes and lot of traffic.
On the other hand, I find useful what Marius proposed as new features.
Cheers, Daniel
Hello,
I saw the pipelimit commit, and I am going to have a look to better understand the design.
the changes are related to the storage engine (db and in memory) by allowing string names and dynamic add/remove pipes.
The first question is about the new pipelimit module... Does it require(as in it's a must) a working DB connection to get the pipe configuration ?
yes. That's why I said the old module should be kept and maybe export pipe algo in a lib ir from module.
I was thinking at some point more or less to your idea, to combine. But the complexity introduced to adapt from one to another does not pay the effort imo when comparing with having two modules clear targeting different sizes.
Most of the cases are as you say, but there are cases when the limit wants to be for each destination or source ip, and that can be controlled via other meanings, like dispatcher, lcr, etc... new ips can appear/disappear dynamically. Having static array with fixed indexing is not convenient.
I uploaded the module so people can discuss based on it. The sql to create the table is in one of the c files at the top.
Cheers, Daniel
Because this could be a problem for some systems, (like a light stateless proxy in front of other machines) because this will mean the requirement of loading a DB module. If there is indeed the need for a database, than I suggest that we still allow cfg configuration , will the limitation that runtime changes aren't permanent and are lost in case of a restart.
Regarding the static indexing versus dynamic allocation of pipes, I was thinking that most setups only need a few pipes and that number doesn't change very often. With this in mind I was thinking about combining the two approaches: start to a low number of pipes (16 for example which is basicaly an array on the stack or continuous allocation on the heap). In case of need of a 17th pipe double the capacity (allocate a 32 continuous array on the heap ) and deep copy the old 16 pipes (something that std::vector in c++ does when we alter its capacity). This will add the advantage on using static indexing but still provide with the new functionality of a variable number of pipes.
What do you think?
Greetings, Marius
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
-- Daniel-Constantin Mierla
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
On Monday 25 January 2010, Ovidiu Sas wrote:
I'm in favor of keeping this module as a db only module. Combining both init methods (param and db config) is messy due to the current module initialization (the shm mem is not available during the param parsing). For small/embedded systems db_text is always a viable alternative.
Hello Ovidiu,
i did not reviewed the code yet, but some people also like to run their balancers without a database to avoid this additional complexity.
In the future, the ratelimit module should become obsolete and only pipelimit should be maintained.
Sure, if the functionality that most people use in ratelimit is also available there, it makes no sense to keep both.
Also, I'm in favor of keeping the new module as simple as possible and perform any fancy operations in the config file (like regex to pv via textops).
I agree, this should be all done in the configuration, its much easier to maintain.
Cheers,
Henning
On 25-01 15:44, Henning Westerholt wrote:
On Monday 25 January 2010, Ovidiu Sas wrote:
I'm in favor of keeping this module as a db only module. Combining both init methods (param and db config) is messy due to the current module initialization (the shm mem is not available during the param parsing). For small/embedded systems db_text is always a viable alternative.
Hello Ovidiu,
i did not reviewed the code yet, but some people also like to run their balancers without a database to avoid this additional complexity.
Yes, from what I have seen having load balancers without database is common. Right now we could probably use db_text module for such setups.
In the future having a driver for Sqlite or berkeleydb would be a better option. Both Sqlite and Berkeley DB databases appear to be reasonably fast for such setups and they require no special or complicated setup.
-Jan
On Monday 01 February 2010, Jan Janak wrote:
I'm in favor of keeping this module as a db only module. Combining both init methods (param and db config) is messy due to the current module initialization (the shm mem is not available during the param parsing). For small/embedded systems db_text is always a viable alternative.
Hello Ovidiu,
i did not reviewed the code yet, but some people also like to run their balancers without a database to avoid this additional complexity.
Yes, from what I have seen having load balancers without database is common. Right now we could probably use db_text module for such setups.
Hi Jan,
yes, this could be used, of course. But for now i'd like to keep the old simple setup with just a configuration file. ;)
In the future having a driver for Sqlite or berkeleydb would be a better option. Both Sqlite and Berkeley DB databases appear to be reasonably fast for such setups and they require no special or complicated setup.
There is a native berkeleydb driver in kamailio, and I used sqlite for testing with the db_unixodbc driver, but i'm not sure how the state of this is now in the sip-router repository. Marius, did you tested the unixodbc cases with the unit tests?
Henning
Hello Henning,
Having both method of initialization will just complicate the code. When the module parameters are parsed, the shared memory is not available and therefor all the module rate limit parameters will need to be cached into the memory until the shared memory is available. New code needs to be added to deal with conflicts when both method of initialization are provided: init via db and init via module parameters.
Having the ratelimit configured via module parameters is forcing changing the config file every time the default rate limits are changed. Using the simple db_text module has a lot of advantages: - db_text is available for all platforms; - the database is used only during init, after that the database is no longer used and therefor there is no impact on performance; - the rate limits are stored in human readable text file; - a database reload feature can be added and the default rate limits can be changed without touching the config file and server restart will keep the new settings.
These are just my observation from using heavily the existing ratelimit module.
If specifying the rate limits in the config file is a must, I would recommend keeping both modules and moving the common code into a library: - ratelimit with config init - pipelimit with db init
Regards, Ovidiu Sas
On Tue, Feb 2, 2010 at 10:32 AM, Henning Westerholt henning.westerholt@1und1.de wrote:
On Monday 01 February 2010, Jan Janak wrote:
I'm in favor of keeping this module as a db only module. Combining both init methods (param and db config) is messy due to the current module initialization (the shm mem is not available during the param parsing). For small/embedded systems db_text is always a viable alternative.
Hello Ovidiu,
i did not reviewed the code yet, but some people also like to run their balancers without a database to avoid this additional complexity.
Yes, from what I have seen having load balancers without database is common. Right now we could probably use db_text module for such setups.
Hi Jan,
yes, this could be used, of course. But for now i'd like to keep the old simple setup with just a configuration file. ;)
In the future having a driver for Sqlite or berkeleydb would be a better option. Both Sqlite and Berkeley DB databases appear to be reasonably fast for such setups and they require no special or complicated setup.
There is a native berkeleydb driver in kamailio, and I used sqlite for testing with the db_unixodbc driver, but i'm not sure how the state of this is now in the sip-router repository. Marius, did you tested the unixodbc cases with the unit tests?
Henning
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
On Tuesday 02 February 2010, Ovidiu Sas wrote:
Having both method of initialization will just complicate the code. When the module parameters are parsed, the shared memory is not available and therefor all the module rate limit parameters will need to be cached into the memory until the shared memory is available.
Hello Ovidiu,
i agree here with you.
New code needs to be added to deal with conflicts when both method of initialization are provided: init via db and init via module parameters.
This is not a problem when you make this exclusive, either config file or DB mode, like in cr.
Having the ratelimit configured via module parameters is forcing changing the config file every time the default rate limits are changed.
The ratelimit supports changing with FIFO commands.
Using the simple db_text module has a lot of advantages:
- db_text is available for all platforms;
- the database is used only during init, after that the database is
no longer used and therefor there is no impact on performance;
- the rate limits are stored in human readable text file;
Well, this is also true for the configuration file. ;-)
- a database reload feature can be added and the default rate limits
can be changed without touching the config file and server restart will keep the new settings.
discussed above
These are just my observation from using heavily the existing ratelimit module. If specifying the rate limits in the config file is a must, I would recommend keeping both modules and moving the common code into a library:
Hm, it seems that the logic is not that big that moving this to a library is strictly necessary. The complete module is only about 1.5 kLOC. The dbtext module is about 3.5 kLOC, for comparison. In my opinion a switch like "config_mode" could be a good option and should hopefully be not that intrusive in the implementation.
Just some arguments from my side why i not like the dbtext approach that much. We've processes and tools for packaging and deploying our configuration to the servers, we need to adapt documentation and get operational experience with the new database module. So this work i'd like to avoid, especially if I only want to provide a few lines of configuration data. I also don't want to run into eventual corruption problems in the database during an failover situation or a watchdog restart of a loadbalancer.
But, moving a bit away from this argumentation, i thought that this discussion was about an eventual replacement of ratelimit with the pipelimit module? As long as the ratelimit module is available i don't worry about if the pipelimit support only DBs, as we don't have too advanced requirements in this regards.
Best regards,
Henning
On Feb 02, 2010 at 13:41, Ovidiu Sas osas@voipembedded.com wrote:
Hello Henning,
Having both method of initialization will just complicate the code. When the module parameters are parsed, the shared memory is not available and therefor all the module rate limit parameters will need to be cached into the memory until the shared memory is available.
The shared memory is available for module params (recent change).
New code needs to be added to deal with conflicts when both method of initialization are provided: init via db and init via module parameters.
Having the ratelimit configured via module parameters is forcing changing the config file every time the default rate limits are changed.
One could change them via rpc/fifo/mi at runtime, without restart.
Using the simple db_text module has a lot of advantages:
- db_text is available for all platforms;
- the database is used only during init, after that the database is
no longer used and therefor there is no impact on performance;
- the rate limits are stored in human readable text file;
- a database reload feature can be added and the default rate limits
can be changed without touching the config file and server restart will keep the new settings.
These are just my observation from using heavily the existing ratelimit module.
If specifying the rate limits in the config file is a must, I would recommend keeping both modules and moving the common code into a library:
- ratelimit with config init
- pipelimit with db init
I would prefer having the option of using modpram+rpc method instead of db.
Andrei
The major disadvantage that I encounter with modparam (in the current ratelimit module) is related to changes done via rpc/fifo/mi: on server restart, all these changes are lost (unless the config is also updated). I much prefer to have the ability to change the db and then perform a reload. This will ensure the same behavior on server restart. One may argue that it is the same with modparam, but the difference is that changing the config and performing changes via rpc/fifo/mi interface are two independent operations vs changing the db and after that perform a reload, which is a cause/effect operation, ensure synchronicity between server restarts. I also much prefer to have the ability to create a generic config file and customize the deployment via the db.
Maybe keeping the ratelimit as is and having the pipelimit with db only config will be the best solution for now. It time, we will see which one has more traction.
Regards, Ovidiu Sas
On Wed, Feb 3, 2010 at 6:15 PM, Andrei Pelinescu-Onciul andrei@iptel.org wrote:
On Feb 02, 2010 at 13:41, Ovidiu Sas osas@voipembedded.com wrote:
Hello Henning,
Having both method of initialization will just complicate the code. When the module parameters are parsed, the shared memory is not available and therefor all the module rate limit parameters will need to be cached into the memory until the shared memory is available.
The shared memory is available for module params (recent change).
New code needs to be added to deal with conflicts when both method of initialization are provided: init via db and init via module parameters.
Having the ratelimit configured via module parameters is forcing changing the config file every time the default rate limits are changed.
One could change them via rpc/fifo/mi at runtime, without restart.
Using the simple db_text module has a lot of advantages: - db_text is available for all platforms; - the database is used only during init, after that the database is no longer used and therefor there is no impact on performance; - the rate limits are stored in human readable text file; - a database reload feature can be added and the default rate limits can be changed without touching the config file and server restart will keep the new settings.
These are just my observation from using heavily the existing ratelimit module.
If specifying the rate limits in the config file is a must, I would recommend keeping both modules and moving the common code into a library: - ratelimit with config init - pipelimit with db init
I would prefer having the option of using modpram+rpc method instead of db.
Andrei
summarizing, I am going to merge pipelimit as separate module for now. The future will dedice if both worth to keep or one will take over the other.
Cheers, Daniel
On 02/04/2010 05:15 AM, Ovidiu Sas wrote:
The major disadvantage that I encounter with modparam (in the current ratelimit module) is related to changes done via rpc/fifo/mi: on server restart, all these changes are lost (unless the config is also updated). I much prefer to have the ability to change the db and then perform a reload. This will ensure the same behavior on server restart. One may argue that it is the same with modparam, but the difference is that changing the config and performing changes via rpc/fifo/mi interface are two independent operations vs changing the db and after that perform a reload, which is a cause/effect operation, ensure synchronicity between server restarts. I also much prefer to have the ability to create a generic config file and customize the deployment via the db.
Maybe keeping the ratelimit as is and having the pipelimit with db only config will be the best solution for now. It time, we will see which one has more traction.
Regards, Ovidiu Sas
On Wed, Feb 3, 2010 at 6:15 PM, Andrei Pelinescu-Onciul andrei@iptel.org wrote:
On Feb 02, 2010 at 13:41, Ovidiu Sasosas@voipembedded.com wrote:
Hello Henning,
Having both method of initialization will just complicate the code. When the module parameters are parsed, the shared memory is not available and therefor all the module rate limit parameters will need to be cached into the memory until the shared memory is available.
The shared memory is available for module params (recent change).
New code needs to be added to deal with conflicts when both method of initialization are provided: init via db and init via module parameters.
Having the ratelimit configured via module parameters is forcing changing the config file every time the default rate limits are changed.
One could change them via rpc/fifo/mi at runtime, without restart.
Using the simple db_text module has a lot of advantages:
- db_text is available for all platforms;
- the database is used only during init, after that the database is
no longer used and therefor there is no impact on performance;
- the rate limits are stored in human readable text file;
- a database reload feature can be added and the default rate limits
can be changed without touching the config file and server restart will keep the new settings.
These are just my observation from using heavily the existing ratelimit module.
If specifying the rate limits in the config file is a must, I would recommend keeping both modules and moving the common code into a library:
- ratelimit with config init
- pipelimit with db init
I would prefer having the option of using modpram+rpc method instead of db.
Andrei
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
Hello
There is a native berkeleydb driver in kamailio, and I used sqlite for testing with the db_unixodbc driver, but i'm not sure how the state of this is now in the sip-router repository. Marius, did you tested the unixodbc cases with the unit tests?
Henning
I have only tested so far the mysql and postresql cases. I will try to test the unixodbc unit test by the end of the week
Marius
On Wednesday 03 February 2010, marius zbihlei wrote:
I have only tested so far the mysql and postresql cases. I will try to test the unixodbc unit test by the end of the week
Hi marius,
thank you, i was just interested in the latest state in this regards, as i did not remembered it from our last discussions.
Cheers,
Henning
-
Andrei Pelinescu-Onciul -
Daniel-Constantin Mierla -
Henning Westerholt -
Jan Janak -
marius zbihlei -
Ovidiu Sas