Hi everyone,
We are looking at offloading routing logic out of our existing softswitch using openser as an external redirect server. Our routing logic is not terribly complicated and I think both carrierroute and LCR modules would be able to handle it but we have a pretty large route list table with around 30 million records (our routing table is 300k codes but keep different tables per client), 25-30 calls per sec would be enough.
From a performance point of view, what would be the best bet within both modules? also, are both modules actively maintained?
Thanks,
Anthony
On Monday 06 September 2010, anthony thomas wrote:
We are looking at offloading routing logic out of our existing softswitch using openser as an external redirect server. Our routing logic is not terribly complicated and I think both carrierroute and LCR modules would be able to handle it but we have a pretty large route list table with around 30 million records (our routing table is 300k codes but keep different tables per client), 25-30 calls per sec would be enough.
From a performance point of view, what would be the best bet within both modules? also, are both modules actively maintained?
Hi Anthony,
both modules are actively maintained, in the case of cr this is done from us. 30 Million records is indeed a bit larger.. For cr the biggest usage i heard was something like 1 Million routing rules in one instance.
Have you maybe also thought about using some database which supports proper prefix matching (i think postgres is able to do this) with some queries in the script instead of a custom module?
If you don't want to trust a database in this regards, there is also the pdb module available which supports huge data sets for prefix matching with a custom server. With the included data "compiler" you can also do prefix optimisation to save memory, its able to store a number portability database for a country like germany in a few hundreds MB of RAM.
Cheers,
Henning
Hi Henning,
Yes, Indeed we are thinking in using postgres (we already use it for our backoffice databases).
This sencente confuses me a little bit:
"some database which supports proper prefix matching (i think postgres is able to do this)"
Once the db is loaded, the prefix matching is done in memory, right?
And I am not following you here: "with some queries in the script instead of a custom module?"
Thanks,
A
________________________________ From: Henning Westerholt henning.westerholt@1und1.de To: sr-users@lists.sip-router.org Cc: anthony thomas g00tyou@yahoo.com Sent: Mon, September 6, 2010 5:33:36 PM Subject: Re: [SR-Users] High performance routing options
On Monday 06 September 2010, anthony thomas wrote:
We are looking at offloading routing logic out of our existing softswitch using openser as an external redirect server. Our routing logic is not terribly complicated and I think both carrierroute and LCR modules would be able to handle it but we have a pretty large route list table with around 30 million records (our routing table is 300k codes but keep different tables per client), 25-30 calls per sec would be enough.
From a performance point of view, what would be the best bet within both modules? also, are both modules actively maintained?
Hi Anthony,
both modules are actively maintained, in the case of cr this is done from us. 30 Million records is indeed a bit larger.. For cr the biggest usage i heard was something like 1 Million routing rules in one instance.
Have you maybe also thought about using some database which supports proper prefix matching (i think postgres is able to do this) with some queries in the script instead of a custom module?
If you don't want to trust a database in this regards, there is also the pdb module available which supports huge data sets for prefix matching with a custom server. With the included data "compiler" you can also do prefix optimisation to save memory, its able to store a number portability database for a country like germany in a few hundreds MB of RAM.
Cheers,
Henning
On Monday 06 September 2010, anthony thomas wrote:
Yes, Indeed we are thinking in using postgres (we already use it for our backoffice databases).
This sencente confuses me a little bit:
Hello Anthony,
"some database which supports proper prefix matching (i think postgres is able to do this)"
Once the db is loaded, the prefix matching is done in memory, right?
well, i think this depends on the database configuration and memory setup of the machine, but normally this is what you want. I was referring to the fact that in my experience one not insert complete number ranges in the database but certain prefixes, and then do a longest prefix match to find the optimal route. But of course you could do also something like this with some SQL.
And I am not following you here: "with some queries in the script instead of a custom module?"
I was referring to the setup you just described, use a standard DB with the a module like sqlops instead of something more specialized, e.g. cr.
Regards,
Henning
Hello,
On 9/6/10 6:01 PM, Henning Westerholt wrote:
On Monday 06 September 2010, anthony thomas wrote:
Yes, Indeed we are thinking in using postgres (we already use it for our backoffice databases).
This sencente confuses me a little bit:
Hello Anthony,
"some database which supports proper prefix matching (i think postgres is able to do this)"
Once the db is loaded, the prefix matching is done in memory, right?
well, i think this depends on the database configuration and memory setup of the machine, but normally this is what you want. I was referring to the fact that in my experience one not insert complete number ranges in the database but certain prefixes, and then do a longest prefix match to find the optimal route. But of course you could do also something like this with some SQL.
And I am not following you here: "with some queries in the script instead of a custom module?"
I was referring to the setup you just described, use a standard DB with the a module like sqlops instead of something more specialized, e.g. cr.
For postgresql, here is a link to follow for more details: http://prefix.projects.postgresql.org/
Cheers, Daniel
We use this in our solutions and search hundreds of millions of routes with it in 2-3 ms. It works very well, to say the least, because it is an approach that allows application of complex business logic (using stored procedures) to the results, something which is much harder with a more primitive (if faster) in-memory structure. Best of all, it is specifically designed to deal with the problem of variable- length prefixes, so many of the prefix length constraints and/or homogeneity requirements of other routing and LCR engines are eliminated.
It is not possible to say whether a database-backed structure is loaded "from memory"; this is a gross oversimplification of a very complex issue. Clearly, an RDBM cannot load all data into heap; there is plenty of demand-loading of data from disk. RDBM caching, filesystem and I/O caching, disk caching, etc. all play a big role in what the result will actually look like from a median performance perspective.
-- Alex Balashov - Principal Evariste Systems LLC 1170 Peachtree Street 12th Floor, Suite 1200 Atlanta, GA 30309 Tel: +1-678-954-0670 Fax: +1-404-961-1892 Web: http://www.evaristesys.com/
On Sep 6, 2010, at 12:06 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
On 9/6/10 6:01 PM, Henning Westerholt wrote:
On Monday 06 September 2010, anthony thomas wrote:
Yes, Indeed we are thinking in using postgres (we already use it for our backoffice databases).
This sencente confuses me a little bit:
Hello Anthony,
"some database which supports proper prefix matching (i think
postgres is
able to do this)"
Once the db is loaded, the prefix matching is done in memory, right?
well, i think this depends on the database configuration and memory setup of the machine, but normally this is what you want. I was referring to the fact that in my experience one not insert complete number ranges in the database but certain prefixes, and then do a longest prefix match to find the optimal route. But of course you could do also something like this with some SQL.
And I am not following you here: "with some queries in the script instead of a custom module?"
I was referring to the setup you just described, use a standard DB with the a module like sqlops instead of something more specialized, e.g. cr.
For postgresql, here is a link to follow for more details: http://prefix.projects.postgresql.org/
Cheers, Daniel
-- Daniel-Constantin Mierla http://www.asipto.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Thanks Alex,
In our case business logic is out of the system, a different application/server takes care of it and just "dumps" the results (prefix->route) to the database that kamailio will load into memory.
I am surprise that you get such a performance with store procedures, what RDBM are you using?
--- On Mon, 9/6/10, Alex Balashov abalashov@evaristesys.com wrote:
From: Alex Balashov abalashov@evaristesys.com Subject: Re: [SR-Users] High performance routing options To: "Daniel-Constantin Mierla" miconda@gmail.com Cc: "sr-users@lists.sip-router.org" sr-users@lists.sip-router.org Date: Monday, September 6, 2010, 12:57 PM We use this in our solutions and search hundreds of millions of routes with it in 2-3 ms. It works very well, to say the least, because it is an approach that allows application of complex business logic (using stored procedures) to the results, something which is much harder with a more primitive (if faster) in-memory structure. Best of all, it is specifically designed to deal with the problem of variable-length prefixes, so many of the prefix length constraints and/or homogeneity requirements of other routing and LCR engines are eliminated.
It is not possible to say whether a database-backed structure is loaded "from memory"; this is a gross oversimplification of a very complex issue. Clearly, an RDBM cannot load all data into heap; there is plenty of demand-loading of data from disk. RDBM caching, filesystem and I/O caching, disk caching, etc. all play a big role in what the result will actually look like from a median performance perspective.
-- Alex Balashov - Principal Evariste Systems LLC 1170 Peachtree Street 12th Floor, Suite 1200 Atlanta, GA 30309 Tel: +1-678-954-0670 Fax: +1-404-961-1892 Web: http://www.evaristesys.com/
On Sep 6, 2010, at 12:06 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
On 9/6/10 6:01 PM, Henning Westerholt wrote:
On Monday 06 September 2010, anthony thomas
wrote:
Yes, Indeed we are thinking in using postgres
(we already use it for our
backoffice databases).
This sencente confuses me a little bit:
Hello Anthony,
> "some database which
supports proper prefix matching (i think postgres is
able to do this)"
Once the db is loaded, the prefix matching is
done in memory, right?
well, i think this depends on the database
configuration and memory setup of
the machine, but normally this is what you want. I
was referring to the fact
that in my experience one not insert complete
number ranges in the database
but certain prefixes, and then do a longest prefix
match to find the optimal
route. But of course you could do also something
like this with some SQL.
And I am not following you here: "with some queries in the script instead of a
custom module?"
I was referring to the setup you just described,
use a standard DB with the a
module like sqlops instead of something more
specialized, e.g. cr.
For postgresql, here is a link to follow for more
details:
http://prefix.projects.postgresql.org/
Cheers, Daniel
--Daniel-Constantin Mierla http://www.asipto.com
SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list
sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 09/06/2010 01:35 PM, anthony thomas wrote:
I am surprise that you get such a performance with store procedures, what RDBM are you using?
PostgreSQL. My response was to Daniel's link to the 'prefix' operator class module, which is for Postgres only.
On Monday 06 September 2010, Alex Balashov wrote:
We use this in our solutions and search hundreds of millions of routes with it in 2-3 ms. It works very well, to say the least, because it is an approach that allows application of complex business logic (using stored procedures) to the results, something which is much harder with a more primitive (if faster) in-memory structure. Best of all, it is specifically designed to deal with the problem of variable- length prefixes, so many of the prefix length constraints and/or homogeneity requirements of other routing and LCR engines are eliminated.
Hi Alex,
thanks for the interesting informations.
It is not possible to say whether a database-backed structure is loaded "from memory"; this is a gross oversimplification of a very complex issue. [..]
Of course i was overly simplifying here. But if you have a sufficient small database with a proper configuration and enough memory you could actually reach a performance in this area.
Henning
Hello Henning
--- On Mon, 9/6/10, Henning Westerholt henning.westerholt@1und1.de wrote:
On Monday 06 September 2010, anthony thomas wrote:
Yes, Indeed we are thinking in using postgres (we
already use it for our
backoffice databases).
This sencente confuses me a little bit:
Hello Anthony,
"some database which supports proper prefix
matching (i think postgres is
able to do this)"
Once the db is loaded, the prefix matching is done in
memory, right?
well, i think this depends on the database configuration and memory setup of the machine, but normally this is what you want. I was referring to the fact that in my experience one not insert complete number ranges in the database but certain prefixes, and then do a longest prefix match to find the optimal route. But of course you could do also something like this with some SQL.
Ah ok, I got it now. We have an external routing server that will feed the db with prefixes and routes. Since the table is quite large and will take some secs to upload, my idea is to have at least 2 redirect servers so I can load new routes to the "standby" db and sync with the memory and then promote it to master and do the same in the other.
And I am not following you here: "with some queries in the script instead of a custom
module?"
I was referring to the setup you just described, use a standard DB with the a module like sqlops instead of something more specialized, e.g. cr.
Ok, I think we will try with cr, maybe we will need some small modifications but I think is our best shot.
Thanks
Regards,
Henning
good news is my company has a solution that can handle this easily -- it can be configured to handle routing tasks of complex structures at high speed even with large tables. bad news is it is not available open-source :(
-jiri
anthony thomas wrote:
Hi everyone,
We are looking at offloading routing logic out of our existing softswitch using openser as an external redirect server. Our routing logic is not terribly complicated and I think both carrierroute and LCR modules would be able to handle it but we have a pretty large route list table with around 30 million records (our routing table is 300k codes but keep different tables per client), 25-30 calls per sec would be enough.
From a performance point of view, what would be the best bet within both modules? also, are both modules actively maintained?
Thanks,
Anthony
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Jiri Kuthan writes:
good news is my company has a solution that can handle this easily -- it can be configured to handle routing tasks of complex structures at high speed even with large tables. bad news is it is not available open-source :(
jiri,
you should know that it is not appropriate to advertise commercial products on sr-users mailing list.
-- juha
Hello,
On 9/6/10 5:03 PM, anthony thomas wrote:
Hi everyone,
We are looking at offloading routing logic out of our existing softswitch using openser as an external redirect server. Our routing logic is not terribly complicated and I think both carrierroute and LCR modules would be able to handle it but we have a pretty large route list table with around 30 million records (our routing table is 300k codes but keep different tables per client), 25-30 calls per sec would be enough.
From a performance point of view, what would be the best bet within both modules? also, are both modules actively maintained?
Both are actively maintained.
However, depends on what you map to a DID/prefix, there might be other solutions using simpler structures that can be relevant in performances for your case.
For example, if you map a DID to a destination ID or address, look at mtree module: http://sip-router.org/docbook/sip-router/branch/master/modules/mtree/mtree.h...
It is for devel version (upcoming 3.1) but you can just copy it for 3.0 (I use it there and iirc works out of the box) - you can have many tables loaded in memory, each with lot of records.
If you map to destination ID, then dispatcher can be used to map the ID to an address.
Cheers, Daniel
Thanks Daniel,
Looks interesting too, I will test it as well and will post the results
regards,
--- On Mon, 9/6/10, Daniel-Constantin Mierla miconda@gmail.com wrote:
From: Daniel-Constantin Mierla miconda@gmail.com Subject: Re: [SR-Users] High performance routing options To: "anthony thomas" g00tyou@yahoo.com Cc: sr-users@lists.sip-router.org Date: Monday, September 6, 2010, 12:15 PM
Hello,
On 9/6/10 5:03 PM, anthony thomas wrote:
Hi everyone,
We are looking at offloading routing logic out of our existing softswitch using openser as an external redirect server. Our routing logic is not terribly complicated and I think both carrierroute and LCR modules would be able to handle it but we have a pretty large route list table with around 30 million records (our routing table is 300k codes but keep different tables per client), 25-30 calls per sec would be enough.
From a performance point of view, what would be the best bet within both modules? also, are both modules actively maintained?
Both are actively maintained.
However, depends on what you map to a DID/prefix, there might be other solutions using simpler structures that can be relevant in performances for your case.
For example, if you map a DID to a destination ID or address, look at mtree module:
http://sip-router.org/docbook/sip-router/branch/master/modules/mtree/mtree.h...
It is for devel version (upcoming 3.1) but you can just copy it for 3.0 (I use it there and iirc works out of the box) - you can have many tables loaded in memory, each with lot of records.
If you map to destination ID, then dispatcher can be used to map the ID to an address.
Cheers,
Daniel
-- Daniel-Constantin Mierla http://www.asipto.com
-
Alex Balashov -
anthony thomas -
Daniel-Constantin Mierla -
Henning Westerholt -
Jiri Kuthan -
Juha Heinanen