Hello
coming back to the topic related to mtree, to be able to set values via mi/rpc -- it won't be that difficult to add such functionality. Usually with a tree is mainly reading, due to fast matching on tree indexing. The question is how many times/often do you need to change values and how many of them at the same time (more or less).
I was also thinking about that: our application works in such a way that
the full tree is recalculated every 15 minutes. Currently we have aound 60 branches with 150000 entries each.
I assume many times the changes will be somewhere down the tree, since the first part of the number is usually the same (e.g., country code and operator prefix). To update the tree at runtime, while there are reads on it, there must be used a lock to be safe an consistent. If you do lot of writes and very often, then you keep the tree structure locked a lot, slowing the search.
That's the case, tname is rarely updated, it's tvalue the column that we normally update.
Can you estimate the number of writes and how often they would be on a daily basis? There might be other solutions to look at, if writes are very often.
From the numbers above, let's say that we need to update around 8M records
every 15 minutes (we expect this number will keep steadily increasing...)
As a side note, I've looked at how to implement the mt_match equivalent in redis and it does't look that hard using kamailio s.prefixed transformation (as you suggested) and redis sorted sets. I'll need to make more tests to check the performance, I'll share the results.
Thanks!
Javi
Cheers, Daniel
Regards
Javi
Hello,
On 11/15/11 10:09 PM, Javier Gallart wrote:
Hello coming back to the topic related to mtree, to be able to set values via mi/rpc -- it won't be that difficult to add such functionality. Usually with a tree is mainly reading, due to fast matching on tree indexing. The question is how many times/often do you need to change values and how many of them at the same time (more or less).I was also thinking about that: our application works in such a way that the full tree is recalculated every 15 minutes. Currently we have aound 60 branches with 150000 entries each.
I assume many times the changes will be somewhere down the tree, since the first part of the number is usually the same (e.g., country code and operator prefix). To update the tree at runtime, while there are reads on it, there must be used a lock to be safe an consistent. If you do lot of writes and very often, then you keep the tree structure locked a lot, slowing the search.That's the case, tname is rarely updated, it's tvalue the column that we normally update.
I guess you meant actually tprefix is not updated -- tname is table name, tprefix is the prefix used to build the tree. What would be the percentage of tvalue updates, do you have add/remove of tprefix-es?
Can you estimate the number of writes and how often they would be on a daily basis? There might be other solutions to look at, if writes are very often.From the numbers above, let's say that we need to update around 8M records every 15 minutes (we expect this number will keep steadily increasing...)
But not all of them change the value, right?
As a side note, I've looked at how to implement the mt_match equivalent in redis and it does't look that hard using kamailio s.prefixed transformation (as you suggested) and redis sorted sets. I'll need to make more tests to check the performance, I'll share the results.
I have used this solution with mysql (using a table structure in memory of mysql server), since the changes could have been done in mysql in normal way, in one update query, which is a sync operation. Does redis do any caching or is using always the disk file db?
Cheers, Daniel
Thanks!
Javi
Cheers, Daniel > > Regards > > Javi
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi
On Wed, Nov 16, 2011 at 8:13 AM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
On 11/15/11 10:09 PM, Javier Gallart wrote:
Hello
coming back to the topic related to mtree, to be able to set values via mi/rpc -- it won't be that difficult to add such functionality. Usually with a tree is mainly reading, due to fast matching on tree indexing. The question is how many times/often do you need to change values and how many of them at the same time (more or less).
I was also thinking about that: our application works in such a way
that the full tree is recalculated every 15 minutes. Currently we have aound 60 branches with 150000 entries each.
I assume many times the changes will be somewhere down the tree, since the first part of the number is usually the same (e.g., country code and operator prefix). To update the tree at runtime, while there are reads on it, there must be used a lock to be safe an consistent. If you do lot of writes and very often, then you keep the tree structure locked a lot, slowing the search.
That's the case, tname is rarely updated, it's tvalue the column that we normally update.
I guess you meant actually tprefix is not updated -- tname is table name, tprefix is the prefix used to build the tree. What would be the percentage of tvalue updates, do you have add/remove of tprefix-es?
You're right, sorry for the wrong names. The tprefixes are only updated once a day, and only a tiny percentage of them (less than 1%) actually changes.
Can you estimate the number of writes and how often they would be on a daily basis? There might be other solutions to look at, if writes are very often.
From the numbers above, let's say that we need to update around 8M records every 15 minutes (we expect this number will keep steadily increasing...)
But not all of them change the value, right?
I've been checking this, and on average about 40% are updated in each reload.
As a side note, I've looked at how to implement the mt_match equivalent in redis and it does't look that hard using kamailio s.prefixed transformation (as you suggested) and redis sorted sets. I'll need to make more tests to check the performance, I'll share the results.
I have used this solution with mysql (using a table structure in memory of mysql server), since the changes could have been done in mysql in normal way, in one update query, which is a sync operation. Does redis do any caching or is using always the disk file db?
All the data in Redis stays in memory, you can dump periodic snapshots to disk. Some time ago they worked in a feature called disktore, with some cache mechanism between disk and memory, but as far as I know that line was abandoned.
Cheers, Daniel
Thanks!
Javi
Cheers, Daniel
Regards
Javi
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kathttp://linkedin.com/in/miconda -- http://twitter.com/miconda
Hello,
On 11/16/11 12:45 PM, Javier Gallart wrote:
[...] All the data in Redis stays in memory, you can dump periodic snapshots to disk. Some time ago they worked in a feature called disktore, with some cache mechanism between disk and memory, but as far as I know that line was abandoned.
so if the redis goes down, then the records are lost or is doing dump of memory to disk at that time? I try to figure out if redis makes more sense than using one kamailio as a load balancer in front of two kamailios having the mtree in memory. The two with mtree will be restarted one after the other every 15 min. Load balancer can be configured to use only one, with quick fallback (100ms) to the second, or via kamctl the active kamailio with mtree is set to be used first.
kamailio with mtree will send back a 302 which is handled by load balancer Cheers, Daniel
Hi Daniel
On Thu, Nov 17, 2011 at 8:03 AM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
On 11/16/11 12:45 PM, Javier Gallart wrote:
[...]
All the data in Redis stays in memory, you can dump periodic snapshots to disk. Some time ago they worked in a feature called disktore, with some cache mechanism between disk and memory, but as far as I know that line was abandoned.
so if the redis goes down, then the records are lost or is doing dump of memory to disk at that time? I try to figure out if redis makes more sense than using one kamailio as a load balancer in front of two kamailios having the mtree in memory. The two with mtree will be restarted one after the other every 15 min. Load balancer can be configured to use only one, with quick fallback (100ms) to the second, or via kamctl the active kamailio with mtree is set to be used first.
That's pretty much our current configuration; instead of having a
kamailio as load-balancer we are using our own softswitch that sends the calls to 2 kamailio with mtree in memory. If there is a problem with one of them, the call is quickly rerouted to the backup kamailio. We're happy with that configuration; but as the tree keeps growing the time needed for populating the db + reloading the tree increases (about 4-5 minutes currently, not really under pressure yet...). That's why we were thinking about a full in-memory solution. At this point we're going to focus our tests with this setup: -1 redis Master server for writing, with frequent snapshots -2 redis Slave servers -2 kamailios, each of them reading from a slave redis stored in a shared variable for quick update via rpc -If redis_cmd returns an error, kamailio will respond with a 503 Status so that the softswitch will send the invite to the backup kamailio
Of course this will make sense only if we're able to implement a mt_match equivalent in redis that gives an acceptable performance...
Regards
Javi
kamailio with mtree will send back a 302 which is handled by load balancer Cheers, Daniel
-- Daniel-Constantin Mierla -- http://www.asipto.com Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kat http://linkedin.com/in/miconda -- http://twitter.com/miconda
-
Daniel-Constantin Mierla -
Javier Gallart