Hi list
we've been happily using the mtree module for months now. Lately the size of the tree has grown a lot. The mtree table needs to be fully repopulated and reloaded several times a day, and we are looking for a fastest mechanism (for populating the table, I guess the reload time does not depend much on the db backend...). Does anyone tried with Berkeley DB? Is this combination mtree-berkeley actually feasible...?
Thanks
Javi
Hello,
just mentioning ndb_redis module (in 3.2) - you may want to look at, it is key based access memory system. Otherwise, I haven't used personally berkeley db to comment on this particular subject.
Cheers, Daniel
On 11/11/11 7:50 PM, Javier Gallart wrote:
Hi list
we've been happily using the mtree module for months now. Lately the size of the tree has grown a lot. The mtree table needs to be fully repopulated and reloaded several times a day, and we are looking for a fastest mechanism (for populating the table, I guess the reload time does not depend much on the db backend...). Does anyone tried with Berkeley DB? Is this combination mtree-berkeley actually feasible...?
Thanks
Javi
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello everybody!
What is the difference, advantages or disadvantages of using Asterisk database in Kamailio/Asterisk setup opposed to using Kamailio database?
regards
roman
On 11/12/2011 07:05 AM, roman dmytriv wrote:
Hello everybody!
What is the difference, advantages or disadvantages of using Asterisk database in Kamailio/Asterisk setup opposed to using Kamailio database?
Using it for what?
Using it to deliver IP voice from one person to the other.
What can be accomplished using these two scenarios and what are fundamental differences?
On Sat, 12 Nov 2011 13:49:05 -0500, Alex Balashov wrote:
On 11/12/2011 07:05 AM, roman dmytriv wrote:
Hello everybody!
What is the difference, advantages or disadvantages of using Asterisk database in Kamailio/Asterisk setup opposed to using Kamailio database?
Using it for what?
Roman,
On 11/12/2011 11:18 PM, roman dmytriv wrote:
Using it to deliver IP voice from one person to the other.
What can be accomplished using these two scenarios and what are fundamental differences?
The reason I am giving you nebulous, seemingly intractable responses to your questions, in this thread and the other, is because I am encouraging you to habituate your mind to a slightly different mode of thinking that is more appropriate to the sort of thing that Kamailio is.
In this case, you should really decide what it is that you want to provide and then reason backward to the appropriate tools for it. It is true that to do that, you must first know what it is that can be provided, in principle, but the way to do that is to learn how VoIP networks and application services are built in general, conceptually. Kamailio does have some highly specific features appropriate to certain situations, but fundamentally, it is a particular kind of building block, but a very amorphous one.
What I can tell you with great certainty is that combining elements like Asterisk as an end in itself--that is, purely for the sake of doing so--is a pointless waste of time, and you should first decide what you want to provide and then determine if it is appropriate to do that, not decide to combine them and think about what can be done with that. Integration is a task full of specific details tailored to the intended result; you can't (usefully) generically integrate, in an undifferentiated, unspecialised sort of way.
Kamailio is a SIP proxy at its core, though it has some SIP server features of which it is the logical endpoint, such as the registrar and presence server. It has also acquired some light UAC functionality of late, such as handling registrations and digest authentication, etc. In addition, it is well-suited for translation between various SIP transports, and as a gateway to some other protocols such as XMPP.
You may have noticed by now that Kamailio is driven by a programmatic route script, and that rather SIP message handling is presented within this script in a rather low-level way that requires an understanding of the fundamentals of SIP protocol mechanics, as well as its various state machines.
This aspect of Kamailio makes it rather like an SDK or a framework in many ways. Yes, it's driven by a proxy and UAS core, but the user really scripts their desired SIP outcomes in a detailed way for virtually every type of message, though some are more automatically handled than others. In this sense, Kamailio is not like Asterisk at all; Asterisk is an application with a great deal of pre-determined, built-in functionality, and you can turn it on with just a few configuration file changes. Kamailio's not really like that, although the packaged, stock config file has been getting more and more user-friendly I suppose. Even then, Asterisk has a myriad of integration paths and scriptable possibilities (dial plan, AGI, AMI, etc.) that leave the question of what can be done with it somewhat open, but it is definitely an easier question to answer than for Kamailio.
So, the question of what can be done with Kamailio, or even the ways in which a Kamailio server can accessorise or enhance an Asterisk element, really is a very open-ended one. It is a challenge of imagination; there are literally at least thousands of logically possible ways to combine these two things, just as there are lots of ways that Asterisk+a2billing and Kamailio can interact. What you need to ask yourself is: why do I want to combine them? Where did you get this idea? Did someone tell you? Did you find a page about integration on the web? If so, what were you searching for? Why? What is the problem you are trying to solve? What do you want to accomplish?
Perhaps the answers to these questions can flesh out some sort of broad underlying purpose, some thematic continuity.
Certainly, there are some "typical" ways in which Asterisk and Kamailio are often combined:
- Load balancer: with judicious use of a database or IPC mechanism, applications delivered on Asterisk can be horizontally scaled across N nodes by putting a round-robin load balancer in front of them--a task for which Kamailio's "dispatcher" module is very well-suited.
- Centralised registrar: large numbers of endpoints can be concentrated in one server and serviced by multiple Asterisk servers.
- Carrier interface: business layer of outbound dialing from Asterisk servers (e.g. LCR) can be handled by Kamailio.
- Core platform router: a Kamailio proxy can route calls to application-specific Asterisk feature servers like voicemail, conferencing, etc.
And many others.
But really, none of those represent essential or rigourous answers to your question. They just impose certain premeditated scenarios that may or may not be appropriate to what you want to accomplish, and simply answering with a few examples under the heading of "here's what you can do" would do you a disservice by limiting the paradigms in which you are thinking.
Kamailio does have some very specific features useful to meet very concrete requirements, once you've decided what you want to do broadly. You can get an overview here:
http://www.kamailio.org/w/features/
And you can take a look at the various available modules here:
http://www.kamailio.org/docs/modules/3.2.x/
Some of these modules are de facto "core", in the sense that while they are not formally part of the server core, they are likely to be essential dependencies of any practical or commonplace use of Kamailio. These include 'tm', 'sl', 'rr', 'pv', 'xlog, etc. Some of these modules primarily exist to export APIs or generic mechanisms to other modules that use them in specific ways that expose functionality to the end-user, such as 'usrloc' and 'pua'. And some others of these modules provide very concrete features. For instance:
- dispatcher: Various kinds of load balancing/distribution. - pipelimit: Rate limiting. - drouting & lcr: Some dynamic routing structures. - nathelper & rtpproxy: Far-end NAT traversal fixups and dynamic control of external third-party RTP relays. - memcached & ndb_redis: Support for Memcached and Redis key-value stores. - db_mysql/postgres/unixodbc/oracle & sqlops: Support for SQL interaction with various relational databases. - dialog: Make the proxy statefully aware of dialogs, not just transactions, which is useful in limiting number of concurrent calls, etc. - geoip: Integration with Max Mind GeoIP database for regional identification of IP addresses. - xhttp: Integrated HTTP server.
... and a whole lot more.
You really need to decide what it is you want to do. These modules provide interesting possibilities, but they are, for the most part, implementation details subsumed under larger architectural and service delivery objectives. In other words, as a methodological matter it would be quite uncommon to say: "Oh, Kamailio has a GeoIP module, so now I've figured out what I want to provide!" You could have implemented "geo IP" in some other way if you wanted, even if the module didn't exist; Kamailio's route script is very flexible and quite evolved at this point, and offers many types of connectors, integration paths, API hooks, management interfaces, etc.
You have to have some idea of what you want to do first. I suggest educating yourself about the basic principles of VoIP networks and how services are built. I would learn about:
- The purpose of the SIP protocol, in the context of the history of telephony and signaling.
- SIP protocol structure, syntax, etc.
- Types of SIP network elements: UACs, UASs, proxies, registrars, B2BUAs, SBCs, and what they all do.
- RTP, media and codecs.
- PSTN interworking, billing, rating, etc.
- Various types of application backends, i.e. open-source like Asterisk, FreeSWITCH, SEMS (SIP Express Media Server), etc., and some proprietary approaches like VoiceXML and ccXML -- you should be aware of them, at least.
Then, I would come back to Kamailio with a greater understanding of the "big picture" of what it is and ask: what can it do for me generally and specifically? Generally in terms of being a proxy+UAS, and specifically in terms of the modules and higher-level features it offers.
-- Alex
Alex,
One of the primary problems with your pedagogy is that it fails to impart enough information to allow the student, who possibly knows not enough to formulate it, the "smart" question you're trying to provoke... Leaving just a bad taste and bad feelings behind. The answer you provided below *is* quite useful, as opposed to some of your previous answers. Kudos.
On 11/13/2011 01:26 AM, Alex Balashov wrote:
Roman,
On 11/12/2011 11:18 PM, roman dmytriv wrote:
Using it to deliver IP voice from one person to the other.
What can be accomplished using these two scenarios and what are fundamental differences?
The reason I am giving you nebulous, seemingly intractable responses to your questions, in this thread and the other, is because I am encouraging you to habituate your mind to a slightly different mode of thinking that is more appropriate to the sort of thing that Kamailio is.
In this case, you should really decide what it is that you want to provide and then reason backward to the appropriate tools for it. It is true that to do that, you must first know what it is that can be provided, in principle, but the way to do that is to learn how VoIP networks and application services are built in general, conceptually. Kamailio does have some highly specific features appropriate to certain situations, but fundamentally, it is a particular kind of building block, but a very amorphous one.
What I can tell you with great certainty is that combining elements like Asterisk as an end in itself--that is, purely for the sake of doing so--is a pointless waste of time, and you should first decide what you want to provide and then determine if it is appropriate to do that, not decide to combine them and think about what can be done with that. Integration is a task full of specific details tailored to the intended result; you can't (usefully) generically integrate, in an undifferentiated, unspecialised sort of way.
Kamailio is a SIP proxy at its core, though it has some SIP server features of which it is the logical endpoint, such as the registrar and presence server. It has also acquired some light UAC functionality of late, such as handling registrations and digest authentication, etc. In addition, it is well-suited for translation between various SIP transports, and as a gateway to some other protocols such as XMPP.
You may have noticed by now that Kamailio is driven by a programmatic route script, and that rather SIP message handling is presented within this script in a rather low-level way that requires an understanding of the fundamentals of SIP protocol mechanics, as well as its various state machines.
This aspect of Kamailio makes it rather like an SDK or a framework in many ways. Yes, it's driven by a proxy and UAS core, but the user really scripts their desired SIP outcomes in a detailed way for virtually every type of message, though some are more automatically handled than others. In this sense, Kamailio is not like Asterisk at all; Asterisk is an application with a great deal of pre-determined, built-in functionality, and you can turn it on with just a few configuration file changes. Kamailio's not really like that, although the packaged, stock config file has been getting more and more user-friendly I suppose. Even then, Asterisk has a myriad of integration paths and scriptable possibilities (dial plan, AGI, AMI, etc.) that leave the question of what can be done with it somewhat open, but it is definitely an easier question to answer than for Kamailio.
So, the question of what can be done with Kamailio, or even the ways in which a Kamailio server can accessorise or enhance an Asterisk element, really is a very open-ended one. It is a challenge of imagination; there are literally at least thousands of logically possible ways to combine these two things, just as there are lots of ways that Asterisk+a2billing and Kamailio can interact. What you need to ask yourself is: why do I want to combine them? Where did you get this idea? Did someone tell you? Did you find a page about integration on the web? If so, what were you searching for? Why? What is the problem you are trying to solve? What do you want to accomplish?
Perhaps the answers to these questions can flesh out some sort of broad underlying purpose, some thematic continuity.
Certainly, there are some "typical" ways in which Asterisk and Kamailio are often combined:
- Load balancer: with judicious use of a database or IPC mechanism, applications delivered on Asterisk can be horizontally scaled across N nodes by putting a round-robin load
balancer in front of them--a task for which Kamailio's "dispatcher" module is very well-suited.
Centralised registrar: large numbers of endpoints can be concentrated in one server and serviced by multiple Asterisk servers.
Carrier interface: business layer of outbound dialing from Asterisk servers (e.g. LCR) can be handled by Kamailio.
Core platform router: a Kamailio proxy can route calls to application-specific Asterisk feature servers like voicemail, conferencing, etc.
And many others.
But really, none of those represent essential or rigourous answers to your question. They just impose certain premeditated scenarios that may or may not be appropriate to what you want to accomplish, and simply answering with a few examples under the heading of "here's what you can do" would do you a disservice by limiting the paradigms in which you are thinking.
Kamailio does have some very specific features useful to meet very concrete requirements, once you've decided what you want to do broadly. You can get an overview here:
http://www.kamailio.org/w/features/
And you can take a look at the various available modules here:
http://www.kamailio.org/docs/modules/3.2.x/
Some of these modules are de facto "core", in the sense that while they are not formally part of the server core, they are likely to be essential dependencies of any practical or commonplace use of Kamailio. These include 'tm', 'sl', 'rr', 'pv', 'xlog, etc. Some of these modules primarily exist to export APIs or generic mechanisms to other modules that use them in specific ways that expose functionality to the end-user, such as 'usrloc' and 'pua'. And some others of these modules provide very concrete features. For instance:
- dispatcher: Various kinds of load balancing/distribution.
- pipelimit: Rate limiting.
- drouting & lcr: Some dynamic routing structures.
- nathelper & rtpproxy: Far-end NAT traversal fixups and dynamic control of external third-party RTP relays.
- memcached & ndb_redis: Support for Memcached and Redis key-value stores.
- db_mysql/postgres/unixodbc/oracle & sqlops: Support for SQL interaction with various relational databases.
- dialog: Make the proxy statefully aware of dialogs, not just transactions, which is useful in limiting number of concurrent calls, etc.
- geoip: Integration with Max Mind GeoIP database for regional identification of IP addresses.
- xhttp: Integrated HTTP server.
... and a whole lot more.
You really need to decide what it is you want to do. These modules provide interesting possibilities, but they are, for the most part, implementation details subsumed under larger architectural and service delivery objectives. In other words, as a methodological matter it would be quite uncommon to say: "Oh, Kamailio has a GeoIP module, so now I've figured out what I want to provide!" You could have implemented "geo IP" in some other way if you wanted, even if the module didn't exist; Kamailio's route script is very flexible and quite evolved at this point, and offers many types of connectors, integration paths, API hooks, management interfaces, etc.
You have to have some idea of what you want to do first. I suggest educating yourself about the basic principles of VoIP networks and how services are built. I would learn about:
The purpose of the SIP protocol, in the context of the history of telephony and signaling.
SIP protocol structure, syntax, etc.
Types of SIP network elements: UACs, UASs, proxies, registrars, B2BUAs, SBCs, and what they all do.
RTP, media and codecs.
PSTN interworking, billing, rating, etc.
Various types of application backends, i.e. open-source like Asterisk, FreeSWITCH, SEMS (SIP Express Media Server), etc., and some proprietary approaches like VoiceXML and
ccXML -- you should be aware of them, at least.
Then, I would come back to Kamailio with a greater understanding of the "big picture" of what it is and ask: what can it do for me generally and specifically? Generally in terms of being a proxy+UAS, and specifically in terms of the modules and higher-level features it offers.
-- Alex
On 11/13/2011 10:08 PM, Bruce Ferrell wrote:
One of the primary problems with your pedagogy is that it fails to impart enough information to allow the student, who possibly knows not enough to formulate it, the "smart" question you're trying to provoke... Leaving just a bad taste and bad feelings behind.
That is, indeed, a liability.
The answer you provided below *is* quite useful, as opposed to some of your previous answers. Kudos.
I'm gratified that you thought so.
-- Alex
Thanks Daniel
yes, I thought about that too, I like redis a lot, and the redis module addition to kamailio is excellent news. However in this context it's not trivial to write a function that looks for the best match in a redis tree structure as mt_match does...
Regards
Javi
On Sat, Nov 12, 2011 at 9:28 AM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
just mentioning ndb_redis module (in 3.2) - you may want to look at, it is key based access memory system. Otherwise, I haven't used personally berkeley db to comment on this particular subject.
Cheers, Daniel
On 11/11/11 7:50 PM, Javier Gallart wrote:
Hi list
we've been happily using the mtree module for months now. Lately the size of the tree has grown a lot. The mtree table needs to be fully repopulated and reloaded several times a day, and we are looking for a fastest mechanism (for populating the table, I guess the reload time does not depend much on the db backend...). Does anyone tried with Berkeley DB? Is this combination mtree-berkeley actually feasible...?
Thanks
Javi
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kathttp://linkedin.com/in/miconda -- http://twitter.com/miconda
Hello,
On 11/12/11 1:28 PM, Javier Gallart wrote:
Thanks Daniel
yes, I thought about that too, I like redis a lot, and the redis module addition to kamailio is excellent news. However in this context it's not trivial to write a function that looks for the best match in a redis tree structure as mt_match does...
ok, in the same idea of a remote caching system, we have memcache connector module, but probably it is the same situation as with redis.
Back to initial topic, I am not a user of db_berkeley, but afaik, the module loads the content in memory of kamailio, so if you use mtree, then it is practically a duplicate of content. Besides, I guess db_berkeley will have some internal structure overhead that will use a bit more memory.
What I can think of at this moment for a solution will be adding/removing prefixes from mtree using mi/rpc command, so in case of change of records, instead of loading the database table, the updates can be done from command line or so.
Another option might be using database directly. With mysql, a good solution is to define database table in memory, then add/updates records there as needed. From config file, use sqlops with sql_query, matching using 'IN' operator, against matching number exapnded with s.prefixes transformation:
http://www.kamailio.org/wiki/cookbooks/3.2.x/transformations#sprefixes_len
Also, afaik, if you want postgres, it has some sort of index plugin that can be used to match on longest prefix.
Cheers, Daniel
Regards
Javi
On Sat, Nov 12, 2011 at 9:28 AM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello, just mentioning ndb_redis module (in 3.2) - you may want to look at, it is key based access memory system. Otherwise, I haven't used personally berkeley db to comment on this particular subject. Cheers, Daniel On 11/11/11 7:50 PM, Javier Gallart wrote:Hi list we've been happily using the mtree module for months now. Lately the size of the tree has grown a lot. The mtree table needs to be fully repopulated and reloaded several times a day, and we are looking for a fastest mechanism (for populating the table, I guess the reload time does not depend much on the db backend...). Does anyone tried with Berkeley DB? Is this combination mtree-berkeley actually feasible...? Thanks Javi _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla --http://www.asipto.com Kamailio Advanced Training, Dec 5-8, Berlin:http://asipto.com/u/kat http://linkedin.com/in/miconda -- http://twitter.com/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel-Constantin Mierla writes:
just mentioning ndb_redis module (in 3.2) - you may want to look at, it is key based access memory system.
sad that redis does not support master-master redundancy, but requires manual promotion of slave to master in case master fails.
-- juha
On 11/13/11 9:37 AM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
just mentioning ndb_redis module (in 3.2) - you may want to look at, it is key based access memory system.
sad that redis does not support master-master redundancy, but requires manual promotion of slave to master in case master fails.
are there any other no-sql database systems that have such mechanism? Might not be hard to make a connector when the time will allow -- just to know the best options here.
Otherwise, I haven't searched there might be some tools/scripts out there (e.g., monit, ucarp) that may help to make the detection of failure and switch to standby more automatic.
Cheers, Daniel
Daniel-Constantin Mierla writes:
are there any other no-sql database systems that have such mechanism? Might not be hard to make a connector when the time will allow -- just to know the best options here.
http://www.couchbase.org/ claims out-of-box clustering support, but i don't have personal experience with it.
what comes to adding/removing mtree prefixes using mi/rpc instead of reloading the whole thing from database, would be a good improvement.
-- juha
On Mon, Nov 14, 2011 at 5:10 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
are there any other no-sql database systems that have such mechanism? Might not be hard to make a connector when the time will allow -- just to know the best options here.
mongodb will auto promote. Caveat, (like redis if i understand correctly), is that all writes are directed to a single master (be it chosen dynamically), but reads can happen anywhere to spread the load. Also, you need to accept the distaster scenario of a "network partition" where a minority set of servers find themselves w/o a master. Example: 5 servers in datacenter #1 and 4 servers in datacenter #2. If the link between datacenters is broken, then all servers in datacenter #2 will not have a master and will be read-only until link is restored. Good part about single master is there's no chance of inconsistent data.
Turns out local fail-over v.s. consistent data is a well explored area.
http://blog.nahurst.com/visual-guide-to-nosql-systems
I've worked w/the C++ driver to mongodb is anyone has questions.
Hello
very interesting issue actually...the mtree module fits perfectly well in a key-value model becaue basically is what the mtree table structure defines; that's why redis was the first thing that came to my mind when I saw the redis module. Two problems with redis: -no "native" mt_match function, up to the user to find the best option -replication. Until the cluster feature is ready, we need to change by hand the server ip address, which implies a kamailio restart. There is no mi command for changing the server in the fly, right..(not in the module documentation at least)?
Daniel, I agree that your suggestion about the mi/rpc method would be nice. I will also take a look at Mongo as Douglas suggests, and especially CouchDB, because you can talk to Couch DB via http...
Regards
Javi
On Mon, Nov 14, 2011 at 1:32 PM, Douglas Hubler douglas@hubler.us wrote:
On Mon, Nov 14, 2011 at 5:10 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
are there any other no-sql database systems that have such mechanism?
Might
not be hard to make a connector when the time will allow -- just to know
the
best options here.
mongodb will auto promote. Caveat, (like redis if i understand correctly), is that all writes are directed to a single master (be it chosen dynamically), but reads can happen anywhere to spread the load. Also, you need to accept the distaster scenario of a "network partition" where a minority set of servers find themselves w/o a master. Example: 5 servers in datacenter #1 and 4 servers in datacenter #2. If the link between datacenters is broken, then all servers in datacenter #2 will not have a master and will be read-only until link is restored. Good part about single master is there's no chance of inconsistent data.
Turns out local fail-over v.s. consistent data is a well explored area.
http://blog.nahurst.com/visual-guide-to-nosql-systems
I've worked w/the C++ driver to mongodb is anyone has questions.
Hello,
On 11/14/11 3:24 PM, Javier Gallart wrote:
Hello
very interesting issue actually...the mtree module fits perfectly well in a key-value model becaue basically is what the mtree table structure defines; that's why redis was the first thing that came to my mind when I saw the redis module. Two problems with redis: -no "native" mt_match function, up to the user to find the best option -replication. Until the cluster feature is ready, we need to change by hand the server ip address, which implies a kamailio restart. There is no mi command for changing the server in the fly, right..(not in the module documentation at least)?
you cannot change the redis server attributes on the fly, but you can define many redis servers and based on the name attribute query a specific one. So if you define two, you can do round robin queries to both of them, by using a shared variable $shv(...) to know which one was used last time. In the same way, since $sht(...) can be changed via MI, you can query either first redis or second one, based on the value of $sht(). In this way you can build some failover solution just in config file of kamailio.
Cheers, Daniel
Daniel, I agree that your suggestion about the mi/rpc method would be nice. I will also take a look at Mongo as Douglas suggests, and especially CouchDB, because you can talk to Couch DB via http...
Regards
Javi
On Mon, Nov 14, 2011 at 1:32 PM, Douglas Hubler <douglas@hubler.us mailto:douglas@hubler.us> wrote:
On Mon, Nov 14, 2011 at 5:10 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: > are there any other no-sql database systems that have such mechanism? Might > not be hard to make a connector when the time will allow -- just to know the > best options here. mongodb will auto promote. Caveat, (like redis if i understand correctly), is that all writes are directed to a single master (be it chosen dynamically), but reads can happen anywhere to spread the load. Also, you need to accept the distaster scenario of a "network partition" where a minority set of servers find themselves w/o a master. Example: 5 servers in datacenter #1 and 4 servers in datacenter #2. If the link between datacenters is broken, then all servers in datacenter #2 will not have a master and will be read-only until link is restored. Good part about single master is there's no chance of inconsistent data. Turns out local fail-over v.s. consistent data is a well explored area. http://blog.nahurst.com/visual-guide-to-nosql-systems I've worked w/the C++ driver to mongodb is anyone has questions.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel
sound like a good idea, thanks for the tip
Regards
Javi
On Tue, Nov 15, 2011 at 9:59 AM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
On 11/14/11 3:24 PM, Javier Gallart wrote:
Hello
very interesting issue actually...the mtree module fits perfectly well in a key-value model becaue basically is what the mtree table structure defines; that's why redis was the first thing that came to my mind when I saw the redis module. Two problems with redis: -no "native" mt_match function, up to the user to find the best option -replication. Until the cluster feature is ready, we need to change by hand the server ip address, which implies a kamailio restart. There is no mi command for changing the server in the fly, right..(not in the module documentation at least)?
you cannot change the redis server attributes on the fly, but you can define many redis servers and based on the name attribute query a specific one. So if you define two, you can do round robin queries to both of them, by using a shared variable $shv(...) to know which one was used last time. In the same way, since $sht(...) can be changed via MI, you can query either first redis or second one, based on the value of $sht(). In this way you can build some failover solution just in config file of kamailio.
Cheers, Daniel
Daniel, I agree that your suggestion about the mi/rpc method would be nice. I will also take a look at Mongo as Douglas suggests, and especially CouchDB, because you can talk to Couch DB via http...
Regards
Javi
On Mon, Nov 14, 2011 at 1:32 PM, Douglas Hubler douglas@hubler.us wrote:
On Mon, Nov 14, 2011 at 5:10 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
are there any other no-sql database systems that have such mechanism?
Might
not be hard to make a connector when the time will allow -- just to
know the
best options here.
mongodb will auto promote. Caveat, (like redis if i understand correctly), is that all writes are directed to a single master (be it chosen dynamically), but reads can happen anywhere to spread the load. Also, you need to accept the distaster scenario of a "network partition" where a minority set of servers find themselves w/o a master. Example: 5 servers in datacenter #1 and 4 servers in datacenter #2. If the link between datacenters is broken, then all servers in datacenter #2 will not have a master and will be read-only until link is restored. Good part about single master is there's no chance of inconsistent data.
Turns out local fail-over v.s. consistent data is a well explored area.
http://blog.nahurst.com/visual-guide-to-nosql-systems
I've worked w/the C++ driver to mongodb is anyone has questions.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- http://www.asipto.com Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kathttp://linkedin.com/in/miconda -- http://twitter.com/miconda
Hello,
On 11/14/11 3:24 PM, Javier Gallart wrote:
Hello
very interesting issue actually...the mtree module fits perfectly well in a key-value model becaue basically is what the mtree table structure defines; that's why redis was the first thing that came to my mind when I saw the redis module. Two problems with redis: -no "native" mt_match function, up to the user to find the best option -replication. Until the cluster feature is ready, we need to change by hand the server ip address, which implies a kamailio restart. There is no mi command for changing the server in the fly, right..(not in the module documentation at least)?
Daniel, I agree that your suggestion about the mi/rpc method would be nice. I will also take a look at Mongo as Douglas suggests, and especially CouchDB, because you can talk to Couch DB via http...
coming back to the topic related to mtree, to be able to set values via mi/rpc -- it won't be that difficult to add such functionality. Usually with a tree is mainly reading, due to fast matching on tree indexing. The question is how many times/often do you need to change values and how many of them at the same time (more or less).
I assume many times the changes will be somewhere down the tree, since the first part of the number is usually the same (e.g., country code and operator prefix). To update the tree at runtime, while there are reads on it, there must be used a lock to be safe an consistent. If you do lot of writes and very often, then you keep the tree structure locked a lot, slowing the search.
Can you estimate the number of writes and how often they would be on a daily basis? There might be other solutions to look at, if writes are very often.
Cheers, Daniel
Regards
Javi
On Mon, Nov 14, 2011 at 1:32 PM, Douglas Hubler <douglas@hubler.us mailto:douglas@hubler.us> wrote:
On Mon, Nov 14, 2011 at 5:10 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: > are there any other no-sql database systems that have such mechanism? Might > not be hard to make a connector when the time will allow -- just to know the > best options here. mongodb will auto promote. Caveat, (like redis if i understand correctly), is that all writes are directed to a single master (be it chosen dynamically), but reads can happen anywhere to spread the load. Also, you need to accept the distaster scenario of a "network partition" where a minority set of servers find themselves w/o a master. Example: 5 servers in datacenter #1 and 4 servers in datacenter #2. If the link between datacenters is broken, then all servers in datacenter #2 will not have a master and will be read-only until link is restored. Good part about single master is there's no chance of inconsistent data. Turns out local fail-over v.s. consistent data is a well explored area. http://blog.nahurst.com/visual-guide-to-nosql-systems I've worked w/the C++ driver to mongodb is anyone has questions.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Daniel-Constantin Mierla writes:
I assume many times the changes will be somewhere down the tree, since the first part of the number is usually the same (e.g., country code and operator prefix). To update the tree at runtime, while there are reads on it, there must be used a lock to be safe an consistent. If you do lot of writes and very often, then you keep the tree structure locked a lot, slowing the search.
daniel,
could the lock be from the node that requires modification downwards, which would leave all other branches unlocked?
Can you estimate the number of writes and how often they would be on a daily basis? There might be other solutions to look at, if writes are very often.
i was thinking that perhaps mtree would be used to store numbers that are ported to other operators instead of the current carrier route pdb server, where the whole mmap file needs to be reloaded. in that application i would imagine a few hundred updates per hour, but many of them would only affect the value associated with the node, not the structure of the three.
-- juha
Hello,
On 11/15/11 12:59 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
I assume many times the changes will be somewhere down the tree, since the first part of the number is usually the same (e.g., country code and operator prefix). To update the tree at runtime, while there are reads on it, there must be used a lock to be safe an consistent. If you do lot of writes and very often, then you keep the tree structure locked a lot, slowing the search.
daniel,
could the lock be from the node that requires modification downwards, which would leave all other branches unlocked?
for large trees it would be not possible to have a lock for each node, there will be too many and different operating systems have limits. On a hash table there is a lock per slot, no matter how many items are in the table. In a tree is hard to define a set of locks and just going with one per node is not that feasible. For eaxample one record with prefix 0123456789 creates 10 nodes, then some nodes are reused, but still there are a greater or equal number of nodes than the number of records loaded.
Can you estimate the number of writes and how often they would be on a daily basis? There might be other solutions to look at, if writes are very often.
i was thinking that perhaps mtree would be used to store numbers that are ported to other operators instead of the current carrier route pdb server, where the whole mmap file needs to be reloaded. in that application i would imagine a few hundred updates per hour, but many of them would only affect the value associated with the node, not the structure of the three.
Adding a record (i.e., prefix/value) in a tree should be fast if building the missing sub-tree part of it and then just linking it to the main tree. Removing might need more time, a solution for it can be just setting the value to NULL and keep the tree structure.. Cheers, Daniel
-
Alex Balashov -
Bruce Ferrell -
Daniel-Constantin Mierla -
Douglas Hubler -
Javier Gallart -
Juha Heinanen -
roman dmytriv