[SR-Users] RFC: db_redisusrloc module - review and how to deal with

[Julien Chavanton]

Hi, just sharing my experience from last year while adding some features to
db_postgres and db_sqlite

On Wed, Feb 21, 2018 at 1:33 AM, Daniel-Constantin Mierla <miconda at gmail.com
> wrote:

> Hello,
>
> (cross-posting being something I want to get developers and users
> express their opinions).
>
> We were recently in a (lucky) situation to have two new modules
> submissions targeting more or less the same purpose: allowing to use
> Kamailio with a Redis backend via database API.
>
> One was submitted by Andreas Granig (Sipwise) and already merged with
> the name db_redis, because it was designed from the beginning as a
> generic DB connector, so the module can be used with auth_db, acc,
> usrloc, ...
>
>   * https://www.kamailio.org/docs/modules/devel/modules/db_redis.html
>
> The second one was submitted by Surendra Tiwari (Plivo), initially
> having a naming conflict with db_redis, but renamed to db_redisusrloc,
> see the pull request at:
>
>   * https://github.com/kamailio/kamailio/pull/1446
>
> Now, this email is about deciding the way to go forward with the second
> module.
>
> It was designed to be used only for usrloc in the first phase, with many
> specific usrloc attributes hard coded inside db_redisusrloc. Surendra
> said (in a private chat) that the long term plan is to make it work for
> other modules. Anyhow, at this moment is very tied to usrloc, therefore
> the name of the module.
>
> Given that the backend (Redis) is not an SQL engine, mapping over
> Kamailio's DB API needs some schema definition (see the readme of
> db_redis) in order to make it generic and work for all our modules that
> use a DB backend.
>
> On the other hand, to squeeze the best of the backend, specially in
> no-SQL cases, having a dedicated DB connector module optimized for a
> specific module might help to get more
> performances/high-availability/scalability from the backend itself.
>
> In this case, for example the expires value for a contact record can be
> set inside redis, so kamailio module doesn't have to run a timer routine
> to clean up (it doesn't mean db_redisusrloc does it right now, I didn't
> have time for a proper review, just giving an example). Surendra said
> they use it in production for couple of months now and it is several
> times faster than using usrloc with db_postgres (iirc, not db_mysql) for
> db_mode=3 (database only mode).
>
> But of course, the reverse of the medal with a dedicated db connector
> for a module: it adds overhead to code maintenance (besides generic
> updates due to external library changes, I expect changing something
> relevant in usrloc, like adding new columns, would require updates in
> this module as well).
>
> So, there are few things I want feedback on:
>
> 1) how do you fill about splitting from a generic-only DB connectors to
> have also some dedicated ones? This is more from confusion point of
> view, as a general rule so far, we do not deny contributions if there
> are other options for same kind of feature (e.g., many lcr or nat
> traversal options). As long as the contributor is willing to maintain
> the code, we were fine.
>


Generic is adding a lot of value in reusability but dedicated ones may also
be needed, especialy with NoSQL.

Even with SQL DB abstraction we are sometimes facing limitations.
Since the db abstraction is generic it may prevent the use of some back-end
specific features, even with real SQL back-end when something is *statement
specific*, one example :
when adding the postgres upsert last year, introduced in Postgres 9.5 the
postgres team decided that the conflict/constraints must be specified
explicitely in the statement, in this case I decided to do automatic
constraint detections so that db_postgres could automaticaly always do
upsert when insert_update is used, this was working fine with  mysql
because insert for update is generic.

In such case may still have several options like extendeding the DB
abstraction or bypassing it by presseting specific values/decisions in the
db_x module (per connection, table, statement type) but if it is per
statement I beleive the only option is to extend the db abstraction.



>
> 2) I guess usrloc, presence and dialog modules would be the main
> suspects that would benefit for such dedicated connectors, in other
> cases might not worth adding dedicated connectors. Missing any other
> module one would like to squeeze more performances with a dedicated
> connector?
>

 I agree, dedicated ones may be valuable, Kamailio is able to do many
things but it seems valuable to also be specialised in doing some tasks
extremely well.


> 3) should we set a different naming policy for such modules, for
> example: use *dbs_* prefix instead *db_*, to suggest better it is a
> DataBase Specific connector?
>
> 4) Andreas said he plans to do some performance testing of usrloc module
> with the two modules and see the differences. Anyone else that wants to
> do it? It can be a good metric to see if it worth going one way or another?
>
> 5) Helping to review the pull request, specially if you use Redis, is
> appreciated. Personally I am very short in available time these days,
> next week I plan do to new Kamailio stable releases, so the schedule is
> not getting lighter in my side.
>
> Cheers,
> Daniel
>
> --
> Daniel-Constantin Mierla
> www.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio Advanced Training - March 5-7, 2018, Berlin - www.asipto.com
> Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20180223/e4e543ee/attachment.html>