[SR-Users] Avoid full table scan for dispatcher table on db_redis using ds_reload() from config script

Mon Nov 25 19:18:38 CET 2019

Hi Henning,

I did read the lengthy explanation.. I also tried your suggestion for the
dispatcher keys modparam but still same problem:

Nov 25 18:01:13 test COPS[17825]: DEBUG: db_redis [redis_dbase.c:1761]:
db_redis_query(): querying prefix (table) 'dispatcher'
Nov 25 18:01:13 test COPS[17825]: DEBUG: db_redis [redis_dbase.c:1811]:
db_redis_query(): no columns given to build query keys, falling back to
full table scan
Nov 25 18:01:13 test COPS[17825]: DEBUG: <core> [db_res.c:119]:
db_new_result(): allocate 56 bytes for result set at 0x7efe768040b0
Nov 25 18:01:13 test COPS[17825]: DEBUG: <core> [db_res.c:156]:
db_allocate_columns(): allocate 40 bytes for result names at 0x7efe76804150
Nov 25 18:01:13 test COPS[17825]: DEBUG: <core> [db_res.c:167]:
db_allocate_columns(): allocate 20 bytes for result types at 0x7efe768041e0
Nov 25 18:01:13 test COPS[17825]: WARNING: db_redis [redis_dbase.c:1098]:
db_redis_perform_query(): performing full table scan on table 'dispatcher'
while performing query

I've tried:

modparam("db_redis", "keys", "version=entry:table_name;dispatcher=entry:id")

And also separated in different declarations:

modparam("db_redis", "keys", "version=entry:table_name")
modparam("db_redis", "keys", "dispatcher=entry:id")

In both cases it's still doing a full table scan and I continue getting the
warning in the logfile.

For the sake of testing I did one more attempt with :

modparam("db_redis", "keys", "version=entry:table_name")
modparam("db_redis", "keys", "dispatcher=entry:destination")

But same results...

I do see in the logs that the modparam is applied correctly:

Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:200]:
db_redis_print_all_tables():   table dispatcher
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:203]:
db_redis_print_all_tables():     schema:
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:208]:
db_redis_print_all_tables():       description: 3
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:208]:
db_redis_print_all_tables():       attrs: 3
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:208]:
db_redis_print_all_tables():       flags: 0
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:208]:
db_redis_print_all_tables():       destination: 3
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:208]:
db_redis_print_all_tables():       priority: 0
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:208]:
db_redis_print_all_tables():       setid: 0
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:208]:
db_redis_print_all_tables():       id: 0
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:214]:
db_redis_print_all_tables():     entry keys:
Nov 25 18:16:09 test COPS[18195]: DEBUG: db_redis [redis_table.c:217]:
db_redis_print_all_tables():       id

But it seems not to change the full table scan behavior...

Any other suggestions I can try?

I appreciate your help on this.

Thanks,
Joel.

On Mon, Nov 25, 2019 at 12:21 AM Henning Westerholt <hw at skalatan.de> wrote:

> Hello Joel,
>
>
>
> You probably read already the lengthy explanation at the module overview
> docs:
>
>
>
>
> https://kamailio.org/docs/modules/devel/modules/db_redis.html#db_redis.sec.overview
>
>
>
> A small correction about my previous comment – as the dispatcher module
> does a full table load anyway during start/re-load, the warning is indeed
> harmless, the number of records in the table does not matter here.
>
>
>
> Haven’t tried it myself right now, but what about simply:
>
>
>
> modparam("db_redis", "keys", "version=entry:table_name;dispatcher=entry:id")
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
> Kamailio Merchandising – https://skalatan.de/merchandising
>
>
>
> *From:* Joel Serrano <joel at textplus.com>
> *Sent:* Monday, November 25, 2019 12:39 AM
> *To:* Henning Westerholt <hw at skalatan.de>
> *Cc:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
> *Subject:* Re: [SR-Users] Avoid full table scan for dispatcher table on
> db_redis using ds_reload() from config script
>
>
>
> Hi Henning,
>
>
>
> That’s the thing, I don’t really understand the docs, thus way opened this
> thread.
>
>
>
> Can you give me an example of what the keys modparam for dispatcher table
> would look like? I would really appreciate it as I don’t get it after
> reading the docs.
>
>
>
> I don’t have many gateways as this is a test env but still I want to do
> things the correct way. Otherwise every time I reload I get a warning in
> the logs and I rather avoid that warning if I can.
>
>
>
> Thanks,
>
> Joel.
>
>
>
> On Sat, Nov 23, 2019 at 00:34 Henning Westerholt <hw at skalatan.de> wrote:
>
> Hello Joel,
>
>
>
> you already quoted the keys modparam docs. 😊 You need to add an entry
> for the dispatcher table to keys as well.
>
>
>
> How many records do you actually have in the dispatcher table? If it is a
> low number (like < 100) or so, it should work also fine without it. IMHO
> the warning is there in case people forget to add it on huge tables. And of
> course, it is “the right thing to do” from an implementation point of view.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
> Kamailio Merchandising – https://skalatan.de/merchandising
>
>
>
> *From:* sr-users <sr-users-bounces at lists.kamailio.org> *On Behalf Of *Joel
> Serrano
> *Sent:* Saturday, November 23, 2019 1:18 AM
> *To:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
> *Subject:* Re: [SR-Users] Avoid full table scan for dispatcher table on
> db_redis using ds_reload() from config script
>
>
>
> Just in case:
>
>
>
> Kam version - latest nightly deb (5.4.0~dev1+0~20191122005600.1540+buster)
>
>
>
> Redis config:
>
>
>
> loadmodule "db_redis.so"
>
> modparam("db_redis", "keys", "version=entry:table_name")
>
> ...
>
> modparam("dispatcher", "db_url", "redis://X.X.X.X:6379/2")
>
>
>
>
>
> Thanks!
>
> Joel.
>
>
>
> On Fri, Nov 22, 2019 at 1:48 PM Joel Serrano <joel at textplus.com> wrote:
>
> Hello,
>
>
>
> I'm trying out redis as db backend, and right now I only have dispatcher
> records there... Every so often, I do a ds_reload() from within Kam config
> script, and I see in logs:
>
>
> Nov 22 20:36:35 test COPS[25531]: WARNING: db_redis [redis_dbase.c:1098]:
> db_redis_perform_query(): performing full table scan on table 'dispatcher'
> while performing query
>
> After enabling debug logs:
>
>
>
> Nov 22 21:15:35 test COPS[26661]: DEBUG: db_redis [redis_dbase.c:1761]:
> db_redis_query(): querying prefix (table) 'dispatcher'
> Nov 22 21:15:35 test COPS[26661]: DEBUG: db_redis [redis_dbase.c:1811]:
> db_redis_query(): no columns given to build query keys, falling back to
> full table scan
> Nov 22 21:15:35 test COPS[26661]: DEBUG: <core> [db_res.c:119]:
> db_new_result(): allocate 56 bytes for result set at 0x7f0be4273898
> Nov 22 21:15:35 test COPS[26661]: DEBUG: <core> [db_res.c:156]:
> db_allocate_columns(): allocate 40 bytes for result names at 0x7f0be4273938
> Nov 22 21:15:35 test COPS[26661]: DEBUG: <core> [db_res.c:167]:
> db_allocate_columns(): allocate 20 bytes for result types at 0x7f0be42739c8
> Nov 22 21:15:35 test COPS[26661]: WARNING: db_redis [redis_dbase.c:1098]:
> db_redis_perform_query(): performing full table scan on table 'dispatcher'
> while performing query
>
> Given the msg... "no columns given to build query keys, falling back to
> full table scan", I assume I'm missing a keys modparam.. but I don't know
> how to build it, and the docs (to me) seem confusing?
>
>
>
>
> https://kamailio.org/docs/modules/devel/modules/db_redis.html#db_redis.p.keys
>
>
>
> Can anyone help me understand how to build the keys modparam for
> dispatcher table? Or if the reason for this is different, what do I need to
> do to avoid the full table scan and the warning in the log?
>
>
>
> Thanks,
>
> Joel.
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20191125/0b0ac700/attachment.html>