25 Feb
2015
25 Feb
'15
12:38 p.m.
Hello,
following some discussions on github issues, mailing list and irc
recently, I am opening a topic and see if we can get something to
accommodate the needs out there at the best for everyone.
These are more in the context with the increase of sizes for
deployments, but also having the option to use a no-sql, some internal
bits of usrloc may need tunings.
The points to be considered:
1) discovering what is the server that wrote the record
2) efficiently getting the records that require nat keepalive
The two are sometime related, but not necessary always.
Victor Seva on irc channel was saying that is planning to propose a
patch for 1), relying on socket value in the location table. The use
case presented was many servers saving and looking up the records in the
same table, but fetch only the records written by current instance for
sending keepalives. It makes sense not send keepalives from all
instances, for load as well and bandwidth considerations. This could
work relying on local socket, but I am not sure how efficient will be
when listening on many sockets (different ips, but also different ports
or protocols - ipv4/6, tcp, tls, udp -- all add sockets).
On 2), it's being for a while on my list to review, as the module uses
custom sql queries (or even functions) to be able to match the the
records for sending keepalives -- it does matching with bitwise
operations to see if flags for nat are set. That doesn't work with
no-sql databases (including db_text, db_mongodb and I expect
db_cassandra). Even for sql, that kind of query is not efficient,
especially when dealing with a large amount of records.
The solutions that came in my mind for now:
- for 1) -- add the server_id as a new column in the location table. It
was pointed to be that ser location had it. This will turn a query
matching on many socket-representation strings to one expression done on
an integer. The value for server_id can be set via the core parameter
with the same name.
- for 2) -- add a new column 'keepalive' to be set internally by the
module to 1 if any of the flags for sending keepalive was set. Then the
query to fetch it will just use a match on it, rather that bitwise
operations in the match expression. Furthermore, this can be set to a
different value when enabling the keepalive partitioning (ie., sending
keepalives from different timer processes, each doing for a part of
natted users) - right now, for the db only mode, the selection is done
by column id % (number of keepalive processes).
Opinions, improvements, other proposals?
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany - http://www.kamailioworld.com
25 Feb
25 Feb
4:07 p.m.
On 02/25/2015 12:38 PM, Daniel-Constantin Mierla wrote:
[snip]
The solutions that came in my mind for now:
- for 1) -- add the server_id as a new column in the location table. It
was pointed to be that ser location had it. This will turn a query
matching on many socket-representation strings to one expression done on
an integer. The value for server_id can be set via the core parameter
with the same name.
way better than my approach of building the string for the query from
get_sock_info_list(proto)
- for 2) -- add a new column 'keepalive' to be
set internally by the
module to 1 if any of the flags for sending keepalive was set. Then the
query to fetch it will just use a match on it, rather that bitwise
operations in the match expression. Furthermore, this can be set to a
different value when enabling the keepalive partitioning (ie., sending
keepalives from different timer processes, each doing for a part of
natted users) - right now, for the db only mode, the selection is done
by column id % (number of keepalive processes).
really nice idea
Cheers,
Victor
26 Feb
26 Feb
4:55 a.m.
Victor Seva writes:
yes, i once tested and there was no performance boost due to indexing
because of the bitiwise operation.
-- juha
- for 2) --
add a new column 'keepalive' to be set internally by the
module to 1 if any of the flags for sending keepalive was set. Then the
query to fetch it will just use a match on it, rather that bitwise
operations in the match expression. Furthermore, this can be set to a
different value when enabling the keepalive partitioning (ie., sending
keepalives from different timer processes, each doing for a part of
natted users) - right now, for the db only mode, the selection is done
by column id % (number of keepalive processes).
really nice idea 
25 Feb
25 Feb
5:15 p.m.
#2 is great
Best
________________________________________
From: sr-dev [sr-dev-bounces(a)lists.sip-router.org] on behalf of Daniel-Constantin Mierla
[miconda(a)gmail.com]
Sent: Wednesday, February 25, 2015 3:38 AM
To: sr-dev
Subject: [sr-dev] usrloc and db only mode
Hello,
following some discussions on github issues, mailing list and irc
recently, I am opening a topic and see if we can get something to
accommodate the needs out there at the best for everyone.
These are more in the context with the increase of sizes for
deployments, but also having the option to use a no-sql, some internal
bits of usrloc may need tunings.
The points to be considered:
1) discovering what is the server that wrote the record
2) efficiently getting the records that require nat keepalive
The two are sometime related, but not necessary always.
Victor Seva on irc channel was saying that is planning to propose a
patch for 1), relying on socket value in the location table. The use
case presented was many servers saving and looking up the records in the
same table, but fetch only the records written by current instance for
sending keepalives. It makes sense not send keepalives from all
instances, for load as well and bandwidth considerations. This could
work relying on local socket, but I am not sure how efficient will be
when listening on many sockets (different ips, but also different ports
or protocols - ipv4/6, tcp, tls, udp -- all add sockets).
On 2), it's being for a while on my list to review, as the module uses
custom sql queries (or even functions) to be able to match the the
records for sending keepalives -- it does matching with bitwise
operations to see if flags for nat are set. That doesn't work with
no-sql databases (including db_text, db_mongodb and I expect
db_cassandra). Even for sql, that kind of query is not efficient,
especially when dealing with a large amount of records.
The solutions that came in my mind for now:
- for 1) -- add the server_id as a new column in the location table. It
was pointed to be that ser location had it. This will turn a query
matching on many socket-representation strings to one expression done on
an integer. The value for server_id can be set via the core parameter
with the same name.
- for 2) -- add a new column 'keepalive' to be set internally by the
module to 1 if any of the flags for sending keepalive was set. Then the
query to fetch it will just use a match on it, rather that bitwise
operations in the match expression. Furthermore, this can be set to a
different value when enabling the keepalive partitioning (ie., sending
keepalives from different timer processes, each doing for a part of
natted users) - right now, for the db only mode, the selection is done
by column id % (number of keepalive processes).
Opinions, improvements, other proposals?
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany - http://www.kamailioworld.com
_______________________________________________
sr-dev mailing list
sr-dev(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
26 Feb
26 Feb
9:03 a.m.
On Wednesday 25 February 2015, Daniel-Constantin Mierla wrote:
1) discovering what is the server that wrote the
record
Victor Seva on irc channel was saying that is planning
to propose a
patch for 1), relying on socket value in the location table. The use
case presented was many servers saving and looking up the records in the
same table, but fetch only the records written by current instance for
sending keepalives. It makes sense not send keepalives from all
instances, for load as well and bandwidth considerations. This could
work relying on local socket, but I am not sure how efficient will be
when listening on many sockets (different ips, but also different ports
or protocols - ipv4/6, tcp, tls, udp -- all add sockets).
The solutions that came in my mind for now:
- for 1) -- add the server_id as a new column in the location table. It
was pointed to be that ser location had it. This will turn a query
matching on many socket-representation strings to one expression done on
an integer. The value for server_id can be set via the core parameter
with the same name.
What would be the use case to share a location table between multiple
kamailio's?
And if it is shared, why would you want to filter per server? This seems
contradictory to sharing the table in the first place.
--
Greetings,
Alex Hermann
10:03 a.m.
On 02/26/2015 09:03 AM, Alex Hermann wrote:
What would be the use case to share a location table
between multiple
kamailio's?
Having a bunch of kamailio's working together?
And if it is shared, why would you want to filter per
server? This seems
contradictory to sharing the table in the first place.
We are talking about nathelper sending keepalives. Not for query a contact.
27 Mar
27 Mar
4:34 p.m.
Hello,
follow up to my original post, as the other replies diverted on
particular topics.
I pushed the first part of changes to usrloc to deal with the proposed
updates.
There are three new columns for each record:
- server_id - set to server_id global parameters
- connection_id - set to tcp connection id when it is the case. This is
because of some other discussions recently on mailing list to make the
tcp related optimizations possible for db-only mode. The value is not
restored on restart, given the connections were lost
- keepalive - right now is set to 1 if nat bflag is set. Optimization to
get the records for nat keepalive (and partitioning of them) to follow
At least the current version should unlock the plans for the others,
like Victor, so can extend on features based on the server id or
connection id when db-only is used.
The version of location table has been increased, therefore be careful
if you update from master branch, you need to adjust the values -- see:
-
http://kamailio.org/wiki/install/upgrade/stable-to-devel?&#mysql_databa…
Cheers,
Daniel
On 25/02/15 12:38, Daniel-Constantin Mierla wrote:
Hello,
following some discussions on github issues, mailing list and irc
recently, I am opening a topic and see if we can get something to
accommodate the needs out there at the best for everyone.
These are more in the context with the increase of sizes for
deployments, but also having the option to use a no-sql, some internal
bits of usrloc may need tunings.
The points to be considered:
1) discovering what is the server that wrote the record
2) efficiently getting the records that require nat keepalive
The two are sometime related, but not necessary always.
Victor Seva on irc channel was saying that is planning to propose a
patch for 1), relying on socket value in the location table. The use
case presented was many servers saving and looking up the records in the
same table, but fetch only the records written by current instance for
sending keepalives. It makes sense not send keepalives from all
instances, for load as well and bandwidth considerations. This could
work relying on local socket, but I am not sure how efficient will be
when listening on many sockets (different ips, but also different ports
or protocols - ipv4/6, tcp, tls, udp -- all add sockets).
On 2), it's being for a while on my list to review, as the module uses
custom sql queries (or even functions) to be able to match the the
records for sending keepalives -- it does matching with bitwise
operations to see if flags for nat are set. That doesn't work with
no-sql databases (including db_text, db_mongodb and I expect
db_cassandra). Even for sql, that kind of query is not efficient,
especially when dealing with a large amount of records.
The solutions that came in my mind for now:
- for 1) -- add the server_id as a new column in the location table. It
was pointed to be that ser location had it. This will turn a query
matching on many socket-representation strings to one expression done on
an integer. The value for server_id can be set via the core parameter
with the same name.
- for 2) -- add a new column 'keepalive' to be set internally by the
module to 1 if any of the flags for sending keepalive was set. Then the
query to fetch it will just use a match on it, rather that bitwise
operations in the match expression. Furthermore, this can be set to a
different value when enabling the keepalive partitioning (ie., sending
keepalives from different timer processes, each doing for a part of
natted users) - right now, for the db only mode, the selection is done
by column id % (number of keepalive processes).
Opinions, improvements, other proposals?
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany - http://www.kamailioworld.com
14 Apr
14 Apr
11:36 a.m.
Hi,
We were talking about the get_all_db_ucontacts function some weeks ago
on IRC. I finally had
sometime to think about that.
On 03/27/2015 04:34 PM, Daniel-Constantin Mierla wrote:
- keepalive - right now is set to 1 if nat bflag is
set. Optimization to
get the records for nat keepalive (and partitioning of them) to follow
Is something like this[0] what you had in mind? No idea how is suppose
to partition the
query ( limit offset ??)
[0]
https://github.com/linuxmaniac/kamailio/commit/d3b4ae71911239b69bd8c3f2bae4…
PS: Not tested
15 Apr
15 Apr
11 a.m.
Hello,
On 14/04/15 11:36, Victor Seva wrote:
Hi,
We were talking about the get_all_db_ucontacts function some weeks ago
on IRC. I finally had
sometime to think about that.
On 03/27/2015 04:34 PM, Daniel-Constantin Mierla wrote:
yes, something like that.
I haven't checked the code, but if the flags for filter were 0, all
records were returned, isn't it? To cover the case of ping all contacts.
Then, just to keep it in mind, the partitioning of the records is not
covered - I can take care of it later.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany - http://www.kamailioworld.com
- keepalive - right now is set to 1 if nat bflag
is set. Optimization to
get the records for nat keepalive (and partitioning of them) to follow
Is
something like this[0] what you had in mind? No idea how is suppose
to partition the
query ( limit offset ??)
[0]
https://github.com/linuxmaniac/kamailio/commit/d3b4ae71911239b69bd8c3f2bae4…
PS: Not tested
5:09 p.m.
On 04/15/2015 11:00 AM, Daniel-Constantin Mierla wrote:
yes, something like that.
I haven't checked the code, but if the flags for filter were 0, all
records were returned, isn't it? To cover the case of ping all contacts.
Now yes. I didn't know if we wanted that. I made some small changes too.
Then, just to keep it in mind, the partitioning of the
records is not
covered - I can take care of it later.
Yes, it is not covered.
4:40 p.m.
On 21/04/15 16:32, Victor Seva wrote:
On 04/15/2015 11:00 AM, Daniel-Constantin Mierla
wrote:
You can merge it if you want. It is anyhow a fix for the existing code,
which can be dealt during the testing period as well.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, May 27-29, 2015
Berlin, Germany - http://www.kamailioworld.com
Then, just to keep it in mind, the partitioning
of the records is not
covered - I can take care of it later.
Should I merge this before freeze?
3297
days inactive
3352
days old
11 comments
5 participants
participants (5)
-
Alex Hermann -
Daniel-Constantin Mierla -
Juha Heinanen -
Luis Azedo -
Victor Seva