12 Dec
2008
12 Dec
'08
1:25 a.m.
Hello,
I think we need to decide about AVPs. Although not deeply investigated,
ser seems to have couple of AVPs lists. In kamailio/openser it is one list.
Can a ser developer describe a bit the AVP implementation as it is now
there?
In openser/kamailio we have more or less same architecture as for ser
0.9.6 with couple extensions, but I will detail here so we can have full
picture:
- avps are kept in shared memory
- they are bound to each sip message and moved to transaction if that
message create a transaction
- avp can have two types of names
- integer id, referred as $avp(i:number)
- string id, referred as $avp(s:string)
- there can be aliases for avp names, defined as "alias=[is]:id", so the
script writer can use $avp(alias)
- a value of an avp can be either integer or string
- not related to avp core part, but important to mention here -- tm,
controlled by a parameter, can make the avps from transaction (request)
available to onreply_route.
Also, just to have in mind, for the future we got into discussion:
- moving from linked list to a structure that allows faster search
(e.g., hash table) as number of used avps can be big some times
- adding support for structure-like values (the value to be another list
of (name,value) pairs). A typical use case is serial forking, where one
needs to store data in avps. If for one destination an avp is missing,
things get screwed. Example:
$avp(uri) = {A, B, C}
$avp(RPID) = {X, not needed, Y}
Internally will look like:
$avp(uri) = {A, B, C}
$avp(RPID) = {X, Y}
So actually, Y will be used for destination B. Workarounds are possible,
or course, but maybe we can do it nicer. so it will be like:
$avp(dst) = {{u=A,r=Y}, {u=B},{u=C,r=Y}}
A possibility to specify the access is $avp(d:dst=>u)
Finally, as pointed in another email, it is about making possible to
hold "value null" (a mark/flag) and differentiate that from
"undefined"
(not existing).
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com
12 Dec
12 Dec
6:22 p.m.
New subject: [SR-Dev] [Serdev] AVPs
On 12-12 00:25, Daniel-Constantin Mierla wrote:
Hello,
I think we need to decide about AVPs. Although not deeply investigated,
ser seems to have couple of AVPs lists. In kamailio/openser it is one
list.
Can a ser developer describe a bit the AVP implementation as it is now
there?
A long time ago we started using AVPs to store all kinds of configuration
information. We needed some sort of general configuration mechanism because
real-world scripts were getting too complex and at that time the AVPs were
readily available.
We started by loading attribute names and value from a database table into
AVPs. This mechanism was/is same as in k. We would always load a set of
attributes this way for a particular user (identified by username and domain
from his/her SIP URI). With this mechanism in place, we were able to load a
generic set of configuration options for a particular user of the server, I
guess you have the same.
To make the AVPs easily accessible from the configuration script, we added
support for AVP identifiers in the config script, so that you can access AVP
with name foo as $foo.
As we migrated more and more configuration options to AVPs, it became clear
that having just one set of AVPs was not enough. Although we could load all
the config options for the caller (From URI) from the database into a set of
AVPs, we could not do the same for the callee (Request-URI) at the same
time. We could not do it because they would conflict with the AVPs of the
caller as both users could have AVPs with same names but different values.
To get around this issue we added another list of AVPs. The new AVP list
works the same way, it can contain AVPs with same names as AVPs in the
original list and they do not conflict. All functions that work with lists
of AVPs now take the list to work on as parameter.
To make both AVP lists available in the configuration script, we extended
the syntax of AVPs identifiers so the script write can choose the AVP list
to work with. SO instead of just $foo you can write either
$f.foo or $t.foo
$f refers to the original AVP list which is commonly associated with the
caller. The address/uid of the caller is taken from From header, hence the
'f' in the identifier. $t refers to the AVP list which contains
configuration settings of the callee. The address/uid of the calle can be
taken either from the Request-URI or To headers, hence the 't' in the
identifier.
The original syntax without any list identifier is still available, in other
words you can still write $foo, this is defined as a shorthand for
$f.foo. If you do not specify the AVP list to be used then you are referring
to the list containing AVPs of the caller (From).
It also turned out that in some cases we would be having too many attributes
in the database table storing AVPs. This can happen in bigger setups, having
tens or hundreds of thousands users or serving multiple domains. This can
slow the database down and makes SER load too much data. Based on my
observations it is common that a large number of users have AVPs with same
values. If ten thousands of users have attribute named 'foo' with value
'bar' then the attribute will be stored in the database ten thousand times.
As a remedy for this problem, we introduced the concept of AVP levels. The
AVPs described so far and stored in user_attrs table are called user-level
attributes and they store configuration specific to particular users. Then
we added another two AVP lists to store so called domain-level AVPs.
Domain-level AVPs are used to store configuration information that is shared
by a group of users. Domain-level AVPs are stored in a separate database
table, the name of the table is domain_attrs and its contents is cached in
memory by 'domain' SER module. This is an important difference, while
user-level AVPs need to be loaded every time a SIP request is processed,
domain-level AVPs are only loaded when the contents of domain_attrs table
has changed.
The domain-level AVPs are called 'domain' because they are tied to a
particular domain handled by the SIP server. That could be 'iptel.org',
'sip-router.org', and so on. This mainly useful for multi-domain
setups. There are two domain-level AVP lists because it is common that SER
works with two domains at the same time, the domain from From header
(caller's domain) and the domain from Request-URI or To (callee's domain).
Again, we extended the syntax of AVP identifiers in the configuration
file. So you can write:
$fu.foo -- this way you are asking for the value of the user-level foo AVP.
$fd.foo -- This will return the value of the domain-level foo AVP.
And similarly there is $tu and $td for callee's user-level and domain-level
AVPs. If you specify 'u' in the AVP identifiers then SER searches only the
list of user-level attributes. If you specify 'd' then SER searches only the
list of domain-level attributes.
This behavior changes if you do NOT specify the level to be searched. In
that case SER searches the user-level list first and if no match is found
then the domain-level list will be searched. Thus if you write:
$f.foo
then you are telling SER to search for the value of 'foo' AVP in the
user-level list and continue with the domain-level list if no match is
found. In other words, user-level AVPs have higher priority than
domain-level AVPs. With this system you can efficiently set a configuration
option for all users within a domain by creating a domain-level attribute
and yet you can override that option with user-level attributes for
particular users within that domain.
Then there are the global AVPs. Global AVPs can be used to store settings
applicable to the whole SIP server and all domains on that server. Global
AVPs are stored in global_attrs table and the contents of that table is
cached in memory by gflags module. There is only one list of global
AVPs. Global AVPs can be accessed with:
$g.foo
(note that there is no 'f' or 't' in the identifier). The list of
global
AVPs is searched after the list of domain-level AVPs. So if you write:
$f.foo
Then SER searches user-level AVPs first, then domain-level and then the
global AVPs.
And finally there are also so called URI-level AVPs. They work just
like user-level AVPs, but a single user can have multiple sets of
uri-level AVPs. They are tied to SIP URIs of that user. Uri-level AVPs are
denoted by 'r' in the AVP identifier, for example:
$fr.foo
In openser/kamailio we have more or less same
architecture as for ser
0.9.6 with couple extensions, but I will detail here so we can have full
picture:
- avps are kept in shared memory
In our case global and domain level AVPs are stored in shared memory.
- they are bound to each sip message and moved to
transaction if that
message create a transaction
This is same.
- avp can have two types of names
- integer id, referred as $avp(i:number)
- string id, referred as $avp(s:string)
- there can be aliases for avp names, defined as "alias=[is]:id", so the
script writer can use $avp(alias)
We've kind of dropped support for integer avps, we've been using string
names pretty much everywhere, although the original code still exists.
- a value of an avp can be either integer or string
- not related to avp core part, but important to mention here -- tm,
controlled by a parameter, can make the avps from transaction (request)
available to onreply_route.
In SER tm restores all AVPs stored in a transaction in all route blocks
called from tm, such as onreply_route, failure_route. At least I hope I
remember it correctly.
Jan.
14 Dec
14 Dec
9:59 a.m.
New subject: [SR-Dev] [Kamailio-Devel] [Serdev] AVPs
Jan Janak writes:
A long time ago we started using AVPs to store all
kinds of configuration
information. We needed some sort of general configuration mechanism because
real-world scripts were getting too complex and at that time the AVPs were
readily available.
jan,
i have been doing in kamailio exactly what you describe, but using radius.
a function that loads caller's attributes does not do anything to the
avp name (integer) in db, but a function that loads callee's attributes
adds a constant (say 100) to the name. in cfg file, i then define m4
macros, like this:
define(`CALLER_LANGUAGE_AVP', i:102)
define(`CALLEE_LANGUAGE_AVP', i:202)
i too have domain level avps in a separate table and when avp's of a
user (caller or callee) are loaded, domain level table is access first,
then user level table. this means that reply attributes returned from
radius are loaded into k avps in the same order. if there is only a
domain level attribute, that gets used, but if there are both, the user
level one that was loaded later overrides the domain one.
in addition to user and domain level, i have uri level, i.e., if an uri
of a user has uri level attribute, it overrides the corresponding user
level avp value.
all this has worked fine for a long time and for this purpose, i don't
need any additions whatsoever to k's avp implementation. for other
purposes (as daniel mentioned), it would be nice if avp value could be a
structure.
-- juha
16 Dec
16 Dec
1:36 a.m.
New subject: [SR-Dev] [Serdev] AVPs
Hello,
On 12/12/08 17:22, Jan Janak wrote:
--
Daniel-Constantin Mierla
http://www.asipto.com
On 12-12 00:25, Daniel-Constantin Mierla wrote:
so, if I understood correctly, there are couple of combinations between
the dot, with one or two letters, right?
Like:
$xy.name
where x = f|t|g and y may be missing or u|d|r
Then we can get rid of overlapping in the namespace if we enforce usage
of $f.foo for $foo . In K there are couple of PV that use dot in
classname (a PV can be $classname or $classname(innername)), but they
are proposed to be removed.
In the same idea as in ser, thenK AVPs will map to From list.
Cheers,
Daniel
Hello,
I think we need to decide about AVPs. Although not deeply investigated,
ser seems to have couple of AVPs lists. In kamailio/openser it is one
list.
Can a ser developer describe a bit the AVP implementation as it is now
there?
A long time ago we started using AVPs to store all kinds of configuration
information. We needed some sort of general configuration mechanism because
real-world scripts were getting too complex and at that time the AVPs were
readily available.
We started by loading attribute names and value from a database table into
AVPs. This mechanism was/is same as in k. We would always load a set of
attributes this way for a particular user (identified by username and domain
from his/her SIP URI). With this mechanism in place, we were able to load a
generic set of configuration options for a particular user of the server, I
guess you have the same.
To make the AVPs easily accessible from the configuration script, we added
support for AVP identifiers in the config script, so that you can access AVP
with name foo as $foo.
As we migrated more and more configuration options to AVPs, it became clear
that having just one set of AVPs was not enough. Although we could load all
the config options for the caller (From URI) from the database into a set of
AVPs, we could not do the same for the callee (Request-URI) at the same
time. We could not do it because they would conflict with the AVPs of the
caller as both users could have AVPs with same names but different values.
To get around this issue we added another list of AVPs. The new AVP list
works the same way, it can contain AVPs with same names as AVPs in the
original list and they do not conflict. All functions that work with lists
of AVPs now take the list to work on as parameter.
To make both AVP lists available in the configuration script, we extended
the syntax of AVPs identifiers so the script write can choose the AVP list
to work with. SO instead of just $foo you can write either
$f.foo or $t.foo
$f refers to the original AVP list which is commonly associated with the
caller. The address/uid of the caller is taken from From header, hence the
'f' in the identifier. $t refers to the AVP list which contains
configuration settings of the callee. The address/uid of the calle can be
taken either from the Request-URI or To headers, hence the 't' in the
identifier.
The original syntax without any list identifier is still available, in other
words you can still write $foo, this is defined as a shorthand for
$f.foo. If you do not specify the AVP list to be used then you are referring
to the list containing AVPs of the caller (From).
It also turned out that in some cases we would be having too many attributes
in the database table storing AVPs. This can happen in bigger setups, having
tens or hundreds of thousands users or serving multiple domains. This can
slow the database down and makes SER load too much data. Based on my
observations it is common that a large number of users have AVPs with same
values. If ten thousands of users have attribute named 'foo' with value
'bar' then the attribute will be stored in the database ten thousand times.
As a remedy for this problem, we introduced the concept of AVP levels. The
AVPs described so far and stored in user_attrs table are called user-level
attributes and they store configuration specific to particular users. Then
we added another two AVP lists to store so called domain-level AVPs.
Domain-level AVPs are used to store configuration information that is shared
by a group of users. Domain-level AVPs are stored in a separate database
table, the name of the table is domain_attrs and its contents is cached in
memory by 'domain' SER module. This is an important difference, while
user-level AVPs need to be loaded every time a SIP request is processed,
domain-level AVPs are only loaded when the contents of domain_attrs table
has changed.
The domain-level AVPs are called 'domain' because they are tied to a
particular domain handled by the SIP server. That could be 'iptel.org',
'sip-router.org', and so on. This mainly useful for multi-domain
setups. There are two domain-level AVP lists because it is common that SER
works with two domains at the same time, the domain from From header
(caller's domain) and the domain from Request-URI or To (callee's domain).
Again, we extended the syntax of AVP identifiers in the configuration
file. So you can write:
$fu.foo -- this way you are asking for the value of the user-level foo AVP.
$fd.foo -- This will return the value of the domain-level foo AVP.
And similarly there is $tu and $td for callee's user-level and domain-level
AVPs. If you specify 'u' in the AVP identifiers then SER searches only the
list of user-level attributes. If you specify 'd' then SER searches only the
list of domain-level attributes.
This behavior changes if you do NOT specify the level to be searched. In
that case SER searches the user-level list first and if no match is found
then the domain-level list will be searched. Thus if you write:
$f.foo
then you are telling SER to search for the value of 'foo' AVP in the
user-level list and continue with the domain-level list if no match is
found. In other words, user-level AVPs have higher priority than
domain-level AVPs. With this system you can efficiently set a configuration
option for all users within a domain by creating a domain-level attribute
and yet you can override that option with user-level attributes for
particular users within that domain.
Then there are the global AVPs. Global AVPs can be used to store settings
applicable to the whole SIP server and all domains on that server. Global
AVPs are stored in global_attrs table and the contents of that table is
cached in memory by gflags module. There is only one list of global
AVPs. Global AVPs can be accessed with:
$g.foo
(note that there is no 'f' or 't' in the identifier). The list of
global
AVPs is searched after the list of domain-level AVPs. So if you write:
$f.foo
Then SER searches user-level AVPs first, then domain-level and then the
global AVPs.
And finally there are also so called URI-level AVPs. They work just
like user-level AVPs, but a single user can have multiple sets of
uri-level AVPs. They are tied to SIP URIs of that user. Uri-level AVPs are
denoted by 'r' in the AVP identifier, for example:
$fr.foo
In openser/kamailio we have more or less same
architecture as for ser
0.9.6 with couple extensions, but I will detail here so we can have full
picture:
- avps are kept in shared memory
In our case global and domain level AVPs are stored in shared memory.
- they are bound to each sip message and moved to
transaction if that
message create a transaction
This is same.
- avp can have two types of names
- integer id, referred as $avp(i:number)
- string id, referred as $avp(s:string)
- there can be aliases for avp names, defined as "alias=[is]:id", so the
script writer can use $avp(alias)
We've kind of dropped support for integer avps, we've been using string
names pretty much everywhere, although the original code still exists.
- a value of an avp can be either integer or
string
- not related to avp core part, but important to mention here -- tm,
controlled by a parameter, can make the avps from transaction (request)
available to onreply_route.
In SER tm restores all AVPs stored in a transaction in all route blocks
called from tm, such as onreply_route, failure_route. At least I hope I
remember it correctly.
Jan.
_______________________________________________
sr-dev mailing list
sr-dev(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
11:15 a.m.
New subject: [SR-Dev] [Serdev] AVPs
On Dec 16, 2008 at 00:36, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
On 12/12/08 17:22, Jan Janak wrote:
x = f|t ("track" in ser avp terminology
y= r|u|d|g ("class")
g is a special case (it cannot be combined with f or t).
Here is the parse_avp_ident() usage comment:
/** parse an avp indentifier.
*
* Parses the following avp indentifier forms:
* - "i:<number>" - old form, deprecated (e.g. i:42)
* - "s:<string>" - old form, deprecated (e.g. s:foo)
* - "<track>.<name>" (e.g.: f.bar)
* - "<track>.<name>[<index>]" (e.g.:
f.bar[1])
* - "<track><class>.<name>" (e.g:
tu.bar)
* - "<track><class>.<name>[<index>]" (e.g:
fd.bar[2])
* - "<string>" (e.g.: foo)
* Where:
* <string> = ascii string
* <id> = ascii string w/o '[', ']', '.' and
'/'
* <name> = <id> | '/' regex '/'
* (Note: regex use is deprecated)
* <track> = 'f' | 't'
* (from or to)
* <class> = 'r' | 'u' | 'd' | 'g'
* (uri, user, domain or global)
* <index> = <number> | '-' <number> | ''
* (the avp index, if missing it means AVP_INDEX_ALL, but
* it's use is deprecated)
* More examples:
* "fr.bar[1]" - from track, uri class, avp "bar", the value
1.
* "tu./^foo/" - to track, user class, all avps for which the name
* starts with foo (note RE in avp names are deprecated).
* "t.did" - to track, "did" avp
*
* @param name - avp identifier
* @param *attr - the result will be stored here
* @return 0 on success, -1 on error
*/
On 12-12 00:25, Daniel-Constantin Mierla wrote:
so, if I understood correctly, there are couple of combinations between
the dot, with one or two letters, right?
Like:
$xy.name
where x = f|t|g and y may be missing or u|d|r Hello,
I think we need to decide about AVPs. Although not deeply investigated,
ser seems to have couple of AVPs lists. In kamailio/openser it is one
list.
Can a ser developer describe a bit the AVP implementation as it is now
there?
A long time ago we started using AVPs to store all kinds of configuration
information. We needed some sort of general configuration mechanism because
real-world scripts were getting too complex and at that time the AVPs were
readily available.
We started by loading attribute names and value from a database table into
AVPs. This mechanism was/is same as in k. We would always load a set of
attributes this way for a particular user (identified by username and domain
from his/her SIP URI). With this mechanism in place, we were able to load a
generic set of configuration options for a particular user of the server, I
guess you have the same.
To make the AVPs easily accessible from the configuration script, we added
support for AVP identifiers in the config script, so that you can access AVP
with name foo as $foo.
As we migrated more and more configuration options to AVPs, it became clear
that having just one set of AVPs was not enough. Although we could load all
the config options for the caller (From URI) from the database into a set of
AVPs, we could not do the same for the callee (Request-URI) at the same
time. We could not do it because they would conflict with the AVPs of the
caller as both users could have AVPs with same names but different values.
To get around this issue we added another list of AVPs. The new AVP list
works the same way, it can contain AVPs with same names as AVPs in the
original list and they do not conflict. All functions that work with lists
of AVPs now take the list to work on as parameter.
To make both AVP lists available in the configuration script, we extended
the syntax of AVPs identifiers so the script write can choose the AVP list
to work with. SO instead of just $foo you can write either
$f.foo or $t.foo
$f refers to the original AVP list which is commonly associated with the
caller. The address/uid of the caller is taken from From header, hence the
'f' in the identifier. $t refers to the AVP list which contains
configuration settings of the callee. The address/uid of the calle can be
taken either from the Request-URI or To headers, hence the 't' in the
identifier.
The original syntax without any list identifier is still available, in other
words you can still write $foo, this is defined as a shorthand for
$f.foo. If you do not specify the AVP list to be used then you are referring
to the list containing AVPs of the caller (From).
It also turned out that in some cases we would be having too many attributes
in the database table storing AVPs. This can happen in bigger setups, having
tens or hundreds of thousands users or serving multiple domains. This can
slow the database down and makes SER load too much data. Based on my
observations it is common that a large number of users have AVPs with same
values. If ten thousands of users have attribute named 'foo' with value
'bar' then the attribute will be stored in the database ten thousand times.
As a remedy for this problem, we introduced the concept of AVP levels. The
AVPs described so far and stored in user_attrs table are called user-level
attributes and they store configuration specific to particular users. Then
we added another two AVP lists to store so called domain-level AVPs.
Domain-level AVPs are used to store configuration information that is shared
by a group of users. Domain-level AVPs are stored in a separate database
table, the name of the table is domain_attrs and its contents is cached in
memory by 'domain' SER module. This is an important difference, while
user-level AVPs need to be loaded every time a SIP request is processed,
domain-level AVPs are only loaded when the contents of domain_attrs table
has changed.
The domain-level AVPs are called 'domain' because they are tied to a
particular domain handled by the SIP server. That could be 'iptel.org',
'sip-router.org', and so on. This mainly useful for multi-domain
setups. There are two domain-level AVP lists because it is common that SER
works with two domains at the same time, the domain from From header
(caller's domain) and the domain from Request-URI or To (callee's domain).
Again, we extended the syntax of AVP identifiers in the configuration
file. So you can write:
$fu.foo -- this way you are asking for the value of the user-level foo AVP.
$fd.foo -- This will return the value of the domain-level foo AVP.
And similarly there is $tu and $td for callee's user-level and domain-level
AVPs. If you specify 'u' in the AVP identifiers then SER searches only the
list of user-level attributes. If you specify 'd' then SER searches only the
list of domain-level attributes.
This behavior changes if you do NOT specify the level to be searched. In
that case SER searches the user-level list first and if no match is found
then the domain-level list will be searched. Thus if you write:
$f.foo
then you are telling SER to search for the value of 'foo' AVP in the
user-level list and continue with the domain-level list if no match is
found. In other words, user-level AVPs have higher priority than
domain-level AVPs. With this system you can efficiently set a configuration
option for all users within a domain by creating a domain-level attribute
and yet you can override that option with user-level attributes for
particular users within that domain.
Then there are the global AVPs. Global AVPs can be used to store settings
applicable to the whole SIP server and all domains on that server. Global
AVPs are stored in global_attrs table and the contents of that table is
cached in memory by gflags module. There is only one list of global
AVPs. Global AVPs can be accessed with:
$g.foo
(note that there is no 'f' or 't' in the identifier). The list of
global
AVPs is searched after the list of domain-level AVPs. So if you write:
$f.foo
Then SER searches user-level AVPs first, then domain-level and then the
global AVPs.
And finally there are also so called URI-level AVPs. They work just
like user-level AVPs, but a single user can have multiple sets of
uri-level AVPs. They are tied to SIP URIs of that user. Uri-level AVPs are
denoted by 'r' in the AVP identifier, for example:
$fr.foo
Then we can get rid of overlapping in the namespace if we enforce usage
of $f.foo for $foo . In K there are couple of PV that use dot in
classname (a PV can be $classname or $classname(innername)), but they
are proposed to be removed.
For the time being in sip-router this is handled in the following way:
$(...) -> pvar
$foo(...) -> pvar
$foo[...] -> avp
$bar.foo -> avp (if it has a dot in it -> avp)
$foo (no dot, [] or ()) -> try first to resolve it as pvar and if it
fails consider it an avp.
For the current version I think this is the most compatible way. I'll
use the same approach for ser module fixups (so all ser module that use
core fixup functions will be able to use also pvars).
One could also force kamailio or ser behaviour, by starting the
script with #!KAMAILIO or #!SER.
For future versions we should rethink how everything is accessed.
For example I would use $foo for script variables and some other form
for avps or the current pvar $foo. Also we would have to unify selects
with the select like part of the pvars.
(Note for ser users: pvars in kamailio are used to access script vars,
avps, message headers and something which is kind of similar to ser
selects.).
In the same idea as in ser, thenK AVPs will map to From list.
[...]
Andrei
11:52 a.m.
New subject: [SR-Dev] [Serdev] AVPs
On 12/16/08 10:15, Andrei Pelinescu-Onciul wrote:
On Dec 16, 2008 at 00:36, Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
yes, indeed,
thanks!
Hello,
On 12/12/08 17:22, Jan Janak wrote:
x = f|t ("track" in ser avp terminology
y= r|u|d|g ("class")
g is a special case (it cannot be combined with f or t).
Here is the parse_avp_ident() usage comment:
/** parse an avp indentifier.
*
* Parses the following avp indentifier forms:
* - "i:<number>" - old form, deprecated (e.g. i:42)
* - "s:<string>" - old form, deprecated (e.g. s:foo)
* - "<track>.<name>" (e.g.: f.bar)
* - "<track>.<name>[<index>]" (e.g.:
f.bar[1])
* - "<track><class>.<name>" (e.g:
tu.bar)
* - "<track><class>.<name>[<index>]" (e.g:
fd.bar[2])
* - "<string>" (e.g.: foo)
* Where:
* <string> = ascii string
* <id> = ascii string w/o '[', ']', '.' and
'/'
* <name> = <id> | '/' regex '/'
* (Note: regex use is deprecated)
* <track> = 'f' | 't'
* (from or to)
* <class> = 'r' | 'u' | 'd' | 'g'
* (uri, user, domain or global)
* <index> = <number> | '-' <number> | ''
* (the avp index, if missing it means AVP_INDEX_ALL, but
* it's use is deprecated)
* More examples:
* "fr.bar[1]" - from track, uri class, avp "bar", the value
1.
* "tu./^foo/" - to track, user class, all avps for which the name
* starts with foo (note RE in avp names are deprecated).
* "t.did" - to track, "did" avp
*
* @param name - avp identifier
* @param *attr - the result will be stored here
* @return 0 on success, -1 on error
*/
On 12-12 00:25, Daniel-Constantin Mierla wrote:
so, if I understood correctly, there are couple of combinations between
the dot, with one or two letters, right?
Like:
$xy.name
where x = f|t|g and y may be missing or u|d|r
Hello,
I think we need to decide about AVPs. Although not deeply investigated,
ser seems to have couple of AVPs lists. In kamailio/openser it is one
list.
Can a ser developer describe a bit the AVP implementation as it is now
there?
A long time ago we started using AVPs to store all kinds of
configuration
information. We needed some sort of general configuration mechanism because
real-world scripts were getting too complex and at that time the AVPs were
readily available.
We started by loading attribute names and value from a database table into
AVPs. This mechanism was/is same as in k. We would always load a set of
attributes this way for a particular user (identified by username and domain
from his/her SIP URI). With this mechanism in place, we were able to load a
generic set of configuration options for a particular user of the server, I
guess you have the same.
To make the AVPs easily accessible from the configuration script, we added
support for AVP identifiers in the config script, so that you can access AVP
with name foo as $foo.
As we migrated more and more configuration options to AVPs, it became clear
that having just one set of AVPs was not enough. Although we could load all
the config options for the caller (From URI) from the database into a set of
AVPs, we could not do the same for the callee (Request-URI) at the same
time. We could not do it because they would conflict with the AVPs of the
caller as both users could have AVPs with same names but different values.
To get around this issue we added another list of AVPs. The new AVP list
works the same way, it can contain AVPs with same names as AVPs in the
original list and they do not conflict. All functions that work with lists
of AVPs now take the list to work on as parameter.
To make both AVP lists available in the configuration script, we extended
the syntax of AVPs identifiers so the script write can choose the AVP list
to work with. SO instead of just $foo you can write either
$f.foo or $t.foo
$f refers to the original AVP list which is commonly associated with the
caller. The address/uid of the caller is taken from From header, hence the
'f' in the identifier. $t refers to the AVP list which contains
configuration settings of the callee. The address/uid of the calle can be
taken either from the Request-URI or To headers, hence the 't' in the
identifier.
The original syntax without any list identifier is still available, in other
words you can still write $foo, this is defined as a shorthand for
$f.foo. If you do not specify the AVP list to be used then you are referring
to the list containing AVPs of the caller (From).
It also turned out that in some cases we would be having too many attributes
in the database table storing AVPs. This can happen in bigger setups, having
tens or hundreds of thousands users or serving multiple domains. This can
slow the database down and makes SER load too much data. Based on my
observations it is common that a large number of users have AVPs with same
values. If ten thousands of users have attribute named 'foo' with value
'bar' then the attribute will be stored in the database ten thousand times.
As a remedy for this problem, we introduced the concept of AVP levels. The
AVPs described so far and stored in user_attrs table are called user-level
attributes and they store configuration specific to particular users. Then
we added another two AVP lists to store so called domain-level AVPs.
Domain-level AVPs are used to store configuration information that is shared
by a group of users. Domain-level AVPs are stored in a separate database
table, the name of the table is domain_attrs and its contents is cached in
memory by 'domain' SER module. This is an important difference, while
user-level AVPs need to be loaded every time a SIP request is processed,
domain-level AVPs are only loaded when the contents of domain_attrs table
has changed.
The domain-level AVPs are called 'domain' because they are tied to a
particular domain handled by the SIP server. That could be 'iptel.org',
'sip-router.org', and so on. This mainly useful for multi-domain
setups. There are two domain-level AVP lists because it is common that SER
works with two domains at the same time, the domain from From header
(caller's domain) and the domain from Request-URI or To (callee's domain).
Again, we extended the syntax of AVP identifiers in the configuration
file. So you can write:
$fu.foo -- this way you are asking for the value of the user-level foo AVP.
$fd.foo -- This will return the value of the domain-level foo AVP.
And similarly there is $tu and $td for callee's user-level and domain-level
AVPs. If you specify 'u' in the AVP identifiers then SER searches only the
list of user-level attributes. If you specify 'd' then SER searches only the
list of domain-level attributes.
This behavior changes if you do NOT specify the level to be searched. In
that case SER searches the user-level list first and if no match is found
then the domain-level list will be searched. Thus if you write:
$f.foo
then you are telling SER to search for the value of 'foo' AVP in the
user-level list and continue with the domain-level list if no match is
found. In other words, user-level AVPs have higher priority than
domain-level AVPs. With this system you can efficiently set a configuration
option for all users within a domain by creating a domain-level attribute
and yet you can override that option with user-level attributes for
particular users within that domain.
Then there are the global AVPs. Global AVPs can be used to store settings
applicable to the whole SIP server and all domains on that server. Global
AVPs are stored in global_attrs table and the contents of that table is
cached in memory by gflags module. There is only one list of global
AVPs. Global AVPs can be accessed with:
$g.foo
(note that there is no 'f' or 't' in the identifier). The list of
global
AVPs is searched after the list of domain-level AVPs. So if you write:
$f.foo
Then SER searches user-level AVPs first, then domain-level and then the
global AVPs.
And finally there are also so called URI-level AVPs. They work just
like user-level AVPs, but a single user can have multiple sets of
uri-level AVPs. They are tied to SIP URIs of that user. Uri-level AVPs are
denoted by 'r' in the AVP identifier, for example:
$fr.foo
Then we can get rid of overlapping in the
namespace if we enforce usage
of $f.foo for $foo . In K there are couple of PV that use dot in
classname (a PV can be $classname or $classname(innername)), but they
are proposed to be removed.
For the time being in sip-router this is handled in the following way:
$(...) -> pvar
$foo(...) -> pvar
$foo[...] -> avp
$bar.foo -> avp (if it has a dot in it -> avp)
$foo (no dot, [] or ()) -> try first to resolve it as pvar and if it
fails consider it an avp.
For the current version I think this is the most compatible way. I'll
use the same approach for ser module fixups (so all ser module that use
core fixup functions will be able to use also pvars).
One could also force kamailio or ser behaviour, by starting the
script with #!KAMAILIO or #!SER.
For future versions we should rethink how everything is accessed.
For example I would use $foo for script variables and some other form
for avps or the current pvar $foo. Also we would have to unify selects
with the select like part of the pvars.
(Note for ser users: pvars in kamailio are used to access script vars,
avps, message headers and something which is kind of similar to ser
selects.).
Perhaps select implementations can be moved in modules, as well.
Integration has to be done at some point anyhow.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com
4:39 p.m.
New subject: [SR-Dev] [Serdev] AVPs
On 16-12 10:52, Daniel-Constantin Mierla wrote:
On 12/16/08 10:15, Andrei Pelinescu-Onciul wrote:
It already is, the core contains the core selects and shared parts.
Jan.
On Dec 16, 2008 at 00:36, Daniel-Constantin
Mierla <miconda(a)gmail.com> wrote:
yes, indeed,
thanks!
Hello,
On 12/12/08 17:22, Jan Janak wrote:
x = f|t ("track" in ser avp terminology
y= r|u|d|g ("class")
g is a special case (it cannot be combined with f or t).
Here is the parse_avp_ident() usage comment:
/** parse an avp indentifier.
*
* Parses the following avp indentifier forms:
* - "i:<number>" - old form, deprecated (e.g. i:42)
* - "s:<string>" - old form, deprecated (e.g. s:foo)
* - "<track>.<name>" (e.g.: f.bar)
* - "<track>.<name>[<index>]" (e.g.:
f.bar[1])
* - "<track><class>.<name>" (e.g:
tu.bar)
* - "<track><class>.<name>[<index>]" (e.g:
fd.bar[2])
* - "<string>" (e.g.: foo)
* Where:
* <string> = ascii string
* <id> = ascii string w/o '[', ']', '.' and
'/'
* <name> = <id> | '/' regex '/'
* (Note: regex use is deprecated)
* <track> = 'f' | 't'
* (from or to)
* <class> = 'r' | 'u' | 'd' | 'g'
* (uri, user, domain or global)
* <index> = <number> | '-' <number> | ''
* (the avp index, if missing it means AVP_INDEX_ALL, but
* it's use is deprecated)
* More examples:
* "fr.bar[1]" - from track, uri class, avp "bar", the value
1.
* "tu./^foo/" - to track, user class, all avps for which the name
* starts with foo (note RE in avp names are deprecated).
* "t.did" - to track, "did" avp
*
* @param name - avp identifier
* @param *attr - the result will be stored here
* @return 0 on success, -1 on error
*/
On 12-12 00:25, Daniel-Constantin Mierla wrote:
> Hello,
>
> I think we need to decide about AVPs. Although not deeply
> investigated, ser seems to have couple of AVPs lists. In
> kamailio/openser it is one list.
>
> Can a ser developer describe a bit the AVP implementation as it
> is now there?
>
A long time ago we started using AVPs to store all kinds of configuration
information. We needed some sort of general configuration mechanism because
real-world scripts were getting too complex and at that time the AVPs were
readily available.
We started by loading attribute names and value from a database table into
AVPs. This mechanism was/is same as in k. We would always load a set of
attributes this way for a particular user (identified by username and domain
from his/her SIP URI). With this mechanism in place, we were able to load a
generic set of configuration options for a particular user of the server, I
guess you have the same.
To make the AVPs easily accessible from the configuration script, we added
support for AVP identifiers in the config script, so that you can access AVP
with name foo as $foo.
As we migrated more and more configuration options to AVPs, it became clear
that having just one set of AVPs was not enough. Although we could load all
the config options for the caller (From URI) from the database into a set of
AVPs, we could not do the same for the callee (Request-URI) at the same
time. We could not do it because they would conflict with the
AVPs of the caller as both users could have AVPs with same names
but different values.
To get around this issue we added another list of AVPs. The new AVP list
works the same way, it can contain AVPs with same names as AVPs in the
original list and they do not conflict. All functions that work with lists
of AVPs now take the list to work on as parameter.
To make both AVP lists available in the configuration script, we extended
the syntax of AVPs identifiers so the script write can choose the AVP list
to work with. SO instead of just $foo you can write either
$f.foo or $t.foo
$f refers to the original AVP list which is commonly associated with the
caller. The address/uid of the caller is taken from From header, hence the
'f' in the identifier. $t refers to the AVP list which contains
configuration settings of the callee. The address/uid of the calle can be
taken either from the Request-URI or To headers, hence the 't' in the
identifier.
The original syntax without any list identifier is still available, in other
words you can still write $foo, this is defined as a shorthand for
$f.foo. If you do not specify the AVP list to be used then you are referring
to the list containing AVPs of the caller (From).
It also turned out that in some cases we would be having too many attributes
in the database table storing AVPs. This can happen in bigger setups, having
tens or hundreds of thousands users or serving multiple domains. This can
slow the database down and makes SER load too much data. Based on my
observations it is common that a large number of users have AVPs with same
values. If ten thousands of users have attribute named 'foo' with value
'bar' then the attribute will be stored in the database ten thousand times.
As a remedy for this problem, we introduced the concept of AVP levels. The
AVPs described so far and stored in user_attrs table are called user-level
attributes and they store configuration specific to particular users. Then
we added another two AVP lists to store so called domain-level AVPs.
Domain-level AVPs are used to store configuration information that is shared
by a group of users. Domain-level AVPs are stored in a separate database
table, the name of the table is domain_attrs and its contents is cached in
memory by 'domain' SER module. This is an important difference, while
user-level AVPs need to be loaded every time a SIP request is processed,
domain-level AVPs are only loaded when the contents of domain_attrs table
has changed.
The domain-level AVPs are called 'domain' because they are tied to a
particular domain handled by the SIP server. That could be 'iptel.org',
'sip-router.org', and so on. This mainly useful for multi-domain
setups. There are two domain-level AVP lists because it is common that SER
works with two domains at the same time, the domain from From header
(caller's domain) and the domain from Request-URI or To (callee's domain).
Again, we extended the syntax of AVP identifiers in the configuration
file. So you can write:
$fu.foo -- this way you are asking for the value of the user-level foo AVP.
$fd.foo -- This will return the value of the domain-level foo AVP.
And similarly there is $tu and $td for callee's user-level and domain-level
AVPs. If you specify 'u' in the AVP identifiers then SER searches only the
list of user-level attributes. If you specify 'd' then SER searches only the
list of domain-level attributes.
This behavior changes if you do NOT specify the level to be searched. In
that case SER searches the user-level list first and if no match is found
then the domain-level list will be searched. Thus if you write:
$f.foo
then you are telling SER to search for the value of 'foo' AVP in the
user-level list and continue with the domain-level list if no match is
found. In other words, user-level AVPs have higher priority than
domain-level AVPs. With this system you can efficiently set a configuration
option for all users within a domain by creating a domain-level attribute
and yet you can override that option with user-level attributes for
particular users within that domain.
Then there are the global AVPs. Global AVPs can be used to store settings
applicable to the whole SIP server and all domains on that server. Global
AVPs are stored in global_attrs table and the contents of that table is
cached in memory by gflags module. There is only one list of global
AVPs. Global AVPs can be accessed with:
$g.foo
(note that there is no 'f' or 't' in the identifier). The list of
global
AVPs is searched after the list of domain-level AVPs. So if you write:
$f.foo
Then SER searches user-level AVPs first, then domain-level and then the
global AVPs.
And finally there are also so called URI-level AVPs. They work just
like user-level AVPs, but a single user can have multiple sets of
uri-level AVPs. They are tied to SIP URIs of that user. Uri-level AVPs are
denoted by 'r' in the AVP identifier, for example:
$fr.foo
so, if I understood correctly, there are couple of combinations
between the dot, with one or two letters, right?
Like:
$xy.name
where x = f|t|g and y may be missing or u|d|r
Then we can get rid of overlapping in the
namespace if we enforce
usage of $f.foo for $foo . In K there are couple of PV that use dot
in classname (a PV can be $classname or $classname(innername)), but
they are proposed to be removed.
For the time being in sip-router this is handled in the following way:
$(...) -> pvar
$foo(...) -> pvar
$foo[...] -> avp
$bar.foo -> avp (if it has a dot in it -> avp)
$foo (no dot, [] or ()) -> try first to resolve it as pvar and if it
fails consider it an avp.
For the current version I think this is the most compatible way. I'll
use the same approach for ser module fixups (so all ser module that use
core fixup functions will be able to use also pvars).
One could also force kamailio or ser behaviour, by starting the
script with #!KAMAILIO or #!SER.
For future versions we should rethink how everything is accessed.
For example I would use $foo for script variables and some other form
for avps or the current pvar $foo. Also we would have to unify selects
with the select like part of the pvars.
(Note for ser users: pvars in kamailio are used to access script vars,
avps, message headers and something which is kind of similar to ser
selects.).
Perhaps select implementations can be moved in modules, as well.
Integration has to be done at some point anyhow.
4:49 p.m.
New subject: [SR-Dev] [Serdev] AVPs
On Dec 16, 2008 at 14:39, Jan Janak <jan(a)ryngle.com> wrote:
[...]
Perhaps select implementations can be moved
in modules, as well.
Integration has to be done at some point anyhow.
It already is, the core contains the core selects and shared parts.
I think Daniel meant moving also the core selects into a select module
and leave only the shared part in core (like he did for pvars in k).
Andrei
5:33 p.m.
New subject: [SR-Dev] [Serdev] AVPs
On 16-12 14:49, Andrei Pelinescu-Onciul wrote:
On Dec 16, 2008 at 14:39, Jan Janak
<jan(a)ryngle.com> wrote:
[...]
Perhaps select implementations can be moved
in modules, as well.
Integration has to be done at some point anyhow.
It already is, the core contains the core selects and shared parts.
I think Daniel meant moving also the core selects into a select module
and leave only the shared part in core (like he did for pvars in k).
Would this be a smart thing to do? Lots of modules use or depend on
selects. Maybe we could turn it into a library if there is a need to move it
away from the core, but I personally would leave it where it is now.
Jan.
5:44 p.m.
New subject: [SR-Dev] [Serdev] AVPs
On 12/16/08 16:33, Jan Janak wrote:
On 16-12 14:49, Andrei Pelinescu-Onciul wrote:
same was in K with PVs, most of modules depend on them. But the core
gets pretty fat over the time will all there. While couple of them are
pretty used, some are not at all. Because there will be pseudo-variables
and selects, with lot of overlapping, keeping in core just the shared
part is an optimal solution.
This step can be done along with integration with PVs (removal of
duplicates), perhaps second release of sip-router.
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com
On Dec 16, 2008 at 14:39, Jan Janak
<jan(a)ryngle.com> wrote:
[...]
I think Daniel meant moving also the core selects into a select module
and leave only the shared part in core (like he did for pvars in k).
Would this be a smart thing to do? Lots of modules use or depend on
selects. Maybe we could turn it into a library if there is a need to move it
away from the core, but I personally would leave it where it is now.
>
>
Perhaps select implementations can be moved in modules, as well.
Integration has to be done at some point anyhow.
It already is, the core contains the core selects and shared parts.
5:58 p.m.
New subject: [SR-Dev] [Serdev] AVPs
On 16-12 16:44, Daniel-Constantin Mierla wrote:
On 12/16/08 16:33, Jan Janak wrote:
I don't understand. Are you suggesting to move selects somewhere else or do
you want to keep it the way it is now? Note that by "selects" I was refering
to the select framework, i.e. the code that is used to parse select
identifiers, lookup functions to be called and calling those functions.
That code is in ser core now, along with a set of select implementation
functions that didn't fit anywhere else.
Jan.
On 16-12 14:49, Andrei Pelinescu-Onciul wrote:
same was in K with PVs, most of modules depend on them. But the core
gets pretty fat over the time will all there. While couple of them are
pretty used, some are not at all. Because there will be pseudo-variables
and selects, with lot of overlapping, keeping in core just the shared
part is an optimal solution.
This step can be done along with integration with PVs (removal of
duplicates), perhaps second release of sip-router. On Dec 16, 2008 at 14:39, Jan Janak
<jan(a)ryngle.com> wrote:
[...]
Would this be a smart thing to do? Lots of modules use or depend on
selects. Maybe we could turn it into a library if there is a need to move it
away from the core, but I personally would leave it where it is now.
>>
> Perhaps select implementations can be moved in modules, as well.
> Integration has to be done at some point anyhow.
>
It already is, the core contains the core selects and shared parts.
I think Daniel meant moving also the core selects into a select module
and leave only the shared part in core (like he did for pvars in k).
5:59 p.m.
New subject: [SR-Dev] [Serdev] AVPs
On Dec 16, 2008 at 15:58, Jan Janak <jan(a)ryngle.com> wrote:
On 16-12 16:44, Daniel-Constantin Mierla wrote:
No. it's not about the framework, is about the "functions that didn't fit
anywhere else" being moved to some select module in the future (from my
point of view this has very very low priority).
Andrei
On 12/16/08 16:33, Jan Janak wrote:
I don't understand. Are you suggesting to move selects somewhere else or do
you want to keep it the way it is now? Note that by "selects" I was refering
to the select framework, i.e. the code that is used to parse select
identifiers, lookup functions to be called and calling those functions.
That code is in ser core now, along with a set of select implementation
functions that didn't fit anywhere else. On 16-12 14:49, Andrei Pelinescu-Onciul wrote:
same was in K with PVs, most of modules depend on them. But the core
gets pretty fat over the time will all there. While couple of them are
pretty used, some are not at all. Because there will be pseudo-variables
and selects, with lot of overlapping, keeping in core just the shared
part is an optimal solution.
This step can be done along with integration with PVs (removal of
duplicates), perhaps second release of sip-router. On Dec 16, 2008 at 14:39, Jan Janak
<jan(a)ryngle.com> wrote:
[...]
>>>
>> Perhaps select implementations can be moved in modules, as well.
>> Integration has to be done at some point anyhow.
>>
> It already is, the core contains the core selects and shared parts.
>
I think Daniel meant moving also the core selects into a select module
and leave only the shared part in core (like he did for pvars in k).
Would this be a smart thing to do? Lots of modules use or depend on
selects. Maybe we could turn it into a library if there is a need to move it
away from the core, but I personally would leave it where it is now.
6:12 p.m.
New subject: [SR-Dev] [Serdev] AVPs
On 16-12 15:59, Andrei Pelinescu-Onciul wrote:
On Dec 16, 2008 at 15:58, Jan Janak
<jan(a)ryngle.com> wrote:
Ah, I understand now, thanks. Yes, moving those functions away from the core
makes sense to me.
Jan.
On 16-12 16:44, Daniel-Constantin Mierla wrote:
No. it's not about the framework, is about the "functions that didn't fit
anywhere else" being moved to some select module in the future (from my
point of view this has very very low priority). On 12/16/08 16:33, Jan Janak wrote:
I don't understand. Are you suggesting to move selects somewhere else or do
you want to keep it the way it is now? Note that by "selects" I was refering
to the select framework, i.e. the code that is used to parse select
identifiers, lookup functions to be called and calling those functions.
That code is in ser core now, along with a set of select implementation
functions that didn't fit anywhere else. On 16-12 14:49, Andrei Pelinescu-Onciul wrote:
> On Dec 16, 2008 at 14:39, Jan Janak <jan(a)ryngle.com> wrote:
> [...]
>
>>>>
>>> Perhaps select implementations can be moved in modules, as well.
>>> Integration has to be done at some point anyhow.
>>>
>> It already is, the core contains the core selects and shared parts.
>>
> I think Daniel meant moving also the core selects into a select module
> and leave only the shared part in core (like he did for pvars in k).
>
Would this be a smart thing to do? Lots of modules use or depend on
selects. Maybe we could turn it into a library if there is a need to move it
away from the core, but I personally would leave it where it is now.
same was in K with PVs, most of modules depend on them. But the core
gets pretty fat over the time will all there. While couple of them are
pretty used, some are not at all. Because there will be pseudo-variables
and selects, with lot of overlapping, keeping in core just the shared
part is an optimal solution.
This step can be done along with integration with PVs (removal of
duplicates), perhaps second release of sip-router.
5810
days inactive
5815
days old
12 comments
4 participants
participants (4)
-
Andrei Pelinescu-Onciul -
Daniel-Constantin Mierla -
Jan Janak -
jh＠tutpro.com