1 Feb
2011
1 Feb
'11
7:18 p.m.
Hi all,
I'd like to bring forward a proposal for creating CDRs (Call Data
Records) from Kamailio by means of extending both acc and dialog
modules. Feedback is very welcome.
So far, CDR generation requires post-processing of log entries created
by the (transaction-based) acc module or some other means. Looking at
OpenSIPS (which is already capable of producing CDRs), my idea is to
copy the basic notion of tracking dialogs for to-be-accounted calls and
log a proper CDR on termination. Creating basic CDRs including start
time, end time, and call duration can be accomplished quite easily this way.
Apart from this very minimum CDR content, however, one can think of a
number of CDR fields that cannot be filled easily. For instance, caller
identity isn't always contained in the From header (think of CLIR calls)
but need to be determined from other headers depending on the type of
call, possibly even involving a database lookup. That's why I'd like to
provide a way to pass additional, CDR-specific data from the
configuration file to the dialog structure associated with the call.
When the dialog is about to terminate, the extended acc module would
make sure that the added CDR data associated to the dialog is included
in the CDR. This way, users may define CDR fields to their personal
needs from the configuration file in a flexible manner.
In order to accomplish this, the dialog module would need to be extended
such that dialog-specific data may be stored for the duration of a
dialog (which is not possible to this day AFAICS). A new setter function
attached to the dialog interface (struct dlg_binds) could accept a
simple key/value pair for storage purposes. Exporting this function to
the configuration file and accompanying it with a corresponding getter
function will ensure that dialog-specific data may be written and read
generically. Thus, improvements to the dialog module may be leveraged
not just for CDR-accounting purposes.
Regarding the acc module, a couple of new features need to be
implemented: First, the introduction of another module parameter called
something like "cdr_fields" that comprises the set of key names
designated for CDR inclusion. Hence, a line like
modparam("acc", "cdr_fields", "caller, callee, foo, bar")
would require Kamailio configurators to store dialog-specific values for
the keys "caller", "callee", "foo", and "bar",
respectively, during call
processing. The values will be fed to the dialog structure employing the
new dialog store function explained above.
Second, when the acc module is in CDR mode, dialogs shall be tracked
from creation to termination using dialog callbacks. On termination of a
dialog, values from keys referenced in the "cdr_fields" parameter will
be fetched from the terminating dialog structure and associated with the
new, fresh CDR.
Third, the CDR is persisted to either log file, database, or both.
Note that my approach would not touch the existing acc module
functionality: CDR mode is supposed to be working in addition to
transaction-based accounting as people will likely want to keep
transaction logs for debugging or similar needs. A flag would enable and
disable CDR generation at will. I have not fully investigated
re-usability possibilities but believe that acc's persistence functions
may be taken advantage of by the new CDR-producing parts as well.
That's it with regards to the basic mode of operation. I am also
thinking of having a few other features, such as using a switch to set
the call start time to the moment the ACK or OK is seen instead of the
INVITE (because users shouldn't be billed for call setup times).
Looking forward to seeing your feedback on this proposal.
Cheers,
--Timo
1 Feb
1 Feb
8:15 p.m.
Hello,
On 2/1/11 8:18 PM, Timo Reimann wrote:
[...]
Apart from this very minimum CDR content, however, one can think of a
number of CDR fields that cannot be filled easily. For instance, caller
identity isn't always contained in the From header (think of CLIR calls)
but need to be determined from other headers depending on the type of
call, possibly even involving a database lookup.
From header URI is not accounted
automatically, you can specify any
variable that can contain caller id as you need (e.g., it can be an AVP
that you previously set for in config) -- see *_extra parameters of acc
module.
That's why I'd like to
provide a way to pass additional, CDR-specific data from the
configuration file to the dialog structure associated with the call.
When the dialog is about to terminate, the extended acc module would
make sure that the added CDR data associated to the dialog is included
in the CDR.
Not sure what you mean here with "acc module will make sure ...", but I
hope is not going to be cross reference/dependency, so that acc has to
walk through dlg structures.
This way, users may define CDR fields to their
personal
needs from the configuration file in a flexible manner.
In order to accomplish this, the dialog module would need to be extended
such that dialog-specific data may be stored for the duration of a
dialog (which is not possible to this day AFAICS).
Carsen committed some code in this regard in his IMS branch, as I could
see from commit log, check:
http://lists.sip-router.org/pipermail/sr-dev/2011-January/010197.html
Probably it will be merged sometime in the near future.
A new setter function
attached to the dialog interface (struct dlg_binds) could accept a
simple key/value pair for storage purposes. Exporting this function to
the configuration file and accompanying it with a corresponding getter
function will ensure that dialog-specific data may be written and read
generically. Thus, improvements to the dialog module may be leveraged
not just for CDR-accounting purposes.
Regarding the acc module, a couple of new features need to be
implemented: First, the introduction of another module parameter called
something like "cdr_fields" that comprises the set of key names
designated for CDR inclusion. Hence, a line like
modparam("acc", "cdr_fields", "caller, callee, foo,
bar")
The db_extra has the format of 'key=variable', where the 'key' is the db
column name and the 'variable' is the name of PV holding the value to be
stored. I think the format is better than just providing the names of
the dialog keys.
would require Kamailio configurators to store
dialog-specific values for
the keys "caller", "callee", "foo", and "bar",
respectively, during call
processing. The values will be fed to the dialog structure employing the
new dialog store function explained above.
I think exporting anything related to dialog via PV and fetching them
like this will be better from point of view of cross-relationships.
Second, when the acc module is in CDR mode, dialogs
shall be tracked
from creation to termination using dialog callbacks. On termination of a
dialog, values from keys referenced in the "cdr_fields" parameter will
be fetched from the terminating dialog structure and associated with the
new, fresh CDR.
Third, the CDR is persisted to either log file, database, or both.
If this new thing is not going to support what acc module API has for
backends (radius is missing), then will not make sense to tie the two.
dialog module can do its accounting alone.
Note that my approach would not touch the existing acc
module
functionality: CDR mode is supposed to be working in addition to
transaction-based accounting as people will likely want to keep
transaction logs for debugging or similar needs. A flag would enable and
disable CDR generation at will. I have not fully investigated
re-usability possibilities but believe that acc's persistence functions
may be taken advantage of by the new CDR-producing parts as well.
That's it with regards to the basic mode of operation. I am also
thinking of having a few other features, such as using a switch to set
the call start time to the moment the ACK or OK is seen instead of the
INVITE (because users shouldn't be billed for call setup times).
Looking forward to seeing your feedback on this proposal.
As general observation,
there are some modules (I mainly think about
presence) that have cross references/dependencies that should be avoided
normally, and planned to clean it up already. It is not good to have A
uses/needs B and B uses/needs A
Regarding db accounting:
- I have in my todo list the plan to enhance acc to export a functions
like acc_start() and acc_stop(), with the goal of storing the initial
record for start of the call (acc_start() for INVITE) then at the end of
the call (BYE) acc_stop() will just set extra details (end time,
duration, etc)
- practically then dialog module can call acc_start() and acc_stop() via
some inter-module api. The benefit is that even sip server crashes
suddently, there is an acc start event to indicate a call
- such functionality is independent of dlg module or any new call
tracking extension in the future, also writing full CDR can be achieved
from config file by tracking INVITE and BYE, calling
acc_start()/acc_stop() from cfg
Therefore if I would do it:
- enhance acc module to export via cfg exports and inter-module api
three functions:
- acc_start() - write the initial call record at start
- acc_stop() - update the call record at stop, based on a matching
condition specified as parameter
- acc_cdr() - write a full CDR
Data to be written in db (or other backend) is going to be taken from
PVs, independent of who (cfg, dialog, or other module) is calling the
function, specified in a similar form like db_extra.
First two functions will work for db only. Third can work also without
dialog, e.g., I can store the start of a call, a.s.o. in hash table and
get it at BYE time to build the full cdr.
Your needs seem to fit in 3, so if you can (wish to) work on that only,
I will take care of the first two when I can.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com
2 Feb
2 Feb
11:52 a.m.
Hey Daniel,
On 01.02.2011 21:15, Daniel-Constantin Mierla wrote:
If I get you right, by cross reference/dependency you mean that A
requires B and vice versa, i.e., cyclic dependency. I do not intend to
let that happen.
Instead, the acc module would use getter functions attached to the
dialog interface (which is what I mean by "acc module will make sure").
dialog would never touch the acc module or even know about the fact that
the acc module is using it.
Missed that one. Will take a look at it.
See my comment above for why binding PVs to CDR fields does not seem
appropriate in this case.
Of course, I would want to take advantage of acc's existing backend
connectors. There's no need to re-invent the wheel, acc will still be
responsible for writing out CDRs; the difference is that a few
additional calls to a well-defined dialog interface will be used in
order to collect the data that constitute the CDR.
On 2/1/11 8:18 PM, Timo Reimann wrote:
I initially thought about having a cdr_extra parameter similar to
log_extra. However, I see a problem with AVPs which live in single
transactions to work with dialogs that span multiple transactions.
Say you want to have a CDR field that contains combined or concatenated
data from multiple transactions, e.g., all Kamailio flags set in the
INVITE, ACK, and BYE transaction. How could that be accomplished with a
log_extra-like module parameter? At what times would AVP be parsed?
IMHO, you will need to keep such data per dialog which is why I came up
with the idea of storing CDR-specific data in the dialog.
[...]
Apart from this very minimum CDR content, however, one can think of a
number of CDR fields that cannot be filled easily. For instance, caller
identity isn't always contained in the From header (think of CLIR calls)
but need to be determined from other headers depending on the type of
call, possibly even involving a database lookup.
From header URI is not accounted
automatically, you can specify any
variable that can contain caller id as you need (e.g., it can be an AVP
that you previously set for in config) -- see *_extra parameters of acc
module. That's why
I'd like to
provide a way to pass additional, CDR-specific data from the
configuration file to the dialog structure associated with the call.
When the dialog is about to terminate, the extended acc module would
make sure that the added CDR data associated to the dialog is included
in the CDR.
Not sure what you mean here with "acc module will make sure ...", but I
hope is not going to be cross reference/dependency, so that acc has to
walk through dlg structures. In order to
accomplish this, the dialog module would need to be extended
such that dialog-specific data may be stored for the duration of a
dialog (which is not possible to this day AFAICS).
Carsen committed some code in this regard in his IMS branch, as I could
see from commit log, check:
http://lists.sip-router.org/pipermail/sr-dev/2011-January/010197.html Regarding the
acc module, a couple of new features need to be
implemented: First, the introduction of another module parameter called
something like "cdr_fields" that comprises the set of key names
designated for CDR inclusion. Hence, a line like
modparam("acc", "cdr_fields", "caller, callee, foo,
bar")
The db_extra has the format of 'key=variable', where the 'key' is the db
column name and the 'variable' is the name of PV holding the value to be
stored. I think the format is better than just providing the names of
the dialog keys. Third, the CDR
is persisted to either log file, database, or both.
If this new thing is not going to support what acc module API has for
backends (radius is missing), then will not make sense to tie the two.
dialog module can do its accounting alone. Regarding db accounting:
- I have in my todo list the plan to enhance acc to export a functions
like acc_start() and acc_stop(), with the goal of storing the initial
record for start of the call (acc_start() for INVITE) then at the end of
the call (BYE) acc_stop() will just set extra details (end time,
duration, etc)
- practically then dialog module can call acc_start() and acc_stop() via
some inter-module api. The benefit is that even sip server crashes
suddently, there is an acc start event to indicate a call
Regarding server crashes/restarts, instead of persisting events ASAP via
acc module I would recommend using dialog's existing feature to store
dialog data in a database. That way, every piece of CDR-specific data
stored in dialogs in my approach will be saved automatically (as
governed by dialog module configuration), and there will be no need to
produce partial CDRs in the first place.
- such functionality is independent of dlg module or
any new call
tracking extension in the future, also writing full CDR can be achieved
from config file by tracking INVITE and BYE, calling
acc_start()/acc_stop() from cfg
Tracking INVITE and BYE messages from the config file may not be enough
if you want to create CDRs for failed calls too, e.g., 408/487. They may
not be relevant for billing but having them in a CDR database may still
be desired.
Does your approach allow automatic CDR generation, or will you need to
call these acc_* functions explicitly? I'd prefer to have automatic CDR
persisting, similar to how transaction-based accounting in the current
acc module works.
Therefore if I would do it:
- enhance acc module to export via cfg exports and inter-module api
three functions:
- acc_start() - write the initial call record at start
- acc_stop() - update the call record at stop, based on a matching
condition specified as parameter
- acc_cdr() - write a full CDR
Data to be written in db (or other backend) is going to be taken from
PVs, independent of who (cfg, dialog, or other module) is calling the
function, specified in a similar form like db_extra.
First two functions will work for db only. Third can work also without
dialog, e.g., I can store the start of a call, a.s.o. in hash table and
get it at BYE time to build the full cdr.
Where would you maintain that hash table containing the call start time
and similar data if you do not seek to use the dialog module? Such data
seems to be naturally associated to a dialog, so my impression is that
it should be kept there too, i.e., in the dialog module.
Cheers,
--Timo
1:03 p.m.
Hello,
On 2/2/11 12:52 PM, Timo Reimann wrote:
If I get you right, by cross reference/dependency you
mean that A
requires B and vice versa, i.e., cyclic dependency. I do not intend to
let that happen.
Instead, the acc module would use getter functions attached to the
dialog interface (which is what I mean by "acc module will make sure").
dialog would never touch the acc module or even know about the fact that
the acc module is using it.
But then acc has to know the dialog internals, what the getter function
will return and how to access those structures. PV framework is exactly
the same, but available for all components/modules, no need to develop
specif ones each time.
With your solution, dialog module has to know acc api to call the
recording function, and acc has to know dialog module to know its
exported structures. This is cross dependency imo.
Missed that
one. Will take a look at it.
See my comment above for why binding PVs to CDR fields does not
seem
appropriate in this case.
I still don't see your point.
Of course, I would want to take
advantage of acc's existing backend
connectors. There's no need to re-invent the wheel, acc will still be
responsible for writing out CDRs; the difference is that a few
additional calls to a well-defined dialog interface will be used in
order to collect the data that constitute the CDR.
What will happen if a new call tracking module will be available, should
acc be changed again?
What stops tracking
these replies in the cfg?
Anyhow, I think you divert from the initial goal. If you just break the
message in pieces and reply on each word will lose the meaning of the
entire content. I presented my plans above and below I added acc_cdr()
that seems to suit your needs.
Hey Daniel,
On 01.02.2011 21:15, Daniel-Constantin Mierla wrote:
well, some of the avps are transaction persistent, but there are also
global avps which are available as long as kamailio runs. None of these
should be used, I gave avp just as generic PV example. The idea was to
make new PV available with the data stored by dlg module.
On 2/1/11 8:18 PM, Timo Reimann wrote:
I initially thought about having a cdr_extra parameter similar to
log_extra. However, I see a problem with AVPs which live in single
transactions to work with dialogs that span multiple transactions. [...]
Apart from this very minimum CDR content, however, one can think of a
number of CDR fields that cannot be filled easily. For instance, caller
identity isn't always contained in the From header (think of CLIR calls)
but need to be determined from other headers depending on the type of
call, possibly even involving a database lookup.
From header URI is not accounted
automatically, you can specify any
variable that can contain caller id as you need (e.g., it can be an AVP
that you previously set for in config) -- see *_extra parameters of acc
module. Say you want to have a CDR field that contains
combined or concatenated
data from multiple transactions, e.g., all Kamailio flags set in the
INVITE, ACK, and BYE transaction. How could that be accomplished with a
log_extra-like module parameter? At what times would AVP be parsed?
IMHO, you will need to keep such data per dialog which is why I came up
with the idea of storing CDR-specific data in the dialog.
It is not about where the data is store, it is about how acc accesses
that data. I would like to avoid acc module being aware of dialog module
internals. In the future might be different call tracking extensions, I
don't want to change acc each time.
That's why we have PVs, if I want to record something from XYZ module,
acc is not going to be changed and understand what xyz stores
internally, but XYZ should export some PVs for that.
For example, if I want to record something from a HTABLE, there is
nothing to do in acc, just specify $sht(x=>mykey) to some of acc parameters.
That's why I'd like to
provide a way to pass additional, CDR-specific data from the
configuration file to the dialog structure associated with the call.
When the dialog is about to terminate, the extended acc module would
make sure that the added CDR data associated to the dialog is included
in the CDR.
Not sure what you mean here with "acc module will make sure
...", but I
hope is not going to be cross reference/dependency, so that acc has to
walk through dlg structures. In order to accomplish this, the dialog module would
need to be extended
such that dialog-specific data may be stored for the duration of a
dialog (which is not possible to this day AFAICS).
Carsen committed some code in
this regard in his IMS branch, as I could
see from commit log, check:
http://lists.sip-router.org/pipermail/sr-dev/2011-January/010197.html Regarding
the acc module, a couple of new features need to be
implemented: First, the introduction of another module parameter called
something like "cdr_fields" that comprises the set of key names
designated for CDR inclusion. Hence, a line like
modparam("acc", "cdr_fields", "caller, callee, foo,
bar")
The db_extra has the format of 'key=variable', where the
'key' is the db
column name and the 'variable' is the name of PV holding the value to be
stored. I think the format is better than just providing the names of
the dialog keys. Third, the CDR is persisted to either log file,
database, or both.
If this new thing is not going to support what acc module API
has for
backends (radius is missing), then will not make sense to tie the two.
dialog module can do its accounting alone. Regarding db
accounting:
- I have in my todo list the plan to enhance acc to export a functions
like acc_start() and acc_stop(), with the goal of storing the initial
record for start of the call (acc_start() for INVITE) then at the end of
the call (BYE) acc_stop() will just set extra details (end time,
duration, etc)
- practically then dialog module can call acc_start() and acc_stop() via
some inter-module api. The benefit is that even sip server crashes
suddently, there is an acc start event to indicate a call
Regarding server
crashes/restarts, instead of persisting events ASAP via
acc module I would recommend using dialog's existing feature to store
dialog data in a database. That way, every piece of CDR-specific data
stored in dialogs in my approach will be saved automatically (as
governed by dialog module configuration), and there will be no need to
produce partial CDRs in the first place.
- such functionality is independent of dlg module
or any new call
tracking extension in the future, also writing full CDR can be achieved
from config file by tracking INVITE and BYE, calling
acc_start()/acc_stop() from cfg
Tracking INVITE and BYE messages from the config
file may not be enough
if you want to create CDRs for failed calls too, e.g., 408/487. They may
not be relevant for billing but having them in a CDR database may still
be desired.
Does your approach allow automatic CDR generation, or will you need to
call these acc_* functions explicitly? I'd prefer to have automatic CDR
persisting, similar to how transaction-based accounting in the current
acc module works.
The hash table module
(htable) keeps the data in shared memory. It is a
generic container, so you can store anything there. dialog module is
just an automatic call tracker, handling sip messages passing through
kamailio. Most of that can be done from config file. Internally, dialog
module has also a hash table. Also, I can track active calls using a db
table (in memory or not for speed), with a timer process (rtimer module)
to handle missing BYEs, and so there could be other options to track
dialogs. My point here is that acc should be independent of all these
potential options available now or added in the future.
You additions to dialog module can export the new stored data inside
dialog via some PV, e.g.,
$dlgacc(caller), $dlgacc(callee), $dlgacc(duration), a.s.o.
dialog module will call acc_cdr() when the call ended, making sure the
PVs are properly available at that time.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://www.asipto.com
Therefore if I would do it:
- enhance acc module to export via cfg exports and inter-module api
three functions:
- acc_start() - write the initial call record at start
- acc_stop() - update the call record at stop, based on a matching
condition specified as parameter
- acc_cdr() - write a full CDR
Data to be written in db (or other backend) is going to be taken from
PVs, independent of who (cfg, dialog, or other module) is calling the
function, specified in a similar form like db_extra.
First two functions will work for db only. Third can work also without
dialog, e.g., I can store the start of a call, a.s.o. in hash table and
get it at BYE time to build the full cdr.
Where would you maintain that hash table
containing the call start time
and similar data if you do not seek to use the dialog module? Such data
seems to be naturally associated to a dialog, so my impression is that
it should be kept there too, i.e., in the dialog module.
4:05 p.m.
On 02.02.2011 14:03, Daniel-Constantin Mierla wrote:
Ok, so instead of providing getter/setter functions, let's use PVs.
However, users will have varying needs regarding the to-be-defined CDR
fields next to the canonical ones (start time, end time, duration,
etc.). We, for example, use a rather large number of fields, many of
them being too specific for general integration into upstream Kamailio.
That's why I would rather see a way to dynamically specify CDR fields.
Carsten's new code would be the right approach to do this, right? His
dlg_var PV takes arbitrary key/value pairs which looks just like what is
needed.
If I get you right, by cross reference/dependency you
mean that A
requires B and vice versa, i.e., cyclic dependency. I do not intend to
let that happen.
Instead, the acc module would use getter functions attached to the
dialog interface (which is what I mean by "acc module will make sure").
dialog would never touch the acc module or even know about the fact that
the acc module is using it.
But then acc has to know the dialog internals, what the getter function
will return and how to access those structures. PV framework is exactly
the same, but available for all components/modules, no need to develop
specif ones each time.
Agreed, let's use PVs. If you think of Carsten's new dlg_var PV, it's
basically just a way to get/set arbitrary values; so I suppose this is
what I should have expressed.
Of course, I would want to take
advantage of acc's existing backend
connectors. There's no need to re-invent the wheel, acc will still be
responsible for writing out CDRs; the difference is that a few
additional calls to a well-defined dialog interface will be used in
order to collect the data that constitute the CDR.
What will happen if a new call tracking module will be available, should
acc be changed again?
I am not completely convinced that call tracking is such a flexible
thing even though you gave a few examples below. A call is pretty much a
dialog, and proper tracking of a dialog should be dialog module's job.
Moreover, I believe your approach has some dependency issues as well --
see my comment at the very end of this mail.
What stops
tracking these replies in the cfg?
You can do it, but for a large config file with many different paths,
it's easy to miss a spot where you should have added a call to
acc_stop(). On the contrary, if recording happens automatically when the
call ends, you don't have to deal with any manual work.
The hash table module
(htable) keeps the data in shared memory. It is a
generic container, so you can store anything there. dialog module is
just an automatic call tracker, handling sip messages passing through
kamailio. Most of that can be done from config file. Internally, dialog
module has also a hash table. Also, I can track active calls using a db
table (in memory or not for speed), with a timer process (rtimer module)
to handle missing BYEs, and so there could be other options to track
dialogs. My point here is that acc should be independent of all these
potential options available now or added in the future.
You additions to dialog module can export the new stored data inside
dialog via some PV, e.g.,
$dlgacc(caller), $dlgacc(callee), $dlgacc(duration), a.s.o.
dialog module will call acc_cdr() when the call ended, making sure the
PVs are properly available at that time.
If you call an acc module-provided method from the dialog module, you
face a similar independence problem: Should accounting functionality
ever be changed or replaced of by another module, then you'd need to
change the dialog implementation as well.
There's a conceptual issue related to the given usage relationship as
well: If component A requires B to implement a particular functionality,
A should use B. Having the dialog module trigger the recording function
puts this concept upside down -- CDR accounting isn't needed by dialog
tracking, it's the other way around.
Cheers,
--Timo
On 01.02.2011
21:15, Daniel-Constantin Mierla wrote:
well, some of the avps are transaction persistent, but there are also
global avps which are available as long as kamailio runs. None of these
should be used, I gave avp just as generic PV example. The idea was to
make new PV available with the data stored by dlg module. On 2/1/11 8:18 PM, Timo Reimann wrote:
I initially thought about having a cdr_extra parameter similar to
log_extra. However, I see a problem with AVPs which live in single
transactions to work with dialogs that span multiple transactions. [...]
Apart from this very minimum CDR content, however, one can think of a
number of CDR fields that cannot be filled easily. For instance, caller
identity isn't always contained in the From header (think of CLIR
calls)
but need to be determined from other headers depending on the type of
call, possibly even involving a database lookup.
From header URI is not accounted
automatically, you can specify any
variable that can contain caller id as you need (e.g., it can be an AVP
that you previously set for in config) -- see *_extra parameters of acc
module. That's why
I'd like to
provide a way to pass additional, CDR-specific data from the
configuration file to the dialog structure associated with the call.
When the dialog is about to terminate, the extended acc module would
make sure that the added CDR data associated to the dialog is included
in the CDR.
Not sure what you mean here with "acc module will make sure
...", but I
hope is not going to be cross reference/dependency, so that acc has to
walk through dlg structures. With your solution, dialog module has to know acc api
to call the
recording function, and acc has to know dialog module to know its
exported structures. This is cross dependency imo.
I think this point shows the major difference between our solutions: My
approach doesn't make the dialog module call the recording function.
Instead, the recording function is called by the acc module as soon as
the dialog terminates where call termination can is observed by means of
using dialog callbacks. (Sorry, I should have possibly mentioned the
words "dialog callback" before.)
Therefore, module dependency is unidirectional: acc registers for some
dialog callbacks and, upon execution of a callback, records the CDR.
Third, the CDR
is persisted to either log file, database, or both.
If this new thing is not going
to support what acc module API has for
backends (radius is missing), then will not make sense to tie the two.
dialog module can do its accounting alone. - such functionality is independent of dlg module or
any new call
tracking extension in the future, also writing full CDR can be achieved
from config file by tracking INVITE and BYE, calling
acc_start()/acc_stop() from cfg
Tracking INVITE and BYE messages from the config
file may not be enough
if you want to create CDRs for failed calls too, e.g., 408/487. Anyhow, I think you divert from the initial goal. If
you just break the
message in pieces and reply on each word will lose the meaning of the
entire content. I presented my plans above and below I added acc_cdr()
that seems to suit your needs.
Maybe I got you wrong but my understanding is that acc_start() records
one part of a CDR (start time, etc.) when the INVITE is routed while
acc_stop() finishes it off (end time, duration, etc.) on reception of
the BYE. I was trying to stress that a single call to the recording
function is likely preferred (at least by me), which acc_cdr() seems to
accomplish.
Therefore if I would do it:
- enhance acc module to export via cfg exports and inter-module api
three functions:
- acc_start() - write the initial call record at start
- acc_stop() - update the call record at stop, based on a matching
condition specified as parameter
- acc_cdr() - write a full CDR
Data to be written in db (or other backend) is going to be taken from
PVs, independent of who (cfg, dialog, or other module) is calling the
function, specified in a similar form like db_extra.
First two functions will work for db only. Third can work also without
dialog, e.g., I can store the start of a call, a.s.o. in hash table and
get it at BYE time to build the full cdr.
Where would you maintain that hash table
containing the call start time
and similar data if you do not seek to use the dialog module? Such data
seems to be naturally associated to a dialog, so my impression is that
it should be kept there too, i.e., in the dialog module. 
7 Feb
7 Feb
11:04 a.m.
New subject: Proposal to extend acc/dialog modules in order to log CDRs
Hi,
The functionality for storing additional data for a dialog is like this:
$dlg_var(ecscf_server) = ‘sip:ecscf.ng-voice.com’;
xlog(“Emergency CSCF is $dlg_var(ecscf_server)\n”);
However, it might be worth noting, that i haven't added bindings to
the dialog-API and that the Pseudo-Variable currently only supports
Strings.
I think, my branch needs to exists a little longer, since the changes
take quite a lot of time. I am currently doing a lot of cleanup where
many methods get obsolete; so i do not want to add docs for later
obsolete methods and parameters.
I think i will need to check, how i can easily just merge some commits
instead of all of them (any hints welcome, otherwise i will ask
google...).
Kind regards,
Carsten
2011/2/2 Timo Reimann <timo.reimann(a)1und1.de>de>:
On 02.02.2011 14:03, Daniel-Constantin Mierla wrote:
Ok, so instead of providing getter/setter functions, let's use PVs.
However, users will have varying needs regarding the to-be-defined CDR
fields next to the canonical ones (start time, end time, duration,
etc.). We, for example, use a rather large number of fields, many of
them being too specific for general integration into upstream Kamailio.
That's why I would rather see a way to dynamically specify CDR fields.
Carsten's new code would be the right approach to do this, right? His
dlg_var PV takes arbitrary key/value pairs which looks just like what is
needed.
But then acc has to know the dialog internals, what the getter function
will return and how to access those structures. PV framework is exactly
the same, but available for all components/modules, no need to develop
specif ones each time.
Agreed, let's use PVs. If you think of Carsten's new dlg_var PV, it's
basically just a way to get/set arbitrary values; so I suppose this is
what I should have expressed.
What will happen if a new call tracking module will be available, should
acc be changed again?
I am not completely convinced that call tracking is such a flexible
thing even though you gave a few examples below. A call is pretty much a
dialog, and proper tracking of a dialog should be dialog module's job.
Moreover, I believe your approach has some dependency issues as well --
see my comment at the very end of this mail.
What stops
tracking these replies in the cfg?
You can do it, but for a large config file with many different paths,
it's easy to miss a spot where you should have added a call to
acc_stop(). On the contrary, if recording happens automatically when the
call ends, you don't have to deal with any manual work.
The hash table module
(htable) keeps the data in shared memory. It is a
generic container, so you can store anything there. dialog module is
just an automatic call tracker, handling sip messages passing through
kamailio. Most of that can be done from config file. Internally, dialog
module has also a hash table. Also, I can track active calls using a db
table (in memory or not for speed), with a timer process (rtimer module)
to handle missing BYEs, and so there could be other options to track
dialogs. My point here is that acc should be independent of all these
potential options available now or added in the future.
You additions to dialog module can export the new stored data inside
dialog via some PV, e.g.,
$dlgacc(caller), $dlgacc(callee), $dlgacc(duration), a.s.o.
dialog module will call acc_cdr() when the call ended, making sure the
PVs are properly available at that time.
If you call an acc module-provided method from the dialog module, you
face a similar independence problem: Should accounting functionality
ever be changed or replaced of by another module, then you'd need to
change the dialog implementation as well.
There's a conceptual issue related to the given usage relationship as
well: If component A requires B to implement a particular functionality,
A should use B. Having the dialog module trigger the recording function
puts this concept upside down -- CDR accounting isn't needed by dialog
tracking, it's the other way around.
Cheers,
--Timo
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Carsten Bock
Schomburgstr. 80
22767 Hamburg
Germany
Mobile +49 179 2021244
Home +49 40 34927217
Büro (Verl) +49 5246 801427
Fax +49 40 34927218
mailto:carsten@bock.info
On
01.02.2011 21:15, Daniel-Constantin Mierla wrote:
well, some of the avps are transaction persistent, but there are also
global avps which are available as long as kamailio runs. None of these
should be used, I gave avp just as generic PV example. The idea was to
make new PV available with the data stored by dlg module. On 2/1/11 8:18 PM, Timo Reimann wrote:
> [...]
>
> Apart from this very minimum CDR content, however, one can think of a
> number of CDR fields that cannot be filled easily. For instance, caller
> identity isn't always contained in the From header (think of CLIR
> calls)
> but need to be determined from other headers depending on the type of
> call, possibly even involving a database lookup.
From header URI is not accounted automatically, you can specify any
variable that can contain caller id as you need (e.g., it can be an AVP
that you previously set for in config) -- see *_extra parameters of acc
module.
I initially thought about having a cdr_extra parameter similar to
log_extra. However, I see a problem with AVPs which live in single
transactions to work with dialogs that span multiple transactions. > That's why I'd like to
> provide a way to pass additional, CDR-specific data from the
> configuration file to the dialog structure associated with the call.
> When the dialog is about to terminate, the extended acc module would
> make sure that the added CDR data associated to the dialog is included
> in the CDR.
Not sure what you mean here with "acc module will make sure ...", but I
hope is not going to be cross reference/dependency, so that acc has to
walk through dlg structures.
If I get you right, by cross reference/dependency you
mean that A
requires B and vice versa, i.e., cyclic dependency. I do not intend to
let that happen.
Instead, the acc module would use getter functions attached to the
dialog interface (which is what I mean by "acc module will make sure").
dialog would never touch the acc module or even know about the fact that
the acc module is using it. With your solution, dialog module has to know acc
api to call the
recording function, and acc has to know dialog module to know its
exported structures. This is cross dependency imo.
I think this point shows the major difference between our solutions: My
approach doesn't make the dialog module call the recording function.
Instead, the recording function is called by the acc module as soon as
the dialog terminates where call termination can is observed by means of
using dialog callbacks. (Sorry, I should have possibly mentioned the
words "dialog callback" before.)
Therefore, module dependency is unidirectional: acc registers for some
dialog callbacks and, upon execution of a callback, records the CDR.
> Third, the CDR is persisted to either log file,
database, or both.
If this new thing is not going to support what acc module API has for
backends (radius is missing), then will not make sense to tie the two.
dialog module can do its accounting alone.
Of course, I would want to take
advantage of acc's existing backend
connectors. There's no need to re-invent the wheel, acc will still be
responsible for writing out CDRs; the difference is that a few
additional calls to a well-defined dialog interface will be used in
order to collect the data that constitute the CDR. - such functionality is independent of dlg module or
any new call
tracking extension in the future, also writing full CDR can be achieved
from config file by tracking INVITE and BYE, calling
acc_start()/acc_stop() from cfg
Tracking INVITE and BYE messages from the config
file may not be enough
if you want to create CDRs for failed calls too, e.g., 408/487. Anyhow, I think you divert from the initial goal.
If you just break the
message in pieces and reply on each word will lose the meaning of the
entire content. I presented my plans above and below I added acc_cdr()
that seems to suit your needs.
Maybe I got you wrong but my understanding is that acc_start() records
one part of a CDR (start time, etc.) when the INVITE is routed while
acc_stop() finishes it off (end time, duration, etc.) on reception of
the BYE. I was trying to stress that a single call to the recording
function is likely preferred (at least by me), which acc_cdr() seems to
accomplish.
Therefore if I would do it:
- enhance acc module to export via cfg exports and inter-module api
three functions:
- acc_start() - write the initial call record at start
- acc_stop() - update the call record at stop, based on a matching
condition specified as parameter
- acc_cdr() - write a full CDR
Data to be written in db (or other backend) is going to be taken from
PVs, independent of who (cfg, dialog, or other module) is calling the
function, specified in a similar form like db_extra.
First two functions will work for db only. Third can work also without
dialog, e.g., I can store the start of a call, a.s.o. in hash table and
get it at BYE time to build the full cdr.
Where would you maintain that hash table
containing the call start time
and similar data if you do not seek to use the dialog module? Such data
seems to be naturally associated to a dialog, so my impression is that
it should be kept there too, i.e., in the dialog module.
11:46 a.m.
New subject: Proposal to extend acc/dialog modules in order to log CDRs
Hey Carsten,
On 07.02.2011 12:04, Carsten Bock wrote:
The functionality for storing additional data for a
dialog is like this:
$dlg_var(ecscf_server) = ‘sip:ecscf.ng-voice.com’;
xlog(“Emergency CSCF is $dlg_var(ecscf_server)\n”);
However, it might be worth noting, that i haven't added bindings to
the dialog-API and that the Pseudo-Variable currently only supports
Strings.
For CDR generation purposes, I wouldn't need any additional dialog API
bindings. The PV interface should suffice as Daniel and I have concluded
on the mailing-list and at FOSDEM.
I think, my branch needs to exists a little longer,
since the changes
take quite a lot of time. I am currently doing a lot of cleanup where
many methods get obsolete; so i do not want to add docs for later
obsolete methods and parameters.
What aspects of your branch do you plan to change in the future? Would
that affect the PV interface as well?
If you can provide a rough plan in order for me to consider whether the
changes may affect my CDR approach I'd be very glad.
Ideally, if the dialog data storage functionality is somewhat complete,
we may consider putting it into the master branch prior to any
outstanding changes in your branch.
Let me know if there's anything I can do.
Cheers,
--Timo
2011/2/2 Timo Reimann <timo.reimann(a)1und1.de>de>:
> On 02.02.2011 14:03, Daniel-Constantin Mierla wrote:
>>> On 01.02.2011 21:15, Daniel-Constantin Mierla wrote:
>>>> On 2/1/11 8:18 PM, Timo Reimann wrote:
>>>>> [...]
>>>>>
>>>>> Apart from this very minimum CDR content, however, one can think of
a
>>>>> number of CDR fields that cannot be filled easily. For instance,
caller
>>>>> identity isn't always contained in the From header (think of
CLIR
>>>>> calls)
>>>>> but need to be determined from other headers depending on the type
of
>>>>> call, possibly even involving a database lookup.
>>>> From header URI is not accounted automatically, you can specify any
>>>> variable that can contain caller id as you need (e.g., it can be an AVP
>>>> that you previously set for in config) -- see *_extra parameters of acc
>>>> module.
>>> I initially thought about having a cdr_extra parameter similar to
>>> log_extra. However, I see a problem with AVPs which live in single
>>> transactions to work with dialogs that span multiple transactions.
>>
>> well, some of the avps are transaction persistent, but there are also
>> global avps which are available as long as kamailio runs. None of these
>> should be used, I gave avp just as generic PV example. The idea was to
>> make new PV available with the data stored by dlg module.
>
> Ok, so instead of providing getter/setter functions, let's use PVs.
> However, users will have varying needs regarding the to-be-defined CDR
> fields next to the canonical ones (start time, end time, duration,
> etc.). We, for example, use a rather large number of fields, many of
> them being too specific for general integration into upstream Kamailio.
> That's why I would rather see a way to dynamically specify CDR fields.
> Carsten's new code would be the right approach to do this, right? His
> dlg_var PV takes arbitrary key/value pairs which looks just like what is
> needed.
>
>
>>>>> That's why I'd like to
>>>>> provide a way to pass additional, CDR-specific data from the
>>>>> configuration file to the dialog structure associated with the call.
>>>>> When the dialog is about to terminate, the extended acc module would
>>>>> make sure that the added CDR data associated to the dialog is
included
>>>>> in the CDR.
>>>> Not sure what you mean here with "acc module will make sure
...", but I
>>>> hope is not going to be cross reference/dependency, so that acc has to
>>>> walk through dlg structures.
>>> If I get you right, by cross reference/dependency you mean that A
>>> requires B and vice versa, i.e., cyclic dependency. I do not intend to
>>> let that happen.
>>>
>>> Instead, the acc module would use getter functions attached to the
>>> dialog interface (which is what I mean by "acc module will make
sure").
>>> dialog would never touch the acc module or even know about the fact that
>>> the acc module is using it.
>>
>> But then acc has to know the dialog internals, what the getter function
>> will return and how to access those structures. PV framework is exactly
>> the same, but available for all components/modules, no need to develop
>> specif ones each time.
>
> Agreed, let's use PVs. If you think of Carsten's new dlg_var PV, it's
> basically just a way to get/set arbitrary values; so I suppose this is
> what I should have expressed.
>
>
>> With your solution, dialog module has to know acc api to call the
>> recording function, and acc has to know dialog module to know its
>> exported structures. This is cross dependency imo.
>
> I think this point shows the major difference between our solutions: My
> approach doesn't make the dialog module call the recording function.
> Instead, the recording function is called by the acc module as soon as
> the dialog terminates where call termination can is observed by means of
> using dialog callbacks. (Sorry, I should have possibly mentioned the
> words "dialog callback" before.)
>
> Therefore, module dependency is unidirectional: acc registers for some
> dialog callbacks and, upon execution of a callback, records the CDR.
>
>
>>>>> Third, the CDR is persisted to either log file, database, or both.
>>>> If this new thing is not going to support what acc module API has for
>>>> backends (radius is missing), then will not make sense to tie the two.
>>>> dialog module can do its accounting alone.
>>> Of course, I would want to take advantage of acc's existing backend
>>> connectors. There's no need to re-invent the wheel, acc will still be
>>> responsible for writing out CDRs; the difference is that a few
>>> additional calls to a well-defined dialog interface will be used in
>>> order to collect the data that constitute the CDR.
>>
>> What will happen if a new call tracking module will be available, should
>> acc be changed again?
>
> I am not completely convinced that call tracking is such a flexible
> thing even though you gave a few examples below. A call is pretty much a
> dialog, and proper tracking of a dialog should be dialog module's job.
>
> Moreover, I believe your approach has some dependency issues as well --
> see my comment at the very end of this mail.
>
>
>>>> - such functionality is independent of dlg module or any new call
>>>> tracking extension in the future, also writing full CDR can be achieved
>>>> from config file by tracking INVITE and BYE, calling
>>>> acc_start()/acc_stop() from cfg
>>> Tracking INVITE and BYE messages from the config file may not be enough
>>> if you want to create CDRs for failed calls too, e.g., 408/487.
>> What stops tracking these replies in the cfg?
>
> You can do it, but for a large config file with many different paths,
> it's easy to miss a spot where you should have added a call to
> acc_stop(). On the contrary, if recording happens automatically when the
> call ends, you don't have to deal with any manual work.
>
>
>> Anyhow, I think you divert from the initial goal. If you just break the
>> message in pieces and reply on each word will lose the meaning of the
>> entire content. I presented my plans above and below I added acc_cdr()
>> that seems to suit your needs.
>
> Maybe I got you wrong but my understanding is that acc_start() records
> one part of a CDR (start time, etc.) when the INVITE is routed while
> acc_stop() finishes it off (end time, duration, etc.) on reception of
> the BYE. I was trying to stress that a single call to the recording
> function is likely preferred (at least by me), which acc_cdr() seems to
> accomplish.
>
>>>> Therefore if I would do it:
>>>> - enhance acc module to export via cfg exports and inter-module api
>>>> three functions:
>>>> - acc_start() - write the initial call record at start
>>>> - acc_stop() - update the call record at stop, based on a matching
>>>> condition specified as parameter
>>>> - acc_cdr() - write a full CDR
>>>>
>>>> Data to be written in db (or other backend) is going to be taken from
>>>> PVs, independent of who (cfg, dialog, or other module) is calling the
>>>> function, specified in a similar form like db_extra.
>>>>
>>>> First two functions will work for db only. Third can work also without
>>>> dialog, e.g., I can store the start of a call, a.s.o. in hash table and
>>>> get it at BYE time to build the full cdr.
>>> Where would you maintain that hash table containing the call start time
>>> and similar data if you do not seek to use the dialog module? Such data
>>> seems to be naturally associated to a dialog, so my impression is that
>>> it should be kept there too, i.e., in the dialog module.
>> The hash table module (htable) keeps the data in shared memory. It is a
>> generic container, so you can store anything there. dialog module is
>> just an automatic call tracker, handling sip messages passing through
>> kamailio. Most of that can be done from config file. Internally, dialog
>> module has also a hash table. Also, I can track active calls using a db
>> table (in memory or not for speed), with a timer process (rtimer module)
>> to handle missing BYEs, and so there could be other options to track
>> dialogs. My point here is that acc should be independent of all these
>> potential options available now or added in the future.
>>
>> You additions to dialog module can export the new stored data inside
>> dialog via some PV, e.g.,
>>
>> $dlgacc(caller), $dlgacc(callee), $dlgacc(duration), a.s.o.
>>
>> dialog module will call acc_cdr() when the call ended, making sure the
>> PVs are properly available at that time.
>
> If you call an acc module-provided method from the dialog module, you
> face a similar independence problem: Should accounting functionality
> ever be changed or replaced of by another module, then you'd need to
> change the dialog implementation as well.
>
> There's a conceptual issue related to the given usage relationship as
> well: If component A requires B to implement a particular functionality,
> A should use B. Having the dialog module trigger the recording function
> puts this concept upside down -- CDR accounting isn't needed by dialog
> tracking, it's the other way around.
>
>
> Cheers,
>
> --Timo
4845
days inactive
4851
days old
6 comments
3 participants
participants (3)
-
Carsten Bock -
Daniel-Constantin Mierla -
Timo Reimann