[sr-dev] [kamailio/kamailio] acc: new json format with outputs to mqueue or syslog (#1437)
Julien Chavanton
notifications at github.com
Mon Feb 12 21:53:53 CET 2018
<!-- Kamailio Pull Request Template -->
<!--
IMPORTANT:
- for detailed contributing guidelines, read:
https://github.com/kamailio/kamailio/blob/master/.github/CONTRIBUTING.md
- pull requests must be done to master branch, unless they are backports
of fixes from master branch to a stable branch
- backports to stable branches must be done with 'git cherry-pick -x ...'
- code is contributed under BSD for core and main components (tm, sl, auth, tls)
- code is contributed GPLv2 or a compatible license for the other components
- GPL code is contributed with OpenSSL licensing exception
-->
#### Pre-Submission Checklist
<!-- Go over all points below, and after creating the PR, tick all the checkboxes that apply -->
<!-- All points should be verified, otherwise, read the CONTRIBUTING guidelines from above-->
<!-- If you're unsure about any of these, don't hesitate to ask on sr-dev mailing list -->
- [x] Commit message has the format required by CONTRIBUTING guide
- [x] Commits are split per component (core, individual modules, libs, utils, ...)
- [x] Each component has a single commit (if not, squash them into one commit)
- [x] No commits to README files for modules (changes must be done to docbook files
in `doc/` subfolder, the README file is autogenerated)
#### Type Of Change
- [ ] Small bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds new functionality)
- [ ] Breaking change (fix or feature that would change existing functionality)
#### Checklist:
<!-- Go over all points below, and after creating the PR, tick the checkboxes that apply -->
- [ ] PR should be backported to stable branches
- [x] Tested changes locally
- [ ] Related to issue #XXXX (replace XXXX with an open issue number)
#### Description
This commit is introducing JSON format for ACC events.
2 way to output the ACC events are proposed/implemented (syslog and mqueue)
The option to use db_flastore was considered and found less attractive, mainly because the DB abstraction layer is not making much sense, example reformatting the data many times ...
and other outputs could benefit from JSON format.
1- syslog (directly from the module, because it may be valuable to run Kamailio using runit or in docker and not "daemionizing" it anymore while still using syslog for acc events)
2- mqueue this way you may be able to control the acc event from Kamailio routing script, and example is provided using the http_client module to publish to NSQD
This seemed to be the best approach to remain as compatible as possible with the ACC module and benefit from it, while introducing the JSON format.
Many parameters are reused like time_mode, log_extra, log_facility, it seemed like reusing was making sense, this may be questionable.
If jansson or mqueue is not available or loaded the module will still build and run disabling the extra features
----------------------------------------------------------
#### Usage example
where we use mqueue and process acc events in other worker processes.
example also inluded in the doc
```
# example using json_mqueue/http_client to publish to NSQD
max_while_loops=100000
modparam("mqueue", "mqueue", "name=acc_events;size=100000")
modparam("acc", "json_mqueue", "acc_events")
modparam("acc", "json_flag", 2)
modparam("acc", "log_extra", "caller_ip_port=$avp(caller_ip_port);")
modparam("rtimer", "timer", "name=nsqt;interval=1;mode=1;")
modparam("rtimer", "exec", "timer=nsqt;route=RUN_CDR_PUBLISH")
modparam("http_client", "keep_connections", 1)
modparam("http_client", "httpcon", "nsqd=>http://localhost:4151/pub?topic=acc")
route[RUN_CDR_PUBLISH] {
$var(count) = 0;
while (mq_fetch("acc_events")) {
$var(q_size) = mq_size("acc_events");
$var(count) = $var(count) + 1;
xinfo("[RUN_CDR_PUBLISH][$var(q_size)][$var(count)][$mqk(acc_events)][$mqv(acc_events)]\n");
$var(res) = http_connect("nsqd", "", "application/json", $mqv(acc_events), "$var(nsq_res)");
if ($var(res) != "200") {
mq_add("acc_events", "acc_key", "$mqv(acc_events)");
return;
}
}
if ($var(count) > 0 ) {
xinfo("[RUN_CDR_PUBLISH]done count[$var(count)]\n");
}
}
```
#### Load testing
Load tests where conducted with both json_syslog and json_mqueue/http to NSQD, latency was very stable it was high because the test server used was on West coast
California >> New-Jersey >> California
average message/response latency graph in ms/sec
----------------------------------------------------------
Using voip_perf to send SIP compliant traffic (INVITE < 100 < 180 < 200 > ACK || BYE < 200)
7M SIP messages calls sent > 800/sec
https://github.com/jchavanton/voip_perf
```
Total 1000000 INVITE calls sent in 1037082 ms at rate of 819/sec
Total 1000000 responses received in 1037881 ms at rate of 818/sec:
Detailed responses received:
- 200 responses: 1000000 (OK)
------
TOTAL responses: 1000000 (rate=818/sec)
```
All the ACC events where queued in NSQD and written to SYSLOG without introducing delay.
```
curl -v http://127.0.0.1:4151/stats
nsqd v1.0.0-compat (built w/go1.8)
start_time 2018-02-12T17:10:58Z
uptime 33m4.181861175s
Health: OK
[acc ] depth: 3000004 be-depth: 2990004 msgs: 3000004 e2e%:
```
```
wc -l /var/log/json_acc.log
3000004 /var/log/json_acc.log
```
You can view, comment on, or merge this pull request online at:
https://github.com/kamailio/kamailio/pull/1437
-- Commit Summary --
* acc: new json format with outputs to mqueue or syslog
-- File Changes --
M src/modules/acc/Makefile (12)
M src/modules/acc/acc.c (109)
M src/modules/acc/acc.h (1)
M src/modules/acc/acc_logic.c (27)
M src/modules/acc/acc_mod.c (25)
M src/modules/acc/acc_mod.h (3)
M src/modules/acc/doc/acc.xml (10)
M src/modules/acc/doc/acc_admin.xml (121)
-- Patch Links --
https://github.com/kamailio/kamailio/pull/1437.patch
https://github.com/kamailio/kamailio/pull/1437.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/pull/1437
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20180212/ed080423/attachment.html>
More information about the sr-dev
mailing list