26 Sep
2022
26 Sep
'22
1:30 p.m.
Hello,
as I am not a user of dialog variables, I am turning to community to ask
for help testing the current master branch with configurations that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected
crashes, I looked over the code and noticed that the access of the value
for dialog variables was not protected, making them vulnerable of
invalid memory access in case of the variable was updated by another
process or dialog was terminated.
I introduced a couple of new functions to try to cover different use
cases of getting the dlg variable values, dialog management code was not
affected, but given that these commits need to be backported to stable
branch (5.6), I want to get proper feedback from community that things
work fine.
A previous attempt of a simpler fix was not enough, having side effects
to acc module for dialog-based cdr generation, because it was keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc
with dialog-based cdr generation using git master branch.
Cheers,
Daniel
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
28 Sep
28 Sep
4:34 p.m.
Hi Daniel. Thank you for back porting the acc and dialog changes to 5.6 -- I
continue to test, but can report that the updates on the 5.6 branch through
bfc5c2aacb272e91da80096ef27fbe6fcfe9a746 are working properly for acc/dialog-
based CDR generation. -A
On Monday, September 26, 2022 6:30:04 AM CDT Daniel-Constantin Mierla wrote:
Hello,
as I am not a user of dialog variables, I am turning to community to ask
for help testing the current master branch with configurations that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected
crashes, I looked over the code and noticed that the access of the value
for dialog variables was not protected, making them vulnerable of
invalid memory access in case of the variable was updated by another
process or dialog was terminated.
I introduced a couple of new functions to try to cover different use
cases of getting the dlg variable values, dialog management code was not
affected, but given that these commits need to be backported to stable
branch (5.6), I want to get proper feedback from community that things
work fine.
A previous attempt of a simpler fix was not enough, having side effects
to acc module for dialog-based cdr generation, because it was keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc
with dialog-based cdr generation using git master branch.
Cheers,
Daniel
29 Sep
29 Sep
9:24 a.m.
Thanks for testing and reporting back!
Cheers,
Daniel
On 28.09.22 16:34, Anthony Messina wrote:
Hi Daniel. Thank you for back porting the acc and
dialog changes to 5.6 -- I
continue to test, but can report that the updates on the 5.6 branch through
bfc5c2aacb272e91da80096ef27fbe6fcfe9a746 are working properly for acc/dialog-
based CDR generation. -A
On Monday, September 26, 2022 6:30:04 AM CDT Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
Hello,
as I am not a user of dialog variables, I am turning to community to ask
for help testing the current master branch with configurations that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected
crashes, I looked over the code and noticed that the access of the value
for dialog variables was not protected, making them vulnerable of
invalid memory access in case of the variable was updated by another
process or dialog was terminated.
I introduced a couple of new functions to try to cover different use
cases of getting the dlg variable values, dialog management code was not
affected, but given that these commits need to be backported to stable
branch (5.6), I want to get proper feedback from community that things
work fine.
A previous attempt of a simpler fix was not enough, having side effects
to acc module for dialog-based cdr generation, because it was keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc
with dialog-based cdr generation using git master branch.
Cheers,
Daniel
3 Oct
3 Oct
4:41 a.m.
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
as I am not a user of dialog variables, I am turning to community to ask
for help testing the current master branch with configurations that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected
crashes, I looked over the code and noticed that the access of the value
for dialog variables was not protected, making them vulnerable of
invalid memory access in case of the variable was updated by another
process or dialog was terminated.
I introduced a couple of new functions to try to cover different use
cases of getting the dlg variable values, dialog management code was not
affected, but given that these commits need to be backported to stable
branch (5.6), I want to get proper feedback from community that things
work fine.
A previous attempt of a simpler fix was not enough, having side effects
to acc module for dialog-based cdr generation, because it was keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc
with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest
commit 6f400a8074fe60916867596431ca26dff00435d1.
I usually leave a commit load test running for 2 months before consider it
ready for production release.
I will report any crash/problem.
13 Oct
13 Oct
11:19 p.m.
On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi <mayamatakeshi(a)gmail.com>
wrote:
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
After a few hours of load test, all 4 load test environments start to log
memory allocation problems:
[root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log
|head
2022-10-13T01:36:10.429809+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
2022-10-13T01:36:12.923609+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
2022-10-13T01:36:12.961677+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:14.983281+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
2022-10-13T01:36:14.983537+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:14.983665+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:20.861558+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
2022-10-13T01:36:20.864388+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:20.878469+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:23.174159+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
I reverted 2 of the load test envs to previous kamailio 5.6 commit
61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the load tests, then
again, kamailio master commit 6f400a8074fe60916867596431ca26dff00435d1
started to write the above logs but this doesn't happen with 5.6.
Hello,
as I am not a user of dialog variables, I am turning to community to ask
for help testing the current master branch with configurations that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected
crashes, I looked over the code and noticed that the access of the value
for dialog variables was not protected, making them vulnerable of
invalid memory access in case of the variable was updated by another
process or dialog was terminated.
I introduced a couple of new functions to try to cover different use
cases of getting the dlg variable values, dialog management code was not
affected, but given that these commits need to be backported to stable
branch (5.6), I want to get proper feedback from community that things
work fine.
A previous attempt of a simpler fix was not enough, having side effects
to acc module for dialog-based cdr generation, because it was keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc
with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest
commit 6f400a8074fe60916867596431ca26dff00435d1.
I usually leave a commit load test running for 2 months before consider it
ready for production release.
I will report any crash/problem.
14 Oct
14 Oct
8:57 a.m.
Hello,
are you using accounting to generate CDRs with dialog module (records in
acc_cdrs table)? Or only for getting the event records in the acc table?
Cheers,
Daniel
On 13.10.22 23:19, mayamatakeshi wrote:
On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi
<mayamatakeshi(a)gmail.com> wrote:
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
as I am not a user of dialog variables, I am turning to
community to ask
for help testing the current master branch with configurations
that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and
unexpected
crashes, I looked over the code and noticed that the access of
the value
for dialog variables was not protected, making them vulnerable of
invalid memory access in case of the variable was updated by
another
process or dialog was terminated.
I introduced a couple of new functions to try to cover
different use
cases of getting the dlg variable values, dialog management
code was not
affected, but given that these commits need to be backported
to stable
branch (5.6), I want to get proper feedback from community
that things
work fine.
A previous attempt of a simpler fix was not enough, having
side effects
to acc module for dialog-based cdr generation, because it was
keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing
dialog and acc
with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest
commit 6f400a8074fe60916867596431ca26dff00435d1.
I usually leave a commit load test running for 2 months before
consider it ready for production release.
I will report any crash/problem.
After a few hours of load test, all 4 load test environments start to
log memory allocation problems:
[root@lab002107-flip-server ~]$ grep memory
/var/log/kamailio/kamailio.log |head
2022-10-13T01:36:10.429809+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory -
bencode failed
2022-10-13T01:36:12.923609+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory -
bencode failed
2022-10-13T01:36:12.961677+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-13T01:36:14.983281+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory -
bencode failed
2022-10-13T01:36:14.983537+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-13T01:36:14.983665+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-13T01:36:20.861558+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory -
bencode failed
2022-10-13T01:36:20.864388+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-13T01:36:20.878469+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-13T01:36:23.174159+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory -
bencode failed
I reverted 2 of the load test envs to previous kamailio 5.6 commit
61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the load tests,
then again, kamailio master commit
6f400a8074fe60916867596431ca26dff00435d1 started to write the above
logs but this doesn't happen with 5.6.
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
10:12 a.m.
Hi,
I am using acc for CDR generation. Here is the module configuration I'm
using:
modparam("acc", "db_url",
"mysql://kamailio:kamailiorw@localhost/kamailio")
modparam("acc", "db_flag", 1)
modparam("acc", "db_missed_flag", 2)
modparam("acc", "failed_transaction_flag", 3)
modparam("acc", "cdr_enable", 1)
modparam("acc", "cdrs_table", "cdr")
modparam("acc", "cdr_extra",
"callid=$ci;caller_domain=$dlg_var(caller_domain);callee_domain=$dlg_var(callee_domain);caller_username=$dlg_var(caller_username);callee_username=$dlg_var(callee_username);calling_number=$dlg_var(calling_number);destination=$dlg_var(destination);anonymous=$dlg_var(anonymous);forwarding=$dlg_var(forwarding);tracing=$dlg_var(tracing);relay=$dlg_var(relay);sip_code=$dlg_var(sip_code);status_code=$dlg_var(status_code);start_time=$dlg_var(start_time)")
modparam("acc", "cdr_start_on_confirmed", 1)
modparam("acc", "cdr_start_id", "answer_time")
modparam("acc", "log_level", 9)
modparam("acc", "log_facility", "LOG_LOCAL0")
modparam("acc", "log_flag", 10)
modparam("acc", "cdr_facility", "LOG_LOCAL0")
modparam("acc", "cdr_log_enable", 1)
On Fri, Oct 14, 2022 at 3:57 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
are you using accounting to generate CDRs with dialog module (records in
acc_cdrs table)? Or only for getting the event records in the acc table?
Cheers,
Daniel
On 13.10.22 23:19, mayamatakeshi wrote:
On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi <mayamatakeshi(a)gmail.com>
wrote:
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
After a few hours of load test, all 4 load test environments start to log
memory allocation problems:
[root@lab002107-flip-server ~]$ grep memory
/var/log/kamailio/kamailio.log |head
2022-10-13T01:36:10.429809+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
2022-10-13T01:36:12.923609+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
2022-10-13T01:36:12.961677+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:14.983281+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
2022-10-13T01:36:14.983537+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:14.983665+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:20.861558+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
2022-10-13T01:36:20.864388+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:20.878469+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-13T01:36:23.174159+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR: rtpengine
[rtpengine.c:2721]: rtpp_function_call(): out of memory - bencode failed
I reverted 2 of the load test envs to previous kamailio 5.6 commit
61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the load tests, then
again, kamailio master commit 6f400a8074fe60916867596431ca26dff00435d1
started to write the above logs but this doesn't happen with 5.6.
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
Hello,
as I am not a user of dialog variables, I am turning to community to ask
for help testing the current master branch with configurations that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules and unexpected
crashes, I looked over the code and noticed that the access of the value
for dialog variables was not protected, making them vulnerable of
invalid memory access in case of the variable was updated by another
process or dialog was terminated.
I introduced a couple of new functions to try to cover different use
cases of getting the dlg variable values, dialog management code was not
affected, but given that these commits need to be backported to stable
branch (5.6), I want to get proper feedback from community that things
work fine.
A previous attempt of a simpler fix was not enough, having side effects
to acc module for dialog-based cdr generation, because it was keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing dialog and acc
with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest
commit 6f400a8074fe60916867596431ca26dff00435d1.
I usually leave a commit load test running for 2 months before consider
it ready for production release.
I will report any crash/problem.
10:25 a.m.
OK. Can you run the tests for a while and then execute:
kamctl rpc corex.pkg_summary idx 1
The value for idx should be the kamailio process expected to handle sip
traffic. If it is mostly udp and the first socket is the corresponding
udp socket, then the value 1, like above, is ok. If it is tcp/tls or you
have different listen parameters on udp, see the processes index with:
kamctl ps
It should not be required to wait till you get out of memory errors, but
be sure you run it long enough to have many accounting records written
by the process that is going to print the pkg summary. You can
eventually set the children to 1 or 2, to "force" traffic via a smaller
group of processes.
Cheers,
Daniel
On 14.10.22 10:12, mayamatakeshi wrote:
Hi,
I am using acc for CDR generation. Here is the module configuration
I'm using:
modparam("acc", "db_url",
"mysql://kamailio:kamailiorw@localhost/kamailio")
modparam("acc", "db_flag", 1)
modparam("acc", "db_missed_flag", 2)
modparam("acc", "failed_transaction_flag", 3)
modparam("acc", "cdr_enable", 1)
modparam("acc", "cdrs_table", "cdr")
modparam("acc", "cdr_extra",
"callid=$ci;caller_domain=$dlg_var(caller_domain);callee_domain=$dlg_var(callee_domain);caller_username=$dlg_var(caller_username);callee_username=$dlg_var(callee_username);calling_number=$dlg_var(calling_number);destination=$dlg_var(destination);anonymous=$dlg_var(anonymous);forwarding=$dlg_var(forwarding);tracing=$dlg_var(tracing);relay=$dlg_var(relay);sip_code=$dlg_var(sip_code);status_code=$dlg_var(status_code);start_time=$dlg_var(start_time)")
modparam("acc", "cdr_start_on_confirmed", 1)
modparam("acc", "cdr_start_id", "answer_time")
modparam("acc", "log_level", 9)
modparam("acc", "log_facility", "LOG_LOCAL0")
modparam("acc", "log_flag", 10)
modparam("acc", "cdr_facility", "LOG_LOCAL0")
modparam("acc", "cdr_log_enable", 1)
On Fri, Oct 14, 2022 at 3:57 PM Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
are you using accounting to generate CDRs with dialog module
(records in acc_cdrs table)? Or only for getting the event records
in the acc table?
Cheers,
Daniel
On 13.10.22 23:19, mayamatakeshi wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi
<mayamatakeshi(a)gmail.com> wrote:
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
as I am not a user of dialog variables, I am turning to
community to ask
for help testing the current master branch with
configurations that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules
and unexpected
crashes, I looked over the code and noticed that the
access of the value
for dialog variables was not protected, making them
vulnerable of
invalid memory access in case of the variable was updated
by another
process or dialog was terminated.
I introduced a couple of new functions to try to cover
different use
cases of getting the dlg variable values, dialog
management code was not
affected, but given that these commits need to be
backported to stable
branch (5.6), I want to get proper feedback from
community that things
work fine.
A previous attempt of a simpler fix was not enough,
having side effects
to acc module for dialog-based cdr generation, because it
was keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on testing
dialog and acc
with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest
commit 6f400a8074fe60916867596431ca26dff00435d1.
I usually leave a commit load test running for 2 months
before consider it ready for production release.
I will report any crash/problem.
After a few hours of load test, all 4 load test environments
start to log memory allocation problems:
[root@lab002107-flip-server ~]$ grep memory
/var/log/kamailio/kamailio.log |head
2022-10-13T01:36:10.429809+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory
- bencode failed
2022-10-13T01:36:12.923609+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory
- bencode failed
2022-10-13T01:36:12.961677+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:14.983281+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory
- bencode failed
2022-10-13T01:36:14.983537+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:14.983665+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:20.861558+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory
- bencode failed
2022-10-13T01:36:20.864388+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:20.878469+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:23.174159+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of memory
- bencode failed
I reverted 2 of the load test envs to previous kamailio 5.6
commit 61e86a1f502388ffd4dce6e52811ba640337c813 and restarted the
load tests, then again, kamailio master commit
6f400a8074fe60916867596431ca26dff00435d1 started to write the
above logs but this doesn't happen with 5.6.
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
11:21 a.m.
Based on your parameters, it seems that acc should send cdrs also to
syslog. I think I identified an issue on that code and pushed a fix, can
you test with latest master git branch?
Cheers,
Daniel
On 14.10.22 10:25, Daniel-Constantin Mierla wrote:
OK. Can you run the tests for a while and then execute:
kamctl rpc corex.pkg_summary idx 1
The value for idx should be the kamailio process expected to handle
sip traffic. If it is mostly udp and the first socket is the
corresponding udp socket, then the value 1, like above, is ok. If it
is tcp/tls or you have different listen parameters on udp, see the
processes index with:
kamctl ps
It should not be required to wait till you get out of memory errors,
but be sure you run it long enough to have many accounting records
written by the process that is going to print the pkg summary. You can
eventually set the children to 1 or 2, to "force" traffic via a
smaller group of processes.
Cheers,
Daniel
On 14.10.22 10:12, mayamatakeshi wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
Hi,
I am using acc for CDR generation. Here is the module configuration
I'm using:
modparam("acc", "db_url",
"mysql://kamailio:kamailiorw@localhost/kamailio")
modparam("acc", "db_flag", 1)
modparam("acc", "db_missed_flag", 2)
modparam("acc", "failed_transaction_flag", 3)
modparam("acc", "cdr_enable", 1)
modparam("acc", "cdrs_table", "cdr")
modparam("acc", "cdr_extra",
"callid=$ci;caller_domain=$dlg_var(caller_domain);callee_domain=$dlg_var(callee_domain);caller_username=$dlg_var(caller_username);callee_username=$dlg_var(callee_username);calling_number=$dlg_var(calling_number);destination=$dlg_var(destination);anonymous=$dlg_var(anonymous);forwarding=$dlg_var(forwarding);tracing=$dlg_var(tracing);relay=$dlg_var(relay);sip_code=$dlg_var(sip_code);status_code=$dlg_var(status_code);start_time=$dlg_var(start_time)")
modparam("acc", "cdr_start_on_confirmed", 1)
modparam("acc", "cdr_start_id", "answer_time")
modparam("acc", "log_level", 9)
modparam("acc", "log_facility", "LOG_LOCAL0")
modparam("acc", "log_flag", 10)
modparam("acc", "cdr_facility", "LOG_LOCAL0")
modparam("acc", "cdr_log_enable", 1)
On Fri, Oct 14, 2022 at 3:57 PM Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
are you using accounting to generate CDRs with dialog module
(records in acc_cdrs table)? Or only for getting the event
records in the acc table?
Cheers,
Daniel
On 13.10.22 23:19, mayamatakeshi wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
On Mon, Oct 3, 2022 at 11:41 AM mayamatakeshi
<mayamatakeshi(a)gmail.com> wrote:
On Mon, Sep 26, 2022 at 8:32 PM Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
as I am not a user of dialog variables, I am turning to
community to ask
for help testing the current master branch with
configurations that make
use of dialog variables and acc dialog-based cdr generation.
With a few reports of issues related to dialog modules
and unexpected
crashes, I looked over the code and noticed that the
access of the value
for dialog variables was not protected, making them
vulnerable of
invalid memory access in case of the variable was
updated by another
process or dialog was terminated.
I introduced a couple of new functions to try to cover
different use
cases of getting the dlg variable values, dialog
management code was not
affected, but given that these commits need to be
backported to stable
branch (5.6), I want to get proper feedback from
community that things
work fine.
A previous attempt of a simpler fix was not enough,
having side effects
to acc module for dialog-based cdr generation, because
it was keeping
referenced to many dlg variables at the same time.
In short, it would be appreciated any feedback on
testing dialog and acc
with dialog-based cdr generation using git master branch.
I have started 4 load test environments today with latest
commit 6f400a8074fe60916867596431ca26dff00435d1.
I usually leave a commit load test running for 2 months
before consider it ready for production release.
I will report any crash/problem.
After a few hours of load test, all 4 load test environments
start to log memory allocation problems:
[root@lab002107-flip-server ~]$ grep memory
/var/log/kamailio/kamailio.log |head
2022-10-13T01:36:10.429809+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of
memory - bencode failed
2022-10-13T01:36:12.923609+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of
memory - bencode failed
2022-10-13T01:36:12.961677+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:14.983281+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of
memory - bencode failed
2022-10-13T01:36:14.983537+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:14.983665+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:20.861558+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of
memory - bencode failed
2022-10-13T01:36:20.864388+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:20.878469+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
acc [acc_extra.c:234]: extra2strar(): could not allocate private
memory from pkg pool
2022-10-13T01:36:23.174159+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3286370]: ERROR:
rtpengine [rtpengine.c:2721]: rtpp_function_call(): out of
memory - bencode failed
I reverted 2 of the load test envs to previous kamailio 5.6
commit 61e86a1f502388ffd4dce6e52811ba640337c813 and restarted
the load tests, then again, kamailio master commit
6f400a8074fe60916867596431ca26dff00435d1 started to write the
above logs but this doesn't happen with 5.6.
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
16 Oct
16 Oct
11:48 p.m.
On Fri, Oct 14, 2022 at 5:25 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
OK. Can you run the tests for a while and then
execute:
kamctl rpc corex.pkg_summary idx 1
The value for idx should be the kamailio process expected to handle sip
traffic. If it is mostly udp and the first socket is the corresponding udp
socket, then the value 1, like above, is ok. If it is tcp/tls or you have
different listen parameters on udp, see the processes index with:
kamctl ps
I tried calling
kamctl rpc corex.pkg_summary idx 1
before and after the memory problem started to be logged but it always
showed up with result empty:
{
"jsonrpc": "2.0",
"result": {
},
"id": 3824999
}
Here is result for all processes:
https://pastebin.com/AabwyJwx
I tried several times but I never saw a non-empty result.
It should not be required to wait till you get out of
memory errors, but
be sure you run it long enough to have many accounting records written by
the process that is going to print the pkg summary. You can eventually set
the children to 1 or 2, to "force" traffic via a smaller group of processes.
By reducing children from
children=12
to
children=1
I started to get memory allocation error just a after a few minutes of load
test:
[root@lab002107-flip-server ~]$ grep memory /var/log/kamailio/kamailio.log
|head
2022-10-17T06:20:41.314619+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-17T06:20:41.916596+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-17T06:20:41.916786+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-17T06:20:42.518013+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-17T06:20:42.518248+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory from
pkg pool
2022-10-17T06:20:43.114454+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core>
[core/rvalue.c:253]: rval_new_empty(): could not allocate private memory
from pkg pool
2022-10-17T06:20:43.163816+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core>
[core/rvalue.c:253]: rval_new_empty(): could not allocate private memory
from pkg pool
2022-10-17T06:20:43.165863+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core>
[core/rvalue.c:253]: rval_new_empty(): could not allocate private memory
from pkg pool
2022-10-17T06:20:43.167092+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core>
[core/rvalue.c:253]: rval_new_empty(): could not allocate private memory
from pkg pool
2022-10-17T06:20:43.167247+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: <core>
[core/data_lump.c:163]: insert_new_lump_before(): could not allocate
private memory from pkg pool
I collected the pkg summary after 4000 CDRs were written to DB but again, i
could not see a non-empty result:
https://pastebin.com/CpaRYLAR
17 Oct
17 Oct
12:45 a.m.
On Fri, Oct 14, 2022 at 6:21 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Based on your parameters, it seems that acc should
send cdrs also to
syslog. I think I identified an issue on that code and pushed a fix, can
you test with latest master git branch?
OK. I have started 2 load test environments using latest master
commit
0d363cf1c1f09b4920e137eac74e1593e7120531.
3:01 p.m.
The rpc command is to arm the printing of the pkg usage summary in the
syslog file. Then, when the kamailio process handles the next sip
message, it is printing the summary in the syslog file, so look inside
/var/log/syslog or /var/log/messages, you should spot the related log
messages, they should contain "summary" or something similar...
Cheers,
Daniel
On 16.10.22 23:48, mayamatakeshi wrote:
On Fri, Oct 14, 2022 at 5:25 PM Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
OK. Can you run the tests for a while and then execute:
kamctl rpc corex.pkg_summary idx 1
The value for idx should be the kamailio process expected to
handle sip traffic. If it is mostly udp and the first socket is
the corresponding udp socket, then the value 1, like above, is ok.
If it is tcp/tls or you have different listen parameters on udp,
see the processes index with:
kamctl ps
I tried calling
kamctl rpc corex.pkg_summary idx 1
before and after the memory problem started to be logged but it always
showed up with result empty:
{
"jsonrpc": "2.0",
"result": {
},
"id": 3824999
}
Here is result for all processes:
https://pastebin.com/AabwyJwx
I tried several times but I never saw a non-empty result.
It should not be required to wait till you get out of memory
errors, but be sure you run it long enough to have many accounting
records written by the process that is going to print the pkg
summary. You can eventually set the children to 1 or 2, to "force"
traffic via a smaller group of processes.
By reducing children from
children=12
to
children=1
I started to get memory allocation error just a after a few minutes of
load test:
[root@lab002107-flip-server ~]$ grep memory
/var/log/kamailio/kamailio.log |head
2022-10-17T06:20:41.314619+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-17T06:20:41.916596+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-17T06:20:41.916786+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-17T06:20:42.518013+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-17T06:20:42.518248+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR: acc
[acc_extra.c:234]: extra2strar(): could not allocate private memory
from pkg pool
2022-10-17T06:20:43.114454+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR:
<core> [core/rvalue.c:253]: rval_new_empty(): could not allocate
private memory from pkg pool
2022-10-17T06:20:43.163816+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR:
<core> [core/rvalue.c:253]: rval_new_empty(): could not allocate
private memory from pkg pool
2022-10-17T06:20:43.165863+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR:
<core> [core/rvalue.c:253]: rval_new_empty(): could not allocate
private memory from pkg pool
2022-10-17T06:20:43.167092+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR:
<core> [core/rvalue.c:253]: rval_new_empty(): could not allocate
private memory from pkg pool
2022-10-17T06:20:43.167247+09:00 lab002107-flip-server
/usr/local/src/git/kamailio-master/src/kamailio[3576087]: ERROR:
<core> [core/data_lump.c:163]: insert_new_lump_before(): could not
allocate private memory from pkg pool
I collected the pkg summary after 4000 CDRs were written to DB but
again, i could not see a non-empty result:
https://pastebin.com/CpaRYLAR
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online
Nov 7-10, 2022 (Europe Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
18 Oct
18 Oct
7:36 a.m.
On Mon, Oct 17, 2022 at 10:01 PM Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
The rpc command is to arm the printing of the pkg
usage summary in the
syslog file. Then, when the kamailio process handles the next sip message,
it is printing the summary in the syslog file, so look inside
/var/log/syslog or /var/log/messages, you should spot the related log
messages, they should contain "summary" or something similar...
OK. Understood. I will try it again the next time pkg memory happens.
7:40 a.m.
On Mon, Oct 17, 2022 at 7:45 AM mayamatakeshi <mayamatakeshi(a)gmail.com>
wrote:
On Fri, Oct 14, 2022 at 6:21 PM Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Load tests are running for more than 30 hours and the pkg problem doesn't
happen anymore.
So it seems your changes solved the issue.
I will start 2 more load test envs and keep them running for 2 months.
Thanks.
Based on your parameters, it seems that acc
should send cdrs also to
syslog. I think I identified an issue on that code and pushed a fix, can
you test with latest master git branch?
OK. I have started 2 load test environments using latest master
commit
0d363cf1c1f09b4920e137eac74e1593e7120531.
563
days inactive
585
days old
13 comments
3 participants
participants (3)
-
Anthony Messina -
Daniel-Constantin Mierla -
mayamatakeshi