3 Dec
2017
3 Dec
'17
6:41 p.m.
Hi,
I have kamailio v5.0.4 git rev. 5f26a4e8c55cdf5a5a7eaf6864c4f5213d321293
installed on 32bit Debian Wheezy virtual machine (1GB RAM and 1 vCPU) for
development and testing of a project.
We need to inspect sip traces to / from webrtc endpoint on kamailio side,
for which we use siptrace module with following config,
# ----- siptrace params -----
modparam("siptrace", "trace_on", 1)
modparam("siptrace", "hep_mode_on", 0)
modparam("siptrace", "xheaders_write", 1)
modparam("siptrace", "trace_to_database", 0)
modparam("siptrace", "trace_flag", FLT_SIP_TRACE)
modparam("siptrace", "traced_user_avp",
"$avp(traced_user)")
modparam("siptrace", "duplicate_uri",
"WEBRTC_SIP_TRACE_ADDRESS")
We observed multiple problems of memleaks and segfaults, here is a summary,
1. Calling sip_trace method without any arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.f.si…
2. To avoid segfault, either I have to specify both args (i am not sure
what to set as 2nd arg, so i used "$ci" for it) OR disable
"xheaders_write"
module param. However, this causes slow memory leak and in about 48 hours
kamailio runs out of memory (sip traffic is less then 10 concurrent users)
https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.p.xh…
3. If we remove siptrace module and use "network:msg" event route in corex
module with "send_data" method then we ran in to PV buffer overflow problem
(webrtc invites are very big > 10KB). So, I increased "pv_buffer_size" to
16K and "pv_buffer_slots" to 12. Which again crashes kamailio with segfault.
https://kamailio.org/docs/modules/5.0.x/modules/corex.html#async.evr.networ…
https://kamailio.org/docs/modules/5.0.x/modules/corex.html#corex.f.send_data
http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_size
http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_slots
Removing "pv_buffer_slots" from config fixes the crash problem, and right
now we are observing kamailio if stays stable.
Thank you.
3 Dec
3 Dec
9:24 p.m.
Hello,
if you get a crash, then take the backtrace from the corefile with gdb
(inside the gdb session inspecting the corefile, do "bt full"), it is
very useful information to discover where the crash happens.
It is also better to open on issue on bug tracker, so the investigation
of the crash can be done easier as well as reduce the risck of
forgetting about the report:
- https://github.com/kamailio/kamailio/issues
Cheers,
Daniel
On 03.12.17 18:41, M S wrote:
Hi,
I have kamailio v5.0.4 git rev.
5f26a4e8c55cdf5a5a7eaf6864c4f5213d321293 installed on 32bit Debian
Wheezy virtual machine (1GB RAM and 1 vCPU) for development and
testing of a project.
We need to inspect sip traces to / from webrtc endpoint on kamailio
side, for which we use siptrace module with following config,
# ----- siptrace params -----
modparam("siptrace", "trace_on", 1)
modparam("siptrace", "hep_mode_on", 0)
modparam("siptrace", "xheaders_write", 1)
modparam("siptrace", "trace_to_database", 0)
modparam("siptrace", "trace_flag", FLT_SIP_TRACE)
modparam("siptrace", "traced_user_avp",
"$avp(traced_user)")
modparam("siptrace", "duplicate_uri",
"WEBRTC_SIP_TRACE_ADDRESS")
We observed multiple problems of memleaks and segfaults, here is a
summary,
1. Calling sip_trace method without any arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.f.si…
2. To avoid segfault, either I have to specify both args (i am not
sure what to set as 2nd arg, so i used "$ci" for it) OR disable
"xheaders_write" module param. However, this causes slow memory leak
and in about 48 hours kamailio runs out of memory (sip traffic is less
then 10 concurrent users)
https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.p.xh…
3. If we remove siptrace module and use "network:msg" event route in
corex module with "send_data" method then we ran in to PV buffer
overflow problem (webrtc invites are very big > 10KB). So, I increased
"pv_buffer_size" to 16K and "pv_buffer_slots" to 12. Which again
crashes kamailio with segfault.
https://kamailio.org/docs/modules/5.0.x/modules/corex.html#async.evr.networ…
https://kamailio.org/docs/modules/5.0.x/modules/corex.html#corex.f.send_data
http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_size
http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_slots
Removing "pv_buffer_slots" from config fixes the crash problem, and
right now we are observing kamailio if stays stable.
Thank you.
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
--
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com
4 Dec
4 Dec
2:54 a.m.
M S writes:
1. Calling sip_trace method without any arguements or
with only one
arguement (duplicate uri) crashes kamailio with segfault.
My K does not crash when I make call like this:
setflag(TRACE);
sip_trace();
but I have not defined as many vars as you, but only these:
# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)
-- Juha
3:14 a.m.
Thanks @Daniel, i am opening up the tickets.
@Juha, we are not dumping sip trace to db but rather to a udp socket
(typically 127.0.0.1:5060), where we can observer them using ngrep or
sipgrep cli utilities. The whole purpose is to watch sip trace at runtime
while conducting our tests and observe the changes as sip client changes
its behaviour (it is sort of whitebox testing). Dumping in db or Homer is
bit of waste of time and resources for our work. That's why we disabled
both of these via module params. We have been doing it with kamailio v4.x
for last 1-2 years or so and it never caused any trouble. These problems
started happening only after upgrading our dev setup to v5.0.
Thank you.
On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <jh(a)tutpro.com> wrote:
M S writes:
1. Calling sip_trace method without any
arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
My K does not crash when I make call like this:
setflag(TRACE);
sip_trace();
but I have not defined as many vars as you, but only these:
# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)
-- Juha
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
3:49 a.m.
For "pv_buffer_slots" issue, kamailio crashes during startup and there is
no core dump. What should i do about it?
Here is what i see in logs (just one log line).
--
Dec 4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879]
segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so
[b75e2000+161000]
--
Thank you.
On Mon, Dec 4, 2017 at 3:14 AM, M S <shaheryarkh(a)gmail.com> wrote:
Thanks @Daniel, i am opening up the tickets.
@Juha, we are not dumping sip trace to db but rather to a udp socket
(typically 127.0.0.1:5060), where we can observer them using ngrep or
sipgrep cli utilities. The whole purpose is to watch sip trace at runtime
while conducting our tests and observe the changes as sip client changes
its behaviour (it is sort of whitebox testing). Dumping in db or Homer is
bit of waste of time and resources for our work. That's why we disabled
both of these via module params. We have been doing it with kamailio v4.x
for last 1-2 years or so and it never caused any trouble. These problems
started happening only after upgrading our dev setup to v5.0.
Thank you.
On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <jh(a)tutpro.com> wrote:
M S writes:
1. Calling sip_trace method without any
arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
My K does not crash when I make call like this:
setflag(TRACE);
sip_trace();
but I have not defined as many vars as you, but only these:
# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)
-- Juha
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
9:48 a.m.
I think I catch this one, too -- I pushed a patch for it to branch 5.0
as well.
Cheers,
Daniel
On 04.12.17 03:49, M S wrote:
For "pv_buffer_slots" issue, kamailio
crashes during startup and there
is no core dump. What should i do about it?
Here is what i see in logs (just one log line).
--
Dec 4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879]
segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so
<http://libc-2.13.so>[b75e2000+161000]
--
Thank you.
On Mon, Dec 4, 2017 at 3:14 AM, M S <shaheryarkh(a)gmail.com
<mailto:shaheryarkh@gmail.com>> wrote:
Thanks @Daniel, i am opening up the tickets.
@Juha, we are not dumping sip trace to db but rather to a udp
socket (typically 127.0.0.1:5060 <http://127.0.0.1:5060>), where
we can observer them using ngrep or sipgrep cli utilities. The
whole purpose is to watch sip trace at runtime while conducting
our tests and observe the changes as sip client changes its
behaviour (it is sort of whitebox testing). Dumping in db or Homer
is bit of waste of time and resources for our work. That's why we
disabled both of these via module params. We have been doing it
with kamailio v4.x for last 1-2 years or so and it never caused
any trouble. These problems started happening only after upgrading
our dev setup to v5.0.
Thank you.
On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <jh(a)tutpro.com
<mailto:jh@tutpro.com>> wrote:
M S writes:
--
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com
1. Calling sip_trace method without any
arguements or with
only one
arguement (duplicate uri) crashes kamailio with
segfault.
My K does not crash when I make call like this:
setflag(TRACE);
sip_trace();
but I have not defined as many vars as you, but only these:
# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)
-- Juha
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org <mailto:sr-dev@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
<https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev>
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
9:50 a.m.
Super. I will test it and report back.
Many thanks and kind regards.
On Mon, Dec 4, 2017 at 9:48 AM, Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
I think I catch this one, too -- I pushed a patch for
it to branch 5.0 as
well.
Cheers,
Daniel
On 04.12.17 03:49, M S wrote:
For "pv_buffer_slots" issue, kamailio crashes during startup and there is
no core dump. What should i do about it?
Here is what i see in logs (just one log line).
--
Dec 4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879]
segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so
[b75e2000+161000]
--
Thank you.
On Mon, Dec 4, 2017 at 3:14 AM, M S <shaheryarkh(a)gmail.com> wrote:
Thanks @Daniel, i am opening up the tickets.
@Juha, we are not dumping sip trace to db but rather to a udp socket
(typically 127.0.0.1:5060), where we can observer them using ngrep or
sipgrep cli utilities. The whole purpose is to watch sip trace at runtime
while conducting our tests and observe the changes as sip client changes
its behaviour (it is sort of whitebox testing). Dumping in db or Homer is
bit of waste of time and resources for our work. That's why we disabled
both of these via module params. We have been doing it with kamailio v4.x
for last 1-2 years or so and it never caused any trouble. These problems
started happening only after upgrading our dev setup to v5.0.
Thank you.
On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <jh(a)tutpro.com> wrote:
_______________________________________________
Kamailio (SER) - Development Mailing
Listsr-dev@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
--
Daniel-Constantin Mierlawww.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com
M S writes:
1. Calling sip_trace method without any
arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
My K does not crash when I make call like this:
setflag(TRACE);
sip_trace();
but I have not defined as many vars as you, but only these:
# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)
-- Juha
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
2357
days inactive
2358
days old
6 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Juha Heinanen -
M S