Hi,
I have kamailio v5.0.4 git rev. 5f26a4e8c55cdf5a5a7eaf6864c4f5213d321293 installed on 32bit Debian Wheezy virtual machine (1GB RAM and 1 vCPU) for development and testing of a project.
We need to inspect sip traces to / from webrtc endpoint on kamailio side, for which we use siptrace module with following config,
# ----- siptrace params ----- modparam("siptrace", "trace_on", 1) modparam("siptrace", "hep_mode_on", 0) modparam("siptrace", "xheaders_write", 1) modparam("siptrace", "trace_to_database", 0) modparam("siptrace", "trace_flag", FLT_SIP_TRACE) modparam("siptrace", "traced_user_avp", "$avp(traced_user)") modparam("siptrace", "duplicate_uri", "WEBRTC_SIP_TRACE_ADDRESS")
We observed multiple problems of memleaks and segfaults, here is a summary,
1. Calling sip_trace method without any arguements or with only one arguement (duplicate uri) crashes kamailio with segfault.
https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.f.sip...
2. To avoid segfault, either I have to specify both args (i am not sure what to set as 2nd arg, so i used "$ci" for it) OR disable "xheaders_write" module param. However, this causes slow memory leak and in about 48 hours kamailio runs out of memory (sip traffic is less then 10 concurrent users)
https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.p.xhe...
3. If we remove siptrace module and use "network:msg" event route in corex module with "send_data" method then we ran in to PV buffer overflow problem (webrtc invites are very big > 10KB). So, I increased "pv_buffer_size" to 16K and "pv_buffer_slots" to 12. Which again crashes kamailio with segfault.
https://kamailio.org/docs/modules/5.0.x/modules/corex.html#async.evr.network...
https://kamailio.org/docs/modules/5.0.x/modules/corex.html#corex.f.send_data
http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_size
http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_slots
Removing "pv_buffer_slots" from config fixes the crash problem, and right now we are observing kamailio if stays stable.
Thank you.
Hello,
if you get a crash, then take the backtrace from the corefile with gdb (inside the gdb session inspecting the corefile, do "bt full"), it is very useful information to discover where the crash happens.
It is also better to open on issue on bug tracker, so the investigation of the crash can be done easier as well as reduce the risck of forgetting about the report:
- https://github.com/kamailio/kamailio/issues
Cheers, Daniel
On 03.12.17 18:41, M S wrote:
Hi,
I have kamailio v5.0.4 git rev. 5f26a4e8c55cdf5a5a7eaf6864c4f5213d321293 installed on 32bit Debian Wheezy virtual machine (1GB RAM and 1 vCPU) for development and testing of a project.
We need to inspect sip traces to / from webrtc endpoint on kamailio side, for which we use siptrace module with following config,
# ----- siptrace params ----- modparam("siptrace", "trace_on", 1) modparam("siptrace", "hep_mode_on", 0) modparam("siptrace", "xheaders_write", 1) modparam("siptrace", "trace_to_database", 0) modparam("siptrace", "trace_flag", FLT_SIP_TRACE) modparam("siptrace", "traced_user_avp", "$avp(traced_user)") modparam("siptrace", "duplicate_uri", "WEBRTC_SIP_TRACE_ADDRESS")
We observed multiple problems of memleaks and segfaults, here is a summary,
- Calling sip_trace method without any arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.f.sip...
- To avoid segfault, either I have to specify both args (i am not
sure what to set as 2nd arg, so i used "$ci" for it) OR disable "xheaders_write" module param. However, this causes slow memory leak and in about 48 hours kamailio runs out of memory (sip traffic is less then 10 concurrent users)
https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.p.xhe...
- If we remove siptrace module and use "network:msg" event route in
corex module with "send_data" method then we ran in to PV buffer overflow problem (webrtc invites are very big > 10KB). So, I increased "pv_buffer_size" to 16K and "pv_buffer_slots" to 12. Which again crashes kamailio with segfault.
https://kamailio.org/docs/modules/5.0.x/modules/corex.html#async.evr.network...
https://kamailio.org/docs/modules/5.0.x/modules/corex.html#corex.f.send_data
http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_size
http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_slots
Removing "pv_buffer_slots" from config fixes the crash problem, and right now we are observing kamailio if stays stable.
Thank you.
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
M S writes:
- Calling sip_trace method without any arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
My K does not crash when I make call like this:
setflag(TRACE); sip_trace();
but I have not defined as many vars as you, but only these:
# -- siptrace params modparam("siptrace", "db_url", "xxx") modparam("siptrace", "trace_flag", TRACE)
-- Juha
Thanks @Daniel, i am opening up the tickets.
@Juha, we are not dumping sip trace to db but rather to a udp socket (typically 127.0.0.1:5060), where we can observer them using ngrep or sipgrep cli utilities. The whole purpose is to watch sip trace at runtime while conducting our tests and observe the changes as sip client changes its behaviour (it is sort of whitebox testing). Dumping in db or Homer is bit of waste of time and resources for our work. That's why we disabled both of these via module params. We have been doing it with kamailio v4.x for last 1-2 years or so and it never caused any trouble. These problems started happening only after upgrading our dev setup to v5.0.
Thank you.
On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen jh@tutpro.com wrote:
M S writes:
- Calling sip_trace method without any arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
My K does not crash when I make call like this:
setflag(TRACE); sip_trace();but I have not defined as many vars as you, but only these:
# -- siptrace params modparam("siptrace", "db_url", "xxx") modparam("siptrace", "trace_flag", TRACE)
-- Juha
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
For "pv_buffer_slots" issue, kamailio crashes during startup and there is no core dump. What should i do about it?
Here is what i see in logs (just one log line).
-- Dec 4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879] segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so [b75e2000+161000] --
Thank you.
On Mon, Dec 4, 2017 at 3:14 AM, M S shaheryarkh@gmail.com wrote:
Thanks @Daniel, i am opening up the tickets.
@Juha, we are not dumping sip trace to db but rather to a udp socket (typically 127.0.0.1:5060), where we can observer them using ngrep or sipgrep cli utilities. The whole purpose is to watch sip trace at runtime while conducting our tests and observe the changes as sip client changes its behaviour (it is sort of whitebox testing). Dumping in db or Homer is bit of waste of time and resources for our work. That's why we disabled both of these via module params. We have been doing it with kamailio v4.x for last 1-2 years or so and it never caused any trouble. These problems started happening only after upgrading our dev setup to v5.0.
Thank you.
On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen jh@tutpro.com wrote:
M S writes:
- Calling sip_trace method without any arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
My K does not crash when I make call like this:
setflag(TRACE); sip_trace();but I have not defined as many vars as you, but only these:
# -- siptrace params modparam("siptrace", "db_url", "xxx") modparam("siptrace", "trace_flag", TRACE)
-- Juha
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
I think I catch this one, too -- I pushed a patch for it to branch 5.0 as well.
Cheers, Daniel
On 04.12.17 03:49, M S wrote:
For "pv_buffer_slots" issue, kamailio crashes during startup and there is no core dump. What should i do about it?
Here is what i see in logs (just one log line).
-- Dec 4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879] segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so
http://libc-2.13.so[b75e2000+161000]
Thank you.
On Mon, Dec 4, 2017 at 3:14 AM, M S <shaheryarkh@gmail.com mailto:shaheryarkh@gmail.com> wrote:
Thanks @Daniel, i am opening up the tickets. @Juha, we are not dumping sip trace to db but rather to a udp socket (typically 127.0.0.1:5060 <http://127.0.0.1:5060>), where we can observer them using ngrep or sipgrep cli utilities. The whole purpose is to watch sip trace at runtime while conducting our tests and observe the changes as sip client changes its behaviour (it is sort of whitebox testing). Dumping in db or Homer is bit of waste of time and resources for our work. That's why we disabled both of these via module params. We have been doing it with kamailio v4.x for last 1-2 years or so and it never caused any trouble. These problems started happening only after upgrading our dev setup to v5.0. Thank you. On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <jh@tutpro.com <mailto:jh@tutpro.com>> wrote: M S writes: > 1. Calling sip_trace method without any arguements or with only one > arguement (duplicate uri) crashes kamailio with segfault. My K does not crash when I make call like this: setflag(TRACE); sip_trace(); but I have not defined as many vars as you, but only these: # -- siptrace params modparam("siptrace", "db_url", "xxx") modparam("siptrace", "trace_flag", TRACE) -- Juha _______________________________________________ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org <mailto:sr-dev@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev>
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Super. I will test it and report back.
Many thanks and kind regards.
On Mon, Dec 4, 2017 at 9:48 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
I think I catch this one, too -- I pushed a patch for it to branch 5.0 as well.
Cheers, Daniel
On 04.12.17 03:49, M S wrote:
For "pv_buffer_slots" issue, kamailio crashes during startup and there is no core dump. What should i do about it?
Here is what i see in logs (just one log line).
-- Dec 4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879] segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so [b75e2000+161000] --
Thank you.
On Mon, Dec 4, 2017 at 3:14 AM, M S shaheryarkh@gmail.com wrote:
Thanks @Daniel, i am opening up the tickets.
@Juha, we are not dumping sip trace to db but rather to a udp socket (typically 127.0.0.1:5060), where we can observer them using ngrep or sipgrep cli utilities. The whole purpose is to watch sip trace at runtime while conducting our tests and observe the changes as sip client changes its behaviour (it is sort of whitebox testing). Dumping in db or Homer is bit of waste of time and resources for our work. That's why we disabled both of these via module params. We have been doing it with kamailio v4.x for last 1-2 years or so and it never caused any trouble. These problems started happening only after upgrading our dev setup to v5.0.
Thank you.
On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen jh@tutpro.com wrote:
M S writes:
- Calling sip_trace method without any arguements or with only one
arguement (duplicate uri) crashes kamailio with segfault.
My K does not crash when I make call like this:
setflag(TRACE); sip_trace();but I have not defined as many vars as you, but only these:
# -- siptrace params modparam("siptrace", "db_url", "xxx") modparam("siptrace", "trace_flag", TRACE)
-- Juha
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Kamailio (SER) - Development Mailing Listsr-dev@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
-- Daniel-Constantin Mierlawww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training - www.asipto.com Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com
-
Daniel-Constantin Mierla -
Juha Heinanen -
M S