sr-dev February 2023

sr-dev@lists.kamailio.org

10 participants
297 discussions

git:master:a229d230: kamctl: hide errors raised by "which"
by Daniel-Constantin Mierla 17 Feb '23

17 Feb '23

Module: kamailio Branch: master Commit: a229d230bca5e785670edc3901f849e93989f9fe URL: https://github.com/kamailio/kamailio/commit/a229d230bca5e785670edc3901f849e… Author: POIROTTE Francois <francois.poirotte(a)csgroup.eu> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2023-02-17T16:41:08+01:00 kamctl: hide errors raised by "which" Hide error messages from "which" to avoid polluting kamctl's output (e.g. when a JSON document is returned) in environments where STDOUT and STDERR may be combined into a single stream (e.g. containers) --- Modified: utils/kamctl/kamctl.base --- Diff: https://github.com/kamailio/kamailio/commit/a229d230bca5e785670edc3901f849e… Patch: https://github.com/kamailio/kamailio/commit/a229d230bca5e785670edc3901f849e… --- diff --git a/utils/kamctl/kamctl.base b/utils/kamctl/kamctl.base index 40e4f09b905..08cd4412054 100644 --- a/utils/kamctl/kamctl.base +++ b/utils/kamctl/kamctl.base @@ -10,7 +10,7 @@ locate_tool() { while [ -n "$1" ] do if [ -x /usr/bin/which ] ; then - TOOLPATH=`which $1` + TOOLPATH=`which $1 2> /dev/null` if [ -n "$TOOLPATH" ]; then return fi

1 0

[kamailio/kamailio] kamctl: hide errors raised by "which" (PR #3377)
by François Poirotte 17 Feb '23

17 Feb '23

#### Pre-Submission Checklist - [ ] Commit message has the format required by CONTRIBUTING guide - [ ] Commits are split per component (core, individual modules, libs, utils, ...) - [ ] Each component has a single commit (if not, squash them into one commit) - [ ] No commits to README files for modules (changes must be done to docbook files in `doc/` subfolder, the README file is autogenerated) #### Type Of Change - [x] Small bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds new functionality) - [ ] Breaking change (fix or feature that would change existing functionality) #### Checklist: - [ ] PR should be backported to stable branches - [ ] Tested changes locally - [ ] Related to issue #XXXX (replace XXXX with an open issue number) #### Description Hide error messages from "which" to avoid polluting kamctl's output (e.g. when a JSON document is returned) in environments where STDOUT and STDERR may be combined into a single stream (e.g. containers). Without this change, kamctl's output may end up containing error messages, resulting in invalid/unparsable data. This is caused by the stdout & stderr streams being combined into a single stream in some environments, which may be outside the end-user's control (e.g. external cloud provider). You can view, comment on, or merge this pull request online at: https://github.com/kamailio/kamailio/pull/3377 -- Commit Summary -- * kamctl: hide errors raised by "which" -- File Changes -- M utils/kamctl/kamctl.base (2) -- Patch Links -- https://github.com/kamailio/kamailio/pull/3377.patch https://github.com/kamailio/kamailio/pull/3377.diff -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/pull/3377 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/pull/3377(a)github.com>

2 2

git:master:ac0edd34: dialog: Do not trigger bye_reply_cb for in-dialog requests
by Carsten Bock 16 Feb '23

16 Feb '23

Module: kamailio Branch: master Commit: ac0edd34b491dc78bca6979d2f1444c50bfb7399 URL: https://github.com/kamailio/kamailio/commit/ac0edd34b491dc78bca6979d2f1444c… Author: Carsten Bock <carsten(a)ng-voice.com> Committer: Carsten Bock <carsten(a)ng-voice.com> Date: 2023-02-16T16:27:39+01:00 dialog: Do not trigger bye_reply_cb for in-dialog requests --- Modified: src/modules/dialog/dlg_req_within.c --- Diff: https://github.com/kamailio/kamailio/commit/ac0edd34b491dc78bca6979d2f1444c… Patch: https://github.com/kamailio/kamailio/commit/ac0edd34b491dc78bca6979d2f1444c… --- diff --git a/src/modules/dialog/dlg_req_within.c b/src/modules/dialog/dlg_req_within.c index 4f9ebfc693..a91bcec246 100644 --- a/src/modules/dialog/dlg_req_within.c +++ b/src/modules/dialog/dlg_req_within.c @@ -697,7 +697,7 @@ int dlg_request_within(struct sip_msg *msg, struct dlg_cell *dlg, int side, set_uac_req(&uac_r, method, allheaders.len?&allheaders:NULL, (content && content->len)?content:NULL, dialog_info, TMCB_LOCAL_COMPLETED, - bye_reply_cb, (void*)iuid); + NULL, NULL); result = d_tmb.t_request_within(&uac_r);

2 3

git:master:2f7da809: statsc: cast to uint64_t when using time_t value
by Daniel-Constantin Mierla 16 Feb '23

16 Feb '23

Module: kamailio Branch: master Commit: 2f7da809d53ffadff017681e257b8d86bec3a5ce URL: https://github.com/kamailio/kamailio/commit/2f7da809d53ffadff017681e257b8d8… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2023-02-16T12:16:26+01:00 statsc: cast to uint64_t when using time_t value --- Modified: src/modules/statsc/statsc_mod.c --- Diff: https://github.com/kamailio/kamailio/commit/2f7da809d53ffadff017681e257b8d8… Patch: https://github.com/kamailio/kamailio/commit/2f7da809d53ffadff017681e257b8d8… --- diff --git a/src/modules/statsc/statsc_mod.c b/src/modules/statsc/statsc_mod.c index 69cd335b66..3b636b81fc 100644 --- a/src/modules/statsc/statsc_mod.c +++ b/src/modules/statsc/statsc_mod.c @@ -24,6 +24,7 @@ #include <unistd.h> #include <stdlib.h> #include <string.h> +#include <stdint.h> #include <time.h> #include "../../core/sr_module.h" @@ -409,8 +410,8 @@ static void statsc_rpc_report(rpc_t* rpc, void* ctx) return; } if(rpc->struct_add(th, "u[", - "timestamp", (unsigned int)tn, - "stats", &ts )<0) { + "timestamp", (uint64_t)tn, + "stats", &ts)<0) { rpc->fault(ctx, 500, "Error creating rpc (2)"); return; }

1 0

git:master:c78910a7: uac: cast to uint64_t when using time_t value
by Daniel-Constantin Mierla 16 Feb '23

16 Feb '23

Module: kamailio Branch: master Commit: c78910a71226f3a567ad396ce5e834036ce75f97 URL: https://github.com/kamailio/kamailio/commit/c78910a71226f3a567ad396ce5e8340… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2023-02-16T12:07:26+01:00 uac: cast to uint64_t when using time_t value --- Modified: src/modules/uac/uac_reg.c --- Diff: https://github.com/kamailio/kamailio/commit/c78910a71226f3a567ad396ce5e8340… Patch: https://github.com/kamailio/kamailio/commit/c78910a71226f3a567ad396ce5e8340… --- diff --git a/src/modules/uac/uac_reg.c b/src/modules/uac/uac_reg.c index 02138101c07..c9df873ab16 100644 --- a/src/modules/uac/uac_reg.c +++ b/src/modules/uac/uac_reg.c @@ -26,6 +26,7 @@ */ #include <time.h> +#include <stdint.h> #include "../../core/dprint.h" #include "../../core/timer.h" @@ -1882,7 +1883,7 @@ static int rpc_uac_reg_add_node_helper(rpc_t* rpc, void* ctx, reg_uac_t *reg, ti rpc->fault(ctx, 500, "Internal error creating rpc"); return -1; } - if (rpc->struct_add(th, "SSSSSSSSSSddddddSS", + if (rpc->struct_add(th, "SSSSSSSSSSJdddJdSS", "l_uuid", &reg->l_uuid, "l_username", &reg->l_username, "l_domain", &reg->l_domain, @@ -1896,11 +1897,11 @@ static int rpc_uac_reg_add_node_helper(rpc_t* rpc, void* ctx, reg_uac_t *reg, ti &reg->auth_ha1:&none, "auth_proxy", (reg->auth_proxy.len)? &reg->auth_proxy:&none, - "expires", (int)reg->expires, + "expires", (uint64_t)reg->expires, "flags", (int)reg->flags, "diff_expires", (int)(reg->timer_expires - tn), "timer_expires", (int)reg->timer_expires, - "reg_init", (int)reg->reg_init, + "reg_init", (uint64_t)reg->reg_init, "reg_delay", (int)reg->reg_delay, "contact_addr", (reg->contact_addr.len)? &reg->contact_addr:&none,

1 0

git:5.5:a9deaa50: rtpengine: avoid repeated queries to same node
by Henning Westerholt 16 Feb '23

16 Feb '23

Module: kamailio Branch: 5.5 Commit: a9deaa503ff20a3384fe74e37093464c21885e44 URL: https://github.com/kamailio/kamailio/commit/a9deaa503ff20a3384fe74e37093464… Author: Richard Fuchs <rfuchs(a)sipwise.com> Committer: Henning Westerholt <hw(a)gilawa.com> Date: 2023-02-16T11:10:10Z rtpengine: avoid repeated queries to same node If we know which node handles a particular call from the hash table, only query that node once and then return error if it's dead, instead of going into a pointless loop looking for other nodes to query which don't exist. closes #3370 (cherry picked from commit bd0ce126109f40f5217053f20e687b93b2268164) (cherry picked from commit ea782be2cc29589f4dd55f0d91c11049b5280fe0) --- Modified: src/modules/rtpengine/rtpengine.c --- Diff: https://github.com/kamailio/kamailio/commit/a9deaa503ff20a3384fe74e37093464… Patch: https://github.com/kamailio/kamailio/commit/a9deaa503ff20a3384fe74e37093464… --- diff --git a/src/modules/rtpengine/rtpengine.c b/src/modules/rtpengine/rtpengine.c index 62665205787..9cd3bd4061c 100644 --- a/src/modules/rtpengine/rtpengine.c +++ b/src/modules/rtpengine/rtpengine.c @@ -3320,6 +3320,12 @@ select_rtpp_node(str callid, str viabranch, int do_test, struct rtpp_node **quer // lookup node node = select_rtpp_node_old(callid, viabranch, do_test, op); + if (node && is_queried_node(node, queried_nodes_ptr, queried_nodes)) { + LM_ERR("rtpengine node for callid=%.*s is known (%.*s) but it has already been queried, therefore not returning it\n", + callid.len, callid.s, node->rn_url.len, node->rn_url.s); + return NULL; + } + // check node if (!node || (node_in_set(node, active_rtpp_set) == 0)) { // run the selection algorithm

1 0

git:5.6:ea782be2: rtpengine: avoid repeated queries to same node
by Henning Westerholt 16 Feb '23

16 Feb '23

Module: kamailio Branch: 5.6 Commit: ea782be2cc29589f4dd55f0d91c11049b5280fe0 URL: https://github.com/kamailio/kamailio/commit/ea782be2cc29589f4dd55f0d91c1104… Author: Richard Fuchs <rfuchs(a)sipwise.com> Committer: Henning Westerholt <hw(a)gilawa.com> Date: 2023-02-16T11:09:42Z rtpengine: avoid repeated queries to same node If we know which node handles a particular call from the hash table, only query that node once and then return error if it's dead, instead of going into a pointless loop looking for other nodes to query which don't exist. closes #3370 (cherry picked from commit bd0ce126109f40f5217053f20e687b93b2268164) --- Modified: src/modules/rtpengine/rtpengine.c --- Diff: https://github.com/kamailio/kamailio/commit/ea782be2cc29589f4dd55f0d91c1104… Patch: https://github.com/kamailio/kamailio/commit/ea782be2cc29589f4dd55f0d91c1104… --- diff --git a/src/modules/rtpengine/rtpengine.c b/src/modules/rtpengine/rtpengine.c index ab6920c01d6..1b1f8b0ca92 100644 --- a/src/modules/rtpengine/rtpengine.c +++ b/src/modules/rtpengine/rtpengine.c @@ -3349,6 +3349,12 @@ select_rtpp_node(str callid, str viabranch, int do_test, struct rtpp_node **quer // lookup node node = select_rtpp_node_old(callid, viabranch, do_test, op); + if (node && is_queried_node(node, queried_nodes_ptr, queried_nodes)) { + LM_ERR("rtpengine node for callid=%.*s is known (%.*s) but it has already been queried, therefore not returning it\n", + callid.len, callid.s, node->rn_url.len, node->rn_url.s); + return NULL; + } + // check node if (!node || (node_in_set(node, active_rtpp_set) == 0)) { // run the selection algorithm

1 0

[kamailio/kamailio] topoh: th_mask_decode (th_mask_encode (x)) ≠ x (Issue #3358)
by Дилян Палаузов 15 Feb '23

15 Feb '23

The file [th-test.c.txt](https://github.com/kamailio/kamailio/files/10608862/th-test.… contains the exact copy of the th_mask_encode() and th_mask_decode() functions from src/modules/topoh/th_mask.c . (except LM_ERR is removed and pkg_malloc/pkg_free are replaced with malloc/free). When I run Kamilio under topoh and the websocket client calls, at some moment `sip:test-gnome-calls@bapha.be;gr=urn:uuid:f2f5a3cf-a0fb-0047-be50-740fb9bdc562;alias=87.118.146.153~60472~2>;+sip.instance="<urn:uuid:f2f5a3cf-a0fb-0047-be50-740fb9bdc562>";+org.linphone.specs="conference/1.0,ephemeral/1.1,groupchat/1.1,groupchat/1.2,lime"` is passed to th_mask_encode(). The output it `sip:127.3.4.84;line=sr-if7s1mg7i36PNf0AbdwPpfzlbqWEpLzsSGItpLwyN39ZMY4t1mCTSd6DNo4LWdIOpfpPp5FLpUQscX63Kd47W5EPWO6sNL90pLgoW5p-1fzlSdzO25n3KoIk1vHkWXptc5wOeopsWXi-eo9*`. This is put in a Contact: header. When that last string is passed to th_mask_decode(), e.g. because BYE or ACK have it as R-URI, the output is `sip:test-gnome-calls@bapha.be;gr=urn:uuid:f2f5a3cf-a0fb-0047-be50-740fb9bdc562;alias=87.118.1`. From the initial input the final output misses at least `46.153`. The attached file contains under `b2` another input, which is passed ot th_mask_decode. This is the actual input Kamailio gets at runtime and passes to th_mask_decode. The result is not a string — callu.b.p...b.;.r...n:s..d..2lQ — where the dots represent unprintable bytes. The error report is that th_mask_encode() produces output, which th_mask_decode() does not convert to its origin. N.B.: The above problem happens, when a websocket client calls TCP-client with topoh-module. When a TLS-client calls TCP-client (no websockets involved), topoh does not make problems — BYE is delivered properly. See also: * https://lists.kamailio.org/mailman3/hyperkitty/list/sr-users@lists.kamailio… * https://lists.kamailio.org/mailman3/hyperkitty/list/sr-users@lists.kamailio… This might also be related: * https://lists.kamailio.org/mailman3/hyperkitty/list/sr-users@lists.kamailio… -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3358 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3358(a)github.com>

2 4

Build failed in Jenkins: kamailio56-nightly-binaries » amd64,bookworm #50
by kamailio＠sipwise.com 15 Feb '23

15 Feb '23

See <https://kamailio.sipwise.com/job/kamailio56-nightly-binaries/architecture=a…> Changes: ------------------------------------------ [...truncated 1.82 MB...] ../../core/dprint.h:326:99: note: in definition of macro 'LOG_FX' 326 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1845:58: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1845 | (pframe && pframe->f_code)?PyBytes_AsString(pframe->f_code->co_name):"", | ^~ ../../core/dprint.h:326:99: note: in definition of macro 'LOG_FX' 326 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1845:91: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1845 | (pframe && pframe->f_code)?PyBytes_AsString(pframe->f_code->co_name):"", | ^~ ../../core/dprint.h:326:99: note: in definition of macro 'LOG_FX' 326 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1846:58: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1846 | (pframe && pframe->f_code)?PyCode_Addr2Line(pframe->f_code, pframe->f_lasti):0); | ^~ ../../core/dprint.h:326:99: note: in definition of macro 'LOG_FX' 326 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1846:91: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1846 | (pframe && pframe->f_code)?PyCode_Addr2Line(pframe->f_code, pframe->f_lasti):0); | ^~ ../../core/dprint.h:326:99: note: in definition of macro 'LOG_FX' 326 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1846:107: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1846 | (pframe && pframe->f_code)?PyCode_Addr2Line(pframe->f_code, pframe->f_lasti):0); | ^~ ../../core/dprint.h:326:99: note: in definition of macro 'LOG_FX' 326 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1844:58: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1844 | (pframe && pframe->f_code)?PyBytes_AsString(pframe->f_code->co_filename):"", | ^~ ../../core/dprint.h:337:99: note: in definition of macro 'LOG_FX' 337 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1844:91: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1844 | (pframe && pframe->f_code)?PyBytes_AsString(pframe->f_code->co_filename):"", | ^~ ../../core/dprint.h:337:99: note: in definition of macro 'LOG_FX' 337 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1845:58: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1845 | (pframe && pframe->f_code)?PyBytes_AsString(pframe->f_code->co_name):"", | ^~ ../../core/dprint.h:337:99: note: in definition of macro 'LOG_FX' 337 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1845:91: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1845 | (pframe && pframe->f_code)?PyBytes_AsString(pframe->f_code->co_name):"", | ^~ ../../core/dprint.h:337:99: note: in definition of macro 'LOG_FX' 337 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1846:58: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1846 | (pframe && pframe->f_code)?PyCode_Addr2Line(pframe->f_code, pframe->f_lasti):0); | ^~ ../../core/dprint.h:337:99: note: in definition of macro 'LOG_FX' 337 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1846:91: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1846 | (pframe && pframe->f_code)?PyCode_Addr2Line(pframe->f_code, pframe->f_lasti):0); | ^~ ../../core/dprint.h:337:99: note: in definition of macro 'LOG_FX' 337 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ apy_kemi.c:1846:107: error: invalid use of incomplete typedef 'PyFrameObject' {aka 'struct _frame'} 1846 | (pframe && pframe->f_code)?PyCode_Addr2Line(pframe->f_code, pframe->f_lasti):0); | ^~ ../../core/dprint.h:337:99: note: in definition of macro 'LOG_FX' 337 | LOGV_FUNCSUFFIX_STR(funcname), ## args); \ | ^~~~ ../../core/dprint.h:351:25: note: in expansion of macro 'LOG_FL' 351 | LOG_FL(facility, level, NULL, prefix, fmt, ## args) | ^~~~~~ ../../core/dprint.h:354:25: note: in expansion of macro 'LOG_FP' 354 | LOG_FP(DEFAULT_FACILITY, (level), LOC_INFO, fmt, ## args) | ^~~~~~ apy_kemi.c:1839:25: note: in expansion of macro 'LOG' 1839 | LOG(cfg_get(core, core_cfg, latency_log), | ^~~ make[3]: *** [../../Makefile.rules:100: apy_kemi.o] Error 1 make[2]: *** [Makefile:511: modules] Error 1 make[2]: Leaving directory '/build/kamailio-5.6.4~bpo12.20230211004740.50/src' make[1]: *** [Makefile:34: every-module] Error 2 make[1]: Leaving directory '/build/kamailio-5.6.4~bpo12.20230211004740.50' make: *** [debian/rules:140: build-stamp] Error 2 dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2 I: copying local configuration E: Failed autobuilding of package I: user script /var/cache/pbuilder/build/cow.18611/tmp/hooks/C10shell starting ┌──────────────────────────────────────────────────────────────────────────────┐ │ FTBFS - problem with building Debian package │ └──────────────────────────────────────────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────────────────────────┐ │ DEB_KEEP_BUILD_ENV is not set to 'true', not keeping build environment │ └──────────────────────────────────────────────────────────────────────────────┘ I: user script /var/cache/pbuilder/build/cow.18611/tmp/hooks/C10shell finished I: unmounting /tmp/apt-jdg_tcCqiU filesystem I: unmounting dev/ptmx filesystem I: unmounting dev/pts filesystem I: unmounting dev/shm filesystem I: unmounting proc filesystem I: unmounting sys filesystem I: Cleaning COW directory I: forking: rm -rf /var/cache/pbuilder/build/cow.18611 + '[' 1 -eq 0 ']' + bailout 1 'Error: Failed to build with cowbuilder.' + '[' -n 1 ']' + EXIT=1 + '[' -n 'Error: Failed to build with cowbuilder.' ']' + echo 'Error: Failed to build with cowbuilder.' Error: Failed to build with cowbuilder. + rm -f /run/lock/bookworm-amd64.building.18278 + '[' -r /run/lock/bookworm-amd64.update.18278 ']' + '[' '' = true ']' + echo '*** Getting rid of files in <https://kamailio.sipwise.com/job/kamailio56-nightly-binaries/architecture=a…> to avoid problems in next run. ***' *** Getting rid of files in <https://kamailio.sipwise.com/job/kamailio56-nightly-binaries/architecture=a…> to avoid problems in next run. *** + rm -f '<https://kamailio.sipwise.com/job/kamailio56-nightly-binaries/architecture=a…'> + '[' -n /tmp/tmp.9odg8EyHnU ']' + rm -rf /tmp/tmp.9odg8EyHnU + '[' -n /tmp/apt-jdg_tcCqiU ']' + sudo rm -rf /tmp/apt-jdg_tcCqiU + '[' -n 300 ']' ++ sed -e 's/^$[0-9]*$.*/\1/' + SECONDS=620 ++ date + echo '*** Finished execution of /usr/bin/build-and-provide-package at Sat Feb 11 01:05:41 UTC 2023 [running 620 seconds] ***' *** Finished execution of /usr/bin/build-and-provide-package at Sat Feb 11 01:05:41 UTC 2023 [running 620 seconds] *** + exit 1 Build step 'Execute shell' marked build as failure Archiving artifacts Recording fingerprints TAP Reports Processing: START Looking for TAP results report in workspace using pattern: report/*.tap Did not find any matching files. Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?

1 2

[kamailio/kamailio] rtpengine: queried_nodes_limit incorrectly applies to delete commands which can cause all processes to be blocked (Issue #3370)
by shaneqld 15 Feb '23

15 Feb '23

## Description When one rtpengine instance becomes unavailable on a Kamailio instance with moderate call traffic, even where there are multiple rtpengine instances configured for high availability, all processes can become blocked and Kamailio stops responding to all requests. We produced this in a real world test, with two Kamailio servers running (20 child processes) and two rtpengine servers running. We loaded the Kamailio servers with calls, some answered, some ringing/busy/etc, and then made one rtpengine server drop all packets. Both Kamailio servers were able to briefly detect that the rtpengine instance was unavailable when it attempted to send an offer to it, but then as existing calls began to complete, both Kamailio servers became congested attempting to send "delete" commands to rtpengine. Both Kamailio servers then became unavailable and stopped responding to all messages for 10 minutes, until we restored rtpengine. As such, an rtpengine server that crashes can cause a complete Kamailio outage. ## Troubleshooting By using the reproduction steps below, we were easily able to make Kamailio stop responding to all incoming SIP messages when taking down an rtpengine instance. The logs below show the sequence of events when this is reproduced for a single call. Configuration: - `rtpengine_tout_ms` was not configured and had the default value of 1000ms. - `rtpengine_retr` was not configured and had the default value of 5. - `queried_nodes_limit` was not configured and had the default value (of 30?). - TM's `exec_time_check` was not configured and had the default value of 5000ms. **A call is forwarded using t_relay and rtpengine_manage. rtpengine is killed mid-call. The call ends with 486.** When the t_on_failure route is called, the config makes a singular call to rtpengine_manage, however what follows is 30 errors, all mentioning the same rtpengine instance: ``` ERROR: rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> ``` Followed by the below: ``` ERROR: rtpengine [rtpengine.c:2729]: rtpp_function_call(): queried nodes limit reached WARNING: tm [t_reply.c:1094]: run_failure_handlers(): failure route execution took too long: 150153261 us ``` Each timeout is 5 seconds (5x retries of 1000ms), which corresponds to about 150 seconds over 30 attempts. This matches the warning generated by tm. In this case, there was about 5 minutes of time where a process was blocked. 150 seconds for the child leg's failure route, and 150 seconds for the parent leg's failure route. When this happens for many calls at once, congestion is guaranteed. The 30 attempts to delete the call from rtpengine comes from the default value of `queried_nodes_limit`. This behavior doesn't make sense to me -- it's a retry loop that runs a retry loop. ## Reproduction _tldr: kill rtpengine via iptables mid-call with the default rtpengine settings and you'll see lots of errors._ You can reproduce this with one or multiple rtpengine instances configured in Kamailio. It's easier with just one. Standard setup, but importantly: - forwarding a call using t_relay - t_on_branch is configured which calls rtpengine_manage - t_on_failure is configured which calls rtpengine_manage To reproduce easily, we configured the destination server to send a 100, a 180, wait 20 seconds, then send a 486. Place a number of calls to Kamailio (we used sipp) -- importantly, exceeding the number of child processes configured. You can reproduce the issue with just one call, but this wont make Kamailio unresponsive. Once the calls are placed, and still in progress (ie before the 486 is returned), use iptables on the rtpengine server to drop all traffic on its control port. When the call(s) start receiving the 486, you will notice a flood of error messages `timeout waiting reply for command "delete" from RTPEngine` for approximately 5 minutes per call. This blocks the process. If it's now attempting to send the delete command to rtpengine for a number of calls greater than your child process count: attempt to send any other SIP messages to Kamailio and you will not receive a response. ## Log Messages This is a log for a single call, forwarded using t_relay. All calls to rtpengine_manage are immediately surrounded by info log messages "calling rtpengine_manage" and "completed calling rtpengine_manage". ``` Feb 12 23:40:54 debian-kamailio kamailio[27715]: INFO: {1 1 INVITE 1-27864(a)127.0.1.1} <script>: new branch [0] to sip:0421XXXXXX@192.168.X.X:9060 Feb 12 23:40:54 debian-kamailio kamailio[27715]: INFO: {1 1 INVITE 1-27864(a)127.0.1.1} <script>: branch_route: calling rtpengine_manage Feb 12 23:40:54 debian-kamailio kamailio[27715]: INFO: {1 1 INVITE 1-27864(a)127.0.1.1} <script>: branch_route: completed calling rtpengine_manage Feb 12 23:40:54 debian-kamailio kamailio[27732]: INFO: {2 1 INVITE 1-27864(a)127.0.1.1} <script>: onreply_route: calling rtpengine_manage Feb 12 23:40:54 debian-kamailio kamailio[27732]: INFO: {2 1 INVITE 1-27864(a)127.0.1.1} <script>: onreply_route: completed calling rtpengine_manage Feb 12 23:40:55 debian-kamailio kamailio[27730]: INFO: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: new branch [0] to sip:61421764018@192.168.X.X:9092;transport=udp Feb 12 23:40:55 debian-kamailio kamailio[27730]: INFO: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: branch_route: calling rtpengine_manage Feb 12 23:40:55 debian-kamailio kamailio[27730]: INFO: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: branch_route: completed calling rtpengine_manage Feb 12 23:40:55 debian-kamailio kamailio[27717]: INFO: {2 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: onreply_route: calling rtpengine_manage Feb 12 23:40:55 debian-kamailio kamailio[27717]: INFO: {2 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: onreply_route: completed calling rtpengine_manage Feb 12 23:40:55 debian-kamailio kamailio[27714]: INFO: {2 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: onreply_route: calling rtpengine_manage Feb 12 23:40:55 debian-kamailio kamailio[27714]: INFO: {2 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: onreply_route: completed calling rtpengine_manage Feb 12 23:40:55 debian-kamailio kamailio[27735]: INFO: {2 1 INVITE 1-27864(a)127.0.1.1} <script>: onreply_route: calling rtpengine_manage Feb 12 23:40:55 debian-kamailio kamailio[27735]: INFO: {2 1 INVITE 1-27864(a)127.0.1.1} <script>: onreply_route: completed calling rtpengine_manage Feb 12 23:41:15 debian-kamailio kamailio[27713]: INFO: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: failure_route: calling rtpengine_manage Feb 12 23:41:20 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:41:25 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:41:30 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:41:35 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:41:40 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:41:45 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:41:50 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:41:55 debian-kamailio kamailio[27740]: INFO: {2 1 INVITE 1-27864(a)127.0.1.1} <script>: onreply_route: calling rtpengine_manage Feb 12 23:41:55 debian-kamailio kamailio[27740]: INFO: {2 1 INVITE 1-27864(a)127.0.1.1} <script>: onreply_route: completed calling rtpengine_manage Feb 12 23:41:55 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:00 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:05 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:10 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:16 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:21 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:26 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:31 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:36 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:41 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:46 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:51 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:42:55 debian-kamailio kamailio[27734]: INFO: {2 1 INVITE 1-27864(a)127.0.1.1} <script>: onreply_route: calling rtpengine_manage Feb 12 23:42:55 debian-kamailio kamailio[27734]: INFO: {2 1 INVITE 1-27864(a)127.0.1.1} <script>: onreply_route: completed calling rtpengine_manage Feb 12 23:42:56 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:01 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:06 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:11 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:16 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:21 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:26 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:31 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:36 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:41 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:46 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:46 debian-kamailio kamailio[27713]: ERROR: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} rtpengine [rtpengine.c:2729]: rtpp_function_call(): queried nodes limit reached Feb 12 23:43:46 debian-kamailio kamailio[27713]: INFO: {1 63577835 INVITE 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: failure_route: completed calling rtpengine_manage Feb 12 23:43:46 debian-kamailio kamailio[27713]: WARNING: tm [t_reply.c:1094]: run_failure_handlers(): failure route execution took too long: 150153207 us Feb 12 23:43:46 debian-kamailio kamailio[27736]: INFO: {1 63577835 ACK 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: ACK: calling rtpengine_manage Feb 12 23:43:46 debian-kamailio kamailio[27736]: INFO: {1 63577835 ACK 720bf6d9-25fb-123c-cb91-80fa5d718101} <script>: ACK: completed calling rtpengine_manage Feb 12 23:43:46 debian-kamailio kamailio[27738]: INFO: {1 1 INVITE 1-27864(a)127.0.1.1} <script>: failure_route: calling rtpengine_manage Feb 12 23:43:51 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:43:56 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:01 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:06 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:11 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:16 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:21 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:26 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:31 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:36 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:41 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:46 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:51 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:44:56 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:01 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:06 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:11 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:16 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:21 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:26 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:31 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:36 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:41 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:46 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:51 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:45:56 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:46:01 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:46:06 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:46:11 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:46:16 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:3129]: send_rtpp_command(): timeout waiting reply for command "delete" from RTPEngine <udp:192.168.X.X:2223> Feb 12 23:46:16 debian-kamailio kamailio[27738]: ERROR: {1 1 INVITE 1-27864(a)127.0.1.1} rtpengine [rtpengine.c:2729]: rtpp_function_call(): queried nodes limit reached Feb 12 23:46:16 debian-kamailio kamailio[27738]: INFO: {1 1 INVITE 1-27864(a)127.0.1.1} <script>: failure_route: completed calling rtpengine_manage Feb 12 23:46:16 debian-kamailio kamailio[27738]: WARNING: tm [t_reply.c:1094]: run_failure_handlers(): failure route execution took too long: 150153261 us ``` ### Possible Solutions To minimize the issue, `rtpengine_tout_ms` and `rtpengine_retr` can be reduced. This doesn't exactly the fix the problem, as you need reasonable values here. Additionally, `queried_nodes_limit` can be reduced to limit the number of unnecessary retries, at the expense of the functionality it gives you to initially select candidate nodes. **Actual solution:** I think that, when a call has already selected an rtpengine instance successfully, and that's in the hash table, then `queried_nodes_limit` should not apply. It should only attempt to send commands with a total number of attempts = `rtpengine_retr`, not `rtpengine_retr * queried_nodes_limit`. By changing the behavior as above, tweaking `rtpengine_tout_ms` and `rtpengine_retr`, and increasing the number of child workers, an rtpengine becoming unavailable should cause much less impactful blocking. ## Additional Information **Kamailio version:** (attempted on 5.4.1 and latest master) ``` version: kamailio 5.4.1 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 23:22:15 Aug 15 2022 with gcc 8.3.0 ``` ``` version: kamailio 5.7.0-dev3 (x86_64/linux) 2384c3 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 2384c3 compiled on 23:22:22 Feb 12 2023 with gcc 8.3.0 ``` **Operating System**:  ``` Distributor ID: Debian Description: Debian GNU/Linux 10 (buster) Release: 10 Codename: buster Linux debian-kamailio 4.19.0-20-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64 GNU/Linux ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3370 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3370(a)github.com>

4 11

← Newer
1
...
13
14
15
16
17
18
19
...
30
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

sr-dev February 2023