sr-dev April 2020

sr-dev@lists.kamailio.org

33 participants
421 discussions

[kamailio/kamailio] async_http_client crash (#2286)
by Alex Balashov 22 Apr '20

22 Apr '20

Version: 5.3.3 `065668` Platform: CentOS 7 (3.10.0-862.14.4.el7.x86_64) I seem to have a persistent intermittent crash on all five gateways which use `http_async_client` to query an HTTP API. In all cases, the crash is on attempt to access or reference an AVP called `$avp(is_subscribe)`: ``` Program terminated with signal 11, Segmentation fault. #0 0x00000000005d479b in match_by_name (avp=0xabcdefed, id=45, name=0x7fff9379c668) at core/usr_avp.c:378 378 if (id==avp->id && avp->flags&AVP_NAME_STR && hMissing separate debuginfos, use: debuginfo-install libuuid-2.23.2-52.el7_5.1.x86_64 nss-pem-1.0.3-4.el7.x86_64 (gdb) where #0 0x00000000005d479b in match_by_name (avp=0xabcdefed, id=45, name=0x7fff9379c668) at core/usr_avp.c:378 #1 0x00000000005d54bf in search_next_avp (s=0x7fff9379c660, val=0x7fff9379c630) at core/usr_avp.c:499 #2 0x00000000005d4f04 in search_avp (ident=..., val=0x7fff9379c630, state=0x7fff9379c660) at core/usr_avp.c:465 #3 0x00000000005d4913 in search_first_avp (flags=1, name=..., val=0x7fff9379c630, s=0x7fff9379c660) at core/usr_avp.c:414 #4 0x00007f3f6ba529fd in ops_is_avp_set (msg=0xb04f60 <_faked_msg>, ap=0x7f3f73296ed8) at avpops_impl.c:1821 #5 0x00007f3f6ba60d47 in w_is_avp_set (msg=0xb04f60 <_faked_msg>, param=0x7f3f73296ed8 "", op=0x0) at avpops.c:1059 #6 0x0000000000480f3d in do_action (h=0x7fff9379ce70, a=0x7f3f732a8b30, msg=0xb04f60 <_faked_msg>) at core/action.c:1077 #7 0x000000000048da7c in run_actions (h=0x7fff9379ce70, a=0x7f3f732a8b30, msg=0xb04f60 <_faked_msg>) at core/action.c:1576 #8 0x000000000048e13d in run_actions_safe (h=0x7fff9379dae0, a=0x7f3f732a8b30, msg=0xb04f60 <_faked_msg>) at core/action.c:1640 #9 0x0000000000438ff3 in rval_get_int (h=0x7fff9379dae0, msg=0xb04f60 <_faked_msg>, i=0x7fff9379d318, rv=0x7f3f732a8d70, cache=0x0) at core/rvalue.c:915 #10 0x000000000043d5a3 in rval_expr_eval_int (h=0x7fff9379dae0, msg=0xb04f60 <_faked_msg>, res=0x7fff9379d318, rve=0x7f3f732a8d68) at core/rvalue.c:1913 #11 0x000000000043d9f6 in rval_expr_eval_int (h=0x7fff9379dae0, msg=0xb04f60 <_faked_msg>, res=0x7fff9379d7cc, rve=0x7f3f732a9598) at core/rvalue.c:1921 #12 0x000000000048097b in do_action (h=0x7fff9379dae0, a=0x7f3f732aa9c0, msg=0xb04f60 <_faked_msg>) at core/action.c:1047 #13 0x000000000048da7c in run_actions (h=0x7fff9379dae0, a=0x7f3f732aa9c0, msg=0xb04f60 <_faked_msg>) at core/action.c:1576 #14 0x000000000048e205 in run_top_route (a=0x7f3f732aa9c0, msg=0xb04f60 <_faked_msg>, c=0x0) at core/action.c:1661 #15 0x00007f3f705f7d0e in async_http_cb (reply=0x7f3f2b7ee620, param=0x7f3f2b7eca18) at async_http.c:256 #16 0x00007f3f705f0dab in check_multi_info (g=0x7f3f2abeb9e8) at http_multi.c:597 #17 0x00007f3f705e84ec in event_cb (fd=6, kind=2, userp=0x1d7cdf0) at http_multi.c:145 #18 0x00007f3f70131a14 in event_process_active_single_queue (activeq=0x1c6f820, base=0x1d3ad90) at event.c:1350 #19 event_process_active (base=<optimized out>) at event.c:1420 #20 event_base_loop (base=0x1d3ad90, flags=0) at event.c:1621 #21 0x00007f3f705f490f in async_http_run_worker (worker=0x7f3f2abd1bb0) at async_http.c:92 #22 0x00007f3f705dcb17 in child_init (rank=0) at http_async_client_mod.c:366 #23 0x000000000057a810 in init_mod_child (m=0x7f3f73245168, rank=0) at core/sr_module.c:780 #24 0x000000000057a4ac in init_mod_child (m=0x7f3f73245d58, rank=0) at core/sr_module.c:776 #25 0x000000000057a4ac in init_mod_child (m=0x7f3f73246340, rank=0) at core/sr_module.c:776 #26 0x000000000057a4ac in init_mod_child (m=0x7f3f732467f8, rank=0) at core/sr_module.c:776 #27 0x000000000057a4ac in init_mod_child (m=0x7f3f73246e38, rank=0) at core/sr_module.c:776 #28 0x000000000057a4ac in init_mod_child (m=0x7f3f73247210, rank=0) at core/sr_module.c:776 #29 0x000000000057a4ac in init_mod_child (m=0x7f3f73247978, rank=0) at core/sr_module.c:776 #30 0x000000000057a4ac in init_mod_child (m=0x7f3f73247da0, rank=0) at core/sr_module.c:776 #31 0x000000000057af5a in init_child (rank=0) at core/sr_module.c:825 #32 0x0000000000426bc1 in main_loop () at main.c:1753 #33 0x000000000042e63a in main (argc=13, argv=0x7fff9379edb8) at main.c:2802 ``` Most recently, it is here: ``` route { $avp(is_subscribe) = 0; ... $http_req(method) = 'GET'; $http_req(suspend) = 1; $http_req(hdr) = "x-api-key: " + $sel(cfg_get.api.whitelist_key); t_newtran(); http_async_query("https://some_host/check_ip_in_whi telist?ip=$(si{s.escape.common})", "WHITELIST_CHECK"); } ... route[WHITELIST_CHECK] { if(!is_avp_set("$avp(is_subscribe)")) { xlog("L_INFO", "[R-HTTP-REPLY:$ci] !> AVP is_subscribe is undefined -- cannot process further\n"); sl_send_reply("500", "Internal Server Error"); exit; } ``` However, adding `is_avp_set()` was my attempt to put a safe-guard around unsafe access of this AVP. It didn't work: ``` (gdb) print *name $1 = {s = 0x7f3f73261835 "is_subscribe)", len = 12} (gdb) print avp $2 = (avp_t *) 0xabcdefed (gdb) print avp->id Cannot access memory at address 0xabcdefed ``` Note that I am making the assumption above that AVPs created in the main request route will be available in the `http_async_client` callback, because it is just a suspended TM transaction underneath -- right? So, transaction-persistent variables such as AVP should be visible throughout the lifetime of the transaction just as for a conventional async TM transaction, I would think. Digging further into the mesage being handled in `async_http_cb()`, it is always a normal HTTP `200 OK` reply: ``` (gdb) set print elements 0 (gdb) frame 15 #15 0x00007f3f705f7d0e in async_http_cb (reply=0x7f3f2b7ee620, param=0x7f3f2b7eca18) at async_http.c:256 256 if (run_top_route(act, fmsg, 0)<0) { (gdb) print *reply $5 = {retcode = 200, result = 0x7f3f2b7ec560, error = '\000' <repeats 255 times>} (gdb) print *(reply->result) $6 = { s = 0x7f3f2b7ee798 "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 16\r\nConnection: keep-alive\r\nDate: Wed, 15 Apr 2020 15:06:52 GMT\r\nx-amzn-RequestId: d3a44941-e0c4-4cb9-9144-ea54d929c300\r\nx-amz-apigw-id: LCJqBF7CIAMFXCw=\r\nX-Amzn-Trace-Id: Root=1-5e97230c-4fe1af7d072275d95c231a4c;Sampled=0\r\nX-Cache: Miss from cloudfront\r\nHia: 1.1 ade18dc841d2e1cc8ef49611c5d4c93e.cloudfront.net (CloudFront)\r\nX-Amz-Cf-Pop: IAD89-C2\r\nX-Amz-Cf-Id: 3s1S2kvOgL7L4ZKMlWwQnfIxHmWFGz0PHbcNFefYA1tZ1j0AGy3JHg==\r\n\r\n{\"status\":false}\300\300\300\300", len = 504} ``` Any insights would be appreciated! These crashes are happening basically hourly from nowhere, after over a year of undisturbed operation. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2286

5 15

git:master:a59dbb7a: presence: safety checks in timer callbacks to cope with storage modes
by Daniel-Constantin Mierla 22 Apr '20

22 Apr '20

Module: kamailio Branch: master Commit: a59dbb7a81ea7909cd542acbba85d231ccd519a4 URL: https://github.com/kamailio/kamailio/commit/a59dbb7a81ea7909cd542acbba85d23… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2020-04-22T08:42:48+02:00 presence: safety checks in timer callbacks to cope with storage modes --- Modified: src/modules/presence/hash.c Modified: src/modules/presence/presence.c Modified: src/modules/presence/publish.c Modified: src/modules/presence/publish.h Modified: src/modules/presence/subscribe.c Modified: src/modules/presence/subscribe.h --- Diff: https://github.com/kamailio/kamailio/commit/a59dbb7a81ea7909cd542acbba85d23… Patch: https://github.com/kamailio/kamailio/commit/a59dbb7a81ea7909cd542acbba85d23…

1 0

git:master:eb5c1a41: presence: use separate structure to match presentity to replace or update
by Daniel-Constantin Mierla 22 Apr '20

22 Apr '20

Module: kamailio Branch: master Commit: eb5c1a41ced14f502648c6bda9ae0ef585b0cb37 URL: https://github.com/kamailio/kamailio/commit/eb5c1a41ced14f502648c6bda9ae0ef… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2020-04-22T08:11:36+02:00 presence: use separate structure to match presentity to replace or update --- Modified: src/modules/presence/hash.c Modified: src/modules/presence/hash.h Modified: src/modules/presence/presentity.c Modified: src/modules/presence/presentity.h Modified: src/modules/presence/publish.c --- Diff: https://github.com/kamailio/kamailio/commit/eb5c1a41ced14f502648c6bda9ae0ef… Patch: https://github.com/kamailio/kamailio/commit/eb5c1a41ced14f502648c6bda9ae0ef…

1 0

git:master:8077f387: presence: init pres and subs hash tables for non db-only mode and memory index
by Daniel-Constantin Mierla 22 Apr '20

22 Apr '20

Module: kamailio Branch: master Commit: 8077f387f9253a486a980c65a776507a7ad6ee7f URL: https://github.com/kamailio/kamailio/commit/8077f387f9253a486a980c65a776507… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2020-04-21T22:42:12+02:00 presence: init pres and subs hash tables for non db-only mode and memory index --- Modified: src/modules/presence/presence.c --- Diff: https://github.com/kamailio/kamailio/commit/8077f387f9253a486a980c65a776507… Patch: https://github.com/kamailio/kamailio/commit/8077f387f9253a486a980c65a776507… --- diff --git a/src/modules/presence/presence.c b/src/modules/presence/presence.c index cbbc23feb3..7120dd7951 100644 --- a/src/modules/presence/presence.c +++ b/src/modules/presence/presence.c @@ -350,10 +350,11 @@ static int mod_init(void) } if(publ_cache_mode==PS_PCACHE_HYBRID || publ_cache_mode==PS_PCACHE_RECORD) { - if(phtable_size < 1) + if(phtable_size < 1) { phtable_size = 256; - else + } else { phtable_size = 1 << phtable_size; + } } if(publ_cache_mode==PS_PCACHE_RECORD) { @@ -362,6 +363,28 @@ static int mod_init(void) } } + if(pres_subs_dbmode != DB_ONLY) { + if(shtable_size < 1) { + shtable_size = 512; + } else { + shtable_size = 1 << shtable_size; + } + + subs_htable = new_shtable(shtable_size); + if(subs_htable == NULL) { + LM_ERR(" initializing subscribe hash table\n"); + goto dberror; + } + } + + if(publ_cache_mode==PS_PCACHE_HYBRID) { + pres_htable = new_phtable(); + if(pres_htable == NULL) { + LM_ERR("initializing presentity hash table\n"); + goto dberror; + } + } + if(publ_cache_mode != PS_PCACHE_RECORD || pres_subs_dbmode != NO_DB) { if(pres_db_url.s == NULL) { LM_ERR("database url not set!\n"); @@ -405,17 +428,8 @@ static int mod_init(void) goto dberror; } + if(pres_subs_dbmode != DB_ONLY) { - if(shtable_size < 1) - shtable_size = 512; - else - shtable_size = 1 << shtable_size; - - subs_htable = new_shtable(shtable_size); - if(subs_htable == NULL) { - LM_ERR(" initializing subscribe hash table\n"); - goto dberror; - } if(restore_db_subs() < 0) { LM_ERR("restoring subscribe info from database\n"); goto dberror; @@ -423,13 +437,7 @@ static int mod_init(void) } if(publ_cache_mode==PS_PCACHE_HYBRID) { - pres_htable = new_phtable(); - if(pres_htable == NULL) { - LM_ERR("initializing presentity hash table\n"); - goto dberror; - } - - if(pres_htable_restore() < 0) { + if(pres_htable_db_restore() < 0) { LM_ERR("filling in presentity hash table from database\n"); goto dberror; }

1 0

Re: [sr-dev] [kamailio/kamailio] presence: functions to keep presentity records in memory (f917318)
by Daniel-Constantin Mierla 21 Apr '20

21 Apr '20

@eschmidbauer I created a dedicated issue on tracker for troubleshooting related issues to in-memory presentity records, let's continue there: * https://github.com/kamailio/kamailio/issues/2294 Attach there the output for `bt full` taken from core file with gdb. Also, add the parameters you set for presence* modules. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/commit/f917318c1e5fefa4106eaaf46123506…

1 0

git:master:174cc5e8: presence: delete expired in-memory presentity records
by Daniel-Constantin Mierla 21 Apr '20

21 Apr '20

Module: kamailio Branch: master Commit: 174cc5e8d1800312eaa6d1bc4fea8df7934c5b55 URL: https://github.com/kamailio/kamailio/commit/174cc5e8d1800312eaa6d1bc4fea8df… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2020-04-21T19:45:01+02:00 presence: delete expired in-memory presentity records --- Modified: src/modules/presence/publish.c --- Diff: https://github.com/kamailio/kamailio/commit/174cc5e8d1800312eaa6d1bc4fea8df… Patch: https://github.com/kamailio/kamailio/commit/174cc5e8d1800312eaa6d1bc4fea8df… --- diff --git a/src/modules/presence/publish.c b/src/modules/presence/publish.c index 4662965f50..dfa37c07e8 100644 --- a/src/modules/presence/publish.c +++ b/src/modules/presence/publish.c @@ -265,7 +265,6 @@ void ps_ptable_timer_clean(unsigned int ticks, void *param) if(ptlist==NULL) { return; } - for(ptn = ptlist; ptn != NULL; ptn = ptn->next) { memset(&pres, 0, sizeof(presentity_t)); @@ -316,6 +315,12 @@ void ps_ptable_timer_clean(unsigned int ticks, void *param) } error: + for(ptn = ptlist; ptn != NULL; ptn = ptn->next) { + if(ps_ptable_remove(ptn) <0) { + LM_ERR("failed deleting presentity item\n"); + } + } + if(ptlist != NULL) { ps_presentity_list_free(ptlist, 1); }

1 0

git:master:7ada66b1: presence: user core macro for hash slot index
by Daniel-Constantin Mierla 21 Apr '20

21 Apr '20

Module: kamailio Branch: master Commit: 7ada66b12254b809faaf265d21e75e7f347f695d URL: https://github.com/kamailio/kamailio/commit/7ada66b12254b809faaf265d21e75e7… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2020-04-21T17:31:44+02:00 presence: user core macro for hash slot index --- Modified: src/modules/presence/hash.c --- Diff: https://github.com/kamailio/kamailio/commit/7ada66b12254b809faaf265d21e75e7… Patch: https://github.com/kamailio/kamailio/commit/7ada66b12254b809faaf265d21e75e7… --- diff --git a/src/modules/presence/hash.c b/src/modules/presence/hash.c index dc72a366ac..f6abcdcc79 100644 --- a/src/modules/presence/hash.c +++ b/src/modules/presence/hash.c @@ -992,7 +992,7 @@ int ps_ptable_insert(ps_presentity_t *pt) return -1; } - idx = ptn->hashid & (_ps_ptable->ssize - 1); + idx = core_hash_idx(ptn->hashid, _ps_ptable->ssize); lock_get(&_ps_ptable->slots[idx].lock); if(_ps_ptable->slots[idx].plist == NULL) { @@ -1028,7 +1028,7 @@ int ps_ptable_replace(ps_presentity_t *pt) ptc.ruid = pres_sruid.uid; } - idx = ptc.hashid & (_ps_ptable->ssize - 1); + idx = core_hash_idx(ptc.hashid, _ps_ptable->ssize); lock_get(&_ps_ptable->slots[idx].lock); ptn = _ps_ptable->slots[idx].plist; @@ -1090,7 +1090,7 @@ int ps_ptable_update(ps_presentity_t *pt) ptc.ruid = pres_sruid.uid; } - idx = ptc.hashid & (_ps_ptable->ssize - 1); + idx = core_hash_idx(ptc.hashid, _ps_ptable->ssize); lock_get(&_ps_ptable->slots[idx].lock); ptn = _ps_ptable->slots[idx].plist; @@ -1146,7 +1146,7 @@ int ps_ptable_remove(ps_presentity_t *pt) memcpy(&ptc, pt, sizeof(ps_presentity_t)); ptc.hashid = core_case_hash(&pt->user, &pt->domain, 0); - idx = ptc.hashid & (_ps_ptable->ssize - 1); + idx = core_hash_idx(ptc.hashid, _ps_ptable->ssize); lock_get(&_ps_ptable->slots[idx].lock); ptn = _ps_ptable->slots[idx].plist; @@ -1189,7 +1189,7 @@ ps_presentity_t *ps_ptable_get_list(str *user, str *domain) ptc.user = *user; ptc.domain = *domain; ptc.hashid = core_case_hash(&ptc.user, &ptc.domain, 0); - idx = ptc.hashid & (_ps_ptable->ssize - 1); + idx = core_hash_idx(ptc.hashid, _ps_ptable->ssize); lock_get(&_ps_ptable->slots[idx].lock); ptn = _ps_ptable->slots[idx].plist; @@ -1236,7 +1236,7 @@ ps_presentity_t *ps_ptable_get_item(str *user, str *domain, str *event, str *eta ptc.event = *event; ptc.etag = *etag; ptc.hashid = core_case_hash(&ptc.user, &ptc.domain, 0); - idx = ptc.hashid & (_ps_ptable->ssize - 1); + idx = core_hash_idx(ptc.hashid, _ps_ptable->ssize); lock_get(&_ps_ptable->slots[idx].lock); ptn = _ps_ptable->slots[idx].plist;

1 0

git:master:d4f8674d: core: hashes - small helper macro to compute hash slot index
by Daniel-Constantin Mierla 21 Apr '20

21 Apr '20

Module: kamailio Branch: master Commit: d4f8674d0068a31bed7097804f37b6bff5fe21fb URL: https://github.com/kamailio/kamailio/commit/d4f8674d0068a31bed7097804f37b6b… Author: Daniel-Constantin Mierla <miconda(a)gmail.com> Committer: Daniel-Constantin Mierla <miconda(a)gmail.com> Date: 2020-04-21T17:30:37+02:00 core: hashes - small helper macro to compute hash slot index --- Modified: src/core/hashes.h --- Diff: https://github.com/kamailio/kamailio/commit/d4f8674d0068a31bed7097804f37b6b… Patch: https://github.com/kamailio/kamailio/commit/d4f8674d0068a31bed7097804f37b6b… --- diff --git a/src/core/hashes.h b/src/core/hashes.h index 5b8e2f2773..2afb8361c8 100644 --- a/src/core/hashes.h +++ b/src/core/hashes.h @@ -27,7 +27,7 @@ #include "str.h" - +#define core_hash_idx(hid, hsize) ((hid)&(hsize-1)) /** internal use: hash update * params: char* s - string start,

1 0

Re: [sr-dev] [kamailio/kamailio] presence: functions to keep presentity records in memory (f917318)
by Emmanuel Schmidbauer 21 Apr '20

21 Apr '20

update, if i set `modparam("presence", "subs_db_mode", 3)` there is no segfault -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/commit/f917318c1e5fefa4106eaaf46123506…

1 0

Re: [sr-dev] [kamailio/kamailio] presence: functions to keep presentity records in memory (f917318)
by Emmanuel Schmidbauer 21 Apr '20

21 Apr '20

here is a trace from the coredump: ``` Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f12607da547 in atomic_cmpxchg_int (var=0x8, old=0, new_v=1) at ../../core/mem/../atomic/atomic_x86.h:224 224 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) (gdb) up #1 0x00007f12607da57b in futex_get (lock=0x8) at ../../core/mem/../futexlock.h:99 99 v=atomic_cmpxchg(lock, 0, 1); /* lock if 0 */ (gdb) #2 0x00007f12607fe0d0 in update_db_subs_timer_dbnone (no_lock=0) at subscribe.c:2079 2079 lock_get(&subs_htable[i].lock); (gdb) up #3 0x00007f1260802121 in timer_db_update (ticks=61825161, param=0x0) at subscribe.c:2441 2441 update_db_subs_timer_dbnone(no_lock); (gdb) up #4 0x00005568ade05b2e in sr_wtimer_exec (ticks=61825161, param=0x0) at core/timer_proc.c:433 433 wt->f(ticks, wt->param); (gdb) up #5 0x00005568ade04b58 in fork_sync_timer (child_id=-1, desc=0x5568ae01a3a1 "secondary timer", make_sock=1, f=0x5568ade057a8 <sr_wtimer_exec>, param=0x0, interval=1000) at core/timer_proc.c:267 267 f(TICKS_TO_S(ts1), param); /* ticks in sec for compatibility with old (gdb) up #6 0x00005568ade05ece in sr_wtimer_start () at core/timer_proc.c:459 459 if(fork_sync_timer(-1 /*PROC_TIMER*/, "secondary timer", 1, (gdb) up #7 0x00005568adc05e24 in main_loop () at main.c:1452 1452 if(sr_wtimer_start()<0) { (gdb) up #8 0x00005568adc123e7 in main (argc=9, argv=0x7ffd9501a8a8) at main.c:2845 2845 ret=main_loop(); (gdb) up ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/commit/f917318c1e5fefa4106eaaf46123506…

1 0

← Newer
1
...
16
17
18
19
20
21
22
...
43
Older →

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

sr-dev April 2020