[SR-Users] Re: Depending on topoh.mask_key Kamailio cannot parse RURIs or modifies magically the SIP uri by changing some bytes in it (2)

Hello Henning, thanks for your answer. The symptoms are that the BYE cannot be handled properly, when the caller is behind websockets. When the caller has established TLS connection, then BYE from caller is delivered properly to the TCP callee. In December in this thread https://lists.kamailio.org/mailman3/hyperkitty/list/sr-users@lists.kamailio… it was again reported that with topoh just BYE does not work. But it says nothing about websockets. I enable system (normal) malloc/free by adding to src/Makefile.defs: diff --git a/src/Makefile.defs b/src/Makefile.defs index eb71d6cd0b..648102d63d 100644 --- a/src/Makefile.defs +++ b/src/Makefile.defs @@ -118,6 +118,8 @@ MEMMNG ?= 0 # 0 - off (no-debug mode) # 1 - on (debug mode) MEMDBG ?= 1 +MEMPKG = sys +MEMDBGSYS = 1 VERSIONVAL = $(shell expr $(VERSION) \* 1000000 + $(PATCHLEVEL) \* 1000 + \ $(SUBLEVEL) ) kamailio -I prints: Print out of kamailio internals Version: kamailio 5.6.3 (x86_64/linux) 641148 Default config: /etc/kamailio/kamailio.cfg Default paths to modules: /lib64/kamailio/modules Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED MAX_RECV_BUFFER_SIZE=262144 MAX_URI_SIZE=1024 BUF_SIZE=65535 DEFAULT PKG_SIZE=8MB DEFAULT SHM_SIZE=64MB ADAPTIVE_WAIT_LOOPS=1024 TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select Source code revision ID: 641148 Compiled with: gcc 12.2.1 Compiled architecture: x86_64 Compiled on: 15:50:37 Feb 3 2023 Thank you for flying kamailio! as the output includes Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY how can I know that the system malloc/free are used? Running under valgrind shows no problems, so my changes to src/Makefile.defs might not trigger system malloc/free use. Any suggestions? To sum up: - without TOPOH, there are no problems - with TOPOH, caller is behind TLS, callee is TCP-connected, there are no problems - with TOPOH, caller is behind websocket, calle is TCP-connected, there are problems: some bytes in the URL of the destination are replaced with unprintable characters, or the R-URI is reported as unparsable (depending on the value of topoh.mask_key). Moreover I have both loaded outbound-module and have configured the websocket provisions, when websockets is used without outbound. As last speciality, the websocket client connects over secured-TLS (https/wss) to the webserver and the latter connects to Kamailio over insecure HTTP/TCP. That is the webclient includes Via:/WSS, but Kamailio adds transport=ws (not transport=wss). Any hints? Greetings Dilyan -----Original Message----- From: Henning Westerholt &lt;hw(a)gilawa.com&gt; To: Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Cc: Дилян Палаузов &lt;dpa-kamailio(a)bapha.be&gt; Subject: RE: [SR-Users] Depending on topoh.mask_key Kamailio cannot parse RURIs or modifies magically the SIP uri by changing some bytes in it (2) Date: 02/02/2023 11:27:47 PM Hello, regarding shared memory, Kamailio uses usually shared memory, regardless of how many workers are configured. About compiling Kamailio without its internal memory pool, its possible. This was discussed last month in a thread about memory performance, if I remember correctly, on the list. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com -----Original Message----- From: Дилян Палаузов &lt;dpa-kamailio(a)bapha.be&gt; Sent: Wednesday, February 1, 2023 4:11 PM To: Kamailio - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: [SR-Users] Depending on topoh.mask_key Kamailio cannot parse RURIs or modifies magically the SIP uri by changing some bytes in it (2) Hello, does somebody has success, when using websockets and topoh? If I run a single Kamailio process, I guess I do not need real shared memory.  Can I adjust Kamailio somehow to use just malloc()/free() in this case, so that debugging is easier? As it turned out dns_cache_init=off leads to crashes - https://github.com/kamailio/kamailio/issues/3350 .  With dns_cache_init=on and use_dns_cache=on, does not lead to reading invalid data. Running Kamailio under valgrind: When I do not set topoh.mask_key (so the default is used), for the yesterday mentioned workflow, for incoming packet: 18(19) ERROR: <core> [core/kemi.c:96]: sr_kemi_core_err(): ksr_sipdump_event 1--------- sipdump:msg src 144.76.142.78:48294 dst 144.76.142.78:5060 ACK sip:127.3.4.84;line=sr-n6iazhainwp1g6s5orj136tdouwy3etacsf73ejxgwvqp9e7zhkucrp- gbeewrfl36313.0e3j4amxpworeiw.y1wlpagevs3eabw.3lz6tdcrtlh.cwobffzgzfwx37m.jlibmlzxczibv* SIP/2.0 Route: <sip:77jSK02EGC2JHwWQTI5OE8SQTI5OvKY=@144.76.142.78;transport=ws;r2=on;lr;nat=yes> Route: <sip:127.3.4.84;line=sr-N6IAzBNwCIWOMxeta4MLmqrwKutjm.KHa.r.jKaeWjQ6muqQPxF4WGZwWJZfWxV7WlcXpUerohWAowe4H9aBNx1LMBu5oB1dNB1739PQkRKl> Via: SIP/2.0/WSS lvqj9uhih64a.invalid;branch=z9hG4bK962314 Max-Forwards: 70 To: <sip:test-gnome-calls@bapha.be>;tag=5Ml7bJz From: "Online https://sip.bapha.be" <sip:online@bapha.be>;tag=kab7lqjdgi Call-ID: i7jjtim7h775c6vqqudg CSeq: 9846 ACK Supported: outbound User-Agent: SIP.js/0.7.8 Content-Length: 0 it logs: 18(19) ERROR: kemix [kemix_mod.c:229]: ki_kx_get_ruri_attr(): failed to parse the R-URI 18(19) ERROR: rr [loose.c:1011]: loose_route_mode(): failed to parse Request URI With topoh.mask_key "TEAI32l)- eauiDEUIA!?()" (in practice without the quotes) the same happens. With topoh.mask_key "TEAI32l" (in practive without the quotes) for the same workflow: the above message does not appear, but KSR.tm.t_relay() produces for this input 8(19) ERROR: <core> [core/kemi.c:96]: sr_kemi_core_err(): ksr_sipdump_event 1--------- sipdump:msg src 144.76.142.78:52284 dst 144.76.142.78:5060 ACK sip:127.3.4.84;    line=sr-if7s1mg7i36pnf0abdwppfzlbqweplzssgitplwyn39zmy4t1mctsd6dno4lwdiopfppp5flpuqscx63kd47w5epwo6snl90plgow5p- 1fzlsdzo25n3koik1vhkwxptc5woeoc-159seo9* SIP/2.0 Route: <sip:wCkxt90ui5/O5wWQTI5OE8SQTI5OzDw=@144.76.142.78;transport=ws;r2=on;lr;nat=yes> Route: <sip:127.3.4.84;line=sr-if7s1mMXS3RQ15FTS5wA5OC3CTzwV5C2g5R5wCg4WwZDgqiZ6XIQWvH3WUHkWX9tWOnyMq4RbmWsb34Q2YgoiXP-coTAboPlioPtpY6Z8dCO> Via: SIP/2.0/WSS do52j57mf93b.invalid;branch=z9hG4bK1262183 Max-Forwards: 70 To: <sip:test-gnome-calls@bapha.be>;tag=~QBNVNr From: "Online https://sip.bapha.be" <sip:online@bapha.be>;tag=rfucq55gg3 Call-ID: 0up52op7q7ilpa4mrq7e CSeq: 3376 ACK Supported: outbound User-Agent: SIP.js/0.7.8 Content-Length: 0 18(19) ERROR: <core> [core/forward.c:508]: forward_request(): resolving "testnimecallubp.b" failed: unresolvable A or AAAA request [-7] In the above string there are some invisible bytes.  When I pass it over hexdump -C the output is: 00000000  31 38 28 31 39 29 20 45  52 52 4f 52 3a 20 3c 63  |18(19) ERROR: <c| 00000010  6f 72 65 3e 20 5b 63 6f  72 65 2f 66 6f 72 77 61  |ore> [core/forwa| 00000020  72 64 2e 63 3a 35 30 38  5d 3a 20 66 6f 72 77 61  |rd.c:508]: forwa| 00000030  72 64 5f 72 65 71 75 65  73 74 28 29 3a 20 72 65  |rd_request(): re| 00000040  73 6f 6c 76 69 6e 67 20  22 74 65 73 74 18 83 6e  |solving "test..n| 00000050  69 6d 65 18 63 61 6c 6c  75 1f 62 c1 70 c1 17 2e  |ime.callu.b.p...| 00000060  62 c5 22 20 66 61 69 6c  65 64 3a 20 75 6e 72 65  |b." failed: unre| 00000070  73 6f 6c 76 61 62 6c 65  20 41 20 6f 72 20 41 41  |solvable A or AA| 00000080  41 41 20 72 65 71 75 65  73 74 20 5b 2d 37 5d 0d  |AA request [-7].| 00000090  0a                                                |.| 00000091 As you can see the strings test-gnome-calls(a)bapha.be and test..nime.callu.b.p are similar: many octets are identacal.  I have stopped and started Kamailio several times, the results do repeat. These are my topoh parameters #modparam("topoh", "mask_key", "TEAI32l)- eauiDEUIA!?()")                                                                               #modparam("topoh", "mask_key", "TEAI32l")                                                                                               modparam("topoh", "mask_ip", "127.3.4.84")                                                                                             modparam("topoh", "sanity_checks", 1)                                                                                                  modparam("topoh", "event_mode", 1)                                                                                                     When I do not load topoh for the otherwise same configuration, I guess this is the corresponding ACK (avoiding topoh): ACK sip:test-gnome-calls@bapha.be;gr=urn:uuid:f2f5a3cf-a0fb-0047-be50-740fb9bdc562 SIP/2.0 Via: SIP/2.0/TCP 144.76.142.78;branch=z9hG4bKeb9b.c2d7e402c761cb6fd66ef32158f8027e.0 Via: SIP/2.0/WSS fcgmb3ib9e9m.invalid;rport=33508;received=144.76.142.78;branch=z9hG4bK1390926 Max-Forwards: 69 To: <sip:test-gnome-calls@bapha.be>;tag=uPOadoD From: "Online https://sip.bapha.be" <sip:online@bapha.be>;tag=cka2tgnhpn Call-ID: scvdj9ijv4khq7lg68jl CSeq: 5381 ACK Supported: outbound User-Agent: SIP.js/0.7.8 Content-Length: 0 Then things work without problems. Greetings   Dilyan -----Original Message----- From: Дилян Палаузов &lt;dpa-kamailio(a)bapha.be&gt; To: Kamailio - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Depending on topoh.mask_key Kamailio either cannot parse RURIs or reads invalid data Date: 01/31/2023 10:28:05 AM Hello, when I use dns_try_naptr=on corelog=-1 debug=-1 enable_tls=yes use_dns_cache=off dns_cache_init=off modparam("topoh", "mask_key", "TEAI32l)- eauiDEUIA!?()") and run Kamailio under valgrind, Kamailio logs: 20(21) ERROR: kemix [kemix_mod.c:229]: ki_kx_get_ruri_attr(): failed to parse the R-URI 20(21) ERROR: rr [loose.c:1011]: loose_route_mode(): failed to parse Request URI The workflow is: I start Kamailio.  A UDP-client registers.  Then a websocket client calls the UDP-client.  Finally the websocket client hangs up. Valgrind does not report anything suspicious. When I use instead modparam("topoh", "mask_key", "TEAI32l") for the same workflow, with the same configuration, valgrind logs: ==14== Invalid read of size 4 ==14==    at 0x4867E5: atomic_cmpxchg_int (atomic_x86.h:224) ==14==    by 0x486830: futex_get (futexlock.h:99) ==14==    by 0x490CE4: dns_hash_get (dns_cache.c:673) ==14==    by 0x4972A6: dns_get_entry (dns_cache.c:2001) ==14==    by 0x499288: dns_srv_get_he (dns_cache.c:2455) ==14==    by 0x597AAD: no_naptr_srv_sip_resolvehost (resolve.c:1599) ==14==    by 0x598332: naptr_sip_resolvehost (resolve.c:1675) ==14==    by 0x5983C5: _sip_resolvehost (resolve.c:1707) ==14==    by 0x49943B: dns_srv_sip_resolvehost (dns_cache.c:2516) ==14==    by 0x49B3DD: dns_sip_resolvehost (dns_cache.c:2738) ==14==    by 0x59846A: sip_hostport2su (resolve.c:1727) ==14==    by 0x4CB951: forward_request (forward.c:515) ==14==    by 0x99993CD: t_relay_to (t_funcs.c:300) ==14==    by 0x99EBA32: _w_t_relay_to (tm.c:1764) ==14==    by 0x99F4DEB: ki_t_relay (tm.c:2917) ==14==    by 0xA80A94B: sr_kemi_lua_exec_func_ex (app_lua_api.c:1022) ==14==    by 0xA81237D: sr_kemi_lua_exec_func (app_lua_api.c:1706) ==14==    by 0xA81B93F: sr_kemi_lua_exec_func_209 (app_lua_kemi_export.c:1717) ==14==    by 0xA8383C0: luaD_precall (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA84C4CA: luaV_execute (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA838C90: luaD_callnoyield (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA837029: luaD_rawrunprotected (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA8391DF: luaD_pcall (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA8333DE: lua_pcallk (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA806F0C: app_lua_run_ex (app_lua_api.c:773) ==14==    by 0xA825F06: sr_kemi_config_engine_lua (app_lua_mod.c:119) ==14==    by 0x5048BE: sr_kemi_route (kemi.c:3784) ==14==    by 0x588909: receive_msg (receive.c:502) ==14==    by 0xA609ADF: ws_frame_receive (ws_frame.c:644) ==14==    by 0x4BCAAB: sr_event_exec (events.c:299) ==14==  Address 0x0 is not stack'd, malloc'd or (recently) free'd ==14== ==14== ==14== Process terminating with default action of signal 11 (SIGSEGV): dumping core ==14==  Access not within mapped region at address 0x0 ==14==    at 0x4867E5: atomic_cmpxchg_int (atomic_x86.h:224) ==14==    by 0x486830: futex_get (futexlock.h:99) ==14==    by 0x490CE4: dns_hash_get (dns_cache.c:673) ==14==    by 0x4972A6: dns_get_entry (dns_cache.c:2001) ==14==    by 0x499288: dns_srv_get_he (dns_cache.c:2455) ==14==    by 0x597AAD: no_naptr_srv_sip_resolvehost (resolve.c:1599) ==14==    by 0x598332: naptr_sip_resolvehost (resolve.c:1675) ==14==    by 0x5983C5: _sip_resolvehost (resolve.c:1707) ==14==    by 0x49943B: dns_srv_sip_resolvehost (dns_cache.c:2516) ==14==    by 0x49B3DD: dns_sip_resolvehost (dns_cache.c:2738) ==14==    by 0x59846A: sip_hostport2su (resolve.c:1727) ==14==    by 0x4CB951: forward_request (forward.c:515) ==14==    by 0x99993CD: t_relay_to (t_funcs.c:300) ==14==    by 0x99EBA32: _w_t_relay_to (tm.c:1764) ==14==    by 0x99F4DEB: ki_t_relay (tm.c:2917) ==14==    by 0xA80A94B: sr_kemi_lua_exec_func_ex (app_lua_api.c:1022) ==14==    by 0xA81237D: sr_kemi_lua_exec_func (app_lua_api.c:1706) ==14==    by 0xA81B93F: sr_kemi_lua_exec_func_209 (app_lua_kemi_export.c:1717) ==14==    by 0xA8383C0: luaD_precall (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA84C4CA: luaV_execute (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA838C90: luaD_callnoyield (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA837029: luaD_rawrunprotected (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA8391DF: luaD_pcall (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA8333DE: lua_pcallk (in /lib64/kamailio/modules/app_lua.so) ==14==    by 0xA806F0C: app_lua_run_ex (app_lua_api.c:773) ==14==    by 0xA825F06: sr_kemi_config_engine_lua (app_lua_mod.c:119) ==14==    by 0x5048BE: sr_kemi_route (kemi.c:3784) ==14==    by 0x588909: receive_msg (receive.c:502) ==14==    by 0xA609ADF: ws_frame_receive (ws_frame.c:644) ==14==    by 0x4BCAAB: sr_event_exec (events.c:299) The stacktrace with the values of the variables is available at https://github.com/kamailio/kamailio/issues/3350 .  Or at least these things seem very similar to me. Any idea? I can share the OCI-image and the full configuration. Greetings   Дилян __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: