[kamailio/kamailio] nghttp2 module causing Kamailio P-CSCF to crash on connect (Issue #3917)
14 Jul
2024
14 Jul
'24
1:07 a.m.
### Description
I am doing some experiments with NGHTTP2 Module to make use of it in some scenarios, where
Kamailio receives HTTP2 requests and has t react on them, unfortunately the module is
causing Kamailio to crash upon receiving a request and need to be restarted.
Also the Docs for the module need some correction :
https://www.kamailio.org/docs/modules/5.9.x/modules/nghttp2.html
Missing required Parameters :
modparam("nghttp2", "tls_private_key", "key_in_PEM_Format")
modparam("nghttp2", "tls_public_key",
"certificate_in_PEM_Format")
also $nghttp2(url) is wrong, $nghttp2(path) or $nghttp2(pathfull) is the correct ones.
its also annoying to have it only working in TLS, would it possible to make it also works
without TLS ?
#### Reproduction
the fastest way it to connect to NGHTTP2 server with curl :
curl --http2-prior-knowledge -v -k https://server_ip:port/
#### Debugging Data
```
92(132) DEBUG: nghttp2 [nghttp2_server.c:717]: eventcb(): 172.22.0.1 connected
92(132) DEBUG: nghttp2 [nghttp2_server.c:507]: on_request_recv(): 172.22.0.1 GET /
92(132) DEBUG: nghttp2 [nghttp2_mod.c:522]: ksr_event_route(): executing
event_route[nghttp2:request] (1)
free(): invalid pointer
102(142) CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 7
102(142) DEBUG: <core> [core/tcp_main.c:3984]: handle_ser_child(): dead child 92,
pid 132 (shutting down?)
102(142) DEBUG: <core> [core/io_wait.h:599]: io_watch_del(): DBG: io_watch_del
(0x5b85ddc10ae0, 7, -1, 0x0) fd_no=130 called
0(40) ALERT: <core> [main.c:806]: handle_sigs(): child process 132 exited by a
signal 6
0(40) ALERT: <core> [main.c:810]: handle_sigs(): core was generated
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them
next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
https://pastebin.com/MQe4F5sv
```
### Additional Information
* **Kamailio Version** 5.8
* **Operating System**:
Ubuntu 24.04
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917(a)github.com>
15 Jul
15 Jul
9:55 a.m.
Please install debug packages (symbols) and check the generated core using GDB utility.
https://stackoverflow.com/questions/5115613/core-dump-file-analysis
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2227812082
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2227812082(a)github.com>
5:33 p.m.
Hi @sergey-safarov,
here is the core dump file.
[core.zip](https://github.com/user-attachments/files/16236064/core.zip).
Best Regards
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2228652067
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2228652067(a)github.com>
5:43 p.m.
I have just tested on Ubuntu 22.04 and all seems fine. What is the libnghttp2 version?
Anyhow, I have no Ubuntu 24.04 at hand that can test with.
The core file is not useful without the entire system where it was generated. You have to
extract the full backtrace with `gdb` from it and post it here -- if you are not familiar
with, search on the web, there are many articles on this topic.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2228674377
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2228674377(a)github.com>
6:19 p.m.
Hi @miconda,
I run Kamailio in container and using ubuntu:latest (24:04).
I use libnghttp2-14 package this contains the version 1.59.0, you used 22.04 this distro
use 1.43.0, which cloud explains why its running in your setup, I will try to force
downgrade it to 1.43 and retest.
I hope this one has more info i use the following command, I am not so familiar with GDB
:
gdb kamailio
set args -f /etc/kamailio_pcscf/kamailio_pcscf.cfg -P /kamailio_pcscf.pid -DD -E -e
start
bt full
bt -frame-arguments all full
continue
(reproduce the issue )
[core975.zip](https://github.com/user-attachments/files/16236563/core975.zip)
Best regards
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2228758442
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2228758442(a)github.com>
16 Jul
16 Jul
1:21 a.m.
Hi @miconda
I saw your last update on nghttp2, I tested it, still the same error, with my poor
experience i think that the module has problem freeing the memory, either it is not
allocated or already free, the problem is exactlly with the part :
"static int ksr_nghttp2_reply_header_clear(void)"
removing the changes done in this
[commit](https://github.com/kamailio/kamailio/commit/671b874a31fb10c6c9e9a24…
to the file "nghttp2_mod.c" fix the issue, so I would suggest to look at this
part of the code and try to find the issue with the free() function.
here is the code dump with the latest changes you made.
[core17152.zip](https://github.com/user-attachments/files/16240868/core17152…
Best regards
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2229534519
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2229534519(a)github.com>
12:34 p.m.
My previous commit was not related to this issue, it was a cleanup in a function used at
startup.
As I said in the comment above, get the backtrace with gdb from the generated core file,
do not run kamailio with gdb. After it crashes and the core file is generated, use gdb to
extract the backtrace from that core file. The backtrace can be taken only on the system
where the core file was generated, providing the core file to others is useless.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2230450339
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2230450339(a)github.com>
1:46 p.m.
Hi @miconda,
I hope this is the correct one now :
[coredump.txt](https://github.com/user-attachments/files/16247429/coredump.t…
Best Regards
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2230587534
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2230587534(a)github.com>
22 Jul
22 Jul
2 p.m.
In the comments is written that the Kamailio version is 5.8, but nghttp2 module is not
part of that series. Can you clarify about the version and how you use the module?
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2242679523
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2242679523(a)github.com>
2:43 p.m.
Hi @miconda,
I cloned Kamailio Master( 5.8 or?), it comes with the module already, then I put it inside
the docker setup from this [repo](https://github.com/herlesupreeth/docker_open5gs) the
plan is to try to build basic N5 Connectivity between P-CSCF and PCF see
[Here](https://github.com/herlesupreeth/docker_open5gs/discussions/333), the main use in
my setup is currently, to answer the SCP, as the P-CSCF sent a NotifyURI, after the user
detaches in my setup the SCP is sending to the URI a request with PATH /terminate, this
could be lead to more logic, but it still very experimental, as I am missing traces to see
N5 traffic in real Network.
I had the issue above, it its related to this part of the code
"ksr_nghttp2_reply_header_clear", I then removed the added code from commit
mentioned above
[(here)](https://github.com/kamailio/kamailio/commit/671b874a31fb10c6c9e9a246…)
and the module works now, looks like there is issue here with the use of free() function
here.
Best Regards
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2242754290
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2242754290(a)github.com>
3:12 p.m.
5.8 series definitely does not have nghttp2 module. Maybe you used the git master branch.
However, the coredump backtrace you attached two comments above points to:
```
0x00007fffef76f21f in ksr_nghttp2_send_reply (msg=0x1, rcode=0x555555eeafb0,
sbody=0x555555efc770) at nghttp2_mod.c:328
```
But there is a log message, not a free() -- latest version of code is not matching with
the coredump.
Therefore you have to take the very latest git master branch version as of now and try
again: reproduce and grab the coredump trace so one can match the backtrace info with the
C source code.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2242806815
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2242806815(a)github.com>
3:46 p.m.
Hi @miconda,
I cloned master and compiled the module again, here is the bt full from core dump, please
feel free to write me back if this one also not correct ´:
```
(gdb) bt full
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at
./nptl/pthread_kill.c:44
tid = <optimized out>
ret = 0
pd = <optimized out>
old_mask = {__val = {0}}
ret = <optimized out>
pd = <optimized out>
old_mask = <optimized out>
ret = <optimized out>
tid = <optimized out>
ret = <optimized out>
resultvar = <optimized out>
resultvar = <optimized out>
__arg3 = <optimized out>
__arg2 = <optimized out>
__arg1 = <optimized out>
_a3 = <optimized out>
_a2 = <optimized out>
_a1 = <optimized out>
__futex = <optimized out>
resultvar = <optimized out>
__arg3 = <optimized out>
__arg2 = <optimized out>
__arg1 = <optimized out>
_a3 = <optimized out>
_a2 = <optimized out>
_a1 = <optimized out>
__futex = <optimized out>
__private = <optimized out>
__oldval = <optimized out>
#1 __pthread_kill_internal (signo=6, threadid=<optimized out>) at
./nptl/pthread_kill.c:78
No locals.
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at
./nptl/pthread_kill.c:89
No locals.
#3 0x000074751a55e26e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
ret = <optimized out>
#4 0x000074751a5418ff in __GI_abort () at ./stdlib/abort.c:79
save_stage = 1
act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask =
{__val = {0 <repeats 16 times>}},
sa_flags = -545419008, sa_restorer = 0x0}
#5 0x000074751a5427b6 in __libc_message_impl (fmt=fmt@entry=0x74751a6e78d7
"%s\n") at ../sysdeps/posix/libc_fatal.c:132
ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7ffff8ddb750,
reg_save_area = 0x7ffff8ddb6e0}}
--Type <RET> for more, q to quit, c to continue without paging--
fd = 2
iov = {{iov_base = 0x74751a6e5672, iov_len = 23}, {iov_base = 0x74751a6e78d9,
iov_len = 1}, {iov_base = 0x0, iov_len = 0}, {
iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0,
iov_len = 0}, {iov_base = 0x0, iov_len = 0}}
iovcnt = <optimized out>
total = <optimized out>
cp = <optimized out>
#6 0x000074751a5c1fe5 in malloc_printerr (str=str@entry=0x74751a6e5672 "free():
invalid pointer") at ./malloc/malloc.c:5772
No locals.
#7 0x000074751a5c437c in _int_free (av=<optimized out>, p=<optimized out>,
have_lock=0) at ./malloc/malloc.c:4507
size = <optimized out>
fb = <optimized out>
#8 0x000074751a5c6d9e in __GI___libc_free (mem=0x747519ed33c0) at ./malloc/malloc.c:3398
ar_ptr = <optimized out>
p = 0x747519ed33b0
err = 0
#9 0x0000747511fcb21f in ksr_nghttp2_reply_header_clear () at nghttp2_mod.c:393
i = 0
p = 0x0
__func__ = "ksr_nghttp2_reply_header_clear"
#10 0x0000747511fcda01 in ksr_event_route () at nghttp2_mod.c:524
keng = 0x0
evname = {s = 0x747511fd629a "nghttp2:request", len = 15}
fmsg = 0x0
ctx = {rec_lev = -119686816, run_flags = 32767, last_retcode = 830566416, jmp_env
= {{__jmpbuf = {0, 140737368668656,
140737368668608, 128046302254286, 140737368668512, 128046170554336,
106352810711448, 106352810749968},
__mask_was_saved = 7, __saved_mask = {__val = {4, 128046303660832, 0, 0,
2600446128444772608, 106352810749968,
128046303660736, 4, 18446744073709551552, 106352810759408,
106352810746776, 140737368668672, 128046302259170,
140737368668756, 0, 140737368668704}}}}}
rtb = 32767
__func__ = "ksr_event_route"
#11 0x0000747511fd28ec in on_request_recv (session=0x60ba31815b60,
session_data=0x60ba316c2320, stream_data=0x60ba316c4f30)
at nghttp2_server.c:542
__func__ = "on_request_recv"
#12 0x0000747511fd29ba in on_frame_recv_callback (session=0x60ba31815b60,
frame=0x60ba31815e38, user_data=0x60ba316c2320)
at nghttp2_server.c:568
session_data = 0x60ba316c2320
stream_data = 0x60ba316c4f30
#13 0x0000747511fa2961 in ?? () from /lib/x86_64-linux-gnu/libnghttp2.so.14
No symbol table info available.
#14 0x0000747511fa92ba in nghttp2_session_mem_recv () from
/lib/x86_64-linux-gnu/libnghttp2.so.14
No symbol table info available.
#15 0x0000747511fd03e8 in session_recv (session_data=0x60ba316c2320) at
nghttp2_server.c:263
--Type <RET> for more, q to quit, c to continue without paging--
readlen = 1
input = 0x60ba317f0f40
datalen = 59
data = 0x60ba3180a440 ""
__func__ = "session_recv"
#16 0x0000747511fd3ee0 in readcb (bev=0x60ba317f0ce0, ptr=0x60ba316c2320) at
nghttp2_server.c:676
session_data = 0x60ba316c2320
#17 0x0000747511f5c8b9 in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7
No symbol table info available.
#18 0x0000747511f67783 in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7
No symbol table info available.
#19 0x0000747511f6948f in event_base_loop () from /lib/x86_64-linux-gnu/libevent-2.1.so.7
No symbol table info available.
#20 0x0000747511fd5d44 in nghttp2_server_run () at nghttp2_server.c:841
ssl_ctx = 0x60ba317fe0f0
app_ctx = {ssl_ctx = 0x60ba317fe0f0, evbase = 0x60ba317f0590}
evbase = 0x60ba317f0590
act = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask =
{__val = {0 <repeats 16 times>}}, sa_flags = 0,
sa_restorer = 0x0}
#21 0x0000747511fc8c93 in child_init (rank=0) at nghttp2_mod.c:165
pid = 0
__func__ = "child_init"
#22 0x000060ba308c1670 in init_mod_child (m=0x747519ec0710, rank=0) at
core/sr_module.c:920
ret = 0
__func__ = "init_mod_child"
#23 0x000060ba308c124f in init_mod_child (m=0x747519ec0c70, rank=0) at
core/sr_module.c:912
ret = 0
__func__ = "init_mod_child"
#24 0x000060ba308c124f in init_mod_child (m=0x747519ec11b0, rank=0) at
core/sr_module.c:912
ret = 32767
__func__ = "init_mod_child"
#25 0x000060ba308c124f in init_mod_child (m=0x747519ec1fe0, rank=0) at
core/sr_module.c:912
ret = 0
__func__ = "init_mod_child"
#26 0x000060ba308c124f in init_mod_child (m=0x747519ec2870, rank=0) at
core/sr_module.c:912
ret = 1
__func__ = "init_mod_child"
#27 0x000060ba308c1fd3 in init_child (rank=0) at core/sr_module.c:999
ret = 29813
type = 0x60ba30b3400d "PROC_MAIN"
__func__ = "init_child"
#28 0x000060ba3066519c in main_loop () at main.c:1958
--Type <RET> for more, q to quit, c to continue without paging--
i = 4
pid = 140
si = 0x0
si_desc = "udp receiver child=3
sock=172.22.0.21:6109\000\377`\000\000\000\003\000\000\000i\000\000\000\000\221}\337\371\243\026$8\227\2740\272`\000\000}@\2560\272`\000\000i\000\000\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\200",
'\000' <repeats 13 times>, "
\307\335\370\377\177\000\000W\223\2030\272`\000"
nrprocs = 4
woneinit = 1
__func__ = "main_loop"
#29 0x000060ba306702b3 in main (argc=8, argv=0x7ffff8ddccc8) at main.c:3280
cfg_stream = 0x60ba31505380
c = -1
r = 0
tmp = 0x74751a81bc1a ""
tmp_len = 29813
port = 5060
proto = 0
aproto = 0
ahost = 0x0
aport = 0
options = 0x60ba30ae7290
":f:cm:M:dVIhEeb:B:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 3676012720
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 2
n_lst = 0x74751a5b865c <__pthread_tunables_init+92>
p = 0x74751a81b590 ""
st = {st_dev = 1048630, st_ino = 918430, st_nlink = 1, st_mode = 16832, st_uid =
0, st_gid = 0, __pad0 = 0, st_rdev = 0,
st_size = 4096, st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec =
1721243358, tv_nsec = 578651101}, st_mtim = {
tv_sec = 1721652006, tv_nsec = 624143935}, st_ctim = {tv_sec = 1721652006,
tv_nsec = 624143935}, __glibc_reserved = {0, 0, 0}}
l1 = 0
tbuf = "\000\000\000\000\000\000\000\000\002", '\000'
<repeats 11 times>,
"a\001\000\000\3777@\025\000\000\000\000(R\337\370\377\177\000\000\000\000\000\000\000\000\000\000\001",
'\000' <repeats 119 times>,
"@`b0\272`\000\0008\000\000\000\000\000\000\000\r\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000"...
option_index = 0
long_options = {{name = 0x60ba30ae982f "help", has_arg = 0, flag = 0x0,
val = 104}, {name = 0x60ba30ae4531 "version", has_arg = 0,
flag = 0x0, val = 118}, {name = 0x60ba30ae9834 "alias", has_arg = 1,
flag = 0x0, val = 1024}, {name = 0x60ba30ae983a "subst",
has_arg = 1, flag = 0x0, val = 1025}, {name = 0x60ba30ae9840
"substdef", has_arg = 1, flag = 0x0, val = 1026}, {
name = 0x60ba30ae9849 "substdefs", has_arg = 1, flag = 0x0, val =
1027}, {name = 0x60ba30ae9853 "server-id", has_arg = 1,
flag = 0x0, val = 1028}, {name = 0x60ba30ae985d "loadmodule",
has_arg = 1, flag = 0x0, val = 1029}, {
--Type <RET> for more, q to quit, c to continue without paging--
name = 0x60ba30ae9868 "modparam", has_arg = 1, flag = 0x0, val =
1030}, {name = 0x60ba30ae9871 "log-engine", has_arg = 1,
flag = 0x0, val = 1031}, {name = 0x60ba30ae987c "debug", has_arg =
1, flag = 0x0, val = 1032}, {
name = 0x60ba30ae9882 "cfg-print", has_arg = 0, flag = 0x0, val =
1033}, {name = 0x60ba30ae988c "atexit", has_arg = 1,
flag = 0x0, val = 1034}, {name = 0x60ba30ae9893 "all-errors",
has_arg = 0, flag = 0x0, val = 1035}, {name = 0x0, has_arg = 0,
flag = 0x0, val = 0}}
__func__ = "main"
(gdb)
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2242871768
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2242871768(a)github.com>
4:12 p.m.
Hi @miconda
just noticed that you commit new change to the module, I tried it and it worked! many
thanks for the help.
Best Regards
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2242928166
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2242928166(a)github.com>
4:28 p.m.
Closed #3917 as completed.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#event-13601366680
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issue/3917/issue_event/13601366680(a)github.com>
4:28 p.m.
Good, thanks for testing and reporting back.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3917#issuecomment-2242964017
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3917/2242964017(a)github.com>
109
days inactive
118
days old
14 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
NUCLEAR-WAR
-
sergey-safarov