[kamailio/kamailio] nghttp2 module causing Kamailio P-CSCF to crash on connect (Issue #3917)
### Description I am doing some experiments with NGHTTP2 Module to make use of it in some scenarios, where Kamailio receives HTTP2 requests and has t react on them, unfortunately the module is causing Kamailio to crash upon receiving a request and need to be restarted. Also the Docs for the module need some correction : https://www.kamailio.org/docs/modules/5.9.x/modules/nghttp2.html Missing required Parameters : modparam("nghttp2", "tls_private_key", "key_in_PEM_Format") modparam("nghttp2", "tls_public_key", "certificate_in_PEM_Format")
also $nghttp2(url) is wrong, $nghttp2(path) or $nghttp2(pathfull) is the correct ones.
its also annoying to have it only working in TLS, would it possible to make it also works without TLS ?
#### Reproduction
the fastest way it to connect to NGHTTP2 server with curl :
curl --http2-prior-knowledge -v -k https://server_ip:port/
#### Debugging Data
``` 92(132) DEBUG: nghttp2 [nghttp2_server.c:717]: eventcb(): 172.22.0.1 connected 92(132) DEBUG: nghttp2 [nghttp2_server.c:507]: on_request_recv(): 172.22.0.1 GET / 92(132) DEBUG: nghttp2 [nghttp2_mod.c:522]: ksr_event_route(): executing event_route[nghttp2:request] (1) free(): invalid pointer 102(142) CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 7 102(142) DEBUG: <core> [core/tcp_main.c:3984]: handle_ser_child(): dead child 92, pid 132 (shutting down?) 102(142) DEBUG: <core> [core/io_wait.h:599]: io_watch_del(): DBG: io_watch_del (0x5b85ddc10ae0, 7, -1, 0x0) fd_no=130 called 0(40) ALERT: <core> [main.c:806]: handle_sigs(): child process 132 exited by a signal 6 0(40) ALERT: <core> [main.c:810]: handle_sigs(): core was generated
```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` https://pastebin.com/MQe4F5sv
```
### Additional Information
* **Kamailio Version** 5.8
* **Operating System**:
Ubuntu 24.04
Please install debug packages (symbols) and check the generated core using GDB utility. https://stackoverflow.com/questions/5115613/core-dump-file-analysis
Hi @sergey-safarov,
here is the core dump file.
[core.zip](https://github.com/user-attachments/files/16236064/core.zip).
Best Regards
I have just tested on Ubuntu 22.04 and all seems fine. What is the libnghttp2 version? Anyhow, I have no Ubuntu 24.04 at hand that can test with.
The core file is not useful without the entire system where it was generated. You have to extract the full backtrace with `gdb` from it and post it here -- if you are not familiar with, search on the web, there are many articles on this topic.
Hi @miconda,
I run Kamailio in container and using ubuntu:latest (24:04).
I use libnghttp2-14 package this contains the version 1.59.0, you used 22.04 this distro use 1.43.0, which cloud explains why its running in your setup, I will try to force downgrade it to 1.43 and retest.
I hope this one has more info i use the following command, I am not so familiar with GDB :
gdb kamailio set args -f /etc/kamailio_pcscf/kamailio_pcscf.cfg -P /kamailio_pcscf.pid -DD -E -e start bt full bt -frame-arguments all full continue
(reproduce the issue )
[core975.zip](https://github.com/user-attachments/files/16236563/core975.zip)
Best regards
Hi @miconda
I saw your last update on nghttp2, I tested it, still the same error, with my poor experience i think that the module has problem freeing the memory, either it is not allocated or already free, the problem is exactlly with the part : "static int ksr_nghttp2_reply_header_clear(void)" removing the changes done in this [commit](https://github.com/kamailio/kamailio/commit/671b874a31fb10c6c9e9a246358928f2...) to the file "nghttp2_mod.c" fix the issue, so I would suggest to look at this part of the code and try to find the issue with the free() function.
here is the code dump with the latest changes you made. [core17152.zip](https://github.com/user-attachments/files/16240868/core17152.zip)
Best regards
My previous commit was not related to this issue, it was a cleanup in a function used at startup.
As I said in the comment above, get the backtrace with gdb from the generated core file, do not run kamailio with gdb. After it crashes and the core file is generated, use gdb to extract the backtrace from that core file. The backtrace can be taken only on the system where the core file was generated, providing the core file to others is useless.
Hi @miconda,
I hope this is the correct one now : [coredump.txt](https://github.com/user-attachments/files/16247429/coredump.txt)
Best Regards
In the comments is written that the Kamailio version is 5.8, but nghttp2 module is not part of that series. Can you clarify about the version and how you use the module?
Hi @miconda,
I cloned Kamailio Master( 5.8 or?), it comes with the module already, then I put it inside the docker setup from this [repo](https://github.com/herlesupreeth/docker_open5gs) the plan is to try to build basic N5 Connectivity between P-CSCF and PCF see [Here](https://github.com/herlesupreeth/docker_open5gs/discussions/333), the main use in my setup is currently, to answer the SCP, as the P-CSCF sent a NotifyURI, after the user detaches in my setup the SCP is sending to the URI a request with PATH /terminate, this could be lead to more logic, but it still very experimental, as I am missing traces to see N5 traffic in real Network.
I had the issue above, it its related to this part of the code "ksr_nghttp2_reply_header_clear", I then removed the added code from commit mentioned above [(here)](https://github.com/kamailio/kamailio/commit/671b874a31fb10c6c9e9a246358928f2...) and the module works now, looks like there is issue here with the use of free() function here.
Best Regards
5.8 series definitely does not have nghttp2 module. Maybe you used the git master branch.
However, the coredump backtrace you attached two comments above points to:
``` 0x00007fffef76f21f in ksr_nghttp2_send_reply (msg=0x1, rcode=0x555555eeafb0, sbody=0x555555efc770) at nghttp2_mod.c:328 ```
But there is a log message, not a free() -- latest version of code is not matching with the coredump.
Therefore you have to take the very latest git master branch version as of now and try again: reproduce and grab the coredump trace so one can match the backtrace info with the C source code.
Hi @miconda,
I cloned master and compiled the module again, here is the bt full from core dump, please feel free to write me back if this one also not correct ´:
```
(gdb) bt full #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44 tid = <optimized out> ret = 0 pd = <optimized out> old_mask = {__val = {0}} ret = <optimized out> pd = <optimized out> old_mask = <optimized out> ret = <optimized out> tid = <optimized out> ret = <optimized out> resultvar = <optimized out> resultvar = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> __futex = <optimized out> resultvar = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> __futex = <optimized out> __private = <optimized out> __oldval = <optimized out> #1 __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78 No locals. #2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 No locals. #3 0x000074751a55e26e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 ret = <optimized out> #4 0x000074751a5418ff in __GI_abort () at ./stdlib/abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = -545419008, sa_restorer = 0x0} #5 0x000074751a5427b6 in __libc_message_impl (fmt=fmt@entry=0x74751a6e78d7 "%s\n") at ../sysdeps/posix/libc_fatal.c:132 ap = {{gp_offset = 16, fp_offset = 0, overflow_arg_area = 0x7ffff8ddb750, reg_save_area = 0x7ffff8ddb6e0}} --Type <RET> for more, q to quit, c to continue without paging-- fd = 2 iov = {{iov_base = 0x74751a6e5672, iov_len = 23}, {iov_base = 0x74751a6e78d9, iov_len = 1}, {iov_base = 0x0, iov_len = 0}, { iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}} iovcnt = <optimized out> total = <optimized out> cp = <optimized out> #6 0x000074751a5c1fe5 in malloc_printerr (str=str@entry=0x74751a6e5672 "free(): invalid pointer") at ./malloc/malloc.c:5772 No locals. #7 0x000074751a5c437c in _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at ./malloc/malloc.c:4507 size = <optimized out> fb = <optimized out> #8 0x000074751a5c6d9e in __GI___libc_free (mem=0x747519ed33c0) at ./malloc/malloc.c:3398 ar_ptr = <optimized out> p = 0x747519ed33b0 err = 0 #9 0x0000747511fcb21f in ksr_nghttp2_reply_header_clear () at nghttp2_mod.c:393 i = 0 p = 0x0 __func__ = "ksr_nghttp2_reply_header_clear" #10 0x0000747511fcda01 in ksr_event_route () at nghttp2_mod.c:524 keng = 0x0 evname = {s = 0x747511fd629a "nghttp2:request", len = 15} fmsg = 0x0 ctx = {rec_lev = -119686816, run_flags = 32767, last_retcode = 830566416, jmp_env = {{__jmpbuf = {0, 140737368668656, 140737368668608, 128046302254286, 140737368668512, 128046170554336, 106352810711448, 106352810749968}, __mask_was_saved = 7, __saved_mask = {__val = {4, 128046303660832, 0, 0, 2600446128444772608, 106352810749968, 128046303660736, 4, 18446744073709551552, 106352810759408, 106352810746776, 140737368668672, 128046302259170, 140737368668756, 0, 140737368668704}}}}} rtb = 32767 __func__ = "ksr_event_route" #11 0x0000747511fd28ec in on_request_recv (session=0x60ba31815b60, session_data=0x60ba316c2320, stream_data=0x60ba316c4f30) at nghttp2_server.c:542 __func__ = "on_request_recv" #12 0x0000747511fd29ba in on_frame_recv_callback (session=0x60ba31815b60, frame=0x60ba31815e38, user_data=0x60ba316c2320) at nghttp2_server.c:568 session_data = 0x60ba316c2320 stream_data = 0x60ba316c4f30 #13 0x0000747511fa2961 in ?? () from /lib/x86_64-linux-gnu/libnghttp2.so.14 No symbol table info available. #14 0x0000747511fa92ba in nghttp2_session_mem_recv () from /lib/x86_64-linux-gnu/libnghttp2.so.14 No symbol table info available. #15 0x0000747511fd03e8 in session_recv (session_data=0x60ba316c2320) at nghttp2_server.c:263 --Type <RET> for more, q to quit, c to continue without paging-- readlen = 1 input = 0x60ba317f0f40 datalen = 59 data = 0x60ba3180a440 "" __func__ = "session_recv" #16 0x0000747511fd3ee0 in readcb (bev=0x60ba317f0ce0, ptr=0x60ba316c2320) at nghttp2_server.c:676 session_data = 0x60ba316c2320 #17 0x0000747511f5c8b9 in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7 No symbol table info available. #18 0x0000747511f67783 in ?? () from /lib/x86_64-linux-gnu/libevent-2.1.so.7 No symbol table info available. #19 0x0000747511f6948f in event_base_loop () from /lib/x86_64-linux-gnu/libevent-2.1.so.7 No symbol table info available. #20 0x0000747511fd5d44 in nghttp2_server_run () at nghttp2_server.c:841 ssl_ctx = 0x60ba317fe0f0 app_ctx = {ssl_ctx = 0x60ba317fe0f0, evbase = 0x60ba317f0590} evbase = 0x60ba317f0590 act = {__sigaction_handler = {sa_handler = 0x1, sa_sigaction = 0x1}, sa_mask = {__val = {0 <repeats 16 times>}}, sa_flags = 0, sa_restorer = 0x0} #21 0x0000747511fc8c93 in child_init (rank=0) at nghttp2_mod.c:165 pid = 0 __func__ = "child_init" #22 0x000060ba308c1670 in init_mod_child (m=0x747519ec0710, rank=0) at core/sr_module.c:920 ret = 0 __func__ = "init_mod_child" #23 0x000060ba308c124f in init_mod_child (m=0x747519ec0c70, rank=0) at core/sr_module.c:912 ret = 0 __func__ = "init_mod_child" #24 0x000060ba308c124f in init_mod_child (m=0x747519ec11b0, rank=0) at core/sr_module.c:912 ret = 32767 __func__ = "init_mod_child" #25 0x000060ba308c124f in init_mod_child (m=0x747519ec1fe0, rank=0) at core/sr_module.c:912 ret = 0 __func__ = "init_mod_child" #26 0x000060ba308c124f in init_mod_child (m=0x747519ec2870, rank=0) at core/sr_module.c:912 ret = 1 __func__ = "init_mod_child" #27 0x000060ba308c1fd3 in init_child (rank=0) at core/sr_module.c:999 ret = 29813 type = 0x60ba30b3400d "PROC_MAIN" __func__ = "init_child" #28 0x000060ba3066519c in main_loop () at main.c:1958 --Type <RET> for more, q to quit, c to continue without paging-- i = 4 pid = 140 si = 0x0 si_desc = "udp receiver child=3 sock=172.22.0.21:6109\000\377`\000\000\000\003\000\000\000i\000\000\000\000\221}\337\371\243\026$8\227\2740\272`\000\000}@\2560\272`\000\000i\000\000\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\200", '\000' <repeats 13 times>, " \307\335\370\377\177\000\000W\223\2030\272`\000" nrprocs = 4 woneinit = 1 __func__ = "main_loop" #29 0x000060ba306702b3 in main (argc=8, argv=0x7ffff8ddccc8) at main.c:3280 cfg_stream = 0x60ba31505380 c = -1 r = 0 tmp = 0x74751a81bc1a "" tmp_len = 29813 port = 5060 proto = 0 aproto = 0 ahost = 0x0 aport = 0 options = 0x60ba30ae7290 ":f:cm:M:dVIhEeb:B:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3676012720 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0x74751a5b865c <__pthread_tunables_init+92> p = 0x74751a81b590 "" st = {st_dev = 1048630, st_ino = 918430, st_nlink = 1, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 4096, st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec = 1721243358, tv_nsec = 578651101}, st_mtim = { tv_sec = 1721652006, tv_nsec = 624143935}, st_ctim = {tv_sec = 1721652006, tv_nsec = 624143935}, __glibc_reserved = {0, 0, 0}} l1 = 0 tbuf = "\000\000\000\000\000\000\000\000\002", '\000' <repeats 11 times>, "a\001\000\000\3777@\025\000\000\000\000(R\337\370\377\177\000\000\000\000\000\000\000\000\000\000\001", '\000' <repeats 119 times>, "@`b0\272`\000\0008\000\000\000\000\000\000\000\r\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000"... option_index = 0 long_options = {{name = 0x60ba30ae982f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x60ba30ae4531 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x60ba30ae9834 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x60ba30ae983a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x60ba30ae9840 "substdef", has_arg = 1, flag = 0x0, val = 1026}, { name = 0x60ba30ae9849 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x60ba30ae9853 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x60ba30ae985d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, { --Type <RET> for more, q to quit, c to continue without paging-- name = 0x60ba30ae9868 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x60ba30ae9871 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x60ba30ae987c "debug", has_arg = 1, flag = 0x0, val = 1032}, { name = 0x60ba30ae9882 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x60ba30ae988c "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x60ba30ae9893 "all-errors", has_arg = 0, flag = 0x0, val = 1035}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" (gdb) ```
Hi @miconda
just noticed that you commit new change to the module, I tried it and it worked! many thanks for the help.
Best Regards
Closed #3917 as completed.
Good, thanks for testing and reporting back.
-
Daniel-Constantin Mierla -
NUCLEAR-WAR
-
sergey-safarov