[kamailio/kamailio] Kamailio 5 coredump 11 (SEGV) upcon receiving MESSAGE via jsonrpcs (#1030)
10 Mar
2017
10 Mar
'17
9:12 p.m.
Using python-requests to generate a MESSAGE injected via jsonrpcs, Kamailio (built from
the 5.0 branch @ 49bafac) dumps core. This did not occur in 4.4.5. See attached gdb
output.
[kamailio-coredump-jsonrpcs-message.txt](https://github.com/kamailio/kamaili…
```
"""
Generate a SIP MESSAGE via Kamailio's jsonrpcs module
"""
headers = ("From: <{0}>\r\n"
"To: <{1}>\r\n"
"Max-Forwards: 1\r\n"
"X-Kamailio-Message: noreply\r\n"
"Content-Type: text/plain; "
"charset=utf-8\r\n").format(self.sender, self.recipient)
content = {
'id': 1,
'jsonrpc': '2.0',
'method': 'tm.t_uac_start',
'params': [
'MESSAGE',
self.recipient,
SIP_NEXTHOP,
'.',
headers,
self.body
]
}
# http://docs.python-requests.org/en/latest/user/quickstart/
try:
r = requests.post(KAMAILIO_RPC_URL, allow_redirects=False,
json=content, timeout=5)
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030
11 Mar
11 Mar
3:32 a.m.
Can this be reproduced? If you get other corefile, are the details of frame 0 missing
again? Without it, no details of the code it effective crashed are shown.
From this corefile, can you get the output of:
```
frame 0
p *ctx
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-285852714
8:43 a.m.
This is reproducable every time. Here is the output you requested:
```
(gdb) frame 1
#1 0x00007f1426c9f5d1 in jsonrpc_send (ctx=0x7f1426ec88e0 <_jsonrpc_ctx>) at
jsonrpcs_mod.c:395
395 xhttp_api.reply(ctx->msg, ctx->http_code,
&ctx->http_text,
(gdb) p *ctx
$1 = {msg = 0x7ffdd76deda0, msg_shm_block_size = 0, method = 0x23fe850
"tm.t_uac_start", flags = 0, jreq = 0x2402810, req_node = 0x0, jrpl = 0x2402f40,
rpl_node = 0x0, reply_sent = 1, error_code = 0, http_code = 200, http_text = {
s = 0x7f1426cbec14 "OK", len = 2}, transport = 1}
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-285867501
13 Mar
13 Mar
10:37 a.m.
Can you also get the content of msg field, respectively:
```
p *ctx->msg
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-286126055
7:11 p.m.
See attached. I have replaced my domain with "example.com" and my username with
"user".
[kamailio-coredump-jsonrpcs-message_2.txt](https://github.com/kamailio/kamai…
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-286272391
14 Mar
14 Mar
9:16 a.m.
Couldn't spot anything relevant wrong for sending the response.
Can you grab all the log messages with debug=3 in kamailio.cfg? Hopefully it will show
more leads about what frame 0 can be.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-286417462
6:18 p.m.
I had to send another message to do this, but here it is:
[kamailio-coredump-jsonrpcs-message-script-debug.txt](https://github.com/kam…
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-286578930
6:21 p.m.
The strange thing is, the MESSAGE is received and transmitted to the recipient, then
Kamailio fails, apparently on the reply.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-286579620
16 Mar
16 Mar
6:05 a.m.
Can you try with latest branch 5.0 and grab again the log messages with the debug=3 in
case the issue is still there.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287010991
5:13 p.m.
I see a97701575863993f8a19f9dd611156b37ffca247 in the master branch, but not the 5.0
branch. Do you want me to include that in the testing as well?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287192751
5:31 p.m.
@miconda attached is the debug log using the 5.0 branch at
f5889fcc9e42cae01c58357a622d2270f994b8d7.
[kamailio.script.debug.3.txt](https://github.com/kamailio/kamailio/files/849…
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287197443
5:50 p.m.
The log messages are not from syslog, so they are missing date/time and which PID printed
them. Can you redo the test and grab via syslog?
You can also test with a977015 .
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287201871
6 p.m.
The new log is from a rebuild of the 5.0 branch as before with
0350885c076fa691de6106eb821ec5890572cee6, fa50c85a9e61dd6b91ce88f3f74c433af5d414e2, and
a97701575863993f8a19f9dd611156b37ffca247 cherry-picked onto it. Still dumps.
[kamailio.script.stateless.debug.3.txt](https://github.com/kamailio/kamailio…
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287204075
6:04 p.m.
Just in case, these are some of the script settings:
```
# ----- jsonrpcs params -----
modparam("jsonrpcs", "pretty_format", 1)
/* set the path to RPC fifo control file */
# modparam("jsonrpcs", "fifo_name",
"/run/kamailio/kamailio_rpc.fifo")
/* set the path to RPC unix socket control file */
# modparam("jsonrpcs", "dgram_socket",
"/run/kamailio/kamailio_rpc.sock")
# ----- ctl params -----
/* set the path to RPC unix socket control file */
# modparam("ctl", "binrpc",
"unix:/run/kamailio/kamailio_ctl")
```
Probably unrelated, but a few xhttp-related compiler warnings:
```
make[2]: Entering directory
'/builddir/build/BUILD/kamailio-5.0.0/src/modules/xhttp_rpc'
Makefile.defs defs skipped
Makefile.defs defs skipped
gcc -fPIC -DPIC -g -funroll-loops -Wcast-align -m64 -minline-all-stringops -falign-loops
-ftree-vectorize -fno-strict-overflow -Wall -DNAME='"kamailio"'
-DVERSION='"5.0.0"' -DARCH='"x86_64"'
-DOS='linux_' -DOS_QUOTED='"linux"' -DCOMPILER='"gcc
6.3.1"' -D__CPU_x86_64 -D__OS_linux -DSER_VER=5000000
-DCFG_DIR='"/etc/kamailio/"'
-DRUN_DIR='"/var/run/kamailio/"' -DPKG_MALLOC -DSHM_MEM -DSHM_MMAP
-DDNS_IP_HACK -DUSE_MCAST -DUSE_TCP -DDISABLE_NAGLE -DHAVE_RESOLV_RES -DUSE_DNS_CACHE
-DUSE_DNS_FAILOVER -DUSE_DST_BLACKLIST -DUSE_NAPTR -DWITH_XAVP -DMEM_JOIN_FREE -DF_MALLOC
-DQ_MALLOC -DTLSF_MALLOC -DDBG_SR_MEMORY -DUSE_TLS -DTLS_HOOKS -DUSE_CORE_STATS
-DSTATISTICS -DMALLOC_STATS -DWITH_AS_SUPPORT -DUSE_SCTP -DFAST_LOCK -DADAPTIVE_WAIT
-DADAPTIVE_WAIT_LOOPS=1024 -DCC_GCC_LIKE_ASM -DHAVE_GETHOSTBYNAME2 -DHAVE_UNION_SEMUN
-DHAVE_SCHED_YIELD -DHAVE_MSG_NOSIGNAL -DHAVE_MSGHDR_MSG_CONTROL -DHAVE_ALLOCA_H
-DHAVE_TIMEGM -DHAVE_SCHED_SETSCHEDULER -DHAVE_IP_MREQN -DHAVE_EPOLL -DHAVE_SIGIO_RT
-DSIGINFO64_WORKARROUND -DUSE_FUTEX -DHAVE_SELECT -DKAMAILIO_MOD_INTERFACE
-DMOD_NAME='"xhttp_rpc"' -c xhttp_rpc.c -o xhttp_rpc.o -MMD -MP
gcc -fPIC -DPIC -g -funroll-loops -Wcast-align -m64 -minline-all-stringops -falign-loops
-ftree-vectorize -fno-strict-overflow -Wall -DNAME='"kamailio"'
-DVERSION='"5.0.0"' -DARCH='"x86_64"'
-DOS='linux_' -DOS_QUOTED='"linux"' -DCOMPILER='"gcc
6.3.1"' -D__CPU_x86_64 -D__OS_linux -DSER_VER=5000000
-DCFG_DIR='"/etc/kamailio/"'
-DRUN_DIR='"/var/run/kamailio/"' -DPKG_MALLOC -DSHM_MEM -DSHM_MMAP
-DDNS_IP_HACK -DUSE_MCAST -DUSE_TCP -DDISABLE_NAGLE -DHAVE_RESOLV_RES -DUSE_DNS_CACHE
-DUSE_DNS_FAILOVER -DUSE_DST_BLACKLIST -DUSE_NAPTR -DWITH_XAVP -DMEM_JOIN_FREE -DF_MALLOC
-DQ_MALLOC -DTLSF_MALLOC -DDBG_SR_MEMORY -DUSE_TLS -DTLS_HOOKS -DUSE_CORE_STATS
-DSTATISTICS -DMALLOC_STATS -DWITH_AS_SUPPORT -DUSE_SCTP -DFAST_LOCK -DADAPTIVE_WAIT
-DADAPTIVE_WAIT_LOOPS=1024 -DCC_GCC_LIKE_ASM -DHAVE_GETHOSTBYNAME2 -DHAVE_UNION_SEMUN
-DHAVE_SCHED_YIELD -DHAVE_MSG_NOSIGNAL -DHAVE_MSGHDR_MSG_CONTROL -DHAVE_ALLOCA_H
-DHAVE_TIMEGM -DHAVE_SCHED_SETSCHEDULER -DHAVE_IP_MREQN -DHAVE_EPOLL -DHAVE_SIGIO_RT
-DSIGINFO64_WORKARROUND -DUSE_FUTEX -DHAVE_SELECT -DKAMAILIO_MOD_INTERFACE
-DMOD_NAME='"xhttp_rpc"' -c xhttp_rpc_fnc.c -o xhttp_rpc_fnc.o -MMD -MP
xhttp_rpc_fnc.c:188:18: warning: 'XHTTP_RPC_Post_1a' defined but not used
[-Wunused-const-variable=]
static const str XHTTP_RPC_Post_1a = str_init("\n"\
^~~~~~~~~~~~~~~~~
xhttp_rpc_fnc.c:176:18: warning: 'XHTTP_RPC_ATTR_VAL_SEPARATOR' defined but not
used [-Wunused-const-variable=]
static const str XHTTP_RPC_ATTR_VAL_SEPARATOR = str_init("=");
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
xhttp_rpc_fnc.c:175:18: warning: 'XHTTP_RPC_ATTR_SEPARATOR' defined but not used
[-Wunused-const-variable=]
static const str XHTTP_RPC_ATTR_SEPARATOR = str_init(" ");
^~~~~~~~~~~~~~~~~~~~~~~~
xhttp_rpc_fnc.c:174:18: warning: 'XHTTP_RPC_NODE_SEPARATOR' defined but not used
[-Wunused-const-variable=]
static const str XHTTP_RPC_NODE_SEPARATOR = str_init(":: ");
^~~~~~~~~~~~~~~~~~~~~~~~
gcc -shared -m64 -Wl,-O2 -Wl,-E xhttp_rpc.o xhttp_rpc_fnc.o -o xhttp_rpc.so
make[2]: Leaving directory
'/builddir/build/BUILD/kamailio-5.0.0/src/modules/xhttp_rpc'
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287205186
18 Mar
18 Mar
1:12 a.m.
After applying f81c128, no coredump, but I get `jsonrpcs [jsonrpcs_mod.c:1174]:
jsonrpc_dispatch(): jsonrpc over http not initialized - check transport param`
FYI, The request is sent in via 127.0.0.1:5060 TCP, with nexthop set as 127.0.0.1:5060 so
the request will route through the script and eventually gets looked up by
lookup_branches--finding the contact which is a SIP/TLS endpoint.
Attached log.
[kamailio-coredump-jsonrpcs-message_3.txt](https://github.com/kamailio/kamai…
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287517348
1:23 a.m.
Well, I'll be damned. Your log message gave me the clue. I was simply missing
`modparam("jsonrpcs", "transport", 1)`. I must have missed that in
the upgrade.
Maybe that parameter should be added to the default kamailio.cfg since the `Default value
is '6' (fifo and datagram transport).`
Sorry @miconda. But I guess it shouldn't segfault either ;) Thanks again.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287517767
1:23 a.m.
Closed #1030.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#event-1005402855
3:04 a.m.
It was a good catch after all, indeed it should not segfault and now should be fixed.
Yesterday I didn't have time to add here the comments about it, but as jsonrpcs
replaced the mi_fifo, its default transports were changed in 5.0, given that xhttp is not
loaded by default.
I also added a note in the upgrade guide from 4.4 to 5.0. Thanks for assisting with
troubleshooting.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1030#issuecomment-287521722
2657
days inactive
2664
days old
17 comments
2 participants
participants (2)
-
Anthony Messina -
Daniel-Constantin Mierla