27 Jul
2022
27 Jul
'22
11:04 a.m.
### Description
There's a regression after enabling Address Space Layout Randomization (ASLR).
Kamailio causes SIGSEGV while loading `app_lua` (or other KEMI Interpreters probably).
### Troubleshooting
#### Reproduction
The regression has appeared after update FreeBSD kernel to
https://github.com/freebsd/freebsd-src/commit/10192e77cfacd1f27601882af6188…
Kamailio falls after `loadmodule "app_lua.so"` in `kamailio.cfg`.
#### Debugging Data
```
root@server:/usr/jails/containers/kamailio/var/coredump/986# jexec kamailio lldb -c
/var/coredump/986/kamailio.13521.core -- /usr/local/sbin/kamailio
(lldb) target create "/usr/local/sbin/kamailio" --core
"/var/coredump/986/kamailio.13521.core"
Core file '/var/coredump/986/kamailio.13521.core' (x86_64) was loaded.
(lldb) bt all
* thread #1, name = 'kamailio', stop reason = signal SIGSEGV
* frame #0: 0x0000000825b61350 libc.so.7`strncmp(s1=<unavailable>,
s2=<unavailable>, n=<unavailable>) at strncmp.c:47:7
frame #1: 0x00000000003fc394 kamailio`sr_kemi_modules_add(klist=0x0000000861afb410) at
kemi.c:3392:8
frame #2: 0x0000000861ae6154
app_lua.so`mod_register(path="/usr/local/lib/kamailio/modules/app_lua.so",
dlflags=0x0000000821805b48, p1=0x0000000000000000, p2=0x0000000000000000) at
app_lua_mod.c:605:2
frame #3: 0x00000000005e4b30 kamailio`load_module(mod_path="app_lua.so") at
sr_module.c:592:7
frame #4: 0x0000000000885936 kamailio`yyparse at cfg.y:1965:8
frame #5: 0x00000000002ff384 kamailio`main(argc=7, argv=0x000000082180a848) at
main.c:2506:6
frame #6: 0x00000000002ddd90 kamailio`_start(ap=<unavailable>,
cleanup=<unavailable>) at crt1_c.c:75:7
```
`_sr_kemi_modules[].mname.s` for `app_lua` points to incorrect data after second call
`mod_register()(a)app_lua_mod.c` from `load_module()(a)sr_module.c` (reloading the module for
setting correct `dlflags`).
#### Log Messages
```
08:35:53.689647 DEBUG: <core> [core/cfg.y:1964]: yyparse(): loading module kemix.so
08:35:53.689684 DEBUG: <core> [core/sr_module.c:516]: ksr_locate_module(): found
module to load </usr/local/lib/kamailio/modules/kemix.so>
08:35:53.689698 DEBUG: <core> [core/sr_module.c:566]: load_module(): trying to load
</usr/local/lib/kamailio/modules/kemix.so>
08:35:53.689796 DEBUG: <core> [core/kemi.c:3398]: sr_kemi_modules_add(): adding
module: kx
08:35:53.689847 DEBUG: <core> [core/cfg.lex:2039]: pp_define(): defining id:
MOD_kemix
08:35:53.689895 DEBUG: <core> [core/cfg.y:1964]: yyparse(): loading module
app_lua.so
08:35:53.689931 DEBUG: <core> [core/sr_module.c:516]: ksr_locate_module(): found
module to load </usr/local/lib/kamailio/modules/app_lua.so>
08:35:53.689948 DEBUG: <core> [core/sr_module.c:566]: load_module(): trying to load
</usr/local/lib/kamailio/modules/app_lua.so>
08:35:53.690418 DEBUG: <core> [core/kemi.c:3494]: sr_kemi_eng_register(): registered
config routing enginge [lua]
08:35:53.690444 DEBUG: <core> [core/kemi.c:3398]: sr_kemi_modules_add(): adding
module: app_lua
```
### Possible Solutions
Temporary solution is disabling ASLR, ex. FreeBSD:
```
# sysctl kern.elf64.aslr.enable=0
# sysctl kern.elf64.aslr.pie_enable=0
```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.6.1 (x86_64/freebsd) b36a13
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST,
HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, select, kqueue.
id: b36a13
compiled on 08:12:34 Jul 27 2022 with cc FreeBSD clang version 14.0.5
(https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386ae247c)
```
5.5.4 is affected too
* **Operating System**:
```
FreeBSD 13/stable
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202(a)github.com>
4 Aug
4 Aug
10:13 a.m.
New subject: [kamailio/kamailio] Regression on app_lua after ASLR enable on FreeBSD (Issue #3202)
@henningw, the problem could appears not only on FreeBSD environment, but on any ASLR
environment.
I've wrote a test case for the regression close to Kamailio behavior:
_libtest.c_:
```c
#include "modules.h"
static char *str= "app_lua";
int mod_register()
{
modules_add(str);
return 0;
}
```
_modules.c_:
```c
#include <stdio.h>
void modules_add(char *msg)
{
printf("modules_add(%p): %s\n", msg, msg);
}
```
_main.c_:
```c
#include <stdio.h>
#include <dlfcn.h>
#include "modules.h"
typedef int (*mod_register_function)();
int testlib(int num) {
mod_register_function mr;
char* error;
void* h = dlopen("libtest.so", RTLD_NOW);
if (h == 0) {
printf("Error loading\n");
return 1;
}
dlerror();
mr = (mod_register_function)dlsym(h, "mod_register");
if ((error = (char*)dlerror()) != 0) {
printf("dlsym error: %s\n", error);
return 1;
}
printf("Call mod_register() #%d: ", num);
mr();
dlclose(h);
return 0;
}
int main()
{
int err;
err = testlib(1);
if (err != 0) return err;
err = testlib(2);
if (err != 0) return err;
return 0;
}
```
And ran it on non-ASLR and ASLR environment:
_non-ASLR_:
```
boris@boris:~/aslr_test% ./aslr_test
Call mod_register() #1: modules_add(0x800646528): app_lua
Call mod_register() #2: modules_add(0x800646528): app_lua
```
_ASLR_:
```
boris@boris:~/aslr_test% ./aslr_test
Call mod_register() #1: modules_add(0x825abc528): app_lua
Call mod_register() #2: modules_add(0x825bfe528): app_lua
```
And how can we see: `str` address is changed on ASLR environment, and we cannot use it
after reloading library.
I suppose using static variable after reloading library is incorrect way.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1204917683
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1204917683(a)github.com>
10:21 a.m.
@drTr0jan thanks for the clarification and the test case. Can you show the exact gcc line
how do you compiled the test cases? You can use "make Q=0" if you do it in the
kamailio make file environment.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1204926317
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1204926317(a)github.com>
10:34 a.m.
@henningw , I've build with clang:
```
clang -o aslr_test modules.c main.c
clang --shared -fPIC -o libtest.so modules.c libtest.c
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1204941107
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1204941107(a)github.com>
10:39 a.m.
Thanks. We could try to disable the PIC mode, it will probably then deactivate the ASRL.
But it will probably also break the library, as it could not re-located anymore. Maybe
@miconda has an idea.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1204947121
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1204947121(a)github.com>
11:27 a.m.
I didn't get the time yet to read properly about ASRL to comment technically, but I
don't think it was supported before and got broken over the time, so this does not
seem to be a regression, but a request for an enhancement (new feature).
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1205000196
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1205000196(a)github.com>
12:42 p.m.
Thanks Daniel. The ASRL is a system-wide feature, and was aparently activated by default
from the FreeBSD developers.
@drTr0jan The strange things is that ASRL seems to be enabled since a rather long time
(just checked an Ubuntu 20.04 and a Debian Buster), both have it activated. I am wondering
why we are not seeing more reports then. Did you executed the tests above on FreeBSD or on
a Linux machine?
```
$ cat /proc/sys/kernel/randomize_va_space
2
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1205084514
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1205084514(a)github.com>
26 Aug
26 Aug
1:14 p.m.
My comment was related to what a regression of an application is supposed to be,
respectively something worked fine and in the same context a new version of the app does
not work anymore. Because the description was saying that the problem showed up after
enabling Address Space Layout Randomization (ASLR), sounded like a new context, plus that
there was no change to app_lua. Similar, we do not say that it is a regression because a
module does not work with a new library version, but rather *lib version N not supported
yet*. But of course, this could be own interpretation.
If ASLR is enabled on Debian/Ubuntu, then it is something specific to the FreeBSD
implementation.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1228367123
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1228367123(a)github.com>
2 Sep
2 Sep
1:27 p.m.
Can you try with master branch or the patch from the commit
43f764cae870b15a96b8ca88f1eb195d4ceb8455 ?
If still not solved, can you try with master branch and use:
```
loadmodule("app_lua.so", "g")
```
for loading `app_lua` module (instead of the classic `loadmodule
"app_lua.so"`)?
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1235383993
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1235383993(a)github.com>
2:38 p.m.
@miconda, thx.
Kamailio continues falls with 43f764cae870b15a96b8ca88f1eb195d4ceb8455 after any
`sr_kemi_route()` call.
But the option "g" has solved the problem. Kamailio works. But I need more
tests.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1235452756
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1235452756(a)github.com>
6 Sep
6 Sep
8:40 a.m.
Hmm, interesting, because 43f764cae870b15a96b8ca88f1eb195d4ceb8455 clones the module name
in the core and it should no longer be influenced by the reload.
Anyhow, good to know that the variant with `g` option makes it work.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1237720427
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1237720427(a)github.com>
11:56 a.m.
@miconda, Kamailio has fallen with 43f764cae870b15a96b8ca88f1eb195d4ceb8455 not after
loading `app_lua` (as before) but after calling `app_lua` methods. I've tried fix the
subject earlier but faced with same problem.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1237930416
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1237930416(a)github.com>
13 Sep
13 Sep
10:40 a.m.
Closing this one, noting that `g` option makes it work.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1245086743
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1245086743(a)github.com>
10:40 a.m.
Closed #3202 as completed.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#event-7374391763
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issue/3202/issue_event/7374391763(a)github.com>
28 Sep
28 Sep
3:29 p.m.
@miconda, what about to merge f5c98a49c98aedcf6e1afec3c42dd862d0eeb9a3 and
69ba64e26e3876ce84053a691dee2f2ad9bb6185 to 5.6?
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1260918081
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1260918081(a)github.com>
4:09 p.m.
Those are changes that introduce nu syntax to config file and do not qualify for back
porting.
As I wrote in previous comments, this report is not about a regression, because there was
not a past commit that introduced such limitation, which could have been eventually
reverted to get it back working. It is about adding support for a new use case.
Others can comment and we can see if there is a better decision that can be taken.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1260973636
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1260973636(a)github.com>
5:55 p.m.
@drTr0jan I agree that its not the optimal solution for the affected OS. But as you
mentioned the easiest workaround is to just deactivate it on the systems until 5.7.0 next
year. OS that build their own packages (like the mentioned FreeBSD) could of course also
add the two patches to their own version as custom backport.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3202#issuecomment-1261117793
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3202/1261117793(a)github.com>
575
days inactive
638
days old
16 comments
3 participants
participants (3)
-
Boris Korzun -
Daniel-Constantin Mierla
-
Henning Westerholt