Daniel-Constantin Mierla writes:

if you are using ctl module, edit its Makefile and comment the line:

DEFS+=-DCTL_SYSTEM_MALLOC

i did that and compiled/installed ctl again. no change. still the same errors at startup.

Also, what is your glib version? Since I did it, all is working fine on my test system.

my system is debian squeeze with this kind of libglib:

ii libglib2.0-0 2.24.2-1 The GLib library of C routines

If not, the problem is somewhere else, send the version of the working instance.

working one is

version: sip-proxy 3.3.0-dev2 (i386/linux) compiled on 13:22:48 Dec 4 2011 with cc 4.4.5

-- juha

Hello,

I will try to test it this evening on a debian and look over the changes from Dec 9 to current to see what could be the cause. There was another change in tcp code to clone the buffer, but was rather small and I double checked.

Can you skip loading the tls module and see if it works without it? Tls mod was touched a bit, but not related to memory at all, but where init of tls happens.

Cheers, Daniel

On 1/16/12 8:17 PM, Juha Heinanen wrote:

daniel,

i copied whole ctl module from working dec 9 version of master to current master and then rebuild sip router. when i start it, i still get

*** glibc detected *** /usr/sbin/sip-proxy: double free or corruption (out): 0x08a6d750 *** Not starting sip-proxy: invalid configuration file!

0(16953) INFO: tls [tls_init.c:375]: tls: init_tls: disabling compression... Inconsistency detected by ld.so: dl-open.c: 221: dl_open_worker: Assertion `_dl_debug_initialize (0, args->nsid)->r_state == RT_CONSISTENT' failed!

so looks like this issue is not related to ctl module, but is something else.

-- juha

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel-Constantin Mierla -- http://www.asipto.com http://linkedin.com/in/miconda -- http://twitter.com/miconda

Hello,

On 1/16/12 8:31 PM, Juha Heinanen wrote:

Daniel-Constantin Mierla writes:

...

Can you skip loading the tls module and see if it works without it? Tls mod was touched a bit, but not related to memory at all, but where init of tls happens.

i get the same errors also when tls module is not loaded and

enable_tls=0

ok, just wanted to exclude that too from the list. I ran on an ubuntu with default config and all goes fine here, sure there is a different set of loaded module.

To understand the proxy is not starting at all, if you run with higher debug mode, where is it stopping? What is the last module/core file that prints messages?

From the man page of malloc, set MALLOC_CHECK_ environment variable to 2 and run again, you should get a coredump and extract the backtrace -- that should give a proper hint.

Also, what is the output of 'ldd --version'?

Cheers, Daniel

daniel,

thanks for the MALLOC_CHECK_ tip. after i set it to 2, i found out that starting failure with current master is due to my local module that contains a small number of non-general purpose functions.

gdb reports on the core file:

(gdb) where #0 0xb7786424 in __kernel_vsyscall () #1 0xb7638751 in raise () from /lib/i686/cmov/libc.so.6 #2 0xb763bb82 in abort () from /lib/i686/cmov/libc.so.6 #3 0xb7679484 in ?? () from /lib/i686/cmov/libc.so.6 #4 0xb767bed4 in ?? () from /lib/i686/cmov/libc.so.6 #5 0xb767c9a8 in ?? () from /lib/i686/cmov/libc.so.6 #6 0xb767de08 in malloc () from /lib/i686/cmov/libc.so.6 #7 0xb778de23 in ?? () from /lib/ld-linux.so.2 #8 0xb778df7b in ?? () from /lib/ld-linux.so.2 #9 0xb778e526 in ?? () from /lib/ld-linux.so.2 #10 0xb7793e9c in ?? () from /lib/ld-linux.so.2 #11 0xb77947f6 in ?? () from /lib/ld-linux.so.2 #12 0xb7793192 in ?? () from /lib/ld-linux.so.2 #13 0xb7798b81 in ?? () from /lib/ld-linux.so.2 #14 0xb77947f6 in ?? () from /lib/ld-linux.so.2 #15 0xb77985c6 in ?? () from /lib/ld-linux.so.2 #16 0xb776cc0b in ?? () from /lib/i686/cmov/libdl.so.2 #17 0xb77947f6 in ?? () from /lib/ld-linux.so.2 #18 0xb776d09c in ?? () from /lib/i686/cmov/libdl.so.2 #19 0xb776cb41 in dlopen () from /lib/i686/cmov/libdl.so.2 #20 0x080f72e4 in load_module (mod_path=0xb7221478 "local") at sr_module.c:569 #21 0x0817e332 in yyparse () at cfg.y:1709 #22 0x08095a1f in main (argc=18, argv=0xbfb55c24) at main.c:2084

which does not tell much to me. Makefile of the module is like this:

include ../../Makefile.defs

auto_gen= NAME=local.so

DEFS+=-DOPENSER_MOD_INTERFACE

SERLIBPATH=../../lib SER_LIBS+=$(SERLIBPATH)/kmi/kmi SER_LIBS+=$(SERLIBPATH)/srdb1/srdb1 SER_LIBS+=$(SERLIBPATH)/kcore/kcore

include ../../Makefile.modules

the module contains mod_init, child_init, mi_child_init, and destroy functions:

struct module_exports exports = { "local", DEFAULT_DLFLAGS, /* dlopen flags */ cmds, /* Exported functions */ params, /* Exported parameters */ 0, /* exported statistics */ mi_cmds, /* exported MI functions */ 0, /* exported pseudo-variables */ 0, /* extra processes */ mod_init, /* module initialization function */ 0, /* response function*/ destroy, /* destroy function */ child_init /* child initialization function */ };

i added LM_INFO calls to the beginning of each, and nothing gets printed to syslog, which makes me think that the crash happens before any of them is called.

-- juha

i tried to start sip router without loading my local module. crash still happens at the same point as before, i.e., when sip router tries to compile modparam that contains a regular expression:

modparam("auth_db|dialplan|domain|htable|lcr|msilo|mtree|permissions|usrloc", "db_url", "MYSQL_SIP_PROXY_URL")

(gdb) where #0 0xb78bd424 in __kernel_vsyscall () #1 0xb776f751 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #2 0xb7772b82 in *__GI_abort () at abort.c:92 #3 0xb77b0484 in __malloc_assert (assertion=<value optimized out>, file=<value optimized out>, line=4636, function=0xb7867490 "_int_malloc") at malloc.c:352 #4 0xb77b2ed4 in _int_malloc (av=<value optimized out>, bytes=<value optimized out>) at malloc.c:4636 #5 0xb77b4c8c in *__GI___libc_malloc (bytes=0) at malloc.c:3661 #6 0xb77fb2cc in analyze (preg=<value optimized out>, pattern=<value optimized out>, length=67, syntax=4436732) at regcomp.c:1135 #7 re_compile_internal (preg=<value optimized out>, pattern=<value optimized out>, length=67, syntax=4436732) at regcomp.c:804 #8 0xb77fbe5e in __regcomp (preg=0xbff52264, pattern=0xb7358610 "^auth_db|dialplan|domain|htable|lcr|msilo|mtree|permissions|usrloc$", cflags=11) at regcomp.c:506 #9 0x0809a500 in set_mod_param_regex ( regex=0xb7358478 "auth_db|dialplan|domain|htable|lcr|msilo|mtree|permissions|usrloc", name=0xb7358500 "db_url", type=1, val=0xb7358588) at modparam.c:86 #10 0x0817e4fd in yyparse () at cfg.y:1733 #11 0x08095a1f in main (argc=18, argv=0xbff52944) at main.c:2084

i think regcomp.c is part of c library in debian squeeze.

-- juha

On Sunday 22 January 2012, Juha Heinanen wrote:

i installed my sip router package on another debian hosts and there it starts fine. so perhaps there is a memory or some other hardware problem on my other host that causes these malloc related crashes.

On Saturday 21 January 2012, Juha Heinanen wrote:

i think regcomp.c is part of c library in debian squeeze.

Are you sure your libc is from squeeze and you do not have a libc version 2.13? If it is 2.13, it might me a wrong use of memcpy somewhere in the code. That problem almost always shows itself as vague, seemingly random, failures.