Hi Daniel,

 

I tried to increase shared memory by setting a parameter in kamctlrc:

STARTOPTIONS="-m 64"

 

When recreating the crash I counted pv_get_ruri_attr in the logfile just to see if the shared memory changed anything:

> grep pv_get_ruri_attr /var/log/kamailio.log | wc -l

75472 with default settings

75436 with shared memory set to 64

75440 with shared memory set to 256

 

But in top I saw a massive increase in memory usage.

 

I don’t know if this is any helpful for you.

 

 

Kind regards

Timo Klecker



 

Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Montag, 8. Dezember 2014 16:25
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: [SR-Users] Kamailio Crash when modifying username of Request Uri

 

Hello,

I noticed that there was an issue with the result stored in r-uri and it should be fixed in the latest git branches -- you can fetch them and try, practically, if the resulting user is empty, then the password is removed as well.

However, I wanted to hunt the real reason of the crash, as kamailio can handle cases when the r-uri is broken (could be an incoming r-uri).

Cheers,
Daniel

On 08/12/14 13:50, Timo Klecker wrote:

Hi Daniel,

 

in the configuration I sent last week is a very simplified version of our loop:

 

route[START_ROUTING] {

    if ($rU == "123") {

        xlog("L_NOTICE", "123 reached\n");

        exit;

    }

    route(MODIFY_NUMBER);

}

 

route[MODIFY_NUMBER] {

    $rU = $(rU{s.strip,6});

        xlog("L_NOTICE", "rU set to: $rU\n");

        xlog("L_NOTICE", "ru set to: $ru\n");

    $rU = '123' + $rU;

        xlog("L_NOTICE", "rU set to: $rU\n");

        xlog("L_NOTICE", "ru set to: $ru\n");

    route(START_ROUTING);

}

 

Due to the parsing Error in $rU the break of the loop is never triggered. So you are right, there is an infinite loop. When breaking this loop, no crash happens.

 

But still one problem remains:

 

Request Uri:  sip:123456:pass@sip.custom.com

$rU = $(rU{s.strip,6});

 

Afterwards the Request Uri is set to

sip::passsip.custom.com

There is an @ missing.

 

Dec  4 13:27:50 /sbin/kamailio[8043]: ERROR: <core> [action.c:805]: do_action(): ERROR: do_action: bad uri <sip::pass10.10.207.20>, dropping packet

 

And $rU and $ru are null due to parsing-errors.

 

 

Kind regards

Timo Klecker




 

Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Freitag, 5. Dezember 2014 18:04
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: AW: AW: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri

 

Hello,

according to the trace, the crash trace is:

#0  0x00007f544c119801 in _IO_default_xsputn_internal () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f544c0e93a9 in vfprintf () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007f544c0f4368 in fprintf () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007f544c189bc8 in __vsyslog_chk () from /lib64/libc.so.6
No symbol table info available.
#4  0x00007f544c18a210 in syslog () from /lib64/libc.so.6
No symbol table info available.
#5  0x00007f54479f2341 in pv_get_ruri_attr (msg=0x7f544bed9c30, param=0x7f544beb2a50, res=0x7fffe1341e88) at pv_core.c:304
        __FUNCTION__ = "pv_get_ruri_attr"

Line 304 in pv_core is:
        LM_ERR("failed to parse the R-URI\n");

Which is a bit strange as a point of crash. I will try to see if I can spot something else...

I noticed that you seem to have somehow kind of infinite loop with calling route blocks - like calling route B from A and route A from B...

Cheers,
Daniel

On 05/12/14 17:45, Timo Klecker wrote:

Hi Daniel,

 

yes, Kamailio 4.1 using syslog:

 

log_facility=LOG_LOCAL4

# modules

loadmodule "xlog.so"

 

 

# kamailio -v

version: kamailio 4.1.6 (x86_64/linux) 010d57

flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB

poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.

id: 010d57

compiled on 17:35:09 Dec  4 2014 with gcc 4.4.6

 

Compiled yesterday because I needed the right debug-rpm and had to recompile.

 

Kind regards

Timo Klecker

 

 

Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Freitag, 5. Dezember 2014 17:40
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: AW: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri

 

I looked at the last gdb trace you sent and the crash is inside syslog() function, which prints a message with static text, if I matched it with the right version of the sources.

Is the trace from running kamailio 4.1? What is the output of:

kamailio -v

Cheers,
Daniel

On 04/12/14 17:23, Daniel-Constantin Mierla wrote:

Btw, you need to install dbg symbols package, in case you installed from some deb/rpm.

Daniel

On 04/12/14 17:22, Daniel-Constantin Mierla wrote:

Hello,

I need the output of the 'bt full' -- I guess it will require a few 'more' paginations, because it is printing the values for local variables which are important to troubleshoot.

Cheers,
daniel

On 04/12/14 17:13, Timo Klecker wrote:

Hi Daniel,

 

gdb /path/to/kamailio /path/to/corefile

sent me into interactive mode. Typing

bt full

now sent me into something like “more” so I had to copy everything by hand, like 46k lines.

 

So I searched the web for a non interactive method and came up with this. If it is not ok, please tell me how to get the “bt full” string out of this interactive gdb shell.

 

I will provide you with the output of gdb itself, I missed that in the last mail.

 

Greetings

Timo




 

 

Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Donnerstag, 4. Dezember 2014 16:47
An: Timo Klecker; 'Kamail
io (SER) - Users Mailing List'
Betreff: Re: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request
Uri

 

Hello,

wondering why not simply sending what I asked for, respectively output for:

gdb /path/to/kamailio /path/to/corefile
bt full

I guess I wanted to get some useful information out of those commands...

Cheers,
Daniel


On 04/12/14 16:41, Timo Klecker wrote:

Hi Daniel,

 

sorry for sending the core files, I now sent the output of:

 

gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" /sbin/kamailio /core.8933

 

Hope this is what you are looking for.

 

Kind regards,

Timo







 

Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Donnerstag, 4. Dezember 2014 16:16
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri

 

Hello,

corefiles themselves are useless. I need the backtrace from them, respectively the output of:

gdb /path/to/kamailio /path/to/corefile
bt full

Cheers,
Daniel

On 04/12/14 15:56, Timo Klecker wrote:

Hello Daniel,

 

yes, with the production config I still get a core, not with the testing config, though. The last line in log with testing config is:

 

Dec  4 13:28:06 lvm-centos-kamailio1 /sbin/kamailio[8045]: ERROR: <core> [action.c:1572]: run_actions(): WARNING: too many recursive routing table lookups (101) giving up!

Dec  4 13:28:06 lvm-centos-kamailio1 /sbin/kamailio[8045]: WARNING: <core> [receive.c:214]: receive_msg(): WARNING: receive_msg: error while trying script

 

But I have 2 cores, each 52M in size from production config. I will bzip them and send them via email.

 

 

Kind regards,

Timo Klecker




 




-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda



-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda