Hello,

kamailio 4.4 is no longer maintained. Upgrade to latest kamailio 5.2.x and try again. The backtrace indicates the race in tm wait handler that was fixed a while ago, but likely after 4.4 series.

If you still get the issue with 5.2.x, then send the new backtrace you get with that version.

Cheers,
Daniel

On 18.06.19 16:29, Efelin Novak wrote:

Hi folks,

I am doing some experiments with Kamailio and Cgrates. To be precise I am testing evapi module. When in testing environment I do not have any problem. After redirecting some production traffic to this kamailio, It crashes in at random times (few minutes to 20). I didn't find any pattern here.

CGRateS doesn't write anything in logs.

I'm on the last 4.4 Kamailio (compiled recently). CGRateS is on the last version.

Can you please point to how to continue with debugging?

Thanks

Efelin

Here is the problem in syslog:

kamailio[9594]: : <core> [mem/f_malloc.c:599]: fm_free(): BUG: fm_free: bad pointer 0x97894818 (out of memory block!), called from tm: h_table.c: free_cell_helper(185) - abortingkamailio[9608]: CRITICAL: <core> [pass_fd.c:277]: receive_fd(): EOF on 16
kamailio[9588]: ALERT: <core> [main.c:743]: handle_sigs(): child process 9594 exited by a signal 6
kamailio[9588]: ALERT: <core> [main.c:746]: handle_sigs(): core was not generated
kamailio[9588]: INFO: <core> [main.c:768]: handle_sigs(): terminating due to SIGCHLD
kamailio[9604]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received
kamailio[9606]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received
kamailio[9603]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received



bt full

#0  0xb77d1424 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7673661 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#2  0xb7676a92 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#3  0x082611a3 in fm_free (qmp=0x56779000, p=0x2d4e4f49, file=0x970932ee "tm: h_table.c",
    func=0x970935b7 "free_cell_helper", line=185, mname=0x97092d20 "tm") at mem/f_malloc.c:601
        qm = 0x56779000
        f = 0x56df046c
        __FUNCTION__ = "fm_free"
#4  0x96fd0559 in free_cell_helper (dead_cell=0x56d935c4, silent=0, fname=0x970b90ca "timer.c",
    fline=648) at h_table.c:185
        b = 0x56df0490 "CANCEL sip:00444123456@192.168.1.3:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.1.2;branch=z9hG4bKb6ae.5d03d7e94e5ec25f34930629f434a006.1\r\nMax-Forwards: 66\r\nFrom: \"0044454321\" <sip:0044454321@i"...
        i = 1
        rpl = 0x9797e818
        tt = 0xbfe44c68
        foo = 0x970ccb14
        cbs = 0x0
        cbs_tmp = 0x56d12b68
        __FUNCTION__ = "free_cell_helper"
#5  0x9706884f in wait_handler (ti=31715179, wait_tl=0x56d9360c, data=0x56d935c4) at timer.c:648
        p_cell = 0x56d935c4
        ret = 0
#6  0x08241e20 in timer_list_expire (t=31715179, h=0x567a3b68, slow_l=0x567a5394, slow_mark=1772)
    at timer.c:873
        tl = 0x56d9360c
        ret = 0
#7  0x08242211 in timer_handler () at timer.c:938
        saved_ticks = 31715179
        run_slow_timer = 0
        i = 748
        __FUNCTION__ = "timer_handler"
#8  0x0824266b in timer_main () at timer.c:977
No locals.
#9  0x080f0005 in main_loop () at main.c:1699
        i = 3
        pid = 0
        si = 0x0
        si_desc = "udp receiver child=2 sock=192.168.1.2:5060\000\270\301\310V\270\301\310V\000\000\000\000\000\000\000\020c\000\000\000\001\000\000\000\000\000\000\000\374Q\344\277HO\344\277\341`&\b\230\071\220\227\004\000\000\000٧5\b$\271\065\bn\000\000\000\324\247\065\b\n\000\000\000\363\320\372\060", '\000' <repeats 11 times>
        nrprocs = 3
        woneinit = 1
        __FUNCTION__ = "main_loop"
#10 0x080f6b3a in main (argc=13, argv=0xbfe45374) at main.c:2644
        cfg_stream = 0x8491008
        c = -1
        r = 0
        tmp = 0xbfe4577c ""
        tmp_len = 1
        port = 0
        proto = 0
        options = 0x8350e74 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:"
        ret = -1
        seed = 1759582648
        rfd = 4
        debug_save = 0
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = 0xbfe452a0
        p = 0x805cc8c "[\201Ä4;"
        st = {st_dev = 14, __pad1 = 0, st_ino = 6161, st_mode = 16877, st_nlink = 2, st_uid = 109,
          st_gid = 111, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0,
          st_atim = {tv_sec = 1560160293, tv_nsec = 983219454}, st_mtim = {tv_sec = 1560165082,
            tv_nsec = 861178334}, st_ctim = {tv_sec = 1560165082, tv_nsec = 861178334},
          __unused4 = 0, __unused5 = 0}
        __FUNCTION__ = "main" 

_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda