Hello,

I didn't know that log messages related to memory operations can be controlled by global parameter, but I like to know if is recommendable I recompile kamailio using q_malloc (default) and not f_malloc... memory operations using q_malloc is more reliable and avoid problems or crashes or it is only more suitable for debugging? What do you suggest Daniel?

The 'get_statistics all' command is avaliable by a specific module? I run 'kamctl fifo get_statistics all' and return '500 command 'get_statistics' not available'

For while is impossible to start a new installation, first because I don't know how much time I will spend to port and second because I am involved with other developments and I have no time  to make this. I know that 3.1 is no longer a official branch but now start a new installation it's very very difficult, my in intention is discover what caused the crash and if exists a way to fix or prevent it.

Best Regards

2012/7/31 Daniel-Constantin Mierla <miconda@gmail.com>
Hello,


On 7/30/12 9:01 PM, Bruno Bresciani wrote:
Hi,

I compiled kamailio with MEMDBG = 0 because I didn't want the memory debug  in kamailio log, but I didn't know these turns on f_malloc and disabling q_malloc...
the log messages related to memory operations can be controlled by global parameters memdbg and memlog.



Probably the size of shared memory that I start kamailio is 32MB because I didn't gave a different -m parameter value, exist a command to verify this information?

kamctl fifo get_statistics all

and see the shared memory total value. It will be interesting to see available shared memory as well.



At moment that crash happened, there were few registered users agents and were being made tests with register and calls with TLS protocol. I got only the two situations that I showed at first email.

If you plan to start a new installation, I strongly recommend 3.3 branch, the code is more actual and easier to debug. 3.1 is no longer an official maintained branch, those being now 3.3 and 3.2. I'm looking at this issue to be sure it is no longer in latest stable.

Cheers,
Daniel




Best Regards

2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com>
Hello,

MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC in the kamailio -V output shows f_malloc is in use.

q_malloc is more suitable for debugging, as it keep trace of location in the source code that allocates/frees the memory chunks, at the expense of a bit more memory usage due to tracking overhead. q_malloc was the default for 3.1, that's why I asked because the trace showed lines from f_malloc c code.

What is the size of shared memory you start kamailio? Is the default 32MB or you give a different -m parameter value? Was the serve very loaded? How many such situations did you get so far?

Cheers,
Daniel



On 7/30/12 7:16 PM, Bruno Bresciani wrote:
Hi,

Bellow is output of 'kamailio -V'...

------------------------------------------------------------------------------------------------------------------
version: kamailio 3.1.2 (i386/linux) 4d9f90
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 4d9f90
compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2
-------------------------------------------------------------------------------------------------------------------

Daniel, I don't remember to enable f_malloc... Actually I do not know the difference between f_malloc and g_malloc to choose compile kamailio with f_malloc instead of g_malloc.
My commands to compile kamailio were:

$ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio include_modules="db_postgres tls" MEMDBG=0 cfg
$ make all
$ make install
 
If you need more informations, I am available...

Best Regards
 
2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com>
Hello,

can you provide output of 'kamailio -V'? Did you enable f_malloc instead of q_malloc which was default in 3.1?

Cheers,
Daniel


On 7/30/12 1:44 PM, Bruno Bresciani wrote:
Thank's for feedback Daniel,

This core is add to the tracker with ID 247...

Best Regards


2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com>
Hello,

it is in my to-do list to investigate, so far I was not able to reproduce.

Vacation time together with other travelings makes a bit more slower process. Please add it also to the tracker:

http://sip-router.org/tracker/

A minor release for 3.3 branch is going to be soon, being there makes sure it will be reviewed to see if it applies.

Cheers,
Daniel


On 7/27/12 3:00 PM, Bruno Bresciani wrote:
Hi Users,

This is my ultimate try to ask for some help to debug the two cores generated by fm_realloc() function. If somebody can give me a feedback about my questions or doubts I will be very grateful, this user list of kamailio always help me to solve my problems.

Best Regards


2012/7/23 Bruno Bresciani <bruno.bresciani@gmail.com>
Hi All,

I imagine that this core doesn't easy to replicate but somebody  can help me to understand why it was generated? There is some way to  prevent it?

Best Regards



2012/7/18 Renan Capaverde <renan.capaverde@digitro.com.br>
I am having the same problem on kamailio 3.1.6. Someone please help.

Cheers,
Renan

Em 17/7/2012 14:26, Bruno Bresciani escreveu:
Excuse-me Daniel, I forgot this information... Kamailio version 3.1.2

Best Regards


2012/7/17 Daniel-Constantin Mierla <miconda@gmail.com>
Hello,

send also the version 'kamailio -V' to match the proper sources.

Cheers,
Daniel


On 7/17/12 7:11 PM, Bruno Bresciani wrote:
Hi all,

I observed two at 0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.

I don't know exactly what may be caused this core because the debug of kamailio is disabled, but looking at backtrace (gdb) and source code I could notice a relationship with TLS connection establishment (handshake). Other important fact is that core was generated by the kamalio itself with signal 6 (abort) because it can't find the pointer to memory realloc.
I'd like to know if someone already observed this issue and if my analyse above is correct?

Bellow is backtrace full with gdb:

Core was generated by `/home2/local/kamailio/sbin/kamailio -P /var/run/kamailio.pid'.
Program terminated with signal 6, Aborted.
#0  0x00f30402 in __kernel_vsyscall ()
(gdb)
(gdb)
(gdb)
(gdb)
(gdb) bt full
#0  0x00f30402 in __kernel_vsyscall ()
No symbol table info available.
#1  0x009fec10 in raise () from /lib/libc.so.6
No symbol table info available.
#2  0x00a00521 in abort () from /lib/libc.so.6
No symbol table info available.
#3  0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536
        f = (struct fm_frag *) 0xb61f0bb8
        pf = <value optimized out>
        orig_size = 18
L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
        ptr = <value optimized out>
        hash = 23548
#4  0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32) at ../../mem/shm_mem.h:266
No locals.
#5  0x00460295 in OPENSSL_cleanse () from /lib/libcrypto.so.6
No symbol table info available.
#6  0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6
No symbol table info available.
#7  0x003efb22 in sk_insert () from /lib/libcrypto.so.6
No symbol table info available.
#8  0x003efbaa in sk_push () from /lib/libcrypto.so.6
No symbol table info available.
#9  0x007c2ac3 in ssl_bytes_to_cipher_list () from /lib/libssl.so.6
No symbol table info available.
#10 0x007aa9f3 in ssl3_get_client_hello () from /lib/libssl.so.6
No symbol table info available.
#11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6
No symbol table info available.
#12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6
No symbol table info available.
#13 0x005278e3 in tls_accept (c=0xb621af88, error=0xbf91dfc8) at tls_server.c:346
        ret = <value optimized out>
        ssl = (SSL *) 0xb61c1878
        cert = <value optimized out>
        tls_c = (struct tls_extra_data *) 0xb6208e58
        tls_log = <value optimized out>
#14 0x0052da8e in tls_read_f (c=0xb621af88, flags=0xbf91e200) at tls_server.c:1028
        r = (struct tcp_req *) 0xb621affc
        bytes_free = 4095
        bytes_read = 70
        read_size = 4095
        ssl_error = 0
        ssl_read = 0
        ssl = (SSL *) 0xb61c1878
        rd_buf = "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃàéEa\023xéJ\206T\204Cà       wr_buf = "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
        rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70, used = 70, size = 65536}
        wr = {
  buf = 0xbf8fdfa4 "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"..., pos = 0, used = 0,
size = 65536}
        tls_c = (struct tls_extra_data *) 0xb6208e58
        enc_rd_buf = (struct tls_rd_buf *) 0x0
        n = 0
        flush_flags = <value optimized out>
        err_src = 0x541a83 "TLS read:"
        x = <value optimized out>
#15 0x08158ade in tcp_read_headers (c=0xb621af88, read_flags=0xbf91e200) at tcp_read.c:406
        bytes = -1
        remaining = <value optimized out>
        p = <value optimized out>
        r = (struct tcp_req *) 0xb621affc
#16 0x08158fd4 in tcp_read_req (con=0xb621af88, bytes_read=0xbf91e204, read_flags=0xbf91e200) at tcp_read.c:871
        bytes = <value optimized out>
        total_bytes = 0
        resp = <value optimized out>
        size = <value optimized out>
        req = (struct tcp_req *) 0xb621affc
        dst = {send_sock = 0x1, to = {s = {sa_family = 57540, sa_data = "\221¿\020\000\000\000\000\000\000\000\020\000\000"}, sin = {sin_family = 57540,
      sin_port = 49041, sin_addr = {s_addr = 16}, sin_zero = "\000\000\000\000\020\000\000"}, sin6 = {sin6_family = 57540, sin6_port = 49041,
      sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8 = "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000", u6_addr16 = {0, 0, 16, 0, 1, 0,
            1, 0}, u6_addr32 = {0, 16, 1, 1}}}, sin6_scope_id = 12}}, id = -1080958468, proto = 4 '\004', send_flags = {f = 0 '\0', blst_imask = 0 '\0'}}
        c = 10 '\n'
#17 0x08159a0f in handle_io (fm=0x83296b0, events=1, idx=-1) at tcp_read.c:1150
        ret = 4
        n = 4
        read_flags = 1
        con = (struct tcp_connection *) 0xb621af88
        s = 12
        resp = <value optimized out>
        t = <value optimized out>
#18 0x0815d653 in tcp_receive_loop (unix_sock=35) at io_wait.h:1091
No locals.
---Type <return> to continue, or q <return> to quit---
#19 0x0812885b in tcp_init_children () at tcp_main.c:4819
        r = 3
        reader_fd_1 = 35
        pid = <value optimized out>
        si = <value optimized out>
#20 0x080ad412 in main_loop () at main.c:1632
        i = 4
        pid = <value optimized out>
        si = (struct socket_info *) 0x0
        si_desc = "udp receiver child=3 sock=192.168.166.31:5060\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
#21 0x080afbe7 in main (argc=3, argv=dwarf2_read_address: Corrupted DWARF expression.
) at main.c:2398
        cfg_stream = (FILE *) 0x8a2d008
        c = <value optimized out>
        r = 0
        tmp = 0xbf91e674 "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221        tmp_len = 10492229
        port = 136192361
        proto = -1080957480
        ret = <value optimized out>
        seed = 21782655
        rfd = 4
        debug_save = <value optimized out>
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = <value optimized out>
        p = <value optimized out>


Best Regards


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw



_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users



_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users





_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw


-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw


-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw


-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw