Hi,
I compiled kamailio with MEMDBG = 0 because I didn't want the memory debug in kamailio log, but I didn't know these turns on f_malloc and disabling q_malloc...
Probably the size of shared memory that I start kamailio is 32MB because I didn't gave a different -m parameter value, exist a command to verify this information?
At moment that crash happened, there were few registered users agents and were being made tests with register and calls with TLS protocol. I got only the two situations that I showed at first email.
Best Regards
Hello,
MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC in the kamailio -V output shows f_malloc is in use.
q_malloc is more suitable for debugging, as it keep trace of location in the source code that allocates/frees the memory chunks, at the expense of a bit more memory usage due to tracking overhead. q_malloc was the default for 3.1, that's why I asked because the trace showed lines from f_malloc c code.
What is the size of shared memory you start kamailio? Is the default 32MB or you give a different -m parameter value? Was the serve very loaded? How many such situations did you get so far?
Cheers,
Daniel
On 7/30/12 7:16 PM, Bruno Bresciani wrote:
Hi,
Bellow is output of 'kamailio -V'...
------------------------------------------------------------------------------------------------------------------
version: kamailio 3.1.2 (i386/linux) 4d9f90
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 4d9f90
compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2
-------------------------------------------------------------------------------------------------------------------
Daniel, I don't remember to enable f_malloc... Actually I do not know the difference between f_malloc and g_malloc to choose compile kamailio with f_malloc instead of g_malloc.
My commands to compile kamailio were:
$ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio include_modules="db_postgres tls" MEMDBG=0 cfg
$ make all
$ make install
If you need more informations, I am available...
Best Regards
2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com>
Hello,
can you provide output of 'kamailio -V'? Did you enable f_malloc instead of q_malloc which was default in 3.1?
Cheers,
Daniel
On 7/30/12 1:44 PM, Bruno Bresciani wrote:
Thank's for feedback Daniel,
This core is add to the tracker with ID 247...
Best Regards
2012/7/30 Daniel-Constantin Mierla <miconda@gmail.com>
Hello,
it is in my to-do list to investigate, so far I was not able to reproduce.
Vacation time together with other travelings makes a bit more slower process. Please add it also to the tracker:
http://sip-router.org/tracker/
A minor release for 3.3 branch is going to be soon, being there makes sure it will be reviewed to see if it applies.
Cheers,
Daniel
On 7/27/12 3:00 PM, Bruno Bresciani wrote:
Hi Users,
This is my ultimate try to ask for some help to debug the two cores generated by fm_realloc() function. If somebody can give me a feedback about my questions or doubts I will be very grateful, this user list of kamailio always help me to solve my problems.
Best Regards
2012/7/23 Bruno Bresciani <bruno.bresciani@gmail.com>
Hi All,
I imagine that this core doesn't easy to replicate but somebody can help me to understand why it was generated? There is some way to prevent it?
Best Regards
2012/7/18 Renan Capaverde <renan.capaverde@digitro.com.br>
I am having the same problem on kamailio 3.1.6. Someone please help.
Cheers,
Renan
Em 17/7/2012 14:26, Bruno Bresciani escreveu:
Excuse-me Daniel, I forgot this information... Kamailio version 3.1.2
Best Regards
2012/7/17 Daniel-Constantin Mierla <miconda@gmail.com>
Hello,
send also the version 'kamailio -V' to match the proper sources.
Cheers,
Daniel
On 7/17/12 7:11 PM, Bruno Bresciani wrote:
Hi all,
I observed two at 0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536.
I don't know exactly what may be caused this core because the debug of kamailio is disabled, but looking at backtrace (gdb) and source code I could notice a relationship with TLS connection establishment (handshake). Other important fact is that core was generated by the kamalio itself with signal 6 (abort) because it can't find the pointer to memory realloc.
I'd like to know if someone already observed this issue and if my analyse above is correct?
Bellow is backtrace full with gdb:
Core was generated by `/home2/local/kamailio/sbin/kamailio -P /var/run/kamailio.pid'.
Program terminated with signal 6, Aborted.
#0 0x00f30402 in __kernel_vsyscall ()
(gdb)
(gdb)
(gdb)
(gdb)
(gdb) bt full
#0 0x00f30402 in __kernel_vsyscall ()
No symbol table info available.
#1 0x009fec10 in raise () from /lib/libc.so.6
No symbol table info available.
#2 0x00a00521 in abort () from /lib/libc.so.6
No symbol table info available.
#3 0x08173bf0 in fm_realloc (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at mem/f_malloc.c:536
f = (struct fm_frag *) 0xb61f0bb8
pf = <value optimized out>
orig_size = 18
L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
ptr = <value optimized out>
hash = 23548
#4 0x0051a87f in ser_realloc (ptr=0xb61f0bc0, size=32) at ../../mem/shm_mem.h:266
No locals.
#5 0x00460295 in OPENSSL_cleanse () from /lib/libcrypto.so.6
No symbol table info available.
#6 0x004609f7 in CRYPTO_realloc () from /lib/libcrypto.so.6
No symbol table info available.
#7 0x003efb22 in sk_insert () from /lib/libcrypto.so.6
No symbol table info available.
#8 0x003efbaa in sk_push () from /lib/libcrypto.so.6
No symbol table info available.
#9 0x007c2ac3 in ssl_bytes_to_cipher_list () from /lib/libssl.so.6
No symbol table info available.
#10 0x007aa9f3 in ssl3_get_client_hello () from /lib/libssl.so.6
No symbol table info available.
#11 0x007ab815 in ssl3_accept () from /lib/libssl.so.6
No symbol table info available.
#12 0x007c1c3a in SSL_accept () from /lib/libssl.so.6
No symbol table info available.
#13 0x005278e3 in tls_accept (c=0xb621af88, error=0xbf91dfc8) at tls_server.c:346
ret = <value optimized out>
ssl = (SSL *) 0xb61c1878
cert = <value optimized out>
tls_c = (struct tls_extra_data *) 0xb6208e58
tls_log = <value optimized out>
#14 0x0052da8e in tls_read_f (c=0xb621af88, flags=0xbf91e200) at tls_server.c:1028
r = (struct tcp_req *) 0xb621affc
bytes_free = 4095
bytes_read = 70
read_size = 4095
ssl_error = 0
ssl_read = 0
ssl = (SSL *) 0xb61c1878
rd_buf = "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
202i<ä<åþ\031ì234 \237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃà éEa\023xéJ\206T\204Cà wr_buf = "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
rd = {buf = 0xbf90dfa4 "\026\003\001", pos = 70, used = 70, size = 65536}
wr = {
buf = 0xbf8fdfa4 "\027\003\001\003 w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"..., pos = 0, used = 0,
size = 65536}
tls_c = (struct tls_extra_data *) 0xb6208e58
enc_rd_buf = (struct tls_rd_buf *) 0x0
n = 0
flush_flags = <value optimized out>
err_src = 0x541a83 "TLS read:"
x = <value optimized out>
#15 0x08158ade in tcp_read_headers (c=0xb621af88, read_flags=0xbf91e200) at tcp_read.c:406
bytes = -1
remaining = <value optimized out>
p = <value optimized out>
r = (struct tcp_req *) 0xb621affc
#16 0x08158fd4 in tcp_read_req (con=0xb621af88, bytes_read=0xbf91e204, read_flags=0xbf91e200) at tcp_read.c:871
bytes = <value optimized out>
total_bytes = 0
resp = <value optimized out>
size = <value optimized out>
req = (struct tcp_req *) 0xb621affc
dst = {send_sock = 0x1, to = {s = {sa_family = 57540, sa_data = "\221¿\020\000\000\000\000\000\000\000\020\000\000"}, sin = {sin_family = 57540,
sin_port = 49041, sin_addr = {s_addr = 16}, sin_zero = "\000\000\000\000\020\000\000"}, sin6 = {sin6_family = 57540, sin6_port = 49041,
sin6_flowinfo = 16, sin6_addr = {in6_u = {u6_addr8 = "\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000", u6_addr16 = {0, 0, 16, 0, 1, 0,
1, 0}, u6_addr32 = {0, 16, 1, 1}}}, sin6_scope_id = 12}}, id = -1080958468, proto = 4 '\004', send_flags = {f = 0 '\0', blst_imask = 0 '\0'}}
c = 10 '\n'
#17 0x08159a0f in handle_io (fm=0x83296b0, events=1, idx=-1) at tcp_read.c:1150
ret = 4
n = 4
read_flags = 1
con = (struct tcp_connection *) 0xb621af88
s = 12
resp = <value optimized out>
t = <value optimized out>
#18 0x0815d653 in tcp_receive_loop (unix_sock=35) at io_wait.h:1091
No locals.
---Type <return> to continue, or q <return> to quit---
#19 0x0812885b in tcp_init_children () at tcp_main.c:4819
r = 3
reader_fd_1 = 35
pid = <value optimized out>
si = <value optimized out>
#20 0x080ad412 in main_loop () at main.c:1632
i = 4
pid = <value optimized out>
si = (struct socket_info *) 0x0
si_desc = "udp receiver child=3 sock=192.168.166.31:5060\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
#21 0x080afbe7 in main (argc=3, argv=dwarf2_read_address: Corrupted DWARF expression.
) at main.c:2398
cfg_stream = (FILE *) 0x8a2d008
c = <value optimized out>
r = 0
tmp = 0xbf91e674 "\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221 tmp_len = 10492229
port = 136192361
proto = -1080957480
ret = <value optimized out>
seed = 21782655
rfd = 4
debug_save = <value optimized out>
debug_flag = 0
dont_fork_cnt = 0
n_lst = <value optimized out>
p = <value optimized out>
Best Regards
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw