30 Jul
2012
30 Jul
'12
8:23 p.m.
Hello,
MEMDBG=0 turns on f_malloc, disabling q_malloc. The token F_MALLOC in
the kamailio -V output shows f_malloc is in use.
q_malloc is more suitable for debugging, as it keep trace of location in
the source code that allocates/frees the memory chunks, at the expense
of a bit more memory usage due to tracking overhead. q_malloc was the
default for 3.1, that's why I asked because the trace showed lines from
f_malloc c code.
What is the size of shared memory you start kamailio? Is the default
32MB or you give a different -m parameter value? Was the serve very
loaded? How many such situations did you get so far?
Cheers,
Daniel
On 7/30/12 7:16 PM, Bruno Bresciani wrote:
Hi,
Bellow is output of 'kamailio -V'...
------------------------------------------------------------------------------------------------------------------
version: kamailio 3.1.2 (i386/linux) 4d9f90
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 4d9f90
compiled on 10:39:54 Apr 29 2011 with gcc 4.1.2
-------------------------------------------------------------------------------------------------------------------
Daniel, I don't remember to enable f_malloc... Actually I do not know
the difference between f_malloc and g_malloc to choose compile
kamailio with f_malloc instead of g_malloc.
My commands to compile kamailio were:
*
$ make PREFIX="/home2/local/kamailio" FLAVOUR=kamailio
include_modules="db_postgres tls" MEMDBG=0 cfg**
$ make all
$ make install*
If you need more informations, I am available...
Best Regards
2012/7/30 Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>>
Hello,
can you provide output of 'kamailio -V'? Did you enable f_malloc
instead of q_malloc which was default in 3.1?
Cheers,
Daniel
On 7/30/12 1:44 PM, Bruno Bresciani wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
Thank's for feedback Daniel,
This core is add to the tracker with ID 247...
Best Regards
2012/7/30 Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>>
Hello,
it is in my to-do list to investigate, so far I was not able
to reproduce.
Vacation time together with other travelings makes a bit more
slower process. Please add it also to the tracker:
http://sip-router.org/tracker/
A minor release for 3.3 branch is going to be soon, being
there makes sure it will be reviewed to see if it applies.
Cheers,
Daniel
On 7/27/12 3:00 PM, Bruno Bresciani wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 -http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 -http://asipto.com/u/kpw
Hi Users,
This is my ultimate try to ask for some help to debug the
two cores generated by fm_realloc() function. If somebody
can give me a feedback about my questions or doubts I will
be very grateful, this user list of kamailio always help me
to solve my problems.
Best Regards
2012/7/23 Bruno Bresciani <bruno.bresciani(a)gmail.com
<mailto:bruno.bresciani@gmail.com>>
Hi All,
I imagine that this core doesn't easy to replicate but
somebody can help me to understand why it was
generated? There is some way to prevent it?
Best Regards
2012/7/18 Renan Capaverde
<renan.capaverde(a)digitro.com.br
<mailto:renan.capaverde@digitro.com.br>>
I am having the same problem on kamailio 3.1.6.
Someone please help.
Cheers,
Renan
Em 17/7/2012 14:26, Bruno Bresciani escreveu:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012
-http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012
-http://asipto.com/u/kpw
Excuse-me Daniel, I forgot this
information...
Kamailio version 3.1.2
Best Regards
2012/7/17 Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>>
Hello,
send also the version 'kamailio -V' to match
the proper sources.
Cheers,
Daniel
On 7/17/12 7:11 PM, Bruno Bresciani wrote:
> Hi all,
>
> I observed two at 0x08173bf0 in fm_realloc
> (qm=0xb5f7d000, p=0xb61f0bc0, size=0) at
> mem/f_malloc.c:536.
>
> I don't know exactly what may be caused this
> core because the debug of kamailio is
> disabled, but looking at backtrace (gdb) and
> source code I could notice a relationship with
> TLS connection establishment (handshake).
> Other important fact is that core was
> generated by the kamalio itself with signal 6
> (abort) because it can't find the pointer to
> memory realloc.
> I'd like to know if someone already observed
> this issue and if my analyse above is correct?
>
> Bellow is backtrace full with gdb:
>
> Core was generated by
> `/home2/local/kamailio/sbin/kamailio -P
> /var/run/kamailio.pid'.
> Program terminated with signal 6, Aborted.
> #0 0x00f30402 in __kernel_vsyscall ()
> (gdb)
> (gdb)
> (gdb)
> (gdb)
> (gdb) bt full
> #0 0x00f30402 in __kernel_vsyscall ()
> No symbol table info available.
> #1 0x009fec10 in raise () from /lib/libc.so.6
> No symbol table info available.
> #2 0x00a00521 in abort () from /lib/libc.so.6
> No symbol table info available.
> #3 0x08173bf0 in fm_realloc (qm=0xb5f7d000,
> p=0xb61f0bc0, size=0) at mem/f_malloc.c:536
> f = (struct fm_frag *) 0xb61f0bb8
> pf = <value optimized out>
> orig_size = 18
> L=Ãb¬Ã2n = (struct fm_frag *) 0xb61f0bd2
> ptr = <value optimized out>
> hash = 23548
> #4 0x0051a87f in ser_realloc (ptr=0xb61f0bc0,
> size=32) at ../../mem/shm_mem.h:266
> No locals.
> #5 0x00460295 in OPENSSL_cleanse () from
> /lib/libcrypto.so.6
> No symbol table info available.
> #6 0x004609f7 in CRYPTO_realloc () from
> /lib/libcrypto.so.6
> No symbol table info available.
> #7 0x003efb22 in sk_insert () from
> /lib/libcrypto.so.6
> No symbol table info available.
> #8 0x003efbaa in sk_push () from
> /lib/libcrypto.so.6
> No symbol table info available.
> #9 0x007c2ac3 in ssl_bytes_to_cipher_list ()
> from /lib/libssl.so.6
> No symbol table info available.
> #10 0x007aa9f3 in ssl3_get_client_hello ()
> from /lib/libssl.so.6
> No symbol table info available.
> #11 0x007ab815 in ssl3_accept () from
> /lib/libssl.so.6
> No symbol table info available.
> #12 0x007c1c3a in SSL_accept () from
> /lib/libssl.so.6
> No symbol table info available.
> #13 0x005278e3 in tls_accept (c=0xb621af88,
> error=0xbf91dfc8) at tls_server.c:346
> ret = <value optimized out>
> ssl = (SSL *) 0xb61c1878
> cert = <value optimized out>
> tls_c = (struct tls_extra_data *)
> 0xb6208e58
> tls_log = <value optimized out>
> #14 0x0052da8e in tls_read_f (c=0xb621af88,
> flags=0xbf91e200) at tls_server.c:1028
> r = (struct tcp_req *) 0xb621affc
> bytes_free = 4095
> bytes_read = 70
> read_size = 4095
> ssl_error = 0
> ssl_read = 0
> ssl = (SSL *) 0xb61c1878
> rd_buf =
> "\026\003\001\000A\001\000\000=\003\001Oÿ\v¡ywP\fkÃDÃ
>
0/\000\a\000\005\001\000ÃÃÃï000\000\026\0009\0008\0005\000\026\000\023\000\n\0003\0002\00
> 202i<ä<åþ\031ì234Â
\237ê030Y){lѶ"...06Ã'Ãr9ìë¸{[©\t\205\037\036ÿHPH5µ\t(é32Ã027æò036¾\"\rÔ¤YÃ215uº\027)ö000ñ215\vÃÃ
éEa\023xéJ\206T\204CÃ
> wr_buf =
> "\027\003\001\003Â
w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
>
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...
> rd = {buf = 0xbf90dfa4 "\026\003\001",
> pos = 70, used = 70, size = 65536}
> wr = {
> buf = 0xbf8fdfa4
> "\027\003\001\003Â
w²'\016ѼÃ033ÃLÃ021Ã)Ã\220hnL¦\020Xê\236\200\tIhøSf\234Ã\rÃ
>
?¯¼\024FÃ\200ª/\2264RÃaäIpXȬÃ035äQh\211Ã026ý4ÃU+\asr£/`¤Oû1AʹqUܿ·v±ü\205"...,
> pos = 0, used = 0,
> size = 65536}
> tls_c = (struct tls_extra_data *)
> 0xb6208e58
> enc_rd_buf = (struct tls_rd_buf *) 0x0
> n = 0
> flush_flags = <value optimized out>
> err_src = 0x541a83 "TLS read:"
> x = <value optimized out>
> #15 0x08158ade in tcp_read_headers
> (c=0xb621af88, read_flags=0xbf91e200) at
> tcp_read.c:406
> bytes = -1
> remaining = <value optimized out>
> p = <value optimized out>
> r = (struct tcp_req *) 0xb621affc
> #16 0x08158fd4 in tcp_read_req
> (con=0xb621af88, bytes_read=0xbf91e204,
> read_flags=0xbf91e200) at tcp_read.c:871
> bytes = <value optimized out>
> total_bytes = 0
> resp = <value optimized out>
> size = <value optimized out>
> req = (struct tcp_req *) 0xb621affc
> dst = {send_sock = 0x1, to = {s =
> {sa_family = 57540, sa_data =
> "\221¿\020\000\000\000\000\000\000\000\020\000\000"},
> sin = {sin_family = 57540,
> sin_port = 49041, sin_addr = {s_addr =
> 16}, sin_zero =
> "\000\000\000\000\020\000\000"}, sin6 =
> {sin6_family = 57540, sin6_port = 49041,
> sin6_flowinfo = 16, sin6_addr = {in6_u =
> {u6_addr8 =
>
"\000\000\000\000\020\000\000\000\001\000\000\000\001\000\000",
> u6_addr16 = {0, 0, 16, 0, 1, 0,
> 1, 0}, u6_addr32 = {0, 16, 1,
> 1}}}, sin6_scope_id = 12}}, id = -1080958468,
> proto = 4 '\004', send_flags = {f = 0 '\0',
> blst_imask = 0 '\0'}}
> c = 10 '\n'
> #17 0x08159a0f in handle_io (fm=0x83296b0,
> events=1, idx=-1) at tcp_read.c:1150
> ret = 4
> n = 4
> read_flags = 1
> con = (struct tcp_connection *) 0xb621af88
> s = 12
> resp = <value optimized out>
> t = <value optimized out>
> #18 0x0815d653 in tcp_receive_loop
> (unix_sock=35) at io_wait.h:1091
> No locals.
> ---Type <return> to continue, or q <return> to
> quit---
> #19 0x0812885b in tcp_init_children () at
> tcp_main.c:4819
> r = 3
> reader_fd_1 = 35
> pid = <value optimized out>
> si = <value optimized out>
> #20 0x080ad412 in main_loop () at main.c:1632
> i = 4
> pid = <value optimized out>
> si = (struct socket_info *) 0x0
> si_desc = "udp receiver child=3
> sock=192.168.166.31:5060
>
<http://192.168.166.31:5060>\000\221¿\002\000\000\000/\200\220\000\001\000\000\000\000\000H\000¨ä21¿\001\000\000\000°¶è026¶\000\000\000\000\000õ\030s2\b\002\000\000\000âf\b\000õ\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000ÿÿÿÿ¸ä21¿"
> #21 0x080afbe7 in main (argc=3,
> argv=dwarf2_read_address: Corrupted DWARF
> expression.
> ) at main.c:2398
> cfg_stream = (FILE *) 0x8a2d008
> c = <value optimized out>
> r = 0
> tmp = 0xbf91e674
>
"\212þ\221¿\233þ\221¿°þ\221¿»þ\221¿Ã\221¿ù1¿\020ÿ\221¿Dÿ\221¿Lÿ\221¿Wÿ\221¿]ÿ\221¿oÿ\221¿{ÿ\221¿\202ÿ\221
> tmp_len = 10492229
> port = 136192361
> proto = -1080957480
> ret = <value optimized out>
> seed = 21782655
> rfd = 4
> debug_save = <value optimized out>
> debug_flag = 0
> dont_fork_cnt = 0
> n_lst = <value optimized out>
> p = <value optimized out>
>
>
> Best Regards
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list
> sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012
-http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012
-http://asipto.com/u/kpw
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users