<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>I looked at the core files when you reported, but due to busy
times traveling, I didn't get the chance to follow up. However,
the core files are truncated, so no useful details there.</p>
<p>We will release 5.1.6 these days, it has few more bits that might
help troubleshooting here, if there will be another good core dump
file.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<br>
<div class="moz-cite-prefix">On 26.09.18 09:25, Jayesh Nambiar
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CANAYozsEA_ZG4dMMesbTkm0EvqjxseXpDwfKri_XzxZUC5quow@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<div dir="ltr">Hello Daniel,
<div><br>
<div>
<div class="gmail_quote">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Just a quick update on this. After running from
the 5.1 branch twice it happened that Server
became unresponsive. It never crashed and on
forceful shutdown it only generated a core with
shutdown routine. Here is what the syslog
contained off while the calls were being sent to
kamailio but kamailio was not processing any
calls:</div>
<div><br>
</div>
<div>
<div>Sep 22 16:09:04 triviamedia
/usr/local/kamailio/sbin/kamailio[16006]:
CRITICAL: <core>
[core/mem/q_malloc.c:502]: qm_free(): BUG:
freeing already freed pointer (0x7fcd23ce5648),
called from tm: h_table.c:
free_cell_helper(235), first free tm: h_table.c:
free_cell_helper(235) - aborting</div>
<div>Sep 22 16:09:04 triviamedia
/usr/local/kamailio/sbin/kamailio[16006]:
CRITICAL: <core>
[core/mem/q_malloc.c:502]: qm_free(): BUG:
freeing already freed pointer (0x7fcd24a38ea8),
called from tm: h_table.c:
free_cell_helper(234), first free tm: h_table.c:
free_cell_helper(234) - aborting</div>
<div>Sep 22 16:09:04 triviamedia
/usr/local/kamailio/sbin/kamailio[16006]:
CRITICAL: <core>
[core/mem/q_malloc.c:502]: qm_free(): BUG:
freeing already freed pointer (0x7fcd23ce5648),
called from tm: h_table.c:
free_cell_helper(235), first free tm: h_table.c:
free_cell_helper(235) - aborting</div>
<div>Sep 22 16:09:04 triviamedia
/usr/local/kamailio/sbin/kamailio[16006]:
CRITICAL: <core>
[core/mem/q_malloc.c:502]: qm_free(): BUG:
freeing already freed pointer (0x7fcd24a38ea8),
called from tm: h_table.c:
free_cell_helper(234), first free tm: h_table.c:
free_cell_helper(234) - aborting</div>
<div>Sep 22 16:09:04 triviamedia
/usr/local/kamailio/sbin/kamailio[16006]:
CRITICAL: <core>
[core/mem/q_malloc.c:502]: qm_free(): BUG:
freeing already freed pointer (0x7fcd23ce5648),
called from tm: h_table.c:
free_cell_helper(235), first free tm: h_table.c:
free_cell_helper(235) - aborting</div>
<div>Sep 22 16:09:04 triviamedia
/usr/local/kamailio/sbin/kamailio[16006]:
CRITICAL: <core>
[core/mem/q_malloc.c:502]: qm_free(): BUG:
freeing already freed pointer (0x7fcd24a38ea8),
called from tm: h_table.c:
free_cell_helper(234), first free tm: h_table.c:
free_cell_helper(234) - aborting</div>
<div>Sep 22 16:09:04 triviamedia
/usr/local/kamailio/sbin/kamailio[16006]:
CRITICAL: <core>
[core/mem/q_malloc.c:502]: qm_free(): BUG:
freeing already freed pointer (0x7fcd23ce5648),
called from tm: h_table.c:
free_cell_helper(235), first fre</div>
</div>
<div><br>
</div>
<div>And here's the "bt full" of the core file:</div>
<div>
<div>[New LWP 16003]</div>
<div>Cannot access memory at address
0x7fcd64e1a148</div>
<div>Cannot access memory at address
0x7fcd64e1a140</div>
<div>Failed to read a valid object file image from
memory.</div>
<div>Core was generated by
`/usr/local/kamailio/sbin/kamailio -P
/var/run/siptrunk.pid -f /usr/local/carrie'.</div>
<div>Program terminated with signal SIGABRT,
Aborted.</div>
<div>#0 0x00007fcd6443f428 in ?? ()</div>
<div>(gdb) bt full</div>
<div>#0 0x00007fcd6443f428 in ?? ()</div>
<div>No symbol table info available.</div>
<div>Backtrace stopped: Cannot access memory at
address 0x7ffed40e9e68</div>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Sat, Sep 15, 2018 at 7:09 PM Jayesh
Nambiar <<a href="mailto:jayesh1017@gmail.com"
target="_blank" moz-do-not-send="true">jayesh1017@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello Daniel,
<div>I have now git cloned from 5.1 branch and
restarted kamailio today. Also an observation;
that when one particular customer with around
40-50cps capacity starts his dialing is when the
kamailio kept crashing. I will observe in the
coming week and report of any crashes
accordingly.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, Sep 12, 2018 at 3:11 AM
Jayesh Nambiar <<a
href="mailto:jayesh1017@gmail.com"
target="_blank" moz-do-not-send="true">jayesh1017@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div dir="ltr">
<div>Thank you Daniel. I will deploy from 5.1
branch. I will be able to update it in next
two days. Since this is a production system,
I will have to have find a proper time slot
to get this updated. Will let you know once
that is done.</div>
<div>Thanks again.</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Sep 11, 2018 at 2:16 PM
Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com"
target="_blank" moz-do-not-send="true">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Jayesh,</p>
<p>I looked over the weekend and seems
like a race or a double free. I
backported some patches to tm module
from master to 5.1 branch (a counter to
see if there is more than one free).
Would you be able to deploy from branch
5.1 or nightly builds for 5.1? With the
next crash I can evaluate (rule out or
go for it) if it is double free or not.</p>
<p>I haven't written you yet, because I
planned to dig more into libev (which is
used by evapi), to see if it creates
some threads that can mess up with some
kamailio globals, although I expect it
is event loop in a single thread. Other
work related project took a lot of time
these days, so not there yet ...<br>
</p>
Cheers,<br>
Daniel<br>
<br>
<div
class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-cite-prefix">On
11.09.18 09:56, Jayesh Nambiar wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Daniel,
<div>Just a gentle follow up on this.
Could you find anything useful?</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
<br>
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF">
<blockquote type="cite">
<div class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div text="#000000"
bgcolor="#FFFFFF"><br>
<div
class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-cite-prefix">On
07.09.18 09:56,
Jayesh Nambiar
wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">Hi
Daniel,
<div>This
happens
randomly. It
is not a
testbed and
never
reproducible
on the test
server. I can
still give you
access if you
wish to
analyse the
core files to
check whats
happening here
exactly.</div>
<div>It simply
feels like on
heavy load the
evapi socket
gives up.
Earlier I was
running 5.0.2
and in that
kamailio would
get stuck by
not sending
any async
events on
evapi socket.
Not even the
evapi:connection-closed
event gets
triggered.
After I
upgraded to
latest stable
version
(5.1.5), it at
least started
crashing. Here
are two core
dumps put onto
pastebin.</div>
<div>
<div><a
href="https://pastebin.com/nn6gJapm"
target="_blank" moz-do-not-send="true">https://pastebin.com/nn6gJapm</a></div>
<div><a
href="https://pastebin.com/ph7b8vFH"
target="_blank" moz-do-not-send="true">https://pastebin.com/ph7b8vFH</a></div>
</div>
<div><br>
</div>
<div>Thanks for
all the
support,</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr">On
Fri, Sep 7, 2018
at 1:55 AM
Daniel-Constantin
Mierla <<a
href="mailto:miconda@gmail.com"
target="_blank" moz-do-not-send="true">miconda@gmail.com</a>> wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div
text="#000000"
bgcolor="#FFFFFF">
<p>Hello,</p>
<p>are you
able to
reproduce it
somehow? Or
just happens
randomly?</p>
<p>Is it on a
testbed where
I could get
access to
investigate
the corefiles?
If not, then I
will ask for
more details
from cores
over email,
first would be
'bt full' for
both cores and
'p *dead_cell'
for the second
one.<br>
</p>
Cheers,<br>
Daniel<br>
<br>
<div
class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-cite-prefix">On
05.09.18
17:47, Jayesh
Nambiar wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi
Daniel,
<div>Have got
these core
dumps. Let me
know if I
should be
doing a bt
full. I'll
pastebin and
send. Thanks,</div>
<div><br>
</div>
<div>
<div>Core was
generated by
`/usr/local/kamailio/sbin/kamailio
-P
/var/run/siptrunk.pid
-f
/usr/local/carrie'.</div>
<div>Program
terminated
with signal
SIGSEGV,
Segmentation
fault.</div>
<div>#0
0x0000000000505d30
in
sip_msg_shm_clone
(org_msg=0x7f6ab0d9f618, sip_msg_len=0x7ffdddb2e8bc, clone_lumps=1) at
core/sip_msg_clone.c:491</div>
<div>491<span style="white-space:pre-wrap"> </span>LUMP_LIST_LEN(len,
org_msg->add_rm);</div>
<div>(gdb) bt</div>
<div>#0
0x0000000000505d30
in
sip_msg_shm_clone
(org_msg=0x7f6ab0d9f618, sip_msg_len=0x7ffdddb2e8bc, clone_lumps=1) at
core/sip_msg_clone.c:491</div>
<div>#1
0x00007f6af0bdf68d
in fake_req
(shmem_msg=0x7f6ab0d9f618,
extra_flags=0,
uac=0x7f6ab1738980, len=0x7ffdddb2e8bc) at t_reply.c:854</div>
<div>#2
0x00007f6af0c3aa27
in
t_continue_helper
(hash_index=58039, label=413633661, rtact=0x7f6af10500f0, cbname=0x0,
cbparam=0x0)
at
t_suspend.c:293</div>
<div>#3
0x00007f6af0c3eed4
in t_continue
(hash_index=58039, label=413633661, route=0x7f6af10500f0) at
t_suspend.c:583</div>
<div>#4
0x00007f6aae4dd010
in
w_t_continue
(msg=0x7ffdddb2fa60,
idx=0x7f6af1098e90 "8\306\t\361j\177", lbl=0x7f6af1098ff0
"\240\264\t\361j\177",
rtn=0x7f6af1099150 "0\275\t\361j\177") at tmx_mod.c:760</div>
<div>#5
0x000000000045b477
in do_action
(h=0x7ffdddb2f850,
a=0x7f6af109ab38, msg=0x7ffdddb2fa60) at core/action.c:1085</div>
<div>#6
0x0000000000467fd5
in run_actions
(h=0x7ffdddb2f850, a=0x7f6af1096630, msg=0x7ffdddb2fa60) at
core/action.c:1565</div>
<div>#7
0x000000000045b234
in do_action
(h=0x7ffdddb2f850,
a=0x7f6af10a0f80, msg=0x7ffdddb2fa60) at core/action.c:1058</div>
<div>#8
0x0000000000467fd5
in run_actions
(h=0x7ffdddb2f850, a=0x7f6af10a0f80, msg=0x7ffdddb2fa60) at
core/action.c:1565</div>
<div>#9
0x0000000000468797
in
run_top_route
(a=0x7f6af10a0f80, msg=0x7ffdddb2fa60, c=0x0) at core/action.c:1654</div>
<div>#10
0x00007f6aabe79370
in
evapi_run_cfg_route
(evenv=0x7ffdddb30250, rt=3, rtname=0x7f6aac08cb18
<_evapi_rts+56>)
at
evapi_dispatch.c:161</div>
<div>#11
0x00007f6aabe7f271
in
evapi_recv_client
(loop=0x7f6aabe698e0, watcher=0x27af5e0, revents=1) at
evapi_dispatch.c:467</div>
<div>#12
0x00007f6aabc5fd73
in
ev_invoke_pending
() from
/usr/lib/x86_64-linux-gnu/libev.so.4</div>
<div>#13
0x00007f6aabc633de
in ev_run ()
from
/usr/lib/x86_64-linux-gnu/libev.so.4</div>
<div>#14
0x00007f6aabe7867c
in ev_loop
(loop=0x7f6aabe698e0,
flags=0) at
/usr/include/ev.h:835</div>
<div>#15
0x00007f6aabe83fc6
in
evapi_run_dispatcher
(laddr=0x7f6af0f72300 "127.0.0.1", lport=8060) at evapi_dispatch.c:705</div>
<div>#16
0x00007f6aabe6e262
in child_init
(rank=0) at
evapi_mod.c:213</div>
<div>#17
0x0000000000542cad
in
init_mod_child
(m=0x7f6af0f71b70, rank=0) at core/sr_module.c:943</div>
<div>#18
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f72968, rank=0) at core/sr_module.c:939</div>
<div>#19
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f73d38, rank=0) at core/sr_module.c:939</div>
<div>#20
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f74670, rank=0) at core/sr_module.c:939</div>
<div>#21
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f76708, rank=0) at core/sr_module.c:939</div>
<div>#22
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f76c08, rank=0) at core/sr_module.c:939</div>
<div>#23
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f770d0, rank=0) at core/sr_module.c:939</div>
<div>#24
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f77cf0, rank=0) at core/sr_module.c:939</div>
<div>#25
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f78808, rank=0) at core/sr_module.c:939</div>
<div>#26
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f78bd8, rank=0) at core/sr_module.c:939</div>
<div>#27
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f794c8, rank=0) at core/sr_module.c:939</div>
<div>#28
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f79920, rank=0) at core/sr_module.c:939</div>
<div>#29
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f7a330, rank=0) at core/sr_module.c:939</div>
<div>#30
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f7afd0, rank=0) at core/sr_module.c:939</div>
<div>#31
0x0000000000542971
in
init_mod_child
(m=0x7f6af0f7bc80, rank=0) at core/sr_module.c:939</div>
<div>#32
0x000000000054303d
in init_child
(rank=0) at
core/sr_module.c:970</div>
<div>#33
0x0000000000425399
in main_loop
() at
main.c:1701</div>
<div>#34
0x000000000042bd5c
in main
(argc=13,
argv=0x7ffdddb31088)
at main.c:2638</div>
</div>
<div><br>
</div>
<div>And this:</div>
<div>
<div>[New LWP
15804]</div>
<div>[Thread
debugging
using
libthread_db
enabled]</div>
<div>Using
host
libthread_db
library
"/lib/x86_64-linux-gnu/libthread_db.so.1".</div>
<div>Core was
generated by
`/usr/local/kamailio/sbin/kamailio
-P
/var/run/siptrunk.pid
-f
/usr/local/carrie'.</div>
<div>Program
terminated
with signal
SIGSEGV,
Segmentation
fault.</div>
<div>#0
0x00007f6af0b90b70
in
free_cell_helper
(dead_cell=0x7f6ab0a6baa8, silent=0, fname=0x7f6af0c8f630 "timer.c",
fline=654) at
h_table.c:230</div>
<div>230<span style="white-space:pre-wrap"> </span>foo
= tt->next;</div>
<div>(gdb) bt</div>
<div>#0
0x00007f6af0b90b70
in
free_cell_helper
(dead_cell=0x7f6ab0a6baa8, silent=0, fname=0x7f6af0c8f630 "timer.c",
fline=654) at
h_table.c:230</div>
<div>#1
0x00007f6af0c24409
in
wait_handler
(ti=932640643,
wait_tl=0x7f6ab0a6bb28, data=0x7f6ab0a6baa8) at timer.c:654</div>
<div>#2
0x00000000004bb445
in
timer_list_expire
(t=932640643,
h=0x7f6ab03ad158, slow_l=0x7f6ab03ae480, slow_mark=271) at
core/timer.c:874</div>
<div>#3
0x00000000004bb8ab
in
timer_handler
() at
core/timer.c:939</div>
<div>#4
0x00000000004bbd30
in timer_main
() at
core/timer.c:978</div>
<div>#5
0x00000000004250f9
in main_loop
() at
main.c:1691</div>
<div>#6
0x000000000042bd5c
in main
(argc=13,
argv=0x7ffdddb31088)
at main.c:2638</div>
</div>
</div>
</div>
</div>
<br>
<div
class="gmail_quote">
<div dir="ltr">On
Wed, Sep 5,
2018 at 3:13
PM
Daniel-Constantin
Mierla <<a
href="mailto:miconda@gmail.com" target="_blank" moz-do-not-send="true">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote
class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
text="#000000"
bgcolor="#FFFFFF">
<p>Hello,</p>
<p>the
backtrace
doesn't show
any hint about
kamailio, only
from
closelog() up.</p>
<p>It may be
the core
generated by
shutdown
procedure,
have you
enabled one
core file per
pid/process?
If not, do it
and reproduce
the issue
again, you may
get two core
files, one
being the
runtime issue
and the other
one from
shutdown
procedure,
which likely
is an effect
of the other
one. The one
from the
runtime is
more relevant.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<br>
<div
class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-cite-prefix">On
05.09.18
10:09, Jayesh
Nambiar wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div dir="ltr">Hello,
<div>I'm using
kamailio 5.1.5
with evapi. I
have a node.js
connecting
with kamailio
evapi to which
I send events
and also
consume events
based on which
I do the
routing. I
have 8 evapi
workers
defined in the
config. </div>
<div>The
problem is
that kamailio
randomly
crashes on
high load. I'm
assuming that
it is related
to the evapi
module as rest
of the config
is pretty
straight
forward. I
could get a
core file and
here's the
core dump:</div>
<div>
<div>[New LWP
14042]</div>
<div>[Thread
debugging
using
libthread_db
enabled]</div>
<div>Using
host
libthread_db
library
“/lib/x86_64-linux-gnu/libthread_db.so.1”.</div>
<div>Core was
generated by
`/usr/local/kamailio/sbin/kamailio
-P
/var/run/siptrunk.pid
-f
/usr/local/carrie’.</div>
<div>Program
terminated
with signal
SIGABRT,
Aborted.</div>
<div>#0
0x00007f9995283428
in __GI_raise
(sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54</div>
<div>54
../sysdeps/unix/sysv/linux/raise.c:
No such file
or directory.</div>
<div>(gdb) bt</div>
<div>#0
0x00007f9995283428
in __GI_raise
(sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54</div>
<div>#1
0x00007f999528502a
in __GI_abort
() at
abort.c:89</div>
<div>#2
0x000000000041a029
in
sig_alarm_abort
(signo=14) at
main.c:646</div>
<div>#3
<signal
handler
called></div>
<div>#4
0x00007f999534f497
in
__libc_cleanup_routine
(f=<optimized out>) at ../sysdeps/nptl/libc-lockP.h:291</div>
<div>#5
closelog () at
../misc/syslog.c:415</div>
<div>#6
0x0000000000000000
in ?? ()</div>
</div>
<div><br>
</div>
<div>Any help
in this
regards is
would allow me
to identify
the reason of
the crash.
Thanks for the
support.</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
</div>
<br>
<fieldset
class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank" moz-do-not-send="true">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank" moz-do-not-send="true">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank" moz-do-not-send="true">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank" moz-do-not-send="true">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank" moz-do-not-send="true">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a></pre>
</body>
</html>