<div dir="ltr">Hello Daniel,<div><br><div><div class="gmail_quote"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Just a quick update on this. After running from the 5.1 branch twice it happened that Server became unresponsive. It never crashed and on forceful shutdown it only generated a core with shutdown routine. Here is what the syslog contained off while the calls were being sent to kamailio but kamailio was not processing any calls:</div><div><br></div><div><div>Sep 22 16:09:04 triviamedia /usr/local/kamailio/sbin/kamailio[16006]: CRITICAL: <core> [core/mem/q_malloc.c:502]: qm_free(): BUG: freeing already freed pointer (0x7fcd23ce5648), called from tm: h_table.c: free_cell_helper(235), first free tm: h_table.c: free_cell_helper(235) - aborting</div><div>Sep 22 16:09:04 triviamedia /usr/local/kamailio/sbin/kamailio[16006]: CRITICAL: <core> [core/mem/q_malloc.c:502]: qm_free(): BUG: freeing already freed pointer (0x7fcd24a38ea8), called from tm: h_table.c: free_cell_helper(234), first free tm: h_table.c: free_cell_helper(234) - aborting</div><div>Sep 22 16:09:04 triviamedia /usr/local/kamailio/sbin/kamailio[16006]: CRITICAL: <core> [core/mem/q_malloc.c:502]: qm_free(): BUG: freeing already freed pointer (0x7fcd23ce5648), called from tm: h_table.c: free_cell_helper(235), first free tm: h_table.c: free_cell_helper(235) - aborting</div><div>Sep 22 16:09:04 triviamedia /usr/local/kamailio/sbin/kamailio[16006]: CRITICAL: <core> [core/mem/q_malloc.c:502]: qm_free(): BUG: freeing already freed pointer (0x7fcd24a38ea8), called from tm: h_table.c: free_cell_helper(234), first free tm: h_table.c: free_cell_helper(234) - aborting</div><div>Sep 22 16:09:04 triviamedia /usr/local/kamailio/sbin/kamailio[16006]: CRITICAL: <core> [core/mem/q_malloc.c:502]: qm_free(): BUG: freeing already freed pointer (0x7fcd23ce5648), called from tm: h_table.c: free_cell_helper(235), first free tm: h_table.c: free_cell_helper(235) - aborting</div><div>Sep 22 16:09:04 triviamedia /usr/local/kamailio/sbin/kamailio[16006]: CRITICAL: <core> [core/mem/q_malloc.c:502]: qm_free(): BUG: freeing already freed pointer (0x7fcd24a38ea8), called from tm: h_table.c: free_cell_helper(234), first free tm: h_table.c: free_cell_helper(234) - aborting</div><div>Sep 22 16:09:04 triviamedia /usr/local/kamailio/sbin/kamailio[16006]: CRITICAL: <core> [core/mem/q_malloc.c:502]: qm_free(): BUG: freeing already freed pointer (0x7fcd23ce5648), called from tm: h_table.c: free_cell_helper(235), first fre</div></div><div><br></div><div>And here's the "bt full" of the core file:</div><div><div>[New LWP 16003]</div><div>Cannot access memory at address 0x7fcd64e1a148</div><div>Cannot access memory at address 0x7fcd64e1a140</div><div>Failed to read a valid object file image from memory.</div><div>Core was generated by `/usr/local/kamailio/sbin/kamailio -P /var/run/siptrunk.pid -f /usr/local/carrie'.</div><div>Program terminated with signal SIGABRT, Aborted.</div><div>#0 0x00007fcd6443f428 in ?? ()</div><div>(gdb) bt full</div><div>#0 0x00007fcd6443f428 in ?? ()</div><div>No symbol table info available.</div><div>Backtrace stopped: Cannot access memory at address 0x7ffed40e9e68</div></div><div><br></div><div>Thanks,</div><div><br></div><div>- Jayesh</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Sep 15, 2018 at 7:09 PM Jayesh Nambiar <<a href="mailto:jayesh1017@gmail.com" target="_blank">jayesh1017@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello Daniel,<div>I have now git cloned from 5.1 branch and restarted kamailio today. Also an observation; that when one particular customer with around 40-50cps capacity starts his dialing is when the kamailio kept crashing. I will observe in the coming week and report of any crashes accordingly.</div><div><br></div><div>Thanks,</div><div><br></div><div>- Jayesh</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Sep 12, 2018 at 3:11 AM Jayesh Nambiar <<a href="mailto:jayesh1017@gmail.com" target="_blank">jayesh1017@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Thank you Daniel. I will deploy from 5.1 branch. I will be able to update it in next two days. Since this is a production system, I will have to have find a proper time slot to get this updated. Will let you know once that is done.</div><div>Thanks again.</div><div><br></div><div>- Jayesh</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Sep 11, 2018 at 2:16 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi Jayesh,</p>
<p>I looked over the weekend and seems like a race or a double free.
I backported some patches to tm module from master to 5.1 branch
(a counter to see if there is more than one free). Would you be
able to deploy from branch 5.1 or nightly builds for 5.1? With the
next crash I can evaluate (rule out or go for it) if it is double
free or not.</p>
<p>I haven't written you yet, because I planned to dig more into
libev (which is used by evapi), to see if it creates some threads
that can mess up with some kamailio globals, although I expect it
is event loop in a single thread. Other work related project took
a lot of time these days, so not there yet ...<br>
</p>
Cheers,<br>
Daniel<br>
<br>
<div class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-cite-prefix">On 11.09.18 09:56, Jayesh Nambiar
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Daniel,
<div>Just a gentle follow up on this. Could you find anything
useful?</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><br>
<div class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-cite-prefix">On
07.09.18 09:56, Jayesh Nambiar wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Daniel,
<div>This happens randomly. It is not a
testbed and never reproducible on the
test server. I can still give you access
if you wish to analyse the core files to
check whats happening here exactly.</div>
<div>It simply feels like on heavy load
the evapi socket gives up. Earlier I was
running 5.0.2 and in that kamailio would
get stuck by not sending any async
events on evapi socket. Not even the
evapi:connection-closed event gets
triggered. After I upgraded to latest
stable version (5.1.5), it at least
started crashing. Here are two core
dumps put onto pastebin.</div>
<div>
<div><a href="https://pastebin.com/nn6gJapm" target="_blank">https://pastebin.com/nn6gJapm</a></div>
<div><a href="https://pastebin.com/ph7b8vFH" target="_blank">https://pastebin.com/ph7b8vFH</a></div>
</div>
<div><br>
</div>
<div>Thanks for all the support,</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Fri, Sep 7, 2018 at 1:55
AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>are you able to reproduce it somehow?
Or just happens randomly?</p>
<p>Is it on a testbed where I could get
access to investigate the corefiles?
If not, then I will ask for more
details from cores over email, first
would be 'bt full' for both cores and
'p *dead_cell' for the second one.<br>
</p>
Cheers,<br>
Daniel<br>
<br>
<div class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-cite-prefix">On
05.09.18 17:47, Jayesh Nambiar wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Daniel,
<div>Have got these core dumps.
Let me know if I should be
doing a bt full. I'll pastebin
and send. Thanks,</div>
<div><br>
</div>
<div>
<div>Core was generated by
`/usr/local/kamailio/sbin/kamailio
-P /var/run/siptrunk.pid -f
/usr/local/carrie'.</div>
<div>Program terminated with
signal SIGSEGV, Segmentation
fault.</div>
<div>#0 0x0000000000505d30 in
sip_msg_shm_clone
(org_msg=0x7f6ab0d9f618,
sip_msg_len=0x7ffdddb2e8bc,
clone_lumps=1) at
core/sip_msg_clone.c:491</div>
<div>491<span style="white-space:pre-wrap"> </span>LUMP_LIST_LEN(len,
org_msg->add_rm);</div>
<div>(gdb) bt</div>
<div>#0 0x0000000000505d30 in
sip_msg_shm_clone
(org_msg=0x7f6ab0d9f618,
sip_msg_len=0x7ffdddb2e8bc,
clone_lumps=1) at
core/sip_msg_clone.c:491</div>
<div>#1 0x00007f6af0bdf68d in
fake_req
(shmem_msg=0x7f6ab0d9f618,
extra_flags=0,
uac=0x7f6ab1738980,
len=0x7ffdddb2e8bc) at
t_reply.c:854</div>
<div>#2 0x00007f6af0c3aa27 in
t_continue_helper
(hash_index=58039,
label=413633661,
rtact=0x7f6af10500f0,
cbname=0x0, cbparam=0x0) at
t_suspend.c:293</div>
<div>#3 0x00007f6af0c3eed4 in
t_continue
(hash_index=58039,
label=413633661,
route=0x7f6af10500f0) at
t_suspend.c:583</div>
<div>#4 0x00007f6aae4dd010 in
w_t_continue
(msg=0x7ffdddb2fa60,
idx=0x7f6af1098e90
"8\306\t\361j\177",
lbl=0x7f6af1098ff0
"\240\264\t\361j\177",
rtn=0x7f6af1099150
"0\275\t\361j\177") at
tmx_mod.c:760</div>
<div>#5 0x000000000045b477 in
do_action (h=0x7ffdddb2f850,
a=0x7f6af109ab38,
msg=0x7ffdddb2fa60) at
core/action.c:1085</div>
<div>#6 0x0000000000467fd5 in
run_actions
(h=0x7ffdddb2f850,
a=0x7f6af1096630,
msg=0x7ffdddb2fa60) at
core/action.c:1565</div>
<div>#7 0x000000000045b234 in
do_action (h=0x7ffdddb2f850,
a=0x7f6af10a0f80,
msg=0x7ffdddb2fa60) at
core/action.c:1058</div>
<div>#8 0x0000000000467fd5 in
run_actions
(h=0x7ffdddb2f850,
a=0x7f6af10a0f80,
msg=0x7ffdddb2fa60) at
core/action.c:1565</div>
<div>#9 0x0000000000468797 in
run_top_route
(a=0x7f6af10a0f80,
msg=0x7ffdddb2fa60, c=0x0)
at core/action.c:1654</div>
<div>#10 0x00007f6aabe79370 in
evapi_run_cfg_route
(evenv=0x7ffdddb30250, rt=3,
rtname=0x7f6aac08cb18
<_evapi_rts+56>) at
evapi_dispatch.c:161</div>
<div>#11 0x00007f6aabe7f271 in
evapi_recv_client
(loop=0x7f6aabe698e0,
watcher=0x27af5e0,
revents=1) at
evapi_dispatch.c:467</div>
<div>#12 0x00007f6aabc5fd73 in
ev_invoke_pending () from
/usr/lib/x86_64-linux-gnu/libev.so.4</div>
<div>#13 0x00007f6aabc633de in
ev_run () from
/usr/lib/x86_64-linux-gnu/libev.so.4</div>
<div>#14 0x00007f6aabe7867c in
ev_loop
(loop=0x7f6aabe698e0,
flags=0) at
/usr/include/ev.h:835</div>
<div>#15 0x00007f6aabe83fc6 in
evapi_run_dispatcher
(laddr=0x7f6af0f72300
"127.0.0.1", lport=8060) at
evapi_dispatch.c:705</div>
<div>#16 0x00007f6aabe6e262 in
child_init (rank=0) at
evapi_mod.c:213</div>
<div>#17 0x0000000000542cad in
init_mod_child
(m=0x7f6af0f71b70, rank=0)
at core/sr_module.c:943</div>
<div>#18 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f72968, rank=0)
at core/sr_module.c:939</div>
<div>#19 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f73d38, rank=0)
at core/sr_module.c:939</div>
<div>#20 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f74670, rank=0)
at core/sr_module.c:939</div>
<div>#21 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f76708, rank=0)
at core/sr_module.c:939</div>
<div>#22 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f76c08, rank=0)
at core/sr_module.c:939</div>
<div>#23 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f770d0, rank=0)
at core/sr_module.c:939</div>
<div>#24 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f77cf0, rank=0)
at core/sr_module.c:939</div>
<div>#25 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f78808, rank=0)
at core/sr_module.c:939</div>
<div>#26 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f78bd8, rank=0)
at core/sr_module.c:939</div>
<div>#27 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f794c8, rank=0)
at core/sr_module.c:939</div>
<div>#28 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f79920, rank=0)
at core/sr_module.c:939</div>
<div>#29 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f7a330, rank=0)
at core/sr_module.c:939</div>
<div>#30 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f7afd0, rank=0)
at core/sr_module.c:939</div>
<div>#31 0x0000000000542971 in
init_mod_child
(m=0x7f6af0f7bc80, rank=0)
at core/sr_module.c:939</div>
<div>#32 0x000000000054303d in
init_child (rank=0) at
core/sr_module.c:970</div>
<div>#33 0x0000000000425399 in
main_loop () at main.c:1701</div>
<div>#34 0x000000000042bd5c in
main (argc=13,
argv=0x7ffdddb31088) at
main.c:2638</div>
</div>
<div><br>
</div>
<div>And this:</div>
<div>
<div>[New LWP 15804]</div>
<div>[Thread debugging using
libthread_db enabled]</div>
<div>Using host libthread_db
library
"/lib/x86_64-linux-gnu/libthread_db.so.1".</div>
<div>Core was generated by
`/usr/local/kamailio/sbin/kamailio
-P /var/run/siptrunk.pid -f
/usr/local/carrie'.</div>
<div>Program terminated with
signal SIGSEGV, Segmentation
fault.</div>
<div>#0 0x00007f6af0b90b70 in
free_cell_helper
(dead_cell=0x7f6ab0a6baa8,
silent=0,
fname=0x7f6af0c8f630
"timer.c", fline=654) at
h_table.c:230</div>
<div>230<span style="white-space:pre-wrap"> </span>foo
= tt->next;</div>
<div>(gdb) bt</div>
<div>#0 0x00007f6af0b90b70 in
free_cell_helper
(dead_cell=0x7f6ab0a6baa8,
silent=0,
fname=0x7f6af0c8f630
"timer.c", fline=654) at
h_table.c:230</div>
<div>#1 0x00007f6af0c24409 in
wait_handler (ti=932640643,
wait_tl=0x7f6ab0a6bb28,
data=0x7f6ab0a6baa8) at
timer.c:654</div>
<div>#2 0x00000000004bb445 in
timer_list_expire
(t=932640643,
h=0x7f6ab03ad158,
slow_l=0x7f6ab03ae480,
slow_mark=271) at
core/timer.c:874</div>
<div>#3 0x00000000004bb8ab in
timer_handler () at
core/timer.c:939</div>
<div>#4 0x00000000004bbd30 in
timer_main () at
core/timer.c:978</div>
<div>#5 0x00000000004250f9 in
main_loop () at main.c:1691</div>
<div>#6 0x000000000042bd5c in
main (argc=13,
argv=0x7ffdddb31088) at
main.c:2638</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Wed, Sep 5, 2018
at 3:13 PM Daniel-Constantin
Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>the backtrace doesn't show
any hint about kamailio, only
from closelog() up.</p>
<p>It may be the core generated
by shutdown procedure, have
you enabled one core file per
pid/process? If not, do it and
reproduce the issue again, you
may get two core files, one
being the runtime issue and
the other one from shutdown
procedure, which likely is an
effect of the other one. The
one from the runtime is more
relevant.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<br>
<div class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-cite-prefix">On
05.09.18 10:09, Jayesh Nambiar
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hello,
<div>I'm using kamailio
5.1.5 with evapi. I have
a node.js connecting
with kamailio evapi to
which I send events and
also consume events
based on which I do the
routing. I have 8 evapi
workers defined in the
config. </div>
<div>The problem is that
kamailio randomly
crashes on high load.
I'm assuming that it is
related to the evapi
module as rest of the
config is pretty
straight forward. I
could get a core file
and here's the core
dump:</div>
<div>
<div>[New LWP 14042]</div>
<div>[Thread debugging
using libthread_db
enabled]</div>
<div>Using host
libthread_db library
“/lib/x86_64-linux-gnu/libthread_db.so.1”.</div>
<div>Core was generated
by
`/usr/local/kamailio/sbin/kamailio
-P
/var/run/siptrunk.pid
-f /usr/local/carrie’.</div>
<div>Program terminated
with signal SIGABRT,
Aborted.</div>
<div>#0
0x00007f9995283428 in
__GI_raise
(sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54</div>
<div>54
../sysdeps/unix/sysv/linux/raise.c:
No such file or
directory.</div>
<div>(gdb) bt</div>
<div>#0
0x00007f9995283428 in
__GI_raise
(sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54</div>
<div>#1
0x00007f999528502a in
__GI_abort () at
abort.c:89</div>
<div>#2
0x000000000041a029 in
sig_alarm_abort
(signo=14) at
main.c:646</div>
<div>#3 <signal
handler called></div>
<div>#4
0x00007f999534f497 in
__libc_cleanup_routine
(f=<optimized
out>) at
../sysdeps/nptl/libc-lockP.h:291</div>
<div>#5 closelog () at
../misc/syslog.c:415</div>
<div>#6
0x0000000000000000 in
?? ()</div>
</div>
<div><br>
</div>
<div>Any help in this
regards is would allow
me to identify the
reason of the crash.
Thanks for the support.</div>
<div><br>
</div>
<div>- Jayesh</div>
</div>
</div>
<br>
<fieldset class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804m_-2303411157393819478moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113m_-7197243762512054804moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755m_-4496335948301825113moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934m_-7809388338774548581m_888952816927642755moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
<br>
<pre class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin -- <a class="m_-378099214799828651m_-5523242438671112591m_4312059096804799736m_-8466257967729247934moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
</div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div>
</div></div></div></div>