[SR-Users] possible TCP deadlock (tls again?) // pike module not releasing IPs
Daniel-Constantin Mierla
miconda at gmail.com
Wed Jan 8 09:54:42 CET 2020
Was this a fresh installation of kamailio directly from the 52-nightly repo?
The jenkins build output shows that the flag was set during compilation:
*
https://kamailio.sipwise.com/view/kam52/job/kamailio52-nightly-binaries/lastBuild/architecture=amd64,distribution=bionic,label=slave/consoleText
The gcc command has the option -DKSR_PTHREAD_MUTEX_SHARED which sets in
the C code the TLS_PTHREAD_MUTEX_SHARED.
Cheers,
Daniel
On 07.01.20 20:33, Andrew Chen wrote:
> So I used the nightly build option and I don't see it.
>
> + sudo add-apt-repository deb http://deb.kamailio.org/kamailio52-nightly bionic main
> Hit:1 https://dl.yarnpkg.com/debian stable InRelease
> Hit:2 http://download.draios.com/stable/deb stable-amd64/ InRelease
> Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
> Hit:4 https://deb.nodesource.com/node_11.x bionic InRelease
> Ign:5 http://fuze-apt-bionic-master.s3-website-us-west-1.amazonaws.com <http://fuze-apt-bionic-master.s3-website-us-west-1.amazonaws.com/> bionic InRelease
> Get:6 http://fuze-apt-bionic-master.s3-website-us-west-1.amazonaws.com <http://fuze-apt-bionic-master.s3-website-us-west-1.amazonaws.com/> bionic Release [2346 B]
> Ign:7 http://fuze-apt-bionic-rc.s3-website-us-west-1.amazonaws.com <http://fuze-apt-bionic-rc.s3-website-us-west-1.amazonaws.com/> bionic InRelease
> Ign:8 http://fuze-apt-bionic-master.s3-website-us-west-1.amazonaws.com <http://fuze-apt-bionic-master.s3-website-us-west-1.amazonaws.com/> bionic Release.gpg
> Hit:9 http://ppa.launchpad.net/maxmind/ppa/ubuntu bionic InRelease
> Get:10 http://fuze-apt-bionic-rc.s3-website-us-west-1.amazonaws.com <http://fuze-apt-bionic-rc.s3-website-us-west-1.amazonaws.com/> bionic Release [2346 B]
> Hit:11 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic InRelease
> Get:12 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
> Get:13 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
> Get:14 http://fuze-apt-bionic-master.s3-website-us-west-1.amazonaws.com <http://fuze-apt-bionic-master.s3-website-us-west-1.amazonaws.com/> bionic/main amd64 Packages [327 kB]
> Ign:15 http://fuze-apt-bionic-rc.s3-website-us-west-1.amazonaws.com <http://fuze-apt-bionic-rc.s3-website-us-west-1.amazonaws.com/> bionic Release.gpg
> Hit:16 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic InRelease
> Get:17 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [605 kB]
> Get:18 http://fuze-apt-bionic-rc.s3-website-us-west-1.amazonaws.com <http://fuze-apt-bionic-rc.s3-website-us-west-1.amazonaws.com/> bionic/main amd64 Packages [233 kB]
> Get:19 http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1035 kB]
> Get:20 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [196 kB]
> Get:21 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [16.8 kB]
> Get:22 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [5028 B]
> Get:23 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [628 kB]
> Get:24 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [210 kB]
> Get:25 http://deb.kamailio.org/kamailio52-nightly bionic InRelease [4223 B]
> Get:26 http://deb.kamailio.org/kamailio52-nightly bionic/main amd64 Packages [14.8 kB]
> root at sjomainkama51:/usr/lib/x86_64-linux-gnu/kamailio/modules # kamailio -I
> Print out of kamailio internals
> Version: kamailio 5.2.5 (x86_64/linux)
> Default config: /etc/kamailio/kamailio.cfg
> Default paths to modules: /usr/lib/x86_64-linux-gnu/kamailio/modules
> Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
> MAX_RECV_BUFFER_SIZE=262144
> MAX_URI_SIZE=1024
> BUF_SIZE=65535
> DEFAULT PKG_SIZE=8MB
> DEFAULT SHM_SIZE=64MB
> ADAPTIVE_WAIT_LOOPS=1024
> TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select
> Source code revision ID: unknown
> Compiled with: gcc 7.4.0
> Compiled on:
> Thank you for flying kamailio!
>
> On Mon, Jan 6, 2020 at 2:52 PM Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> You can download the code from git repository and build the
> openssl_mutex_shared.so locally.
>
> Or install from the nightly builts, there should be the version
> with the fix embedded -- after installation check kamailio -I and
> see if it lists TLS_PTHREAD_MUTEX_SHARED.
>
> Cheers,
> Daniel
>
> On 06.01.20 16:48, Andrew Chen wrote:
>> I have seen similar issues with pike and running 5.2.5 installed
>> using deb repo. I don't see this openssl_mutex_shared directory:
>>
>> root at ashmainkama51:/usr/lib/x86_64-linux-gnu/kamailio # ls -lart
>> total 400
>> -rw-r--r-- 1 root root 22528 Oct 10 12:29 libtrie.so.1.0
>> lrwxrwxrwx 1 root root 14 Oct 10 12:29 libtrie.so.1 ->
>> libtrie.so.1.0
>> lrwxrwxrwx 1 root root 14 Oct 10 12:29 libtrie.so ->
>> libtrie.so.1.0
>> -rw-r--r-- 1 root root 63864 Oct 10 12:29 libsrutils.so.1.0
>> lrwxrwxrwx 1 root root 17 Oct 10 12:29 libsrutils.so.1 ->
>> libsrutils.so.1.0
>> lrwxrwxrwx 1 root root 17 Oct 10 12:29 libsrutils.so ->
>> libsrutils.so.1.0
>> -rw-r--r-- 1 root root 51472 Oct 10 12:29 libsrdb2.so.1.0
>> lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb2.so.1 ->
>> libsrdb2.so.1.0
>> lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb2.so ->
>> libsrdb2.so.1.0
>> -rw-r--r-- 1 root root 211264 Oct 10 12:29 libsrdb1.so.1.0
>> lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb1.so.1 ->
>> libsrdb1.so.1.0
>> lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb1.so ->
>> libsrdb1.so.1.0
>> drwxr-xr-x 4 root root 4096 Dec 3 21:32 .
>> drwxr-xr-x 3 root root 4096 Dec 3 21:33 kamctl
>> drwxr-xr-x 2 root root 4096 Dec 3 21:33 modules
>> drwxr-xr-x 47 root root 36864 Dec 10 20:13 ..
>> root at ashmainkama51:/usr/lib/x86_64-linux-gnu/kamailio #
>>
>> and here are my command outputs:
>>
>> root at ashmainkama51:~ # ldd
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
>> linux-vdso.so.1 (0x00007ffd28de0000)
>> libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> (0x00007f925de6d000)
>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
>> (0x00007f925da7c000)
>> libcrypto.so.1.1 =>
>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f925d5b1000)
>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
>> (0x00007f925d392000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007f925e39d000)
>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
>> (0x00007f925d18e000)
>> root at ashmainkama51:~ # kamailio -I
>> Print out of kamailio internals
>> Version: kamailio 5.2.5 (x86_64/linux)
>> Default config: /etc/kamailio/kamailio.cfg
>> Default paths to modules:
>> /usr/lib/x86_64-linux-gnu/kamailio/modules
>> Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP,
>> TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
>> SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
>> DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>> MAX_RECV_BUFFER_SIZE=262144
>> MAX_URI_SIZE=1024
>> BUF_SIZE=65535
>> DEFAULT PKG_SIZE=8MB
>> DEFAULT SHM_SIZE=64MB
>> ADAPTIVE_WAIT_LOOPS=1024
>> TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select
>> Source code revision ID: unknown
>> Compiled with: gcc 7.4.0
>> Compiled on:
>> Thank you for flying kamailio!
>>
>>
>> On Mon, Dec 16, 2019 at 12:51 PM Aymeric Moizard
>> <amoizard at gmail.com <mailto:amoizard at gmail.com>> wrote:
>>
>> Hi Daniel,
>>
>> The file openssl_mutex_shared.so is there and same
>> installation time than the other files!
>>
>> I have added
>>
>> Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so'
>>
>> To my kamailio.service and I have restarted!
>> I'm sure it will behave better now ;)
>>
>> Tks a lot for the help.
>> Regards,
>> Aymeric
>>
>>
>> Le lun. 16 déc. 2019 à 17:23, Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> a écrit :
>>
>> I pinged Victor to see if he can figure out what happens
>> within the deb building process that makes the libssl
>> mutex fix not enabled.
>>
>> The extra .so preload object should be still installed,
>> try to see if it is at:
>>
>> /usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so
>>
>> Cheers,
>> Daniel
>>
>> On 16.12.19 12:09, Aymeric Moizard wrote:
>>> Good catch!
>>>
>>> As I said in my first mail, I also add the issue with
>>> latest 5.2.X so I suppose the deb package has the same
>>> issue for 52X.
>>>
>>> Is the extra binary to load still there? I will check
>>> that as soon as I'm online...
>>>
>>> Tks a lot!
>>> Aymeric
>>>
>>> Le lun. 16 déc. 2019 à 11:16, Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> a écrit :
>>>
>>> Hello,
>>>
>>> for some reason the binary doesn't seem to have the
>>> libssl mutex fix, in my system with the libssl 1.1
>>> gives:
>>>
>>> # kamailio -I
>>> Print out of kamailio internals
>>> Version: kamailio 5.3.1 (x86_64/linux) f36ac2
>>> Default config:
>>> /tmp/kamailio-5.3/etc/kamailio/kamailio.cfg
>>> Default paths to modules:
>>> /tmp/kamailio-5.3/lib64/kamailio/modules
>>> Compile flags: USE_TCP, USE_TLS, USE_SCTP,
>>> TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
>>> DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
>>> F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
>>> HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
>>> MAX_RECV_BUFFER_SIZE=262144
>>> MAX_URI_SIZE=1024
>>> BUF_SIZE=65535
>>> DEFAULT PKG_SIZE=8MB
>>> DEFAULT SHM_SIZE=64MB
>>> ADAPTIVE_WAIT_LOOPS=1024
>>> TCP poll methods: poll, epoll_lt, epoll_et,
>>> sigio_rt, select
>>> Source code revision ID: f36ac2
>>> Compiled with: gcc 9.2.1
>>> Compiled architecture: x86_64
>>> Compiled on: 11:11:20 Dec 16 2019
>>> Thank you for flying kamailio!
>>>
>>> The important part above is the presence of
>>> TLS_PTHREAD_MUTEX_SHARED compile time flag in the
>>> output.
>>>
>>> Needs to be investigated why the dep packages have
>>> the kamailio binary without the libssl mutex fix
>>> enabled.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 16.12.19 09:22, Aymeric Moizard wrote:
>>>> Hi Daniel,
>>>>
>>>> Tks a lot for lookint at it.
>>>>
>>>> $ ldd /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
>>>> linux-vdso.so.1 (0x00007fff997dd000)
>>>> libssl.so.1.1 =>
>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>>>> (0x00007fe40b53c000)
>>>> libc.so.6 =>
>>>> /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe40b19d000)
>>>> libcrypto.so.1.1 =>
>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>> (0x00007fe40ad03000)
>>>> libdl.so.2 =>
>>>> /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe40aaff000)
>>>> libpthread.so.0 =>
>>>> /lib/x86_64-linux-gnu/libpthread.so.0
>>>> (0x00007fe40a8e2000)
>>>> /lib64/ld-linux-x86-64.so.2
>>>> (0x00007fe40ba4a000)
>>>>
>>>> $ /usr/sbin/kamailio -I
>>>> Print out of kamailio internals
>>>> Version: kamailio 5.3.1 (x86_64/linux)
>>>> Default config: /etc/kamailio/kamailio.cfg
>>>> Default paths to modules:
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules
>>>> Compile flags: USE_TCP, USE_TLS, USE_SCTP,
>>>> TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
>>>> DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
>>>> F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
>>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>>>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
>>>> HAVE_RESOLV_RES
>>>> MAX_RECV_BUFFER_SIZE=262144
>>>> MAX_URI_SIZE=1024
>>>> BUF_SIZE=65535
>>>> DEFAULT PKG_SIZE=8MB
>>>> DEFAULT SHM_SIZE=64MB
>>>> ADAPTIVE_WAIT_LOOPS=1024
>>>> TCP poll methods: poll, epoll_lt, epoll_et,
>>>> sigio_rt, select
>>>> Source code revision ID: unknown
>>>> Compiled with: gcc 6.3.0
>>>> Compiled architecture: x86_64
>>>> Compiled on:
>>>> Thank you for flying kamailio!
>>>>
>>>> Additional note:
>>>> I have tried to better understand the pike module
>>>> and after reading the "end" of the module
>>>> documentation,
>>>> I do better understand the "Tree of IP" and settings.
>>>>
>>>> The pike documentation, for each settins and
>>>> description, should refer to the section "Chapter
>>>> 3. Developer Guide",
>>>> otherwise, the parameters cannot be understood.
>>>> Also, it's not possible to understand, according to
>>>> me, the real time
>>>> for removing an IP from the tree (removing it 100%
>>>> or only last node of IP)
>>>>
>>>> Looking again at my statistics, I feel the first
>>>> graph is definitly showing an issue. This graph is
>>>> showing
>>>> "$stat(location-users)" and
>>>> "$stat(location-contacts)". During the 10 hours,
>>>> many users are banned, unregistred, etc..
>>>> so it is really not expected that the number of
>>>> registred users is maintained. From what I
>>>> understand, the fact
>>>> that the stats went down when deadlock dissapeared
>>>> obviouly means kamailio threads was in a bad state
>>>> for the
>>>> last 10 hours...
>>>>
>>>> https://www.antisip.com/sip-antisip-com-register/status2.htm
>>>>
>>>> If you need more information, let me know...
>>>> Regards
>>>> Aymeric
>>>>
>>>> Le lun. 16 déc. 2019 à 08:22, Daniel-Constantin
>>>> Mierla <miconda at gmail.com
>>>> <mailto:miconda at gmail.com>> a écrit :
>>>>
>>>> Hello,
>>>>
>>>> can you provide output of ldd for tls.so and
>>>> output of "kamailio -I" (that's an uppercase i)?
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 13.12.19 16:39, Aymeric Moizard wrote:
>>>>> Hi List,
>>>>>
>>>>> History:
>>>>> * In the past, I had deadlock which was, most
>>>>> probably, related to ssl1.1.
>>>>> We have discussed this issue, and a fix is
>>>>> supposed to workaround the issue that was
>>>>> detected.
>>>>> * With latest 5.2.X, I have experienced ONCE a
>>>>> similar behavior with TCP and TLS being mostly
>>>>> stuck. I have not been using this version
>>>>> much, but the fix was supposed to be in the
>>>>> core of kamailio.
>>>>>
>>>>> The status of the server this night:
>>>>> * I'm today running version: kamailio 5.3.1
>>>>> (x86_64/linux),
>>>>> * Installed on stretch using
>>>>> http://deb.kamailio.org/kamailio53 repository.
>>>>> * This versions use libssl1.1
>>>>> * A user reported that he can't connect with TCP
>>>>> * An average of 5000 IPs per 10 minutes are
>>>>> being banned by the pike module
>>>>> (could be twice the same)
>>>>> Yesterday/Today:
>>>>> * at the end of the outage, I had 2479 IP in
>>>>> my ipban htable. (which is equivalent to my
>>>>> statistics showing 2 bans/IP every 10 minutes
>>>>> = 5000)
>>>>> * looking at my logs, it appears that most
>>>>> (ALL?) ip being banned... are my regular users.
>>>>> * looking at my logs, I can't understand why
>>>>> pike would block them.
>>>>>
>>>>> This is a graph for statistics on my service
>>>>> for the last 24 hours:
>>>>> https://www.antisip.com/sip-antisip-com-register/status2.html
>>>>>
>>>>> Yesterday, at 22:18:39, kamailio started to
>>>>> BAN some IPs. 52 IPs were banned in a period
>>>>> of 10 minutes. I can confirm this from my logs.
>>>>>
>>>>> My pike configuration is this one:
>>>>>
>>>>> modparam("pike", "sampling_time_unit", 2)
>>>>> modparam("pike", "reqs_density_per_unit", 64)
>>>>> modparam("pike", "remove_latency", 4)
>>>>>
>>>>> When detecting the issue, this morning, I typed:
>>>>>
>>>>> $> sudo kamctl stats
>>>>> $> sudo kamcmd htable.dump ipban
>>>>> //FAILURE (answer too large...)
>>>>> $> sudo kamctl trap
>>>>>
>>>>> Then, I started an agent with TCP and it
>>>>> worked...???
>>>>> Then, a few seconds, may be a minute after:
>>>>>
>>>>> $> sudo kamcmd htable.dump ipban
>>>>> //SUCCESS and shows 2479 banned ip.
>>>>>
>>>>> and... everything is back to normal in a few
>>>>> minutes.
>>>>>
>>>>> I haven't restarted kamailio, and all
>>>>> statistics are as expected, as usual.
>>>>>
>>>>> Thus, it looks that " sudo kamctl trap" has
>>>>> triggered something. I already
>>>>> experienced a similar behavior -when testing
>>>>> my ssl1.1 deadlock last year-.
>>>>>
>>>>> 2 questions:
>>>>> 1/ I beleive my "pike" configuration should
>>>>> not ban users. Is my pike configuration wrong?
>>>>> As an example, pike has banned an IP sending
>>>>> one message/second. I believe my configuration
>>>>> should accept that?
>>>>>
>>>>> 2/ Could there still be a TLS issue with
>>>>> libssl1.1?
>>>>>
>>>>> This is the result of the "kamctl trap":
>>>>>
>>>>> https://sip.antisip.com/kamailio-pike-or-tls-issue-13-12-2019.kamctl-trap
>>>>>
>>>>> Sorry for the long story & hoping to find a
>>>>> long term solution or at least a workaround!
>>>>>
>>>>> Regards
>>>>> Aymeric
>>>>>
>>>>> --
>>>>> Antisip - http://www.antisip.com
>>>>>
>>>>> _______________________________________________
>>>>> Kamailio (SER) - Users Mailing List
>>>>> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>>> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>
>>>>
>>>>
>>>>
>>>> --
>>>> Antisip - http://www.antisip.com
>>>
>>> --
>>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>
>>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>
>>
>>
>>
>> --
>> Antisip - http://www.antisip.com
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> 415 516 5535 (M)
>> achen@ <mailto:achen at thinkingphones.com>fuze.com <http://fuze.com>
>>
>>
>> *Confidentiality Notice: The information contained in this e-mail
>> and any
>> attachments may be confidential. If you are not an intended
>> recipient, you
>> are hereby notified that any dissemination, distribution or
>> copying of this
>> e-mail is strictly prohibited. If you have received this e-mail
>> in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this
>> e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com <http://www.kamailioworld.com>
>
>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <mailto:achen at thinkingphones.com>fuze.com <http://fuze.com>
>
>
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of
> this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200108/92346327/attachment.html>
More information about the sr-users
mailing list