[SR-Users] Info: initialization of libssl 1.1.+ with PTHREAD_PROCESS_SHARED attribute for pthread mutex and rwlock

Daniel-Constantin Mierla miconda at gmail.com
Thu Sep 19 17:03:43 CEST 2019


You do not need to reproduce if you still have the core files and
installed the debug symbols for your version, just run again gdb on the
core and get the backtraces.

Cheers,
Daniel

On 18.09.19 22:35, Andrew Chen wrote:
> Yup understood.  I'll provide the bt when we can reproduce this.
>
> Thanks
>
> On Wed, Sep 18, 2019 at 4:32 PM Henning Westerholt <hw at skalatan.de
> <mailto:hw at skalatan.de>> wrote:
>
>     Hi Andrew,
>
>     the 5.2.4 does not contain the 5.3-pre1 workaround yet.
>
>     So if you test the workaround you need to install the lastest 5.3
>     from the git master debian repository.
>
>     But nevertheless it would be good to look at the backtrace, while
>     you have it.
>
>     Cheers,
>
>     Henning
>
>     Am 18.09.19 um 22:10 schrieb Joel Serrano:
>>     Hi Andrew, 
>>
>>     Download from
>>     here: http://deb.kamailio.org/kamailio52/pool/main/k/kamailio/
>>
>>     You have kamailio-dbg v5.2.4 (latest 5.2) for all distros...
>>
>>     Example for
>>     amd64+stretch: http://deb.kamailio.org/kamailio52/pool/main/k/kamailio/kamailio-dbg_5.2.4%2Bstretch_amd64.deb
>>
>>     Cheers, 
>>     Joel.
>>
>>
>>
>>
>>
>>     On Wed, Sep 18, 2019 at 12:16 PM Andrew Chen <achen at fuze.com
>>     <mailto:achen at fuze.com>> wrote:
>>
>>         Hi Henning,
>>
>>         I'm running 5.2.4.  Will this deb package work?
>>
>>         On Wed, Sep 18, 2019 at 2:39 PM Andrew Chen <achen at fuze.com
>>         <mailto:achen at fuze.com>> wrote:
>>
>>             Thanks Henning
>>
>>             On Wed, Sep 18, 2019 at 2:13 PM Henning Westerholt
>>             <hw at skalatan.de <mailto:hw at skalatan.de>> wrote:
>>
>>                 Hello Andrew,
>>
>>                 strange - it should be there, as it is available on
>>                 deb.kamailio.org <http://deb.kamailio.org>.
>>
>>                 Here e.g. the one for 64bit stretch:
>>
>>                 https://deb.kamailio.org/kamailiodev-nightly/pool/main/k/kamailio/kamailio-dbg_5.3.0~pre1%2B0~20190918005413.1490%2Bstretch_amd64.deb
>>
>>                 Cheers,
>>
>>                 Henning
>>
>>                 Am 18.09.19 um 18:52 schrieb Andrew Chen:
>>>                 I don't see one for debug:
>>>
>>>                 root at ashintgkama51:~ # apt-cache search kamailio
>>>
>>>                 resource-agents - Cluster Resource Agents
>>>
>>>                 kamailio - very fast, dynamic and configurable SIP
>>>                 server
>>>
>>>                 kamailio-autheph-modules - authentication using
>>>                 ephemeral credentials for the Kamailio SIP server
>>>
>>>                 kamailio-berkeley-bin - Berkeley database module for
>>>                 Kamailio - helper program
>>>
>>>                 kamailio-berkeley-modules - Berkeley database module
>>>                 for the Kamailio SIP server
>>>
>>>                 kamailio-carrierroute-modules - carrierroute module
>>>                 for the Kamailio SIP server
>>>
>>>                 kamailio-cnxcc-modules - cnxcc modules for the
>>>                 Kamailio SIP server
>>>
>>>                 kamailio-cpl-modules - CPL module (CPL interpreter
>>>                 engine) for the Kamailio SIP server
>>>
>>>                 kamailio-erlang-modules - Erlang modules for the
>>>                 Kamailio SIP server
>>>
>>>                 kamailio-geoip-modules - geoip module for the
>>>                 Kamailio SIP Server
>>>
>>>                 kamailio-geoip2-modules - geoip2 module for the
>>>                 Kamailio SIP Server
>>>
>>>                 kamailio-ims-modules - IMS module for the Kamailio
>>>                 SIP server
>>>
>>>                 kamailio-json-modules - Json parser and Json-RPC
>>>                 modules for the Kamailio SIP server
>>>
>>>
>>>                 On Wed, Sep 18, 2019 at 11:26 AM Daniel-Constantin
>>>                 Mierla <miconda at gmail.com
>>>                 <mailto:miconda at gmail.com>> wrote:
>>>
>>>                     You have to install the debug symbols package,
>>>                     otherwise the backtrace doesn't show any useful
>>>                     information. It just require to install that
>>>                     package and then get again the backtraces. The
>>>                     package name should be kamailio-dbg.
>>>
>>>                     Cheers,
>>>                     Daniel
>>>
>>>                     On 18.09.19 16:37, Andrew Chen wrote:
>>>>                     Sorry..forgot bt full output
>>>>
>>>>                     (gdb) bt full
>>>>
>>>>                     #0  0x0000562fcc3544c0 in ?? ()
>>>>
>>>>                     No symbol table info available.
>>>>
>>>>                     #1  0x0000562fcc354632 in ?? ()
>>>>
>>>>                     No symbol table info available.
>>>>
>>>>                     #2  0x0000562fcc354974 in local_timer_run ()
>>>>
>>>>                     No symbol table info available.
>>>>
>>>>                     #3  0x0000562fcc3fb256 in ?? ()
>>>>
>>>>                     No symbol table info available.
>>>>
>>>>                     #4  0x0000562fcc3fb64c in tcp_receive_loop ()
>>>>
>>>>                     No symbol table info available.
>>>>
>>>>                     #5  0x0000562fcc2acf52 in tcp_init_children ()
>>>>
>>>>                     No symbol table info available.
>>>>
>>>>                     #6  0x0000562fcc1ad94a in main_loop ()
>>>>
>>>>                     No symbol table info available.
>>>>
>>>>                     #7  0x0000562fcc1b4a7b in main ()
>>>>
>>>>                     No symbol table info available.
>>>>
>>>>                     (gdb) 
>>>>
>>>>
>>>>                     On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen
>>>>                     <achen at fuze.com <mailto:achen at fuze.com>> wrote:
>>>>
>>>>                         backtrace output:
>>>>
>>>>
>>>>                         (gdb) bt
>>>>
>>>>                         #0  0x0000562fcc3544c0 in ?? ()
>>>>
>>>>                         #1  0x0000562fcc354632 in ?? ()
>>>>
>>>>                         #2  0x0000562fcc354974 in local_timer_run ()
>>>>
>>>>                         #3  0x0000562fcc3fb256 in ?? ()
>>>>
>>>>                         #4  0x0000562fcc3fb64c in tcp_receive_loop ()
>>>>
>>>>                         #5  0x0000562fcc2acf52 in tcp_init_children ()
>>>>
>>>>                         #6  0x0000562fcc1ad94a in main_loop ()
>>>>
>>>>                         #7  0x0000562fcc1b4a7b in main ()
>>>>
>>>>                         (gdb) 
>>>>
>>>>
>>>>
>>>>
>>>>                         On Wed, Sep 18, 2019 at 10:01 AM Andrew
>>>>                         Chen <achen at fuze.com
>>>>                         <mailto:achen at fuze.com>> wrote:
>>>>
>>>>                             I'll look for the core.
>>>>
>>>>                             As part of our release process, I don't
>>>>                             think we can release master branch of
>>>>                             Kamailio.
>>>>
>>>>                             On Wed, Sep 18, 2019 at 9:54 AM
>>>>                             Daniel-Constantin Mierla
>>>>                             <miconda at gmail.com
>>>>                             <mailto:miconda at gmail.com>> wrote:
>>>>
>>>>                                 Hello,
>>>>
>>>>                                 did you get core files? If yes, it
>>>>                                 would be good to see the gdb
>>>>                                 bracktraces.
>>>>
>>>>                                 Have you used the pre-loaded
>>>>                                 library workaround?
>>>>
>>>>                                 The code was not backported to
>>>>                                 branch 5.2 yet, I wanted to wait a
>>>>                                 bit and see if there are reports
>>>>                                 with good/bad results. If you want
>>>>                                 to use it, then you have to install
>>>>                                 nightly builds from master branch
>>>>                                 (development version, should be
>>>>                                 5.3.0-pre1 at this moment).
>>>>
>>>>                                 Cheers,
>>>>                                 Daniel
>>>>
>>>>                                 On 18.09.19 15:26, Andrew Chen wrote:
>>>>>                                 Hi Daniel,
>>>>>                                 Which apt repository can we get
>>>>>                                 this workaround?   Today I noticed
>>>>>                                 our Kamailio 5.2.4 cored during
>>>>>                                 one of our load test sessions:
>>>>>
>>>>>                                 Wed Sep 18 09:31:14 2019]
>>>>>                                 kamailio[26997]: segfault at
>>>>>                                 7f8501366000 ip 00007f8501e58585
>>>>>                                 sp 00007fffd5c84248 error 4 in
>>>>>                                 libcrypto.so.1.1[7f8501dd4000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 10:12:29 2019]
>>>>>                                 kamailio[27110]: segfault at
>>>>>                                 7f1a8eea1000 ip 00007f1a8f993585
>>>>>                                 sp 00007ffddc8848c8 error 4 in
>>>>>                                 libcrypto.so.1.1[7f1a8f90f000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 10:12:43 2019]
>>>>>                                 kamailio[11060]: segfault at 20 ip
>>>>>                                 00007f0a85dac128 sp
>>>>>                                 00007fff6847aad0 error 4 in
>>>>>                                 libcrypto.so.1.1[7f0a85c40000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 10:12:44 2019]
>>>>>                                 kamailio[11059]: segfault at f0 ip
>>>>>                                 00007f0a85cc43ea sp
>>>>>                                 00007fff6847a9e8 error 4 in
>>>>>                                 libcrypto.so.1.1[7f0a85c40000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 10:13:02 2019]
>>>>>                                 kamailio[11187]: segfault at
>>>>>                                 7f75e06c8000 ip 00007f75e11ba585
>>>>>                                 sp 00007ffe29b47c88 error 4 in
>>>>>                                 libcrypto.so.1.1[7f75e1136000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 10:13:17 2019]
>>>>>                                 kamailio[11445]: segfault at
>>>>>                                 7f0f3743f000 ip 00007f0f37f31585
>>>>>                                 sp 00007ffde54daa38 error 4 in
>>>>>                                 libcrypto.so.1.1[7f0f37ead000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 10:13:36 2019]
>>>>>                                 kamailio[11644]: segfault at
>>>>>                                 7f804309c000 ip 00007f8043b8e585
>>>>>                                 sp 00007ffe89fe8228 error 4 in
>>>>>                                 libcrypto.so.1.1[7f8043b0a000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 10:13:52 2019]
>>>>>                                 kamailio[11799]: segfault at
>>>>>                                 7ff59c7e2000 ip 00007ff59d2d4585
>>>>>                                 sp 00007ffd69d76bb8 error 4 in
>>>>>                                 libcrypto.so.1.1[7ff59d250000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 10:13:52 2019]
>>>>>                                 kamailio[11792]: segfault at
>>>>>                                 7ff59c7e2000 ip 00007ff59d2d4585
>>>>>                                 sp 00007ffd69d76bb8 error 4 in
>>>>>                                 libcrypto.so.1.1[7ff59d250000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 12:48:01 2019]
>>>>>                                 kamailio[12066]: segfault at
>>>>>                                 7f22dca6d000 ip 00007f22dd55f585
>>>>>                                 sp 00007fffbc766ab8 error 4 in
>>>>>                                 libcrypto.so.1.1[7f22dd4db000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 12:48:14 2019]
>>>>>                                 kamailio[16344]: segfault at
>>>>>                                 7f7cca42d000 ip 00007f7ccaf1f585
>>>>>                                 sp 00007ffe99f16108 error 4 in
>>>>>                                 libcrypto.so.1.1[7f7ccae9b000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 12:48:32 2019]
>>>>>                                 kamailio[16708]: segfault at
>>>>>                                 7fb309141000 ip 00007fb309c33585
>>>>>                                 sp 00007ffd07a5f3c8 error 4 in
>>>>>                                 libcrypto.so.1.1[7fb309baf000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 12:48:49 2019]
>>>>>                                 kamailio[16824]: segfault at f0 ip
>>>>>                                 00007fc59095b3ea sp
>>>>>                                 00007ffe58a490b8 error 4 in
>>>>>                                 libcrypto.so.1.1[7fc5908d7000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:04 2019]
>>>>>                                 kamailio[16956]: segfault at
>>>>>                                 7fcc22e58000 ip 00007fcc2394a585
>>>>>                                 sp 00007fffa9c52258 error 4 in
>>>>>                                 libcrypto.so.1.1[7fcc238c6000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:04 2019]
>>>>>                                 kamailio[16945]: segfault at f0 ip
>>>>>                                 00007fcc2394a3ea sp
>>>>>                                 00007fffa9c52128 error 4 in
>>>>>                                 libcrypto.so.1.1[7fcc238c6000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:16 2019]
>>>>>                                 kamailio[17244]: segfault at
>>>>>                                 7f586ed20000 ip 00007f586f812585
>>>>>                                 sp 00007ffeae1adcd8 error 4 in
>>>>>                                 libcrypto.so.1.1[7f586f78e000+29b000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:33 2019]
>>>>>                                 kamailio[17431]: segfault at 8 ip
>>>>>                                 0000562fcc44c5d5 sp
>>>>>                                 00007ffe825c9dd0 error 4 in
>>>>>                                 kamailio[562fcc17c000+4a2000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:33 2019]
>>>>>                                 kamailio[17386]: segfault at 10 ip
>>>>>                                 00007f96c3735c3e sp
>>>>>                                 00007ffe825ca270 error 4 in
>>>>>                                 dialog.so[7f96c36d6000+be000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:33 2019]
>>>>>                                 kamailio[17423]: segfault at 0 ip
>>>>>                                 0000562fcc3544c0 sp
>>>>>                                 00007ffe825ca1f0 error 6 in
>>>>>                                 kamailio[562fcc17c000+4a2000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:33 2019]
>>>>>                                 kamailio[17378]: segfault at 0 ip
>>>>>                                 00007f96c5dd099f sp
>>>>>                                 00007ffe825c98e0 error 4 in
>>>>>                                 tm.so[7f96c5da5000+13f000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:33 2019]
>>>>>                                 kamailio[17417]: segfault at 0 ip
>>>>>                                 0000562fcc3544c0 sp
>>>>>                                 00007ffe825ca1f0 error 6 in
>>>>>                                 kamailio[562fcc17c000+4a2000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:33 2019]
>>>>>                                 kamailio[17416]: segfault at 0 ip
>>>>>                                 0000562fcc3544c0 sp
>>>>>                                 00007ffe825ca1f0 error 6 in
>>>>>                                 kamailio[562fcc17c000+4a2000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:33 2019]
>>>>>                                 kamailio[17380]: segfault at 0 ip
>>>>>                                 00007f96c5dd099f sp
>>>>>                                 00007ffe825c98e0 error 4 in
>>>>>                                 tm.so[7f96c5da5000+13f000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:33 2019]
>>>>>                                 kamailio[17381]: segfault at 0 ip
>>>>>                                 00007f96c5dd099f sp
>>>>>                                 00007ffe825c98e0 error 4 in
>>>>>                                 tm.so[7f96c5da5000+13f000]
>>>>>
>>>>>                                 [Wed Sep 18 12:49:34 2019]
>>>>>                                 kamailio[17387]: segfault at 8 ip
>>>>>                                 0000562fcc25d1f3 sp
>>>>>                                 00007ffe825ca370 error 6 in
>>>>>                                 kamailio[562fcc17c000+4a2000]
>>>>>
>>>>>
>>>>>                                 and I was wondering if this core
>>>>>                                 is related.
>>>>>
>>>>>
>>>>>                                 Thanks
>>>>>
>>>>>
>>>>>                                 On Fri, Sep 13, 2019 at 5:18 PM
>>>>>                                 Alex Balashov
>>>>>                                 <abalashov at evaristesys.com
>>>>>                                 <mailto:abalashov at evaristesys.com>>
>>>>>                                 wrote:
>>>>>
>>>>>                                     Hi Daniel,
>>>>>
>>>>>                                     Thanks a lot for this work;
>>>>>                                     it's definitely important that
>>>>>                                     Kamailio
>>>>>                                     build against OpenSSL v1.1,
>>>>>                                     and I'll be happy to
>>>>>                                     contribute some testing
>>>>>                                     in a live environment in the
>>>>>                                     coming weeks. Much appreciated.
>>>>>
>>>>>                                     Have a good weekend,
>>>>>
>>>>>                                     -- Alex
>>>>>
>>>>>                                     On Fri, Sep 13, 2019 at
>>>>>                                     01:34:12PM +0200,
>>>>>                                     Daniel-Constantin Mierla wrote:
>>>>>
>>>>>                                     > Hello,
>>>>>                                     > after couple of months
>>>>>                                     waiting for feedback about the
>>>>>                                     workaround with the
>>>>>                                     > preloaded shared library to
>>>>>                                     set the PTHREAD_PROCESS_SHARED
>>>>>                                     attribute for
>>>>>                                     > pthread mutex/rwlock in
>>>>>                                     order to avoid deadlocks and
>>>>>                                     as it seems to be the
>>>>>                                     > solution, I just pushed a
>>>>>                                     few commits to move that code
>>>>>                                     in the core of
>>>>>                                     > kamailio.
>>>>>                                     >
>>>>>                                     > It this way the
>>>>>                                     initialization with the
>>>>>                                     PTHREAD_PROCESS_SHARED
>>>>>                                     attribute is
>>>>>                                     > done by default when libssl
>>>>>                                     v1.1 is detected at compile
>>>>>                                     time, without the
>>>>>                                     > need of using the approach
>>>>>                                     with a preloaded library.
>>>>>                                     >
>>>>>                                     > The shared library is still
>>>>>                                     kept in the tls module:
>>>>>                                     >   *
>>>>>                                     >
>>>>>                                     https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared
>>>>>                                     >
>>>>>                                     > Probably is going to be
>>>>>                                     removed in the near future.
>>>>>                                     >
>>>>>                                     > I am asking the community
>>>>>                                     members that use Kamailio on
>>>>>                                     systems with libssl
>>>>>                                     > v1.1+ to test the master
>>>>>                                     branch and give feedback. The
>>>>>                                     master branch is
>>>>>                                     > anyhow in testing phase for
>>>>>                                     preparing the next major
>>>>>                                     release v5.3.0, so it
>>>>>                                     > will help for that as well.
>>>>>                                     >
>>>>>                                     > If no issue is going to be
>>>>>                                     reported soon, the commits
>>>>>                                     will be backported to
>>>>>                                     > stable branches, to
>>>>>                                     re-enable smooth use of
>>>>>                                     Kamailio with libssl v1.1+.
>>>>>                                     >
>>>>>                                     > When libssl v1.1+ is
>>>>>                                     detected, the core links with
>>>>>                                     pthread, but that was
>>>>>                                     > already present in the
>>>>>                                     Makefiles.defs for different
>>>>>                                     compile options. The
>>>>>                                     > pthread more or less a
>>>>>                                     default library on
>>>>>                                     linuxes/unixes, so this should
>>>>>                                     > not introduce any major
>>>>>                                     dependency to the Kamailio core.
>>>>>                                     >
>>>>>                                     > Cheers,
>>>>>                                     > Daniel
>>>>>                                     >
>>>>>                                     > --
>>>>>                                     > Daniel-Constantin Mierla -
>>>>>                                     http://www.asipto.com
>>>>>                                     >
>>>>>                                     http://twitter.com/#!/miconda
>>>>>                                     -
>>>>>                                     http://www.linkedin.com/in/miconda
>>>>>
>>>>>                                     >
>>>>>                                     _______________________________________________
>>>>>                                     > Kamailio (SER) - Users
>>>>>                                     Mailing List
>>>>>                                     > sr-users at lists.kamailio.org
>>>>>                                     <mailto:sr-users at lists.kamailio.org>
>>>>>                                     >
>>>>>                                     https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>>
>>>>>                                     -- 
>>>>>                                     Alex Balashov | Principal |
>>>>>                                     Evariste Systems LLC
>>>>>
>>>>>                                     Tel: +1-706-510-6800 /
>>>>>                                     +1-800-250-5920 (toll-free)
>>>>>                                     Web:
>>>>>                                     http://www.evaristesys.com/,
>>>>>                                     http://www.csrpswitch.com/
>>>>>
>>>>>                                     _______________________________________________
>>>>>                                     Kamailio (SER) - Users Mailing
>>>>>                                     List
>>>>>                                     sr-users at lists.kamailio.org
>>>>>                                     <mailto:sr-users at lists.kamailio.org>
>>>>>                                     https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>>
>>>>>
>>>>>                                 -- 
>>>>>                                 Andy Chen
>>>>>                                 Sr. Telephony Lead Engineer
>>>>>                                 415 516 5535 (M)
>>>>>                                 achen@
>>>>>                                 <mailto:achen at thinkingphones.com>fuze.com
>>>>>                                 <http://fuze.com>
>>>>>
>>>>>
>>>>>                                 *Confidentiality Notice: The
>>>>>                                 information contained in this
>>>>>                                 e-mail and any
>>>>>                                 attachments may be confidential.
>>>>>                                 If you are not an intended
>>>>>                                 recipient, you
>>>>>                                 are hereby notified that any
>>>>>                                 dissemination, distribution or
>>>>>                                 copying of this
>>>>>                                 e-mail is strictly prohibited. If
>>>>>                                 you have received this e-mail in
>>>>>                                 error,
>>>>>                                 please notify the sender and
>>>>>                                 permanently delete the e-mail and any
>>>>>                                 attachments immediately. You
>>>>>                                 should not retain, copy or use
>>>>>                                 this e-mail or
>>>>>                                 any attachment for any purpose,
>>>>>                                 nor disclose all or any part of the
>>>>>                                 contents to any other person.
>>>>>                                 Thank you.*
>>>>>
>>>>>                                 _______________________________________________
>>>>>                                 Kamailio (SER) - Users Mailing List
>>>>>                                 sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>>>>                                 https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>                                 -- 
>>>>                                 Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>>>                                 www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>>>                                 Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
>>>>
>>>>
>>>>
>>>>                             -- 
>>>>                             Andy Chen
>>>>                             Sr. Telephony Lead Engineer
>>>>                             415 516 5535 (M)
>>>>                             achen@
>>>>                             <mailto:achen at thinkingphones.com>fuze.com
>>>>                             <http://fuze.com>
>>>>
>>>>
>>>>
>>>>                         -- 
>>>>                         Andy Chen
>>>>                         Sr. Telephony Lead Engineer
>>>>                         415 516 5535 (M)
>>>>                         achen@
>>>>                         <mailto:achen at thinkingphones.com>fuze.com
>>>>                         <http://fuze.com>
>>>>
>>>>
>>>>
>>>>                     -- 
>>>>                     Andy Chen
>>>>                     Sr. Telephony Lead Engineer
>>>>                     415 516 5535 (M)
>>>>                     achen@
>>>>                     <mailto:achen at thinkingphones.com>fuze.com
>>>>                     <http://fuze.com>
>>>>
>>>>
>>>>                     *Confidentiality Notice: The information
>>>>                     contained in this e-mail and any
>>>>                     attachments may be confidential. If you are not
>>>>                     an intended recipient, you
>>>>                     are hereby notified that any dissemination,
>>>>                     distribution or copying of this
>>>>                     e-mail is strictly prohibited. If you have
>>>>                     received this e-mail in error,
>>>>                     please notify the sender and permanently delete
>>>>                     the e-mail and any
>>>>                     attachments immediately. You should not retain,
>>>>                     copy or use this e-mail or
>>>>                     any attachment for any purpose, nor disclose
>>>>                     all or any part of the
>>>>                     contents to any other person. Thank you.* 
>>>
>>>                     -- 
>>>                     Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>>                     www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>>                     Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
>>>
>>>
>>>
>>>                 -- 
>>>                 Andy Chen
>>>                 Sr. Telephony Lead Engineer
>>>                 415 516 5535 (M)
>>>                 achen@ <mailto:achen at thinkingphones.com>fuze.com
>>>                 <http://fuze.com>
>>>
>>>
>>>                 *Confidentiality Notice: The information contained
>>>                 in this e-mail and any
>>>                 attachments may be confidential. If you are not an
>>>                 intended recipient, you
>>>                 are hereby notified that any dissemination,
>>>                 distribution or copying of this
>>>                 e-mail is strictly prohibited. If you have received
>>>                 this e-mail in error,
>>>                 please notify the sender and permanently delete the
>>>                 e-mail and any
>>>                 attachments immediately. You should not retain, copy
>>>                 or use this e-mail or
>>>                 any attachment for any purpose, nor disclose all or
>>>                 any part of the
>>>                 contents to any other person. Thank you.*
>>>
>>>                 _______________________________________________
>>>                 Kamailio (SER) - Users Mailing List
>>>                 sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>>                 https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>>                 -- 
>>                 Kamailio Merchandising - https://skalatan.de/merchandising/
>>                 Kamailio services - https://skalatan.de/services
>>                 Henning Westerholt - https://skalatan.de/blog/
>>
>>
>>
>>             -- 
>>             Andy Chen
>>             Sr. Telephony Lead Engineer
>>             415 516 5535 (M)
>>             achen@ <mailto:achen at thinkingphones.com>fuze.com
>>             <http://fuze.com>
>>
>>
>>
>>         -- 
>>         Andy Chen
>>         Sr. Telephony Lead Engineer
>>         415 516 5535 (M)
>>         achen@ <mailto:achen at thinkingphones.com>fuze.com
>>         <http://fuze.com>
>>
>>
>>         *Confidentiality Notice: The information contained in this
>>         e-mail and any
>>         attachments may be confidential. If you are not an intended
>>         recipient, you
>>         are hereby notified that any dissemination, distribution or
>>         copying of this
>>         e-mail is strictly prohibited. If you have received this
>>         e-mail in error,
>>         please notify the sender and permanently delete the e-mail
>>         and any
>>         attachments immediately. You should not retain, copy or use
>>         this e-mail or
>>         any attachment for any purpose, nor disclose all or any part
>>         of the
>>         contents to any other person. Thank
>>         you.*_______________________________________________
>>         Kamailio (SER) - Users Mailing List
>>         sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>         https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>     -- 
>     Kamailio Merchandising - https://skalatan.de/merchandising/
>     Kamailio services - https://skalatan.de/services
>     Henning Westerholt - https://skalatan.de/blog/
>
>
>
> -- 
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <mailto:achen at thinkingphones.com>fuze.com <http://fuze.com>
>
>
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of
> this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190919/73be475e/attachment.html>


More information about the sr-users mailing list