[SR-Users] Info: initialization of libssl 1.1.+ with PTHREAD_PROCESS_SHARED attribute for pthread mutex and rwlock

Andrew Chen achen at fuze.com
Sat Sep 21 03:44:33 CEST 2019


Hi Daniel,

Just checking in to get any updates/findings on the stack trace I provided
above.

Thanks.

On Thu, Sep 19, 2019 at 11:29 AM Andrew Chen <achen at fuze.com> wrote:

> So what I pasted above is the same segfault we got from libcrypto.so
>
> On Thu, Sep 19, 2019 at 11:03 AM Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>> You do not need to reproduce if you still have the core files and
>> installed the debug symbols for your version, just run again gdb on the
>> core and get the backtraces.
>>
>> Cheers,
>> Daniel
>> On 18.09.19 22:35, Andrew Chen wrote:
>>
>> Yup understood.  I'll provide the bt when we can reproduce this.
>>
>> Thanks
>>
>> On Wed, Sep 18, 2019 at 4:32 PM Henning Westerholt <hw at skalatan.de>
>> wrote:
>>
>>> Hi Andrew,
>>>
>>> the 5.2.4 does not contain the 5.3-pre1 workaround yet.
>>>
>>> So if you test the workaround you need to install the lastest 5.3 from
>>> the git master debian repository.
>>>
>>> But nevertheless it would be good to look at the backtrace, while you
>>> have it.
>>>
>>> Cheers,
>>>
>>> Henning
>>> Am 18.09.19 um 22:10 schrieb Joel Serrano:
>>>
>>> Hi Andrew,
>>>
>>> Download from here:
>>> http://deb.kamailio.org/kamailio52/pool/main/k/kamailio/
>>>
>>> You have kamailio-dbg v5.2.4 (latest 5.2) for all distros...
>>>
>>> Example for amd64+stretch:
>>> http://deb.kamailio.org/kamailio52/pool/main/k/kamailio/kamailio-dbg_5.2.4%2Bstretch_amd64.deb
>>>
>>> Cheers,
>>> Joel.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Sep 18, 2019 at 12:16 PM Andrew Chen <achen at fuze.com> wrote:
>>>
>>>> Hi Henning,
>>>>
>>>> I'm running 5.2.4.  Will this deb package work?
>>>>
>>>> On Wed, Sep 18, 2019 at 2:39 PM Andrew Chen <achen at fuze.com> wrote:
>>>>
>>>>> Thanks Henning
>>>>>
>>>>> On Wed, Sep 18, 2019 at 2:13 PM Henning Westerholt <hw at skalatan.de>
>>>>> wrote:
>>>>>
>>>>>> Hello Andrew,
>>>>>>
>>>>>> strange - it should be there, as it is available on deb.kamailio.org.
>>>>>>
>>>>>>
>>>>>> Here e.g. the one for 64bit stretch:
>>>>>>
>>>>>>
>>>>>> https://deb.kamailio.org/kamailiodev-nightly/pool/main/k/kamailio/kamailio-dbg_5.3.0~pre1%2B0~20190918005413.1490%2Bstretch_amd64.deb
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Henning
>>>>>> Am 18.09.19 um 18:52 schrieb Andrew Chen:
>>>>>>
>>>>>> I don't see one for debug:
>>>>>>
>>>>>> root at ashintgkama51:~ # apt-cache search kamailio
>>>>>>
>>>>>> resource-agents - Cluster Resource Agents
>>>>>>
>>>>>> kamailio - very fast, dynamic and configurable SIP server
>>>>>>
>>>>>> kamailio-autheph-modules - authentication using ephemeral credentials
>>>>>> for the Kamailio SIP server
>>>>>>
>>>>>> kamailio-berkeley-bin - Berkeley database module for Kamailio -
>>>>>> helper program
>>>>>>
>>>>>> kamailio-berkeley-modules - Berkeley database module for the Kamailio
>>>>>> SIP server
>>>>>>
>>>>>> kamailio-carrierroute-modules - carrierroute module for the Kamailio
>>>>>> SIP server
>>>>>>
>>>>>> kamailio-cnxcc-modules - cnxcc modules for the Kamailio SIP server
>>>>>>
>>>>>> kamailio-cpl-modules - CPL module (CPL interpreter engine) for the
>>>>>> Kamailio SIP server
>>>>>>
>>>>>> kamailio-erlang-modules - Erlang modules for the Kamailio SIP server
>>>>>>
>>>>>> kamailio-geoip-modules - geoip module for the Kamailio SIP Server
>>>>>>
>>>>>> kamailio-geoip2-modules - geoip2 module for the Kamailio SIP Server
>>>>>>
>>>>>> kamailio-ims-modules - IMS module for the Kamailio SIP server
>>>>>>
>>>>>> kamailio-json-modules - Json parser and Json-RPC modules for the
>>>>>> Kamailio SIP server
>>>>>>
>>>>>> On Wed, Sep 18, 2019 at 11:26 AM Daniel-Constantin Mierla <
>>>>>> miconda at gmail.com> wrote:
>>>>>>
>>>>>>> You have to install the debug symbols package, otherwise the
>>>>>>> backtrace doesn't show any useful information. It just require to install
>>>>>>> that package and then get again the backtraces. The package name should be
>>>>>>> kamailio-dbg.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>> On 18.09.19 16:37, Andrew Chen wrote:
>>>>>>>
>>>>>>> Sorry..forgot bt full output
>>>>>>>
>>>>>>> (gdb) bt full
>>>>>>>
>>>>>>> #0  0x0000562fcc3544c0 in ?? ()
>>>>>>>
>>>>>>> No symbol table info available.
>>>>>>>
>>>>>>> #1  0x0000562fcc354632 in ?? ()
>>>>>>>
>>>>>>> No symbol table info available.
>>>>>>>
>>>>>>> #2  0x0000562fcc354974 in local_timer_run ()
>>>>>>>
>>>>>>> No symbol table info available.
>>>>>>>
>>>>>>> #3  0x0000562fcc3fb256 in ?? ()
>>>>>>>
>>>>>>> No symbol table info available.
>>>>>>>
>>>>>>> #4  0x0000562fcc3fb64c in tcp_receive_loop ()
>>>>>>>
>>>>>>> No symbol table info available.
>>>>>>>
>>>>>>> #5  0x0000562fcc2acf52 in tcp_init_children ()
>>>>>>>
>>>>>>> No symbol table info available.
>>>>>>>
>>>>>>> #6  0x0000562fcc1ad94a in main_loop ()
>>>>>>>
>>>>>>> No symbol table info available.
>>>>>>>
>>>>>>> #7  0x0000562fcc1b4a7b in main ()
>>>>>>>
>>>>>>> No symbol table info available.
>>>>>>>
>>>>>>> (gdb)
>>>>>>>
>>>>>>> On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen <achen at fuze.com> wrote:
>>>>>>>
>>>>>>>> backtrace output:
>>>>>>>>
>>>>>>>>
>>>>>>>> (gdb) bt
>>>>>>>>
>>>>>>>> #0  0x0000562fcc3544c0 in ?? ()
>>>>>>>>
>>>>>>>> #1  0x0000562fcc354632 in ?? ()
>>>>>>>>
>>>>>>>> #2  0x0000562fcc354974 in local_timer_run ()
>>>>>>>>
>>>>>>>> #3  0x0000562fcc3fb256 in ?? ()
>>>>>>>>
>>>>>>>> #4  0x0000562fcc3fb64c in tcp_receive_loop ()
>>>>>>>>
>>>>>>>> #5  0x0000562fcc2acf52 in tcp_init_children ()
>>>>>>>>
>>>>>>>> #6  0x0000562fcc1ad94a in main_loop ()
>>>>>>>>
>>>>>>>> #7  0x0000562fcc1b4a7b in main ()
>>>>>>>>
>>>>>>>> (gdb)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen <achen at fuze.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I'll look for the core.
>>>>>>>>>
>>>>>>>>> As part of our release process, I don't think we can
>>>>>>>>> release master branch of Kamailio.
>>>>>>>>>
>>>>>>>>> On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla <
>>>>>>>>> miconda at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> did you get core files? If yes, it would be good to see the gdb
>>>>>>>>>> bracktraces.
>>>>>>>>>>
>>>>>>>>>> Have you used the pre-loaded library workaround?
>>>>>>>>>>
>>>>>>>>>> The code was not backported to branch 5.2 yet, I wanted to wait a
>>>>>>>>>> bit and see if there are reports with good/bad results. If you want to use
>>>>>>>>>> it, then you have to install nightly builds from master branch (development
>>>>>>>>>> version, should be 5.3.0-pre1 at this moment).
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Daniel
>>>>>>>>>> On 18.09.19 15:26, Andrew Chen wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Daniel,
>>>>>>>>>> Which apt repository can we get this workaround?   Today I
>>>>>>>>>> noticed our Kamailio 5.2.4 cored during one of our load test sessions:
>>>>>>>>>>
>>>>>>>>>> Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at
>>>>>>>>>> 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f8501dd4000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at
>>>>>>>>>> 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f1a8f90f000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip
>>>>>>>>>> 00007f0a85dac128 sp 00007fff6847aad0 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f0a85c40000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip
>>>>>>>>>> 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f0a85c40000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at
>>>>>>>>>> 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f75e1136000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at
>>>>>>>>>> 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f0f37ead000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at
>>>>>>>>>> 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f8043b0a000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at
>>>>>>>>>> 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7ff59d250000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at
>>>>>>>>>> 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7ff59d250000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at
>>>>>>>>>> 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f22dd4db000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at
>>>>>>>>>> 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f7ccae9b000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at
>>>>>>>>>> 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7fb309baf000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip
>>>>>>>>>> 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7fc5908d7000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at
>>>>>>>>>> 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7fcc238c6000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip
>>>>>>>>>> 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7fcc238c6000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at
>>>>>>>>>> 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in
>>>>>>>>>> libcrypto.so.1.1[7f586f78e000+29b000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip
>>>>>>>>>> 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in
>>>>>>>>>> kamailio[562fcc17c000+4a2000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip
>>>>>>>>>> 00007f96c3735c3e sp 00007ffe825ca270 error 4 in
>>>>>>>>>> dialog.so[7f96c36d6000+be000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip
>>>>>>>>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in
>>>>>>>>>> kamailio[562fcc17c000+4a2000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip
>>>>>>>>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip
>>>>>>>>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in
>>>>>>>>>> kamailio[562fcc17c000+4a2000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip
>>>>>>>>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in
>>>>>>>>>> kamailio[562fcc17c000+4a2000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip
>>>>>>>>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip
>>>>>>>>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
>>>>>>>>>>
>>>>>>>>>> [Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip
>>>>>>>>>> 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in
>>>>>>>>>> kamailio[562fcc17c000+4a2000]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> and I was wondering if this core is related.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov <
>>>>>>>>>> abalashov at evaristesys.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Daniel,
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot for this work; it's definitely important that
>>>>>>>>>>> Kamailio
>>>>>>>>>>> build against OpenSSL v1.1, and I'll be happy to contribute some
>>>>>>>>>>> testing
>>>>>>>>>>> in a live environment in the coming weeks. Much appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Have a good weekend,
>>>>>>>>>>>
>>>>>>>>>>> -- Alex
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin
>>>>>>>>>>> Mierla wrote:
>>>>>>>>>>>
>>>>>>>>>>> > Hello,
>>>>>>>>>>> > after couple of months waiting for feedback about the
>>>>>>>>>>> workaround with the
>>>>>>>>>>> > preloaded shared library to set the PTHREAD_PROCESS_SHARED
>>>>>>>>>>> attribute for
>>>>>>>>>>> > pthread mutex/rwlock in order to avoid deadlocks and as it
>>>>>>>>>>> seems to be the
>>>>>>>>>>> > solution, I just pushed a few commits to move that code in the
>>>>>>>>>>> core of
>>>>>>>>>>> > kamailio.
>>>>>>>>>>> >
>>>>>>>>>>> > It this way the initialization with the PTHREAD_PROCESS_SHARED
>>>>>>>>>>> attribute is
>>>>>>>>>>> > done by default when libssl v1.1 is detected at compile time,
>>>>>>>>>>> without the
>>>>>>>>>>> > need of using the approach with a preloaded library.
>>>>>>>>>>> >
>>>>>>>>>>> > The shared library is still kept in the tls module:
>>>>>>>>>>> >   *
>>>>>>>>>>> >
>>>>>>>>>>> https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared
>>>>>>>>>>> >
>>>>>>>>>>> > Probably is going to be removed in the near future.
>>>>>>>>>>> >
>>>>>>>>>>> > I am asking the community members that use Kamailio on systems
>>>>>>>>>>> with libssl
>>>>>>>>>>> > v1.1+ to test the master branch and give feedback. The master
>>>>>>>>>>> branch is
>>>>>>>>>>> > anyhow in testing phase for preparing the next major release
>>>>>>>>>>> v5.3.0, so it
>>>>>>>>>>> > will help for that as well.
>>>>>>>>>>> >
>>>>>>>>>>> > If no issue is going to be reported soon, the commits will be
>>>>>>>>>>> backported to
>>>>>>>>>>> > stable branches, to re-enable smooth use of Kamailio with
>>>>>>>>>>> libssl v1.1+.
>>>>>>>>>>> >
>>>>>>>>>>> > When libssl v1.1+ is detected, the core links with pthread,
>>>>>>>>>>> but that was
>>>>>>>>>>> > already present in the Makefiles.defs for different compile
>>>>>>>>>>> options. The
>>>>>>>>>>> > pthread more or less a default library on linuxes/unixes, so
>>>>>>>>>>> this should
>>>>>>>>>>> > not introduce any major dependency to the Kamailio core.
>>>>>>>>>>> >
>>>>>>>>>>> > Cheers,
>>>>>>>>>>> > Daniel
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > Daniel-Constantin Mierla - http://www.asipto.com
>>>>>>>>>>> > http://twitter.com/#!/miconda -
>>>>>>>>>>> http://www.linkedin.com/in/miconda
>>>>>>>>>>>
>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>> > Kamailio (SER) - Users Mailing List
>>>>>>>>>>> > sr-users at lists.kamailio.org
>>>>>>>>>>> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Alex Balashov | Principal | Evariste Systems LLC
>>>>>>>>>>>
>>>>>>>>>>> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
>>>>>>>>>>> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Kamailio (SER) - Users Mailing List
>>>>>>>>>>> sr-users at lists.kamailio.org
>>>>>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Andy Chen
>>>>>>>>>> Sr. Telephony Lead Engineer
>>>>>>>>>> 415 516 5535 (M)
>>>>>>>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Confidentiality Notice: The information contained in this e-mail
>>>>>>>>>> and any
>>>>>>>>>> attachments may be confidential. If you are not an intended
>>>>>>>>>> recipient, you
>>>>>>>>>> are hereby notified that any dissemination, distribution or
>>>>>>>>>> copying of this
>>>>>>>>>> e-mail is strictly prohibited. If you have received this e-mail
>>>>>>>>>> in error,
>>>>>>>>>> please notify the sender and permanently delete the e-mail and any
>>>>>>>>>> attachments immediately. You should not retain, copy or use this
>>>>>>>>>> e-mail or
>>>>>>>>>> any attachment for any purpose, nor disclose all or any part of
>>>>>>>>>> the
>>>>>>>>>> contents to any other person. Thank you.*
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>>>>>>>> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Andy Chen
>>>>>>>>> Sr. Telephony Lead Engineer
>>>>>>>>> 415 516 5535 (M)
>>>>>>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Andy Chen
>>>>>>>> Sr. Telephony Lead Engineer
>>>>>>>> 415 516 5535 (M)
>>>>>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Andy Chen
>>>>>>> Sr. Telephony Lead Engineer
>>>>>>> 415 516 5535 (M)
>>>>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>>>>
>>>>>>>
>>>>>>> *Confidentiality Notice: The information contained in this e-mail
>>>>>>> and any
>>>>>>> attachments may be confidential. If you are not an intended
>>>>>>> recipient, you
>>>>>>> are hereby notified that any dissemination, distribution or copying
>>>>>>> of this
>>>>>>> e-mail is strictly prohibited. If you have received this e-mail in
>>>>>>> error,
>>>>>>> please notify the sender and permanently delete the e-mail and any
>>>>>>> attachments immediately. You should not retain, copy or use this
>>>>>>> e-mail or
>>>>>>> any attachment for any purpose, nor disclose all or any part of the
>>>>>>> contents to any other person. Thank you.*
>>>>>>>
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>>>>> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Andy Chen
>>>>>> Sr. Telephony Lead Engineer
>>>>>> 415 516 5535 (M)
>>>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>>>
>>>>>>
>>>>>> *Confidentiality Notice: The information contained in this e-mail and
>>>>>> any
>>>>>> attachments may be confidential. If you are not an intended
>>>>>> recipient, you
>>>>>> are hereby notified that any dissemination, distribution or copying
>>>>>> of this
>>>>>> e-mail is strictly prohibited. If you have received this e-mail in
>>>>>> error,
>>>>>> please notify the sender and permanently delete the e-mail and any
>>>>>> attachments immediately. You should not retain, copy or use this
>>>>>> e-mail or
>>>>>> any attachment for any purpose, nor disclose all or any part of the
>>>>>> contents to any other person. Thank you.*
>>>>>>
>>>>>> _______________________________________________
>>>>>> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>> --
>>>>>> Kamailio Merchandising - https://skalatan.de/merchandising/
>>>>>> Kamailio services - https://skalatan.de/services
>>>>>> Henning Westerholt - https://skalatan.de/blog/
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Andy Chen
>>>>> Sr. Telephony Lead Engineer
>>>>> 415 516 5535 (M)
>>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>>
>>>>>
>>>>
>>>> --
>>>> Andy Chen
>>>> Sr. Telephony Lead Engineer
>>>> 415 516 5535 (M)
>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>
>>>>
>>>> *Confidentiality Notice: The information contained in this e-mail and
>>>> any
>>>> attachments may be confidential. If you are not an intended recipient,
>>>> you
>>>> are hereby notified that any dissemination, distribution or copying of
>>>> this
>>>> e-mail is strictly prohibited. If you have received this e-mail in
>>>> error,
>>>> please notify the sender and permanently delete the e-mail and any
>>>> attachments immediately. You should not retain, copy or use this e-mail
>>>> or
>>>> any attachment for any purpose, nor disclose all or any part of the
>>>> contents to any other person. Thank you.*
>>>> _______________________________________________
>>>> Kamailio (SER) - Users Mailing List
>>>> sr-users at lists.kamailio.org
>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>> --
>>> Kamailio Merchandising - https://skalatan.de/merchandising/
>>> Kamailio services - https://skalatan.de/services
>>> Henning Westerholt - https://skalatan.de/blog/
>>>
>>>
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> 415 516 5535 (M)
>> achen@ <achen at thinkingphones.com>fuze.com
>>
>>
>> *Confidentiality Notice: The information contained in this e-mail and any
>> attachments may be confidential. If you are not an intended recipient, you
>> are hereby notified that any dissemination, distribution or copying of
>> this
>> e-mail is strictly prohibited. If you have received this e-mail in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
>>
>>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <achen at thinkingphones.com>fuze.com
>
>

-- 
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <achen at thinkingphones.com>fuze.com

-- 
*Confidentiality Notice: The information contained in this e-mail and any

attachments may be confidential. If you are not an intended recipient, you

are hereby notified that any dissemination, distribution or copying of this

e-mail is strictly prohibited. If you have received this e-mail in error,

please notify the sender and permanently delete the e-mail and any

attachments immediately. You should not retain, copy or use this e-mail or

any attachment for any purpose, nor disclose all or any part of the

contents to any other person. Thank you.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190920/b3d44abc/attachment.html>


More information about the sr-users mailing list