[SR-Users] Using AWS boto3 with KEMI Python routing scripts
Michael Loughrey
mgloughrey at gmail.com
Wed Apr 10 15:11:12 CEST 2019
Thanks for the prompt reply. Yes I do have TLS enabled and in fact the
crash is inside one of the TCP receiver processes when it is handing a
REGISTER message.
I will try with non TLS connections, however it is intermittent so the
absence of crashes hasn't always been a good guide. Also I really need TLS
for the client connections.
I might try something else first though. From Boto3 docs :
The boto3 module acts as a proxy to the default session
A session manages state about a particular configuration. By default a
session is created for you when needed. However it is possible and
recommended to maintain your own session(s) in some scenarios.
It is also possible to manage your own session and create clients or
resources from it:
# Creating your own session
session = boto3.session.Session()
sqs = session.client('sqs')
s3 = session.resource('s3')
Multithreading / Multiprocessing
--------------------------------
It is recommended to create a resource instance for each thread / process
in a multithreaded or multiprocess application rather than sharing a single
instance among the threads / processes. For example:
import boto3
import boto3.session
import threading
class MyTask(threading.Thread):
def run(self):
session = boto3.session.Session()
s3 = session.resource('s3')
# ... do some work with S3 ...
In the example above, each thread would have its own Boto 3 session and its
own instance of the S3 resource. This is a good idea because resources
contain shared data when loaded and calling actions, accessing properties,
or manually loading or reloading the resource can modify this data.
So I could in __init__() initialise thus
self.session = null
self.sns_client = null
and on demand from child processes :
if not self.session:
self.session =
boto3.session.Session(region_name=MY_AWS_REGION,use_ssl=True)
if not self.client:
self.sns_client = self.session.resource('sns')
So each child process that need it gets it's own boto3 session.
Another thing I can try is setting use_ssl=False.
Cheers
Mike
On Wed, 10 Apr 2019 at 03:34, Daniel-Constantin Mierla <miconda at gmail.com>
wrote:
> Hello,
>
> do you have tls module loaded in kamailio.cfg? If yes, can you try without
> it, just to see if there is a conflict between our module and the boto3
> client in use of libssl, because libssl creates global contexts per
> application, not per library/object.
>
> Cheers,
> Daniel
> On 10.04.19 01:26, Michael Loughrey wrote:
>
> Hi All,
>
> looking for some advice regarding the proper way to initialise an AWS API
> boto3 client object for send SNS messages within a KEMI Python 2.7 routing
> script.
>
> I'm sure process forking has a major impact on how this works - from
> kamailio.cfg
>
> fork=yes
> children=4
>
> - but would greatly appreciate some guidance.
>
> I have tried various methods to allocate an client object self.sns_client
> = boto3.client('sns', region_name=MY_AWS_REGION)
>
> 1. within __init__(), only once from module initialisation
> 2. on demand within a function call by any client code e.g. called from
> ksr_route_request(), each creating it's own client object
>
> and so far all have resulted in intermittent crashes within
> botocore/client.py ultimately crashing thus :
>
>
> Core was generated by `/usr/local/sbin/kamailio -P
> /usr/local/kamailio/run/kamailio.pid -f /usr/local/'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007f8f9734f754 in ?? () from
> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
> (gdb) bt
> #0 0x00007f8f9734f754 in ?? () from
> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
> #1 0x00007f8f9734f82e in X509_VERIFY_PARAM_free () from
> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
> #2 0x00007f8f97642f5c in SSL_free () from
> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> #3 0x00007f8f86b09c64 in PySSL_dealloc.lto_priv.5 () at
> ./Modules/_ssl.c:1598
> #4 0x00007f8f99cbb007 in insertdict_by_entry (mp=0x7f8f841f1e88,
> key='_sslobj', hash=<optimized out>, ep=<optimized out>, value=<optimized
> out>) at ../Objects/dictobject.c:519
> #5 0x00007f8f99cbe2cf in insertdict (value=None,
> hash=1051385741686792393, key='_sslobj', mp=0x7f8f841f1e88) at
> ../Objects/dictobject.c:556
> #6 dict_set_item_by_hash_or_entry (value=None, ep=0x0,
> hash=1051385741686792393, key='_sslobj',
> op={'server_hostname': u'sns.us-east-1.amazonaws.com', '_connected':
> True, '_context': <SSLContext at remote 0x7f8f84232398>, 'server_side':
> False, '_makefile_refs': 0, '_closed': False, '_sslobj': None,
> 'do_handshake_on_connect': True, 'suppress_ragged_eofs': True}) at
> ../Objects/dictobject.c:795
> #7 PyDict_SetItem (op=<optimized out>, key=<optimized out>,
> value=<optimized out>) at ../Objects/dictobject.c:848
> #8 0x00007f8f99becec1 in _PyObject_GenericSetAttrWithDict (obj=<optimized
> out>, name='_sslobj', value=None,
> dict={'server_hostname': u'sns.us-east-1.amazonaws.com',
> '_connected': True, '_context': <SSLContext at remote 0x7f8f84232398>,
> 'server_side': False, '_makefile_refs': 0, '_closed': False, '_sslobj':
> None, 'do_handshake_on_connect': True, 'suppress_ragged_eofs': True}) at
> ../Objects/object.c:1529
> #9 0x00007f8f99bed437 in PyObject_SetAttr (
> v=<SSLSocket(server_hostname=u'sns.us-east-1.amazonaws.com',
> _connected=True, _context=<SSLContext at remote 0x7f8f84232398>,
> server_side=False, _makefile_refs=0, _closed=False, _sslobj=None,
> do_handshake_on_connect=True, suppress_ragged_eofs=True) at remote
> 0x7f8f841f62a8>, name=<optimized out>, value=None) at
> ../Objects/object.c:1247
>
> Here's my setup
>
> host : Debian9/stretch
>
> python --version
> Python 2.7.13
>
> kamailio -v
> version: kamailio 5.2.2 (x86_64/linux) 67f967-dirty
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
> USE_DST_BLACKLIST, HAVE_RESOLV_RES, MEM_JOIN_FREE
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024,
> BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: 67f967 -dirty
> compiled on 23:44:31 Apr 8 2019 with gcc 6.3.0
>
>
> Cheers
> Mike
>
>
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190410/2af72345/attachment.html>
More information about the sr-users
mailing list