From igorolhovskiy@gmail.com Tue Aug 29 12:30:06 2023
From: Ihor Olkhovskyi <igorolhovskiy@gmail.com>
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Kamailio stops processing UDP on one interface
Date: Tue, 29 Aug 2023 14:29:32 +0200
Message-ID:
 <CAJTkRNtE5GgbDod0RF5CJYBONc95yLMqmRrhuk164cbp845b0g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============2130123295=="

--===============2130123295==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Hello,

I've faced a bit strange issue, but a bit of preface. I have Kamailio as a
proxy (TLS/WS <-> UDP) and second Kamailio as a presence server. At some
point presence server accepts around 5K PUBLISH within 1 minute and sending
around the same amount of NOTIFY to proxy Kamailio.

Proxy is "transforming" protocol to TLS, but at sime point I'm starting to
get these type of errors

tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
<script>: [RELAY] Relay to <sip:X.X.X.X:51571;transport=tls> failed!
tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed

Some of those messages are 100% valid as client can go away or so. Some are
not, cause I'm sure client is alive and connected.

But the problem comes later. At some moment proxy Kamailio just stops
accept UDP traffic on this interface (where it also accepts all NOTIFY's),
at the start of the "stopping accepting" Kamailio sends OPTIONS via
DISPATCHER but not able to receive 200 OK.

Over TLS on the same interface all is ok. On other (loopback) interface UDP
is being processed fine, so I don't suspert some limit on open files here.

Only restart of Kamailio proxy process helps in this case.

I've tuned net.core.rmem_max and net.core.rmem_default to 25 Mb, so in
theory buffer should not be the case.

Is there some internal "interface buffer" in Kamailio that is not freed
upon failure send or maybe I've missed somethig?

Kamailio 5.6.4

fork=yes
children=12
tcp_children=12

enable_tls=yes

tcp_accept_no_cl=yes
tcp_max_connections=63536
tls_max_connections=63536
tcp_accept_aliases=no
tcp_async=yes
tcp_connect_timeout=10
tcp_conn_wq_max=63536
tcp_crlf_ping=yes
tcp_delayed_ack=yes
tcp_fd_cache=yes
tcp_keepalive=yes
tcp_keepcnt=3
tcp_keepidle=30
tcp_keepintvl=10
tcp_linger2=30
tcp_rd_buf_size=80000
tcp_send_timeout=10
tcp_wq_blk_size=2100
tcp_wq_max=10485760
open_files_limit=63536

Sysctl

# To increase the amount of memory available for socket input/output queues
net.ipv4.tcp_rmem = 4096 25165824 25165824
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824

# To limit the maximum number of requests queued to a listen socket
net.core.somaxconn = 128

# Tells TCP to instead make decisions that would prefer lower latency.
net.ipv4.tcp_low_latency=1

# Optional (it will increase performance)
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_max_syn_backlog = 128

# Flush the routing table to make changes happen instantly.
net.ipv4.route.flush=1
-- 
Best regards,
Ihor (Igor)



--===============2130123295==
Content-Type: text/html
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.html"
MIME-Version: 1.0

PGRpdiBkaXI9Imx0ciI+SGVsbG8sPGJyPjxkaXY+PGJyPjwvZGl2PjxkaXY+SSYjMzk7dmUgZmFj
ZWQgYSBiaXQgc3RyYW5nZSBpc3N1ZSwgYnV0IGEgYml0IG9mIHByZWZhY2UuIEkgaGF2ZSBLYW1h
aWxpbyBhcyBhIHByb3h5IChUTFMvV1MgJmx0Oy0mZ3Q7IFVEUCkgYW5kIHNlY29uZCBLYW1haWxp
byBhcyBhIHByZXNlbmNlIHNlcnZlci4gQXQgc29tZSBwb2ludCBwcmVzZW5jZSBzZXJ2ZXIgYWNj
ZXB0cyBhcm91bmQgNUsgUFVCTElTSCB3aXRoaW4gMSBtaW51dGUgYW5kIHNlbmRpbmcgYXJvdW5k
IHRoZSBzYW1lIGFtb3VudCBvZiBOT1RJRlkgdG8gcHJveHkgS2FtYWlsaW8uPC9kaXY+PGRpdj48
YnI+PC9kaXY+PGRpdj5Qcm94eSBpcyAmcXVvdDt0cmFuc2Zvcm1pbmcmcXVvdDsgcHJvdG9jb2wg
dG8gVExTLCBidXQgYXQgc2ltZSBwb2ludCBJJiMzOTttIHN0YXJ0aW5nIHRvIGdldCB0aGVzZSB0
eXBlIG9mIGVycm9yczwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+dG0gWy4uLy4uL2NvcmUvZm9y
d2FyZC5oOjI5Ml06IG1zZ19zZW5kX2J1ZmZlcigpOiB0Y3Bfc2VuZCBmYWlsZWQ8YnI+dG0gW3Rf
ZndkLmM6MTU4OF06IHRfc2VuZF9icmFuY2goKTogc2VuZGluZyByZXF1ZXN0IG9uIGJyYW5jaCAw
IGZhaWxlZDxicj4mbHQ7c2NyaXB0Jmd0OzogW1JFTEFZXSBSZWxheSB0byAmbHQ7c2lwOlguWC5Y
Llg6NTE1NzE7dHJhbnNwb3J0PXRscyZndDsgZmFpbGVkITxicj50bSBbLi4vLi4vY29yZS9mb3J3
YXJkLmg6MjkyXTogbXNnX3NlbmRfYnVmZmVyKCk6IHRjcF9zZW5kIGZhaWxlZDxicj50bSBbdF9m
d2QuYzoxNTg4XTogdF9zZW5kX2JyYW5jaCgpOiBzZW5kaW5nIHJlcXVlc3Qgb24gYnJhbmNoIDAg
ZmFpbGVkPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5Tb21lIG9mIHRob3NlIG1lc3NhZ2VzIGFy
ZSAxMDAlIHZhbGlkIGFzIGNsaWVudCBjYW4gZ28gYXdheSBvciBzby4gU29tZSBhcmUgbm90LCBj
YXVzZSBJJiMzOTttIHN1cmUgY2xpZW50IGlzIGFsaXZlIGFuZCBjb25uZWN0ZWQuwqA8L2Rpdj48
ZGl2Pjxicj48L2Rpdj48ZGl2PkJ1dCB0aGUgcHJvYmxlbSBjb21lcyBsYXRlci4gQXQgc29tZSBt
b21lbnQgcHJveHkgS2FtYWlsaW8ganVzdCBzdG9wcyBhY2NlcHQgVURQIHRyYWZmaWMgb24gdGhp
cyBpbnRlcmZhY2UgKHdoZXJlIGl0IGFsc28gYWNjZXB0cyBhbGwgTk9USUZZJiMzOTtzKSwgYXQg
dGhlIHN0YXJ0IG9mIHRoZSAmcXVvdDtzdG9wcGluZyBhY2NlcHRpbmcmcXVvdDsgS2FtYWlsaW8g
c2VuZHMgT1BUSU9OUyB2aWEgRElTUEFUQ0hFUiBidXQgbm90IGFibGUgdG8gcmVjZWl2ZSAyMDAg
T0suIDxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pk92ZXIgVExTIG9uIHRoZSBzYW1lIGlu
dGVyZmFjZSBhbGwgaXMgb2suIE9uIG90aGVyIChsb29wYmFjaykgaW50ZXJmYWNlIFVEUCBpcyBi
ZWluZyBwcm9jZXNzZWQgZmluZSwgc28gSSBkb24mIzM5O3Qgc3VzcGVydCBzb21lIGxpbWl0IG9u
IG9wZW4gZmlsZXMgaGVyZS48YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5Pbmx5IHJlc3Rh
cnQgb2YgS2FtYWlsaW8gcHJveHkgcHJvY2VzcyBoZWxwcyBpbiB0aGlzIGNhc2UuIDxicj48L2Rp
dj48ZGl2Pjxicj48L2Rpdj48ZGl2PkkmIzM5O3ZlIHR1bmVkIG5ldC5jb3JlLnJtZW1fbWF4IGFu
ZCBuZXQuY29yZS5ybWVtX2RlZmF1bHQgdG8gMjUgTWIsIHNvIGluIHRoZW9yeSBidWZmZXIgc2hv
dWxkIG5vdCBiZSB0aGUgY2FzZS48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PklzIHRoZXJlIHNv
bWUgaW50ZXJuYWwgJnF1b3Q7aW50ZXJmYWNlIGJ1ZmZlciZxdW90OyBpbiBLYW1haWxpbyB0aGF0
IGlzIG5vdCBmcmVlZCB1cG9uIGZhaWx1cmUgc2VuZCBvciBtYXliZSBJJiMzOTt2ZSBtaXNzZWQg
c29tZXRoaWc/PGJyPjxicj48L2Rpdj48ZGl2PkthbWFpbGlvIDUuNi40PC9kaXY+PGRpdj48YnI+
PC9kaXY+PGRpdj5mb3JrPXllczxicj5jaGlsZHJlbj0xMjxicj50Y3BfY2hpbGRyZW49MTI8YnI+
PGJyPmVuYWJsZV90bHM9eWVzPGJyPjxicj50Y3BfYWNjZXB0X25vX2NsPXllczxicj50Y3BfbWF4
X2Nvbm5lY3Rpb25zPTYzNTM2PGJyPnRsc19tYXhfY29ubmVjdGlvbnM9NjM1MzY8YnI+dGNwX2Fj
Y2VwdF9hbGlhc2VzPW5vPGJyPnRjcF9hc3luYz15ZXM8YnI+dGNwX2Nvbm5lY3RfdGltZW91dD0x
MDxicj50Y3BfY29ubl93cV9tYXg9NjM1MzY8YnI+dGNwX2NybGZfcGluZz15ZXM8YnI+dGNwX2Rl
bGF5ZWRfYWNrPXllczxicj50Y3BfZmRfY2FjaGU9eWVzPGJyPnRjcF9rZWVwYWxpdmU9eWVzPGJy
PnRjcF9rZWVwY250PTM8YnI+dGNwX2tlZXBpZGxlPTMwPGJyPnRjcF9rZWVwaW50dmw9MTA8YnI+
dGNwX2xpbmdlcjI9MzA8YnI+dGNwX3JkX2J1Zl9zaXplPTgwMDAwPGJyPnRjcF9zZW5kX3RpbWVv
dXQ9MTA8YnI+dGNwX3dxX2Jsa19zaXplPTIxMDA8YnI+dGNwX3dxX21heD0xMDQ4NTc2MDxicj5v
cGVuX2ZpbGVzX2xpbWl0PTYzNTM2PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5TeXNjdGw8YnI+
PGJyPiMgVG8gaW5jcmVhc2UgdGhlIGFtb3VudCBvZiBtZW1vcnkgYXZhaWxhYmxlIGZvciBzb2Nr
ZXQgaW5wdXQvb3V0cHV0IHF1ZXVlczxicj5uZXQuaXB2NC50Y3Bfcm1lbSA9IDQwOTYgMjUxNjU4
MjQgMjUxNjU4MjQ8YnI+bmV0LmNvcmUucm1lbV9tYXggPSAyNTE2NTgyNDxicj5uZXQuY29yZS5y
bWVtX2RlZmF1bHQgPSAyNTE2NTgyNDxicj5uZXQuaXB2NC50Y3Bfd21lbSA9IDQwOTYgNjU1MzYg
MjUxNjU4MjQ8YnI+bmV0LmNvcmUud21lbV9tYXggPSAyNTE2NTgyNDxicj5uZXQuY29yZS53bWVt
X2RlZmF1bHQgPSA2NTUzNjxicj5uZXQuY29yZS5vcHRtZW1fbWF4ID0gMjUxNjU4MjQ8YnI+PGJy
PiMgVG8gbGltaXQgdGhlIG1heGltdW0gbnVtYmVyIG9mIHJlcXVlc3RzIHF1ZXVlZCB0byBhIGxp
c3RlbiBzb2NrZXQ8YnI+bmV0LmNvcmUuc29tYXhjb25uID0gMTI4PGJyPjxicj4jIFRlbGxzIFRD
UCB0byBpbnN0ZWFkIG1ha2UgZGVjaXNpb25zIHRoYXQgd291bGQgcHJlZmVyIGxvd2VyIGxhdGVu
Y3kuPGJyPm5ldC5pcHY0LnRjcF9sb3dfbGF0ZW5jeT0xPGJyPjxicj4jIE9wdGlvbmFsIChpdCB3
aWxsIGluY3JlYXNlIHBlcmZvcm1hbmNlKTxicj5uZXQuY29yZS5uZXRkZXZfbWF4X2JhY2tsb2cg
PSAxMDAwPGJyPm5ldC5pcHY0LnRjcF9tYXhfc3luX2JhY2tsb2cgPSAxMjg8YnI+PGJyPiMgRmx1
c2ggdGhlIHJvdXRpbmcgdGFibGUgdG8gbWFrZSBjaGFuZ2VzIGhhcHBlbiBpbnN0YW50bHkuPGJy
Pm5ldC5pcHY0LnJvdXRlLmZsdXNoPTE8YnI+PC9kaXY+PGRpdj48c3BhbiBjbGFzcz0iZ21haWxf
c2lnbmF0dXJlX3ByZWZpeCI+LS0gPC9zcGFuPjxicj48ZGl2IGRpcj0ibHRyIiBjbGFzcz0iZ21h
aWxfc2lnbmF0dXJlIiBkYXRhLXNtYXJ0bWFpbD0iZ21haWxfc2lnbmF0dXJlIj48ZGl2IGRpcj0i
bHRyIj5CZXN0IHJlZ2FyZHMsPGRpdj5JaG9yIChJZ29yKTxicj48L2Rpdj48L2Rpdj48L2Rpdj48
L2Rpdj48L2Rpdj4K

--===============2130123295==--


From igorolhovskiy@gmail.com Tue Aug 29 12:44:22 2023
From: Ihor Olkhovskyi <igorolhovskiy@gmail.com>
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Re: Kamailio stops processing UDP on one interface
Date: Tue, 29 Aug 2023 14:43:47 +0200
Message-ID:
 <CAJTkRNtgiMHejf_obXTXCc_tg=96wPu1jwS0PfFOL_V6Ks+7jA@mail.gmail.com>
In-Reply-To:
 <CAJTkRNtE5GgbDod0RF5CJYBONc95yLMqmRrhuk164cbp845b0g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1529578566=="

--===============1529578566==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit

Just to add some info

netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
    PID/Program name
...
udp   25167616      0 <local_interface>:5060     0.0.0.0:*
          211759/kamailio
...

So I see a huge Receive Queue on UDP for Kamailio which is not clearing.

Le mar. 29 août 2023 à 14:29, Ihor Olkhovskyi <igorolhovskiy(a)gmail.com> a
écrit :

> Hello,
>
> I've faced a bit strange issue, but a bit of preface. I have Kamailio as a
> proxy (TLS/WS <-> UDP) and second Kamailio as a presence server. At some
> point presence server accepts around 5K PUBLISH within 1 minute and sending
> around the same amount of NOTIFY to proxy Kamailio.
>
> Proxy is "transforming" protocol to TLS, but at sime point I'm starting to
> get these type of errors
>
> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
> <script>: [RELAY] Relay to <sip:X.X.X.X:51571;transport=tls> failed!
> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
>
> Some of those messages are 100% valid as client can go away or so. Some
> are not, cause I'm sure client is alive and connected.
>
> But the problem comes later. At some moment proxy Kamailio just stops
> accept UDP traffic on this interface (where it also accepts all NOTIFY's),
> at the start of the "stopping accepting" Kamailio sends OPTIONS via
> DISPATCHER but not able to receive 200 OK.
>
> Over TLS on the same interface all is ok. On other (loopback) interface
> UDP is being processed fine, so I don't suspert some limit on open files
> here.
>
> Only restart of Kamailio proxy process helps in this case.
>
> I've tuned net.core.rmem_max and net.core.rmem_default to 25 Mb, so in
> theory buffer should not be the case.
>
> Is there some internal "interface buffer" in Kamailio that is not freed
> upon failure send or maybe I've missed somethig?
>
> Kamailio 5.6.4
>
> fork=yes
> children=12
> tcp_children=12
>
> enable_tls=yes
>
> tcp_accept_no_cl=yes
> tcp_max_connections=63536
> tls_max_connections=63536
> tcp_accept_aliases=no
> tcp_async=yes
> tcp_connect_timeout=10
> tcp_conn_wq_max=63536
> tcp_crlf_ping=yes
> tcp_delayed_ack=yes
> tcp_fd_cache=yes
> tcp_keepalive=yes
> tcp_keepcnt=3
> tcp_keepidle=30
> tcp_keepintvl=10
> tcp_linger2=30
> tcp_rd_buf_size=80000
> tcp_send_timeout=10
> tcp_wq_blk_size=2100
> tcp_wq_max=10485760
> open_files_limit=63536
>
> Sysctl
>
> # To increase the amount of memory available for socket input/output queues
> net.ipv4.tcp_rmem = 4096 25165824 25165824
> net.core.rmem_max = 25165824
> net.core.rmem_default = 25165824
> net.ipv4.tcp_wmem = 4096 65536 25165824
> net.core.wmem_max = 25165824
> net.core.wmem_default = 65536
> net.core.optmem_max = 25165824
>
> # To limit the maximum number of requests queued to a listen socket
> net.core.somaxconn = 128
>
> # Tells TCP to instead make decisions that would prefer lower latency.
> net.ipv4.tcp_low_latency=1
>
> # Optional (it will increase performance)
> net.core.netdev_max_backlog = 1000
> net.ipv4.tcp_max_syn_backlog = 128
>
> # Flush the routing table to make changes happen instantly.
> net.ipv4.route.flush=1
> --
> Best regards,
> Ihor (Igor)
>


-- 
Best regards,
Ihor (Igor)



--===============1529578566==
Content-Type: text/html
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.html"
MIME-Version: 1.0

PGRpdiBkaXI9Imx0ciI+PGRpdj5KdXN0IHRvIGFkZCBzb21lIGluZm88L2Rpdj48ZGl2Pjxicj48
L2Rpdj48ZGl2Pm5ldHN0YXQgLW5scDxicj5BY3RpdmUgSW50ZXJuZXQgY29ubmVjdGlvbnMgKG9u
bHkgc2VydmVycyk8YnI+UHJvdG8gUmVjdi1RIFNlbmQtUSBMb2NhbCBBZGRyZXNzIMKgIMKgIMKg
IMKgIMKgIEZvcmVpZ24gQWRkcmVzcyDCoCDCoCDCoCDCoCBTdGF0ZSDCoCDCoCDCoCBQSUQvUHJv
Z3JhbSBuYW1lIMKgIMKgPGJyPi4uLjxicj51ZHAgwqAgMjUxNjc2MTYgwqAgwqAgwqAwICZsdDts
b2NhbF9pbnRlcmZhY2UmZ3Q7OjUwNjAgwqAgwqAgMC4wLjAuMDoqIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIDIxMTc1OS9rYW1haWxpbzxicj4uLi48YnI+PGJyPjwvZGl2
PjxkaXY+U28gSSBzZWUgYSBodWdlIFJlY2VpdmUgUXVldWUgb24gVURQIGZvciBLYW1haWxpbyB3
aGljaCBpcyBub3QgY2xlYXJpbmcuPGJyPjwvZGl2PjwvZGl2Pjxicj48ZGl2IGNsYXNzPSJnbWFp
bF9xdW90ZSI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX2F0dHIiPkxlwqBtYXIuIDI5IGFv
w7t0IDIwMjMgw6DCoDE0OjI5LCBJaG9yIE9sa2hvdnNreWkgJmx0OzxhIGhyZWY9Im1haWx0bzpp
Z29yb2xob3Zza2l5QGdtYWlsLmNvbSI+aWdvcm9saG92c2tpeUBnbWFpbC5jb208L2E+Jmd0OyBh
IMOpY3JpdMKgOjxicj48L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxl
PSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQs
MjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBkaXI9Imx0ciI+SGVsbG8sPGJyPjxkaXY+
PGJyPjwvZGl2PjxkaXY+SSYjMzk7dmUgZmFjZWQgYSBiaXQgc3RyYW5nZSBpc3N1ZSwgYnV0IGEg
Yml0IG9mIHByZWZhY2UuIEkgaGF2ZSBLYW1haWxpbyBhcyBhIHByb3h5IChUTFMvV1MgJmx0Oy0m
Z3Q7IFVEUCkgYW5kIHNlY29uZCBLYW1haWxpbyBhcyBhIHByZXNlbmNlIHNlcnZlci4gQXQgc29t
ZSBwb2ludCBwcmVzZW5jZSBzZXJ2ZXIgYWNjZXB0cyBhcm91bmQgNUsgUFVCTElTSCB3aXRoaW4g
MSBtaW51dGUgYW5kIHNlbmRpbmcgYXJvdW5kIHRoZSBzYW1lIGFtb3VudCBvZiBOT1RJRlkgdG8g
cHJveHkgS2FtYWlsaW8uPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5Qcm94eSBpcyAmcXVvdDt0
cmFuc2Zvcm1pbmcmcXVvdDsgcHJvdG9jb2wgdG8gVExTLCBidXQgYXQgc2ltZSBwb2ludCBJJiMz
OTttIHN0YXJ0aW5nIHRvIGdldCB0aGVzZSB0eXBlIG9mIGVycm9yczwvZGl2PjxkaXY+PGJyPjwv
ZGl2PjxkaXY+dG0gWy4uLy4uL2NvcmUvZm9yd2FyZC5oOjI5Ml06IG1zZ19zZW5kX2J1ZmZlcigp
OiB0Y3Bfc2VuZCBmYWlsZWQ8YnI+dG0gW3RfZndkLmM6MTU4OF06IHRfc2VuZF9icmFuY2goKTog
c2VuZGluZyByZXF1ZXN0IG9uIGJyYW5jaCAwIGZhaWxlZDxicj4mbHQ7c2NyaXB0Jmd0OzogW1JF
TEFZXSBSZWxheSB0byAmbHQ7c2lwOlguWC5YLlg6NTE1NzE7dHJhbnNwb3J0PXRscyZndDsgZmFp
bGVkITxicj50bSBbLi4vLi4vY29yZS9mb3J3YXJkLmg6MjkyXTogbXNnX3NlbmRfYnVmZmVyKCk6
IHRjcF9zZW5kIGZhaWxlZDxicj50bSBbdF9md2QuYzoxNTg4XTogdF9zZW5kX2JyYW5jaCgpOiBz
ZW5kaW5nIHJlcXVlc3Qgb24gYnJhbmNoIDAgZmFpbGVkPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRp
dj5Tb21lIG9mIHRob3NlIG1lc3NhZ2VzIGFyZSAxMDAlIHZhbGlkIGFzIGNsaWVudCBjYW4gZ28g
YXdheSBvciBzby4gU29tZSBhcmUgbm90LCBjYXVzZSBJJiMzOTttIHN1cmUgY2xpZW50IGlzIGFs
aXZlIGFuZCBjb25uZWN0ZWQuwqA8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkJ1dCB0aGUgcHJv
YmxlbSBjb21lcyBsYXRlci4gQXQgc29tZSBtb21lbnQgcHJveHkgS2FtYWlsaW8ganVzdCBzdG9w
cyBhY2NlcHQgVURQIHRyYWZmaWMgb24gdGhpcyBpbnRlcmZhY2UgKHdoZXJlIGl0IGFsc28gYWNj
ZXB0cyBhbGwgTk9USUZZJiMzOTtzKSwgYXQgdGhlIHN0YXJ0IG9mIHRoZSAmcXVvdDtzdG9wcGlu
ZyBhY2NlcHRpbmcmcXVvdDsgS2FtYWlsaW8gc2VuZHMgT1BUSU9OUyB2aWEgRElTUEFUQ0hFUiBi
dXQgbm90IGFibGUgdG8gcmVjZWl2ZSAyMDAgT0suIDxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48
ZGl2Pk92ZXIgVExTIG9uIHRoZSBzYW1lIGludGVyZmFjZSBhbGwgaXMgb2suIE9uIG90aGVyIChs
b29wYmFjaykgaW50ZXJmYWNlIFVEUCBpcyBiZWluZyBwcm9jZXNzZWQgZmluZSwgc28gSSBkb24m
IzM5O3Qgc3VzcGVydCBzb21lIGxpbWl0IG9uIG9wZW4gZmlsZXMgaGVyZS48YnI+PC9kaXY+PGRp
dj48YnI+PC9kaXY+PGRpdj5Pbmx5IHJlc3RhcnQgb2YgS2FtYWlsaW8gcHJveHkgcHJvY2VzcyBo
ZWxwcyBpbiB0aGlzIGNhc2UuIDxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkkmIzM5O3Zl
IHR1bmVkIG5ldC5jb3JlLnJtZW1fbWF4IGFuZCBuZXQuY29yZS5ybWVtX2RlZmF1bHQgdG8gMjUg
TWIsIHNvIGluIHRoZW9yeSBidWZmZXIgc2hvdWxkIG5vdCBiZSB0aGUgY2FzZS48L2Rpdj48ZGl2
Pjxicj48L2Rpdj48ZGl2PklzIHRoZXJlIHNvbWUgaW50ZXJuYWwgJnF1b3Q7aW50ZXJmYWNlIGJ1
ZmZlciZxdW90OyBpbiBLYW1haWxpbyB0aGF0IGlzIG5vdCBmcmVlZCB1cG9uIGZhaWx1cmUgc2Vu
ZCBvciBtYXliZSBJJiMzOTt2ZSBtaXNzZWQgc29tZXRoaWc/PGJyPjxicj48L2Rpdj48ZGl2Pkth
bWFpbGlvIDUuNi40PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5mb3JrPXllczxicj5jaGlsZHJl
bj0xMjxicj50Y3BfY2hpbGRyZW49MTI8YnI+PGJyPmVuYWJsZV90bHM9eWVzPGJyPjxicj50Y3Bf
YWNjZXB0X25vX2NsPXllczxicj50Y3BfbWF4X2Nvbm5lY3Rpb25zPTYzNTM2PGJyPnRsc19tYXhf
Y29ubmVjdGlvbnM9NjM1MzY8YnI+dGNwX2FjY2VwdF9hbGlhc2VzPW5vPGJyPnRjcF9hc3luYz15
ZXM8YnI+dGNwX2Nvbm5lY3RfdGltZW91dD0xMDxicj50Y3BfY29ubl93cV9tYXg9NjM1MzY8YnI+
dGNwX2NybGZfcGluZz15ZXM8YnI+dGNwX2RlbGF5ZWRfYWNrPXllczxicj50Y3BfZmRfY2FjaGU9
eWVzPGJyPnRjcF9rZWVwYWxpdmU9eWVzPGJyPnRjcF9rZWVwY250PTM8YnI+dGNwX2tlZXBpZGxl
PTMwPGJyPnRjcF9rZWVwaW50dmw9MTA8YnI+dGNwX2xpbmdlcjI9MzA8YnI+dGNwX3JkX2J1Zl9z
aXplPTgwMDAwPGJyPnRjcF9zZW5kX3RpbWVvdXQ9MTA8YnI+dGNwX3dxX2Jsa19zaXplPTIxMDA8
YnI+dGNwX3dxX21heD0xMDQ4NTc2MDxicj5vcGVuX2ZpbGVzX2xpbWl0PTYzNTM2PC9kaXY+PGRp
dj48YnI+PC9kaXY+PGRpdj5TeXNjdGw8YnI+PGJyPiMgVG8gaW5jcmVhc2UgdGhlIGFtb3VudCBv
ZiBtZW1vcnkgYXZhaWxhYmxlIGZvciBzb2NrZXQgaW5wdXQvb3V0cHV0IHF1ZXVlczxicj5uZXQu
aXB2NC50Y3Bfcm1lbSA9IDQwOTYgMjUxNjU4MjQgMjUxNjU4MjQ8YnI+bmV0LmNvcmUucm1lbV9t
YXggPSAyNTE2NTgyNDxicj5uZXQuY29yZS5ybWVtX2RlZmF1bHQgPSAyNTE2NTgyNDxicj5uZXQu
aXB2NC50Y3Bfd21lbSA9IDQwOTYgNjU1MzYgMjUxNjU4MjQ8YnI+bmV0LmNvcmUud21lbV9tYXgg
PSAyNTE2NTgyNDxicj5uZXQuY29yZS53bWVtX2RlZmF1bHQgPSA2NTUzNjxicj5uZXQuY29yZS5v
cHRtZW1fbWF4ID0gMjUxNjU4MjQ8YnI+PGJyPiMgVG8gbGltaXQgdGhlIG1heGltdW0gbnVtYmVy
IG9mIHJlcXVlc3RzIHF1ZXVlZCB0byBhIGxpc3RlbiBzb2NrZXQ8YnI+bmV0LmNvcmUuc29tYXhj
b25uID0gMTI4PGJyPjxicj4jIFRlbGxzIFRDUCB0byBpbnN0ZWFkIG1ha2UgZGVjaXNpb25zIHRo
YXQgd291bGQgcHJlZmVyIGxvd2VyIGxhdGVuY3kuPGJyPm5ldC5pcHY0LnRjcF9sb3dfbGF0ZW5j
eT0xPGJyPjxicj4jIE9wdGlvbmFsIChpdCB3aWxsIGluY3JlYXNlIHBlcmZvcm1hbmNlKTxicj5u
ZXQuY29yZS5uZXRkZXZfbWF4X2JhY2tsb2cgPSAxMDAwPGJyPm5ldC5pcHY0LnRjcF9tYXhfc3lu
X2JhY2tsb2cgPSAxMjg8YnI+PGJyPiMgRmx1c2ggdGhlIHJvdXRpbmcgdGFibGUgdG8gbWFrZSBj
aGFuZ2VzIGhhcHBlbiBpbnN0YW50bHkuPGJyPm5ldC5pcHY0LnJvdXRlLmZsdXNoPTE8YnI+PC9k
aXY+PGRpdj48c3BhbiBjbGFzcz0iZ21haWxfc2lnbmF0dXJlX3ByZWZpeCI+LS0gPC9zcGFuPjxi
cj48ZGl2IGRpcj0ibHRyIiBjbGFzcz0iZ21haWxfc2lnbmF0dXJlIj48ZGl2IGRpcj0ibHRyIj5C
ZXN0IHJlZ2FyZHMsPGRpdj5JaG9yIChJZ29yKTxicj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48
L2Rpdj4KPC9ibG9ja3F1b3RlPjwvZGl2PjxiciBjbGVhcj0iYWxsIj48YnI+PHNwYW4gY2xhc3M9
ImdtYWlsX3NpZ25hdHVyZV9wcmVmaXgiPi0tIDwvc3Bhbj48YnI+PGRpdiBkaXI9Imx0ciIgY2xh
c3M9ImdtYWlsX3NpZ25hdHVyZSI+PGRpdiBkaXI9Imx0ciI+QmVzdCByZWdhcmRzLDxkaXY+SWhv
ciAoSWdvcik8YnI+PC9kaXY+PC9kaXY+PC9kaXY+Cg==

--===============1529578566==--


From bastian.triller@gmail.com Wed Aug 30 12:23:20 2023
From: Bastian Triller <bastian.triller@gmail.com>
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Re: Kamailio stops processing UDP on one interface
Date: Wed, 30 Aug 2023 14:22:47 +0200
Message-ID:
 <CALJ8HB4u3E5xd5C4Dr8Fheoi3wNS+_FfFi3fxPgU2hybgXAEDQ@mail.gmail.com>
In-Reply-To:
 <CAJTkRNtgiMHejf_obXTXCc_tg=96wPu1jwS0PfFOL_V6Ks+7jA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0457605947=="

--===============0457605947==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Are drops increasing on that socket while it is happening?
ss -l src <local_interface> -u sport 5060 -m

On Tue, Aug 29, 2023 at 3:26=E2=80=AFPM Ihor Olkhovskyi <igorolhovskiy(a)gmai=
l.com>
wrote:

> Just to add some info
>
> netstat -nlp
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address           Foreign Address         State
>       PID/Program name
> ...
> udp   25167616      0 <local_interface>:5060     0.0.0.0:*
>             211759/kamailio
> ...
>
> So I see a huge Receive Queue on UDP for Kamailio which is not clearing.
>
> Le mar. 29 ao=C3=BBt 2023 =C3=A0 14:29, Ihor Olkhovskyi <igorolhovskiy(a)gm=
ail.com> a
> =C3=A9crit :
>
>> Hello,
>>
>> I've faced a bit strange issue, but a bit of preface. I have Kamailio as
>> a proxy (TLS/WS <-> UDP) and second Kamailio as a presence server. At some
>> point presence server accepts around 5K PUBLISH within 1 minute and sending
>> around the same amount of NOTIFY to proxy Kamailio.
>>
>> Proxy is "transforming" protocol to TLS, but at sime point I'm starting
>> to get these type of errors
>>
>> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
>> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
>> <script>: [RELAY] Relay to <sip:X.X.X.X:51571;transport=3Dtls> failed!
>> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
>> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
>>
>> Some of those messages are 100% valid as client can go away or so. Some
>> are not, cause I'm sure client is alive and connected.
>>
>> But the problem comes later. At some moment proxy Kamailio just stops
>> accept UDP traffic on this interface (where it also accepts all NOTIFY's),
>> at the start of the "stopping accepting" Kamailio sends OPTIONS via
>> DISPATCHER but not able to receive 200 OK.
>>
>> Over TLS on the same interface all is ok. On other (loopback) interface
>> UDP is being processed fine, so I don't suspert some limit on open files
>> here.
>>
>> Only restart of Kamailio proxy process helps in this case.
>>
>> I've tuned net.core.rmem_max and net.core.rmem_default to 25 Mb, so in
>> theory buffer should not be the case.
>>
>> Is there some internal "interface buffer" in Kamailio that is not freed
>> upon failure send or maybe I've missed somethig?
>>
>> Kamailio 5.6.4
>>
>> fork=3Dyes
>> children=3D12
>> tcp_children=3D12
>>
>> enable_tls=3Dyes
>>
>> tcp_accept_no_cl=3Dyes
>> tcp_max_connections=3D63536
>> tls_max_connections=3D63536
>> tcp_accept_aliases=3Dno
>> tcp_async=3Dyes
>> tcp_connect_timeout=3D10
>> tcp_conn_wq_max=3D63536
>> tcp_crlf_ping=3Dyes
>> tcp_delayed_ack=3Dyes
>> tcp_fd_cache=3Dyes
>> tcp_keepalive=3Dyes
>> tcp_keepcnt=3D3
>> tcp_keepidle=3D30
>> tcp_keepintvl=3D10
>> tcp_linger2=3D30
>> tcp_rd_buf_size=3D80000
>> tcp_send_timeout=3D10
>> tcp_wq_blk_size=3D2100
>> tcp_wq_max=3D10485760
>> open_files_limit=3D63536
>>
>> Sysctl
>>
>> # To increase the amount of memory available for socket input/output
>> queues
>> net.ipv4.tcp_rmem =3D 4096 25165824 25165824
>> net.core.rmem_max =3D 25165824
>> net.core.rmem_default =3D 25165824
>> net.ipv4.tcp_wmem =3D 4096 65536 25165824
>> net.core.wmem_max =3D 25165824
>> net.core.wmem_default =3D 65536
>> net.core.optmem_max =3D 25165824
>>
>> # To limit the maximum number of requests queued to a listen socket
>> net.core.somaxconn =3D 128
>>
>> # Tells TCP to instead make decisions that would prefer lower latency.
>> net.ipv4.tcp_low_latency=3D1
>>
>> # Optional (it will increase performance)
>> net.core.netdev_max_backlog =3D 1000
>> net.ipv4.tcp_max_syn_backlog =3D 128
>>
>> # Flush the routing table to make changes happen instantly.
>> net.ipv4.route.flush=3D1
>> --
>> Best regards,
>> Ihor (Igor)
>>
>
>
> --
> Best regards,
> Ihor (Igor)
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to
> the sender!
> Edit mailing list options or unsubscribe:
>



--===============0457605947==
Content-Type: text/html
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.html"
MIME-Version: 1.0

PGRpdiBkaXI9Imx0ciI+QXJlIGRyb3BzIGluY3JlYXNpbmcgb24gdGhhdCBzb2NrZXQgd2hpbGUg
aXQgaXMgaGFwcGVuaW5nPzxkaXY+c3MgLWwgc3JjICZsdDs8c3BhbiBzdHlsZT0iY29sb3I6cmdi
KDAsMCwwKSI+bG9jYWxfaW50ZXJmYWNlJmd0Ozwvc3Bhbj7CoC11IHNwb3J0IDUwNjAgLW08L2Rp
dj48L2Rpdj48YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiIGNsYXNz
PSJnbWFpbF9hdHRyIj5PbiBUdWUsIEF1ZyAyOSwgMjAyMyBhdCAzOjI24oCvUE0gSWhvciBPbGto
b3Zza3lpICZsdDs8YSBocmVmPSJtYWlsdG86aWdvcm9saG92c2tpeUBnbWFpbC5jb20iPmlnb3Jv
bGhvdnNraXlAZ21haWwuY29tPC9hPiZndDsgd3JvdGU6PGJyPjwvZGl2PjxibG9ja3F1b3RlIGNs
YXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXIt
bGVmdC13aWR0aDoxcHg7Ym9yZGVyLWxlZnQtc3R5bGU6c29saWQ7Ym9yZGVyLWxlZnQtY29sb3I6
cmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48ZGl2Pkp1
c3QgdG8gYWRkIHNvbWUgaW5mbzwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+bmV0c3RhdCAtbmxw
PGJyPkFjdGl2ZSBJbnRlcm5ldCBjb25uZWN0aW9ucyAob25seSBzZXJ2ZXJzKTxicj5Qcm90byBS
ZWN2LVEgU2VuZC1RIExvY2FsIEFkZHJlc3MgwqAgwqAgwqAgwqAgwqAgRm9yZWlnbiBBZGRyZXNz
IMKgIMKgIMKgIMKgIFN0YXRlIMKgIMKgIMKgIFBJRC9Qcm9ncmFtIG5hbWUgwqAgwqA8YnI+Li4u
PGJyPnVkcCDCoCAyNTE2NzYxNiDCoCDCoCDCoDAgJmx0O2xvY2FsX2ludGVyZmFjZSZndDs6NTA2
MCDCoCDCoCAwLjAuMC4wOiogwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
MjExNzU5L2thbWFpbGlvPGJyPi4uLjxicj48YnI+PC9kaXY+PGRpdj5TbyBJIHNlZSBhIGh1Z2Ug
UmVjZWl2ZSBRdWV1ZSBvbiBVRFAgZm9yIEthbWFpbGlvIHdoaWNoIGlzIG5vdCBjbGVhcmluZy48
YnI+PC9kaXY+PC9kaXY+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IGRpcj0ibHRy
IiBjbGFzcz0iZ21haWxfYXR0ciI+TGXCoG1hci4gMjkgYW/Du3QgMjAyMyDDoMKgMTQ6MjksIElo
b3IgT2xraG92c2t5aSAmbHQ7PGEgaHJlZj0ibWFpbHRvOmlnb3JvbGhvdnNraXlAZ21haWwuY29t
IiB0YXJnZXQ9Il9ibGFuayI+aWdvcm9saG92c2tpeUBnbWFpbC5jb208L2E+Jmd0OyBhIMOpY3Jp
dMKgOjxicj48L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJn
aW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQtd2lkdGg6MXB4O2JvcmRlci1sZWZ0LXN0
eWxlOnNvbGlkO2JvcmRlci1sZWZ0LWNvbG9yOnJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0
OjFleCI+PGRpdiBkaXI9Imx0ciI+SGVsbG8sPGJyPjxkaXY+PGJyPjwvZGl2PjxkaXY+SSYjMzk7
dmUgZmFjZWQgYSBiaXQgc3RyYW5nZSBpc3N1ZSwgYnV0IGEgYml0IG9mIHByZWZhY2UuIEkgaGF2
ZSBLYW1haWxpbyBhcyBhIHByb3h5IChUTFMvV1MgJmx0Oy0mZ3Q7IFVEUCkgYW5kIHNlY29uZCBL
YW1haWxpbyBhcyBhIHByZXNlbmNlIHNlcnZlci4gQXQgc29tZSBwb2ludCBwcmVzZW5jZSBzZXJ2
ZXIgYWNjZXB0cyBhcm91bmQgNUsgUFVCTElTSCB3aXRoaW4gMSBtaW51dGUgYW5kIHNlbmRpbmcg
YXJvdW5kIHRoZSBzYW1lIGFtb3VudCBvZiBOT1RJRlkgdG8gcHJveHkgS2FtYWlsaW8uPC9kaXY+
PGRpdj48YnI+PC9kaXY+PGRpdj5Qcm94eSBpcyAmcXVvdDt0cmFuc2Zvcm1pbmcmcXVvdDsgcHJv
dG9jb2wgdG8gVExTLCBidXQgYXQgc2ltZSBwb2ludCBJJiMzOTttIHN0YXJ0aW5nIHRvIGdldCB0
aGVzZSB0eXBlIG9mIGVycm9yczwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+dG0gWy4uLy4uL2Nv
cmUvZm9yd2FyZC5oOjI5Ml06IG1zZ19zZW5kX2J1ZmZlcigpOiB0Y3Bfc2VuZCBmYWlsZWQ8YnI+
dG0gW3RfZndkLmM6MTU4OF06IHRfc2VuZF9icmFuY2goKTogc2VuZGluZyByZXF1ZXN0IG9uIGJy
YW5jaCAwIGZhaWxlZDxicj4mbHQ7c2NyaXB0Jmd0OzogW1JFTEFZXSBSZWxheSB0byAmbHQ7c2lw
OlguWC5YLlg6NTE1NzE7dHJhbnNwb3J0PXRscyZndDsgZmFpbGVkITxicj50bSBbLi4vLi4vY29y
ZS9mb3J3YXJkLmg6MjkyXTogbXNnX3NlbmRfYnVmZmVyKCk6IHRjcF9zZW5kIGZhaWxlZDxicj50
bSBbdF9md2QuYzoxNTg4XTogdF9zZW5kX2JyYW5jaCgpOiBzZW5kaW5nIHJlcXVlc3Qgb24gYnJh
bmNoIDAgZmFpbGVkPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5Tb21lIG9mIHRob3NlIG1lc3Nh
Z2VzIGFyZSAxMDAlIHZhbGlkIGFzIGNsaWVudCBjYW4gZ28gYXdheSBvciBzby4gU29tZSBhcmUg
bm90LCBjYXVzZSBJJiMzOTttIHN1cmUgY2xpZW50IGlzIGFsaXZlIGFuZCBjb25uZWN0ZWQuwqA8
L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkJ1dCB0aGUgcHJvYmxlbSBjb21lcyBsYXRlci4gQXQg
c29tZSBtb21lbnQgcHJveHkgS2FtYWlsaW8ganVzdCBzdG9wcyBhY2NlcHQgVURQIHRyYWZmaWMg
b24gdGhpcyBpbnRlcmZhY2UgKHdoZXJlIGl0IGFsc28gYWNjZXB0cyBhbGwgTk9USUZZJiMzOTtz
KSwgYXQgdGhlIHN0YXJ0IG9mIHRoZSAmcXVvdDtzdG9wcGluZyBhY2NlcHRpbmcmcXVvdDsgS2Ft
YWlsaW8gc2VuZHMgT1BUSU9OUyB2aWEgRElTUEFUQ0hFUiBidXQgbm90IGFibGUgdG8gcmVjZWl2
ZSAyMDAgT0suIDxicj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pk92ZXIgVExTIG9uIHRoZSBz
YW1lIGludGVyZmFjZSBhbGwgaXMgb2suIE9uIG90aGVyIChsb29wYmFjaykgaW50ZXJmYWNlIFVE
UCBpcyBiZWluZyBwcm9jZXNzZWQgZmluZSwgc28gSSBkb24mIzM5O3Qgc3VzcGVydCBzb21lIGxp
bWl0IG9uIG9wZW4gZmlsZXMgaGVyZS48YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5Pbmx5
IHJlc3RhcnQgb2YgS2FtYWlsaW8gcHJveHkgcHJvY2VzcyBoZWxwcyBpbiB0aGlzIGNhc2UuIDxi
cj48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PkkmIzM5O3ZlIHR1bmVkIG5ldC5jb3JlLnJtZW1f
bWF4IGFuZCBuZXQuY29yZS5ybWVtX2RlZmF1bHQgdG8gMjUgTWIsIHNvIGluIHRoZW9yeSBidWZm
ZXIgc2hvdWxkIG5vdCBiZSB0aGUgY2FzZS48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PklzIHRo
ZXJlIHNvbWUgaW50ZXJuYWwgJnF1b3Q7aW50ZXJmYWNlIGJ1ZmZlciZxdW90OyBpbiBLYW1haWxp
byB0aGF0IGlzIG5vdCBmcmVlZCB1cG9uIGZhaWx1cmUgc2VuZCBvciBtYXliZSBJJiMzOTt2ZSBt
aXNzZWQgc29tZXRoaWc/PGJyPjxicj48L2Rpdj48ZGl2PkthbWFpbGlvIDUuNi40PC9kaXY+PGRp
dj48YnI+PC9kaXY+PGRpdj5mb3JrPXllczxicj5jaGlsZHJlbj0xMjxicj50Y3BfY2hpbGRyZW49
MTI8YnI+PGJyPmVuYWJsZV90bHM9eWVzPGJyPjxicj50Y3BfYWNjZXB0X25vX2NsPXllczxicj50
Y3BfbWF4X2Nvbm5lY3Rpb25zPTYzNTM2PGJyPnRsc19tYXhfY29ubmVjdGlvbnM9NjM1MzY8YnI+
dGNwX2FjY2VwdF9hbGlhc2VzPW5vPGJyPnRjcF9hc3luYz15ZXM8YnI+dGNwX2Nvbm5lY3RfdGlt
ZW91dD0xMDxicj50Y3BfY29ubl93cV9tYXg9NjM1MzY8YnI+dGNwX2NybGZfcGluZz15ZXM8YnI+
dGNwX2RlbGF5ZWRfYWNrPXllczxicj50Y3BfZmRfY2FjaGU9eWVzPGJyPnRjcF9rZWVwYWxpdmU9
eWVzPGJyPnRjcF9rZWVwY250PTM8YnI+dGNwX2tlZXBpZGxlPTMwPGJyPnRjcF9rZWVwaW50dmw9
MTA8YnI+dGNwX2xpbmdlcjI9MzA8YnI+dGNwX3JkX2J1Zl9zaXplPTgwMDAwPGJyPnRjcF9zZW5k
X3RpbWVvdXQ9MTA8YnI+dGNwX3dxX2Jsa19zaXplPTIxMDA8YnI+dGNwX3dxX21heD0xMDQ4NTc2
MDxicj5vcGVuX2ZpbGVzX2xpbWl0PTYzNTM2PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5TeXNj
dGw8YnI+PGJyPiMgVG8gaW5jcmVhc2UgdGhlIGFtb3VudCBvZiBtZW1vcnkgYXZhaWxhYmxlIGZv
ciBzb2NrZXQgaW5wdXQvb3V0cHV0IHF1ZXVlczxicj5uZXQuaXB2NC50Y3Bfcm1lbSA9IDQwOTYg
MjUxNjU4MjQgMjUxNjU4MjQ8YnI+bmV0LmNvcmUucm1lbV9tYXggPSAyNTE2NTgyNDxicj5uZXQu
Y29yZS5ybWVtX2RlZmF1bHQgPSAyNTE2NTgyNDxicj5uZXQuaXB2NC50Y3Bfd21lbSA9IDQwOTYg
NjU1MzYgMjUxNjU4MjQ8YnI+bmV0LmNvcmUud21lbV9tYXggPSAyNTE2NTgyNDxicj5uZXQuY29y
ZS53bWVtX2RlZmF1bHQgPSA2NTUzNjxicj5uZXQuY29yZS5vcHRtZW1fbWF4ID0gMjUxNjU4MjQ8
YnI+PGJyPiMgVG8gbGltaXQgdGhlIG1heGltdW0gbnVtYmVyIG9mIHJlcXVlc3RzIHF1ZXVlZCB0
byBhIGxpc3RlbiBzb2NrZXQ8YnI+bmV0LmNvcmUuc29tYXhjb25uID0gMTI4PGJyPjxicj4jIFRl
bGxzIFRDUCB0byBpbnN0ZWFkIG1ha2UgZGVjaXNpb25zIHRoYXQgd291bGQgcHJlZmVyIGxvd2Vy
IGxhdGVuY3kuPGJyPm5ldC5pcHY0LnRjcF9sb3dfbGF0ZW5jeT0xPGJyPjxicj4jIE9wdGlvbmFs
IChpdCB3aWxsIGluY3JlYXNlIHBlcmZvcm1hbmNlKTxicj5uZXQuY29yZS5uZXRkZXZfbWF4X2Jh
Y2tsb2cgPSAxMDAwPGJyPm5ldC5pcHY0LnRjcF9tYXhfc3luX2JhY2tsb2cgPSAxMjg8YnI+PGJy
PiMgRmx1c2ggdGhlIHJvdXRpbmcgdGFibGUgdG8gbWFrZSBjaGFuZ2VzIGhhcHBlbiBpbnN0YW50
bHkuPGJyPm5ldC5pcHY0LnJvdXRlLmZsdXNoPTE8YnI+PC9kaXY+PGRpdj48c3BhbiBjbGFzcz0i
Z21haWxfc2lnbmF0dXJlX3ByZWZpeCI+LS0gPC9zcGFuPjxicj48ZGl2IGRpcj0ibHRyIiBjbGFz
cz0iZ21haWxfc2lnbmF0dXJlIj48ZGl2IGRpcj0ibHRyIj5CZXN0IHJlZ2FyZHMsPGRpdj5JaG9y
IChJZ29yKTxicj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj4KPC9ibG9ja3F1b3RlPjwv
ZGl2PjxiciBjbGVhcj0iYWxsIj48YnI+PHNwYW4gY2xhc3M9ImdtYWlsX3NpZ25hdHVyZV9wcmVm
aXgiPi0tIDwvc3Bhbj48YnI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSI+
PGRpdiBkaXI9Imx0ciI+QmVzdCByZWdhcmRzLDxkaXY+SWhvciAoSWdvcik8YnI+PC9kaXY+PC9k
aXY+PC9kaXY+Cl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX188YnI+CkthbWFpbGlvIC0gVXNlcnMgTWFpbGluZyBMaXN0IC0gTm9uIENvbW1l
cmNpYWwgRGlzY3Vzc2lvbnM8YnI+ClRvIHVuc3Vic2NyaWJlIHNlbmQgYW4gZW1haWwgdG8gPGEg
aHJlZj0ibWFpbHRvOnNyLXVzZXJzLWxlYXZlQGxpc3RzLmthbWFpbGlvLm9yZyIgdGFyZ2V0PSJf
YmxhbmsiPnNyLXVzZXJzLWxlYXZlQGxpc3RzLmthbWFpbGlvLm9yZzwvYT48YnI+CkltcG9ydGFu
dDoga2VlcCB0aGUgbWFpbGluZyBsaXN0IGluIHRoZSByZWNpcGllbnRzLCBkbyBub3QgcmVwbHkg
b25seSB0byB0aGUgc2VuZGVyITxicj4KRWRpdCBtYWlsaW5nIGxpc3Qgb3B0aW9ucyBvciB1bnN1
YnNjcmliZTo8YnI+CjwvYmxvY2txdW90ZT48L2Rpdj4K

--===============0457605947==--


From igorolhovskiy@gmail.com Wed Aug 30 13:30:47 2023
From: Ihor Olkhovskyi <igorolhovskiy@gmail.com>
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Re: Kamailio stops processing UDP on one interface
Date: Wed, 30 Aug 2023 15:30:21 +0200
Message-ID:
 <CAJTkRNtpBDrSrwPiZpxZpEfFHjLaHAmh81pXF0bYodMoM3OHMg@mail.gmail.com>
In-Reply-To:
 <CALJ8HB4u3E5xd5C4Dr8Fheoi3wNS+_FfFi3fxPgU2hybgXAEDQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============1295552112=="

--===============1295552112==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

They are increasing, actually

# ss -l -u -m src X.X.X.X/Y
State       Recv-Q Send-Q
Local Address:Port
       Peer Address:Port
UNCONN      25167616 0
X.X.X.X:sip
           *:*
         skmem:(r25167616,rb25165824,t0,tb65536,f2304,w0,o0,bl0,d514894)

Le mer. 30 ao=C3=BBt 2023 =C3=A0 15:04, Bastian Triller <bastian.triller(a)gm=
ail.com> a
=C3=A9crit :

> Are drops increasing on that socket while it is happening?
> ss -l src <local_interface> -u sport 5060 -m
>
> On Tue, Aug 29, 2023 at 3:26=E2=80=AFPM Ihor Olkhovskyi <igorolhovskiy(a)gm=
ail.com>
> wrote:
>
>> Just to add some info
>>
>> netstat -nlp
>> Active Internet connections (only servers)
>> Proto Recv-Q Send-Q Local Address           Foreign Address         State
>>       PID/Program name
>> ...
>> udp   25167616      0 <local_interface>:5060     0.0.0.0:*
>>             211759/kamailio
>> ...
>>
>> So I see a huge Receive Queue on UDP for Kamailio which is not clearing.
>>
>> Le mar. 29 ao=C3=BBt 2023 =C3=A0 14:29, Ihor Olkhovskyi <igorolhovskiy(a)g=
mail.com>
>> a =C3=A9crit :
>>
>>> Hello,
>>>
>>> I've faced a bit strange issue, but a bit of preface. I have Kamailio as
>>> a proxy (TLS/WS <-> UDP) and second Kamailio as a presence server. At some
>>> point presence server accepts around 5K PUBLISH within 1 minute and sendi=
ng
>>> around the same amount of NOTIFY to proxy Kamailio.
>>>
>>> Proxy is "transforming" protocol to TLS, but at sime point I'm starting
>>> to get these type of errors
>>>
>>> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
>>> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
>>> <script>: [RELAY] Relay to <sip:X.X.X.X:51571;transport=3Dtls> failed!
>>> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
>>> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
>>>
>>> Some of those messages are 100% valid as client can go away or so. Some
>>> are not, cause I'm sure client is alive and connected.
>>>
>>> But the problem comes later. At some moment proxy Kamailio just stops
>>> accept UDP traffic on this interface (where it also accepts all NOTIFY's),
>>> at the start of the "stopping accepting" Kamailio sends OPTIONS via
>>> DISPATCHER but not able to receive 200 OK.
>>>
>>> Over TLS on the same interface all is ok. On other (loopback) interface
>>> UDP is being processed fine, so I don't suspert some limit on open files
>>> here.
>>>
>>> Only restart of Kamailio proxy process helps in this case.
>>>
>>> I've tuned net.core.rmem_max and net.core.rmem_default to 25 Mb, so in
>>> theory buffer should not be the case.
>>>
>>> Is there some internal "interface buffer" in Kamailio that is not freed
>>> upon failure send or maybe I've missed somethig?
>>>
>>> Kamailio 5.6.4
>>>
>>> fork=3Dyes
>>> children=3D12
>>> tcp_children=3D12
>>>
>>> enable_tls=3Dyes
>>>
>>> tcp_accept_no_cl=3Dyes
>>> tcp_max_connections=3D63536
>>> tls_max_connections=3D63536
>>> tcp_accept_aliases=3Dno
>>> tcp_async=3Dyes
>>> tcp_connect_timeout=3D10
>>> tcp_conn_wq_max=3D63536
>>> tcp_crlf_ping=3Dyes
>>> tcp_delayed_ack=3Dyes
>>> tcp_fd_cache=3Dyes
>>> tcp_keepalive=3Dyes
>>> tcp_keepcnt=3D3
>>> tcp_keepidle=3D30
>>> tcp_keepintvl=3D10
>>> tcp_linger2=3D30
>>> tcp_rd_buf_size=3D80000
>>> tcp_send_timeout=3D10
>>> tcp_wq_blk_size=3D2100
>>> tcp_wq_max=3D10485760
>>> open_files_limit=3D63536
>>>
>>> Sysctl
>>>
>>> # To increase the amount of memory available for socket input/output
>>> queues
>>> net.ipv4.tcp_rmem =3D 4096 25165824 25165824
>>> net.core.rmem_max =3D 25165824
>>> net.core.rmem_default =3D 25165824
>>> net.ipv4.tcp_wmem =3D 4096 65536 25165824
>>> net.core.wmem_max =3D 25165824
>>> net.core.wmem_default =3D 65536
>>> net.core.optmem_max =3D 25165824
>>>
>>> # To limit the maximum number of requests queued to a listen socket
>>> net.core.somaxconn =3D 128
>>>
>>> # Tells TCP to instead make decisions that would prefer lower latency.
>>> net.ipv4.tcp_low_latency=3D1
>>>
>>> # Optional (it will increase performance)
>>> net.core.netdev_max_backlog =3D 1000
>>> net.ipv4.tcp_max_syn_backlog =3D 128
>>>
>>> # Flush the routing table to make changes happen instantly.
>>> net.ipv4.route.flush=3D1
>>> --
>>> Best regards,
>>> Ihor (Igor)
>>>
>>
>>
>> --
>> Best regards,
>> Ihor (Igor)
>> __________________________________________________________
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only to
>> the sender!
>> Edit mailing list options or unsubscribe:
>>
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to
> the sender!
> Edit mailing list options or unsubscribe:
>


--=20
Best regards,
Ihor (Igor)



--===============1295552112==
Content-Type: text/html
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.html"
MIME-Version: 1.0

PGRpdiBkaXI9Imx0ciI+PGRpdj5UaGV5IGFyZSBpbmNyZWFzaW5nLCBhY3R1YWxseTwvZGl2Pjxk
aXY+PGJyPjwvZGl2PjxkaXY+IyBzcyAtbCAtdSAtbSBzcmMgWC5YLlguWC9ZPGJyPlN0YXRlIMKg
IMKgIMKgIFJlY3YtUSBTZW5kLVEgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgTG9jYWwgQWRkcmVzczpQb3J0IMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgUGVlciBBZGRyZXNzOlBvcnQgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqA8YnI+VU5DT05OIMKgIMKgIMKgMjUxNjc2MTYgMCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCBYLlguWC5YOnNpcCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCo6KiDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoDxicj7C
oCDCoCDCoCDCoCDCoHNrbWVtOihyMjUxNjc2MTYscmIyNTE2NTgyNCx0MCx0YjY1NTM2LGYyMzA0
LHcwLG8wLGJsMCxkNTE0ODk0KTwvZGl2PjwvZGl2Pjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90
ZSI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX2F0dHIiPkxlwqBtZXIuIDMwIGFvw7t0IDIw
MjMgw6DCoDE1OjA0LCBCYXN0aWFuIFRyaWxsZXIgJmx0OzxhIGhyZWY9Im1haWx0bzpiYXN0aWFu
LnRyaWxsZXJAZ21haWwuY29tIj5iYXN0aWFuLnRyaWxsZXJAZ21haWwuY29tPC9hPiZndDsgYSDD
qWNyaXTCoDo8YnI+PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0i
bWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIw
NCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgZGlyPSJsdHIiPkFyZSBkcm9wcyBpbmNyZWFz
aW5nIG9uIHRoYXQgc29ja2V0IHdoaWxlIGl0IGlzIGhhcHBlbmluZz88ZGl2PnNzIC1sIHNyYyAm
bHQ7PHNwYW4gc3R5bGU9ImNvbG9yOnJnYigwLDAsMCkiPmxvY2FsX2ludGVyZmFjZSZndDs8L3Nw
YW4+wqAtdSBzcG9ydCA1MDYwIC1tPC9kaXY+PC9kaXY+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1
b3RlIj48ZGl2IGRpcj0ibHRyIiBjbGFzcz0iZ21haWxfYXR0ciI+T24gVHVlLCBBdWcgMjksIDIw
MjMgYXQgMzoyNuKAr1BNIElob3IgT2xraG92c2t5aSAmbHQ7PGEgaHJlZj0ibWFpbHRvOmlnb3Jv
bGhvdnNraXlAZ21haWwuY29tIiB0YXJnZXQ9Il9ibGFuayI+aWdvcm9saG92c2tpeUBnbWFpbC5j
b208L2E+Jmd0OyB3cm90ZTo8YnI+PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3Rl
IiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCBy
Z2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgZGlyPSJsdHIiPjxkaXY+SnVz
dCB0byBhZGQgc29tZSBpbmZvPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5uZXRzdGF0IC1ubHA8
YnI+QWN0aXZlIEludGVybmV0IGNvbm5lY3Rpb25zIChvbmx5IHNlcnZlcnMpPGJyPlByb3RvIFJl
Y3YtUSBTZW5kLVEgTG9jYWwgQWRkcmVzcyDCoCDCoCDCoCDCoCDCoCBGb3JlaWduIEFkZHJlc3Mg
wqAgwqAgwqAgwqAgU3RhdGUgwqAgwqAgwqAgUElEL1Byb2dyYW0gbmFtZSDCoCDCoDxicj4uLi48
YnI+dWRwIMKgIDI1MTY3NjE2IMKgIMKgIMKgMCAmbHQ7bG9jYWxfaW50ZXJmYWNlJmd0Ozo1MDYw
IMKgIMKgIDAuMC4wLjA6KiDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCAy
MTE3NTkva2FtYWlsaW88YnI+Li4uPGJyPjxicj48L2Rpdj48ZGl2PlNvIEkgc2VlIGEgaHVnZSBS
ZWNlaXZlIFF1ZXVlIG9uIFVEUCBmb3IgS2FtYWlsaW8gd2hpY2ggaXMgbm90IGNsZWFyaW5nLjxi
cj48L2Rpdj48L2Rpdj48YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIi
IGNsYXNzPSJnbWFpbF9hdHRyIj5MZcKgbWFyLiAyOSBhb8O7dCAyMDIzIMOgwqAxNDoyOSwgSWhv
ciBPbGtob3Zza3lpICZsdDs8YSBocmVmPSJtYWlsdG86aWdvcm9saG92c2tpeUBnbWFpbC5jb20i
IHRhcmdldD0iX2JsYW5rIj5pZ29yb2xob3Zza2l5QGdtYWlsLmNvbTwvYT4mZ3Q7IGEgw6ljcml0
wqA6PGJyPjwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdp
bjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0
KTtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj5IZWxsbyw8YnI+PGRpdj48YnI+PC9k
aXY+PGRpdj5JJiMzOTt2ZSBmYWNlZCBhIGJpdCBzdHJhbmdlIGlzc3VlLCBidXQgYSBiaXQgb2Yg
cHJlZmFjZS4gSSBoYXZlIEthbWFpbGlvIGFzIGEgcHJveHkgKFRMUy9XUyAmbHQ7LSZndDsgVURQ
KSBhbmQgc2Vjb25kIEthbWFpbGlvIGFzIGEgcHJlc2VuY2Ugc2VydmVyLiBBdCBzb21lIHBvaW50
IHByZXNlbmNlIHNlcnZlciBhY2NlcHRzIGFyb3VuZCA1SyBQVUJMSVNIIHdpdGhpbiAxIG1pbnV0
ZSBhbmQgc2VuZGluZyBhcm91bmQgdGhlIHNhbWUgYW1vdW50IG9mIE5PVElGWSB0byBwcm94eSBL
YW1haWxpby48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PlByb3h5IGlzICZxdW90O3RyYW5zZm9y
bWluZyZxdW90OyBwcm90b2NvbCB0byBUTFMsIGJ1dCBhdCBzaW1lIHBvaW50IEkmIzM5O20gc3Rh
cnRpbmcgdG8gZ2V0IHRoZXNlIHR5cGUgb2YgZXJyb3JzPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRp
dj50bSBbLi4vLi4vY29yZS9mb3J3YXJkLmg6MjkyXTogbXNnX3NlbmRfYnVmZmVyKCk6IHRjcF9z
ZW5kIGZhaWxlZDxicj50bSBbdF9md2QuYzoxNTg4XTogdF9zZW5kX2JyYW5jaCgpOiBzZW5kaW5n
IHJlcXVlc3Qgb24gYnJhbmNoIDAgZmFpbGVkPGJyPiZsdDtzY3JpcHQmZ3Q7OiBbUkVMQVldIFJl
bGF5IHRvICZsdDtzaXA6WC5YLlguWDo1MTU3MTt0cmFuc3BvcnQ9dGxzJmd0OyBmYWlsZWQhPGJy
PnRtIFsuLi8uLi9jb3JlL2ZvcndhcmQuaDoyOTJdOiBtc2dfc2VuZF9idWZmZXIoKTogdGNwX3Nl
bmQgZmFpbGVkPGJyPnRtIFt0X2Z3ZC5jOjE1ODhdOiB0X3NlbmRfYnJhbmNoKCk6IHNlbmRpbmcg
cmVxdWVzdCBvbiBicmFuY2ggMCBmYWlsZWQ8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PlNvbWUg
b2YgdGhvc2UgbWVzc2FnZXMgYXJlIDEwMCUgdmFsaWQgYXMgY2xpZW50IGNhbiBnbyBhd2F5IG9y
IHNvLiBTb21lIGFyZSBub3QsIGNhdXNlIEkmIzM5O20gc3VyZSBjbGllbnQgaXMgYWxpdmUgYW5k
IGNvbm5lY3RlZC7CoDwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+QnV0IHRoZSBwcm9ibGVtIGNv
bWVzIGxhdGVyLiBBdCBzb21lIG1vbWVudCBwcm94eSBLYW1haWxpbyBqdXN0IHN0b3BzIGFjY2Vw
dCBVRFAgdHJhZmZpYyBvbiB0aGlzIGludGVyZmFjZSAod2hlcmUgaXQgYWxzbyBhY2NlcHRzIGFs
bCBOT1RJRlkmIzM5O3MpLCBhdCB0aGUgc3RhcnQgb2YgdGhlICZxdW90O3N0b3BwaW5nIGFjY2Vw
dGluZyZxdW90OyBLYW1haWxpbyBzZW5kcyBPUFRJT05TIHZpYSBESVNQQVRDSEVSIGJ1dCBub3Qg
YWJsZSB0byByZWNlaXZlIDIwMCBPSy4gPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+T3Zl
ciBUTFMgb24gdGhlIHNhbWUgaW50ZXJmYWNlIGFsbCBpcyBvay4gT24gb3RoZXIgKGxvb3BiYWNr
KSBpbnRlcmZhY2UgVURQIGlzIGJlaW5nIHByb2Nlc3NlZCBmaW5lLCBzbyBJIGRvbiYjMzk7dCBz
dXNwZXJ0IHNvbWUgbGltaXQgb24gb3BlbiBmaWxlcyBoZXJlLjxicj48L2Rpdj48ZGl2Pjxicj48
L2Rpdj48ZGl2Pk9ubHkgcmVzdGFydCBvZiBLYW1haWxpbyBwcm94eSBwcm9jZXNzIGhlbHBzIGlu
IHRoaXMgY2FzZS4gPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+SSYjMzk7dmUgdHVuZWQg
bmV0LmNvcmUucm1lbV9tYXggYW5kIG5ldC5jb3JlLnJtZW1fZGVmYXVsdCB0byAyNSBNYiwgc28g
aW4gdGhlb3J5IGJ1ZmZlciBzaG91bGQgbm90IGJlIHRoZSBjYXNlLjwvZGl2PjxkaXY+PGJyPjwv
ZGl2PjxkaXY+SXMgdGhlcmUgc29tZSBpbnRlcm5hbCAmcXVvdDtpbnRlcmZhY2UgYnVmZmVyJnF1
b3Q7IGluIEthbWFpbGlvIHRoYXQgaXMgbm90IGZyZWVkIHVwb24gZmFpbHVyZSBzZW5kIG9yIG1h
eWJlIEkmIzM5O3ZlIG1pc3NlZCBzb21ldGhpZz88YnI+PGJyPjwvZGl2PjxkaXY+S2FtYWlsaW8g
NS42LjQ8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PmZvcms9eWVzPGJyPmNoaWxkcmVuPTEyPGJy
PnRjcF9jaGlsZHJlbj0xMjxicj48YnI+ZW5hYmxlX3Rscz15ZXM8YnI+PGJyPnRjcF9hY2NlcHRf
bm9fY2w9eWVzPGJyPnRjcF9tYXhfY29ubmVjdGlvbnM9NjM1MzY8YnI+dGxzX21heF9jb25uZWN0
aW9ucz02MzUzNjxicj50Y3BfYWNjZXB0X2FsaWFzZXM9bm88YnI+dGNwX2FzeW5jPXllczxicj50
Y3BfY29ubmVjdF90aW1lb3V0PTEwPGJyPnRjcF9jb25uX3dxX21heD02MzUzNjxicj50Y3BfY3Js
Zl9waW5nPXllczxicj50Y3BfZGVsYXllZF9hY2s9eWVzPGJyPnRjcF9mZF9jYWNoZT15ZXM8YnI+
dGNwX2tlZXBhbGl2ZT15ZXM8YnI+dGNwX2tlZXBjbnQ9Mzxicj50Y3Bfa2VlcGlkbGU9MzA8YnI+
dGNwX2tlZXBpbnR2bD0xMDxicj50Y3BfbGluZ2VyMj0zMDxicj50Y3BfcmRfYnVmX3NpemU9ODAw
MDA8YnI+dGNwX3NlbmRfdGltZW91dD0xMDxicj50Y3Bfd3FfYmxrX3NpemU9MjEwMDxicj50Y3Bf
d3FfbWF4PTEwNDg1NzYwPGJyPm9wZW5fZmlsZXNfbGltaXQ9NjM1MzY8L2Rpdj48ZGl2Pjxicj48
L2Rpdj48ZGl2PlN5c2N0bDxicj48YnI+IyBUbyBpbmNyZWFzZSB0aGUgYW1vdW50IG9mIG1lbW9y
eSBhdmFpbGFibGUgZm9yIHNvY2tldCBpbnB1dC9vdXRwdXQgcXVldWVzPGJyPm5ldC5pcHY0LnRj
cF9ybWVtID0gNDA5NiAyNTE2NTgyNCAyNTE2NTgyNDxicj5uZXQuY29yZS5ybWVtX21heCA9IDI1
MTY1ODI0PGJyPm5ldC5jb3JlLnJtZW1fZGVmYXVsdCA9IDI1MTY1ODI0PGJyPm5ldC5pcHY0LnRj
cF93bWVtID0gNDA5NiA2NTUzNiAyNTE2NTgyNDxicj5uZXQuY29yZS53bWVtX21heCA9IDI1MTY1
ODI0PGJyPm5ldC5jb3JlLndtZW1fZGVmYXVsdCA9IDY1NTM2PGJyPm5ldC5jb3JlLm9wdG1lbV9t
YXggPSAyNTE2NTgyNDxicj48YnI+IyBUbyBsaW1pdCB0aGUgbWF4aW11bSBudW1iZXIgb2YgcmVx
dWVzdHMgcXVldWVkIHRvIGEgbGlzdGVuIHNvY2tldDxicj5uZXQuY29yZS5zb21heGNvbm4gPSAx
Mjg8YnI+PGJyPiMgVGVsbHMgVENQIHRvIGluc3RlYWQgbWFrZSBkZWNpc2lvbnMgdGhhdCB3b3Vs
ZCBwcmVmZXIgbG93ZXIgbGF0ZW5jeS48YnI+bmV0LmlwdjQudGNwX2xvd19sYXRlbmN5PTE8YnI+
PGJyPiMgT3B0aW9uYWwgKGl0IHdpbGwgaW5jcmVhc2UgcGVyZm9ybWFuY2UpPGJyPm5ldC5jb3Jl
Lm5ldGRldl9tYXhfYmFja2xvZyA9IDEwMDA8YnI+bmV0LmlwdjQudGNwX21heF9zeW5fYmFja2xv
ZyA9IDEyODxicj48YnI+IyBGbHVzaCB0aGUgcm91dGluZyB0YWJsZSB0byBtYWtlIGNoYW5nZXMg
aGFwcGVuIGluc3RhbnRseS48YnI+bmV0LmlwdjQucm91dGUuZmx1c2g9MTxicj48L2Rpdj48ZGl2
PjxzcGFuIGNsYXNzPSJnbWFpbF9zaWduYXR1cmVfcHJlZml4Ij4tLSA8L3NwYW4+PGJyPjxkaXYg
ZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9zaWduYXR1cmUiPjxkaXYgZGlyPSJsdHIiPkJlc3QgcmVn
YXJkcyw8ZGl2Pklob3IgKElnb3IpPGJyPjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2Pgo8
L2Jsb2NrcXVvdGU+PC9kaXY+PGJyIGNsZWFyPSJhbGwiPjxicj48c3BhbiBjbGFzcz0iZ21haWxf
c2lnbmF0dXJlX3ByZWZpeCI+LS0gPC9zcGFuPjxicj48ZGl2IGRpcj0ibHRyIiBjbGFzcz0iZ21h
aWxfc2lnbmF0dXJlIj48ZGl2IGRpcj0ibHRyIj5CZXN0IHJlZ2FyZHMsPGRpdj5JaG9yIChJZ29y
KTxicj48L2Rpdj48L2Rpdj48L2Rpdj4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fXzxicj4KS2FtYWlsaW8gLSBVc2VycyBNYWlsaW5nIExp
c3QgLSBOb24gQ29tbWVyY2lhbCBEaXNjdXNzaW9uczxicj4KVG8gdW5zdWJzY3JpYmUgc2VuZCBh
biBlbWFpbCB0byA8YSBocmVmPSJtYWlsdG86c3ItdXNlcnMtbGVhdmVAbGlzdHMua2FtYWlsaW8u
b3JnIiB0YXJnZXQ9Il9ibGFuayI+c3ItdXNlcnMtbGVhdmVAbGlzdHMua2FtYWlsaW8ub3JnPC9h
Pjxicj4KSW1wb3J0YW50OiBrZWVwIHRoZSBtYWlsaW5nIGxpc3QgaW4gdGhlIHJlY2lwaWVudHMs
IGRvIG5vdCByZXBseSBvbmx5IHRvIHRoZSBzZW5kZXIhPGJyPgpFZGl0IG1haWxpbmcgbGlzdCBv
cHRpb25zIG9yIHVuc3Vic2NyaWJlOjxicj4KPC9ibG9ja3F1b3RlPjwvZGl2PgpfX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyPgpLYW1h
aWxpbyAtIFVzZXJzIE1haWxpbmcgTGlzdCAtIE5vbiBDb21tZXJjaWFsIERpc2N1c3Npb25zPGJy
PgpUbyB1bnN1YnNjcmliZSBzZW5kIGFuIGVtYWlsIHRvIDxhIGhyZWY9Im1haWx0bzpzci11c2Vy
cy1sZWF2ZUBsaXN0cy5rYW1haWxpby5vcmciIHRhcmdldD0iX2JsYW5rIj5zci11c2Vycy1sZWF2
ZUBsaXN0cy5rYW1haWxpby5vcmc8L2E+PGJyPgpJbXBvcnRhbnQ6IGtlZXAgdGhlIG1haWxpbmcg
bGlzdCBpbiB0aGUgcmVjaXBpZW50cywgZG8gbm90IHJlcGx5IG9ubHkgdG8gdGhlIHNlbmRlciE8
YnI+CkVkaXQgbWFpbGluZyBsaXN0IG9wdGlvbnMgb3IgdW5zdWJzY3JpYmU6PGJyPgo8L2Jsb2Nr
cXVvdGU+PC9kaXY+PGJyIGNsZWFyPSJhbGwiPjxicj48c3BhbiBjbGFzcz0iZ21haWxfc2lnbmF0
dXJlX3ByZWZpeCI+LS0gPC9zcGFuPjxicj48ZGl2IGRpcj0ibHRyIiBjbGFzcz0iZ21haWxfc2ln
bmF0dXJlIj48ZGl2IGRpcj0ibHRyIj5CZXN0IHJlZ2FyZHMsPGRpdj5JaG9yIChJZ29yKTxicj48
L2Rpdj48L2Rpdj48L2Rpdj4K

--===============1295552112==--


From miconda@gmail.com Wed Aug 30 14:32:25 2023
From: Daniel-Constantin Mierla <miconda@gmail.com>
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Re: Kamailio stops processing UDP on one interface
Date: Wed, 30 Aug 2023 16:32:06 +0200
Message-ID: <fb80d501-12fd-bdbc-3aed-0af556698d50@gmail.com>
In-Reply-To:
 <CAJTkRNtpBDrSrwPiZpxZpEfFHjLaHAmh81pXF0bYodMoM3OHMg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0505140767=="

--===============0505140767==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

You should see what the UDP worker processes do when that happens. You
have to take the execution backtrace with gdb, one option is to use:

kamctl trap

which grabs the backtrace for all kamailio processes, saving the data in
a local file.

By looking at backtrace, one can figure out what that process is
execution at that moment.

Cheers,
Daniel

On 30.08.23 15:30, Ihor Olkhovskyi wrote:
> They are increasing, actually
>
> # ss -l -u -m src X.X.X.X/Y
> State =C2=A0 =C2=A0 =C2=A0 Recv-Q Send-Q =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> =C2=A0 Local Address:Port =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Peer Address:Port =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> UNCONN =C2=A0 =C2=A0 =C2=A025167616 0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> =C2=A0 X.X.X.X:sip =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0*:* =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0skmem:(r25167616,rb25165824,t0,tb65536,f2=
304,w0,o0,bl0,d514894)
>
> Le=C2=A0mer. 30 ao=C3=BBt 2023 =C3=A0=C2=A015:04, Bastian Triller
> <bastian.triller(a)gmail.com> a =C3=A9crit=C2=A0:
>
>     Are drops increasing on that socket while it is happening?
>     ss -l src <local_interface>=C2=A0-u sport 5060 -m
>
>     On Tue, Aug 29, 2023 at 3:26=E2=80=AFPM Ihor Olkhovskyi
>     <igorolhovskiy(a)gmail.com> wrote:
>
>         Just to add some info
>
>         netstat -nlp
>         Active Internet connections (only servers)
>         Proto Recv-Q Send-Q Local Address =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 Foreign Address =C2=A0
>         =C2=A0 =C2=A0 =C2=A0 State =C2=A0 =C2=A0 =C2=A0 PID/Program name =
=C2=A0 =C2=A0
>         ...
>         udp =C2=A0 25167616 =C2=A0 =C2=A0 =C2=A00 <local_interface>:5060 =
=C2=A0 =C2=A0 0.0.0.0:* =C2=A0 =C2=A0
>         =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 211759/kamailio
>         ...
>
>         So I see a huge Receive Queue on UDP for Kamailio which is not
>         clearing.
>
>         Le=C2=A0mar. 29 ao=C3=BBt 2023 =C3=A0=C2=A014:29, Ihor Olkhovskyi
>         <igorolhovskiy(a)gmail.com> a =C3=A9crit=C2=A0:
>
>             Hello,
>
>             I've faced a bit strange issue, but a bit of preface. I
>             have Kamailio as a proxy (TLS/WS <-> UDP) and second
>             Kamailio as a presence server. At some point presence
>             server accepts around 5K PUBLISH within 1 minute and
>             sending around the same amount of NOTIFY to proxy Kamailio.
>
>             Proxy is "transforming" protocol to TLS, but at sime point
>             I'm starting to get these type of errors
>
>             tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send
>             failed
>             tm [t_fwd.c:1588]: t_send_branch(): sending request on
>             branch 0 failed
>             <script>: [RELAY] Relay to
>             <sip:X.X.X.X:51571;transport=3Dtls> failed!
>             tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send
>             failed
>             tm [t_fwd.c:1588]: t_send_branch(): sending request on
>             branch 0 failed
>
>             Some of those messages are 100% valid as client can go
>             away or so. Some are not, cause I'm sure client is alive
>             and connected.=C2=A0
>
>             But the problem comes later. At some moment proxy Kamailio
>             just stops accept UDP traffic on this interface (where it
>             also accepts all NOTIFY's), at the start of the "stopping
>             accepting" Kamailio sends OPTIONS via DISPATCHER but not
>             able to receive 200 OK.
>
>             Over TLS on the same interface all is ok. On other
>             (loopback) interface UDP is being processed fine, so I
>             don't suspert some limit on open files here.
>
>             Only restart of Kamailio proxy process helps in this case.
>
>             I've tuned net.core.rmem_max and net.core.rmem_default to
>             25 Mb, so in theory buffer should not be the case.
>
>             Is there some internal "interface buffer" in Kamailio that
>             is not freed upon failure send or maybe I've missed somethig?
>
>             Kamailio 5.6.4
>
>             fork=3Dyes
>             children=3D12
>             tcp_children=3D12
>
>             enable_tls=3Dyes
>
>             tcp_accept_no_cl=3Dyes
>             tcp_max_connections=3D63536
>             tls_max_connections=3D63536
>             tcp_accept_aliases=3Dno
>             tcp_async=3Dyes
>             tcp_connect_timeout=3D10
>             tcp_conn_wq_max=3D63536
>             tcp_crlf_ping=3Dyes
>             tcp_delayed_ack=3Dyes
>             tcp_fd_cache=3Dyes
>             tcp_keepalive=3Dyes
>             tcp_keepcnt=3D3
>             tcp_keepidle=3D30
>             tcp_keepintvl=3D10
>             tcp_linger2=3D30
>             tcp_rd_buf_size=3D80000
>             tcp_send_timeout=3D10
>             tcp_wq_blk_size=3D2100
>             tcp_wq_max=3D10485760
>             open_files_limit=3D63536
>
>             Sysctl
>
>             # To increase the amount of memory available for socket
>             input/output queues
>             net.ipv4.tcp_rmem =3D 4096 25165824 25165824
>             net.core.rmem_max =3D 25165824
>             net.core.rmem_default =3D 25165824
>             net.ipv4.tcp_wmem =3D 4096 65536 25165824
>             net.core.wmem_max =3D 25165824
>             net.core.wmem_default =3D 65536
>             net.core.optmem_max =3D 25165824
>
>             # To limit the maximum number of requests queued to a
>             listen socket
>             net.core.somaxconn =3D 128
>
>             # Tells TCP to instead make decisions that would prefer
>             lower latency.
>             net.ipv4.tcp_low_latency=3D1
>
>             # Optional (it will increase performance)
>             net.core.netdev_max_backlog =3D 1000
>             net.ipv4.tcp_max_syn_backlog =3D 128
>
>             # Flush the routing table to make changes happen instantly.
>             net.ipv4.route.flush=3D1
>             --=20
>             Best regards,
>             Ihor (Igor)
>
>
>
>         --=20
>         Best regards,
>         Ihor (Igor)
>         __________________________________________________________
>         Kamailio - Users Mailing List - Non Commercial Discussions
>         To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
>         Important: keep the mailing list in the recipients, do not
>         reply only to the sender!
>         Edit mailing list options or unsubscribe:
>
>     __________________________________________________________
>     Kamailio - Users Mailing List - Non Commercial Discussions
>     To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
>     Important: keep the mailing list in the recipients, do not reply
>     only to the sender!
>     Edit mailing list options or unsubscribe:
>
>
>
> --=20
> Best regards,
> Ihor (Igor)
>
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to th=
e sender!
> Edit mailing list options or unsubscribe:

--=20
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy - Training Services -- asipto.com
Kamailio World Conference - kamailioworld.com



--===============0505140767==
Content-Type: text/html
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.html"
MIME-Version: 1.0

PGh0bWw+DQogIDxoZWFkPg0KICAgIDxtZXRhIGh0dHAtZXF1aXY9IkNvbnRlbnQtVHlwZSIgY29u
dGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04Ij4NCiAgPC9oZWFkPg0KICA8Ym9keT4NCiAg
ICA8cD5Zb3Ugc2hvdWxkIHNlZSB3aGF0IHRoZSBVRFAgd29ya2VyIHByb2Nlc3NlcyBkbyB3aGVu
IHRoYXQNCiAgICAgIGhhcHBlbnMuIFlvdSBoYXZlIHRvIHRha2UgdGhlIGV4ZWN1dGlvbiBiYWNr
dHJhY2Ugd2l0aCBnZGIsIG9uZQ0KICAgICAgb3B0aW9uIGlzIHRvIHVzZTo8L3A+DQogICAgPHA+
a2FtY3RsIHRyYXA8L3A+DQogICAgPHA+d2hpY2ggZ3JhYnMgdGhlIGJhY2t0cmFjZSBmb3IgYWxs
IGthbWFpbGlvIHByb2Nlc3Nlcywgc2F2aW5nIHRoZQ0KICAgICAgZGF0YSBpbiBhIGxvY2FsIGZp
bGUuPC9wPg0KICAgIDxwPkJ5IGxvb2tpbmcgYXQgYmFja3RyYWNlLCBvbmUgY2FuIGZpZ3VyZSBv
dXQgd2hhdCB0aGF0IHByb2Nlc3MgaXMNCiAgICAgIGV4ZWN1dGlvbiBhdCB0aGF0IG1vbWVudC48
L3A+DQogICAgPHA+Q2hlZXJzLDxicj4NCiAgICAgIERhbmllbDxicj4NCiAgICA8L3A+DQogICAg
PGRpdiBjbGFzcz0ibW96LWNpdGUtcHJlZml4Ij5PbiAzMC4wOC4yMyAxNTozMCwgSWhvciBPbGto
b3Zza3lpDQogICAgICB3cm90ZTo8YnI+DQogICAgPC9kaXY+DQogICAgPGJsb2NrcXVvdGUgdHlw
ZT0iY2l0ZSINCmNpdGU9Im1pZDpDQUpUa1JOdHBCRHJTcndQaVpweFpwRWZGSGpMYUhBbWg4MXBY
RjBiWW9kTW9NM09ITWdAbWFpbC5nbWFpbC5jb20iPg0KICAgICAgPG1ldGEgaHR0cC1lcXVpdj0i
Y29udGVudC10eXBlIiBjb250ZW50PSJ0ZXh0L2h0bWw7IGNoYXJzZXQ9VVRGLTgiPg0KICAgICAg
PGRpdiBkaXI9Imx0ciI+DQogICAgICAgIDxkaXY+VGhleSBhcmUgaW5jcmVhc2luZywgYWN0dWFs
bHk8L2Rpdj4NCiAgICAgICAgPGRpdj48YnI+DQogICAgICAgIDwvZGl2Pg0KICAgICAgICA8ZGl2
PiMgc3MgLWwgLXUgLW0gc3JjIFguWC5YLlgvWTxicj4NCiAgICAgICAgICBTdGF0ZSDCoCDCoCDC
oCBSZWN2LVEgU2VuZC1RIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgDQogICAgICAgICAgwqAgwqAgwqAgwqAgwqAgTG9jYWwgQWRkcmVzczpQb3J0
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgDQogICAg
ICAgICAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqBQZWVyIEFk
ZHJlc3M6UG9ydCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoDxicj4NCiAgICAgICAgICBVTkNPTk4g
wqAgwqAgwqAyNTE2NzYxNiAwIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgDQogICAgICAgICAgwqAgwqAgwqAgwqAgwqAgWC5YLlguWDpz
aXAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqANCiAgICAgICAgICDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCo6KiDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoDxicj4NCiAgICAg
ICAgICDCoCDCoCDCoCDCoA0KICAgICAgICAgIMKgc2ttZW06KHIyNTE2NzYxNixyYjI1MTY1ODI0
LHQwLHRiNjU1MzYsZjIzMDQsdzAsbzAsYmwwLGQ1MTQ4OTQpPC9kaXY+DQogICAgICA8L2Rpdj4N
CiAgICAgIDxicj4NCiAgICAgIDxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj4NCiAgICAgICAgPGRp
diBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX2F0dHIiPkxlwqBtZXIuIDMwIGFvw7t0IDIwMjMgw6DC
oDE1OjA0LA0KICAgICAgICAgIEJhc3RpYW4gVHJpbGxlciAmbHQ7PGEgaHJlZj0ibWFpbHRvOmJh
c3RpYW4udHJpbGxlckBnbWFpbC5jb20iDQogICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRy
dWUiIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiPmJhc3RpYW4udHJpbGxlckBnbWFpbC5j
b208L2E+Jmd0Ow0KICAgICAgICAgIGEgw6ljcml0wqA6PGJyPg0KICAgICAgICA8L2Rpdj4NCiAg
ICAgICAgPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAw
cHggMHB4DQogICAgICAgICAgMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0
LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+DQogICAgICAgICAgPGRpdiBkaXI9Imx0ciI+QXJlIGRy
b3BzIGluY3JlYXNpbmcgb24gdGhhdCBzb2NrZXQgd2hpbGUgaXQgaXMNCiAgICAgICAgICAgIGhh
cHBlbmluZz8NCiAgICAgICAgICAgIDxkaXY+c3MgLWwgc3JjICZsdDs8c3BhbiBzdHlsZT0iY29s
b3I6cmdiKDAsMCwwKSI+bG9jYWxfaW50ZXJmYWNlJmd0Ozwvc3Bhbj7CoC11DQogICAgICAgICAg
ICAgIHNwb3J0IDUwNjAgLW08L2Rpdj4NCiAgICAgICAgICA8L2Rpdj4NCiAgICAgICAgICA8YnI+
DQogICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPg0KICAgICAgICAgICAgPGRpdiBk
aXI9Imx0ciIgY2xhc3M9ImdtYWlsX2F0dHIiPk9uIFR1ZSwgQXVnIDI5LCAyMDIzIGF0DQogICAg
ICAgICAgICAgIDM6MjbigK9QTSBJaG9yIE9sa2hvdnNreWkgJmx0OzxhDQogICAgICAgICAgICAg
ICAgaHJlZj0ibWFpbHRvOmlnb3JvbGhvdnNraXlAZ21haWwuY29tIiB0YXJnZXQ9Il9ibGFuayIN
CiAgICAgICAgICAgICAgICBtb3otZG8tbm90LXNlbmQ9InRydWUiIGNsYXNzPSJtb3otdHh0LWxp
bmstZnJlZXRleHQiPmlnb3JvbGhvdnNraXlAZ21haWwuY29tPC9hPiZndDsNCiAgICAgICAgICAg
ICAgd3JvdGU6PGJyPg0KICAgICAgICAgICAgPC9kaXY+DQogICAgICAgICAgICA8YmxvY2txdW90
ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHgNCiAgICAgICAg
ICAgICAgMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkDQogICAgICAgICAgICAgIHJnYigyMDQs
MjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+DQogICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIi
Pg0KICAgICAgICAgICAgICAgIDxkaXY+SnVzdCB0byBhZGQgc29tZSBpbmZvPC9kaXY+DQogICAg
ICAgICAgICAgICAgPGRpdj48YnI+DQogICAgICAgICAgICAgICAgPC9kaXY+DQogICAgICAgICAg
ICAgICAgPGRpdj5uZXRzdGF0IC1ubHA8YnI+DQogICAgICAgICAgICAgICAgICBBY3RpdmUgSW50
ZXJuZXQgY29ubmVjdGlvbnMgKG9ubHkgc2VydmVycyk8YnI+DQogICAgICAgICAgICAgICAgICBQ
cm90byBSZWN2LVEgU2VuZC1RIExvY2FsIEFkZHJlc3MgwqAgwqAgwqAgwqAgwqAgRm9yZWlnbg0K
ICAgICAgICAgICAgICAgICAgQWRkcmVzcyDCoCDCoCDCoCDCoCBTdGF0ZSDCoCDCoCDCoCBQSUQv
UHJvZ3JhbSBuYW1lIMKgIMKgPGJyPg0KICAgICAgICAgICAgICAgICAgLi4uPGJyPg0KICAgICAg
ICAgICAgICAgICAgdWRwIMKgIDI1MTY3NjE2IMKgIMKgIMKgMCAmbHQ7bG9jYWxfaW50ZXJmYWNl
Jmd0Ozo1MDYwIMKgIMKgDQogICAgICAgICAgICAgICAgICAwLjAuMC4wOiogwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgMjExNzU5L2thbWFpbGlvPGJyPg0KICAgICAgICAg
ICAgICAgICAgLi4uPGJyPg0KICAgICAgICAgICAgICAgICAgPGJyPg0KICAgICAgICAgICAgICAg
IDwvZGl2Pg0KICAgICAgICAgICAgICAgIDxkaXY+U28gSSBzZWUgYSBodWdlIFJlY2VpdmUgUXVl
dWUgb24gVURQIGZvciBLYW1haWxpbw0KICAgICAgICAgICAgICAgICAgd2hpY2ggaXMgbm90IGNs
ZWFyaW5nLjxicj4NCiAgICAgICAgICAgICAgICA8L2Rpdj4NCiAgICAgICAgICAgICAgPC9kaXY+
DQogICAgICAgICAgICAgIDxicj4NCiAgICAgICAgICAgICAgPGRpdiBjbGFzcz0iZ21haWxfcXVv
dGUiPg0KICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9hdHRyIj5M
ZcKgbWFyLiAyOSBhb8O7dCAyMDIzDQogICAgICAgICAgICAgICAgICDDoMKgMTQ6MjksIElob3Ig
T2xraG92c2t5aSAmbHQ7PGENCiAgICAgICAgICAgICAgICAgICAgaHJlZj0ibWFpbHRvOmlnb3Jv
bGhvdnNraXlAZ21haWwuY29tIg0KICAgICAgICAgICAgICAgICAgICB0YXJnZXQ9Il9ibGFuayIg
bW96LWRvLW5vdC1zZW5kPSJ0cnVlIg0KICAgICAgICAgICAgICAgICAgICBjbGFzcz0ibW96LXR4
dC1saW5rLWZyZWV0ZXh0Ij5pZ29yb2xob3Zza2l5QGdtYWlsLmNvbTwvYT4mZ3Q7DQogICAgICAg
ICAgICAgICAgICBhIMOpY3JpdMKgOjxicj4NCiAgICAgICAgICAgICAgICA8L2Rpdj4NCiAgICAg
ICAgICAgICAgICA8YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46
MHB4IDBweA0KICAgICAgICAgICAgICAgICAgMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xp
ZA0KICAgICAgICAgICAgICAgICAgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij4N
CiAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiPkhlbGxvLDxicj4NCiAgICAgICAgICAg
ICAgICAgICAgPGRpdj48YnI+DQogICAgICAgICAgICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAg
ICAgICAgICAgICA8ZGl2PkkndmUgZmFjZWQgYSBiaXQgc3RyYW5nZSBpc3N1ZSwgYnV0IGEgYml0
IG9mDQogICAgICAgICAgICAgICAgICAgICAgcHJlZmFjZS4gSSBoYXZlIEthbWFpbGlvIGFzIGEg
cHJveHkgKFRMUy9XUw0KICAgICAgICAgICAgICAgICAgICAgICZsdDstJmd0OyBVRFApIGFuZCBz
ZWNvbmQgS2FtYWlsaW8gYXMgYSBwcmVzZW5jZQ0KICAgICAgICAgICAgICAgICAgICAgIHNlcnZl
ci4gQXQgc29tZSBwb2ludCBwcmVzZW5jZSBzZXJ2ZXIgYWNjZXB0cw0KICAgICAgICAgICAgICAg
ICAgICAgIGFyb3VuZCA1SyBQVUJMSVNIIHdpdGhpbiAxIG1pbnV0ZSBhbmQgc2VuZGluZw0KICAg
ICAgICAgICAgICAgICAgICAgIGFyb3VuZCB0aGUgc2FtZSBhbW91bnQgb2YgTk9USUZZIHRvIHBy
b3h5DQogICAgICAgICAgICAgICAgICAgICAgS2FtYWlsaW8uPC9kaXY+DQogICAgICAgICAgICAg
ICAgICAgIDxkaXY+PGJyPg0KICAgICAgICAgICAgICAgICAgICA8L2Rpdj4NCiAgICAgICAgICAg
ICAgICAgICAgPGRpdj5Qcm94eSBpcyAidHJhbnNmb3JtaW5nIiBwcm90b2NvbCB0byBUTFMsIGJ1
dCBhdA0KICAgICAgICAgICAgICAgICAgICAgIHNpbWUgcG9pbnQgSSdtIHN0YXJ0aW5nIHRvIGdl
dCB0aGVzZSB0eXBlIG9mDQogICAgICAgICAgICAgICAgICAgICAgZXJyb3JzPC9kaXY+DQogICAg
ICAgICAgICAgICAgICAgIDxkaXY+PGJyPg0KICAgICAgICAgICAgICAgICAgICA8L2Rpdj4NCiAg
ICAgICAgICAgICAgICAgICAgPGRpdj50bSBbLi4vLi4vY29yZS9mb3J3YXJkLmg6MjkyXToNCiAg
ICAgICAgICAgICAgICAgICAgICBtc2dfc2VuZF9idWZmZXIoKTogdGNwX3NlbmQgZmFpbGVkPGJy
Pg0KICAgICAgICAgICAgICAgICAgICAgIHRtIFt0X2Z3ZC5jOjE1ODhdOiB0X3NlbmRfYnJhbmNo
KCk6IHNlbmRpbmcNCiAgICAgICAgICAgICAgICAgICAgICByZXF1ZXN0IG9uIGJyYW5jaCAwIGZh
aWxlZDxicj4NCiAgICAgICAgICAgICAgICAgICAgICAmbHQ7c2NyaXB0Jmd0OzogW1JFTEFZXSBS
ZWxheSB0bw0KICAgICAgICAgICAgICAgICAgICAgICZsdDtzaXA6WC5YLlguWDo1MTU3MTt0cmFu
c3BvcnQ9dGxzJmd0OyBmYWlsZWQhPGJyPg0KICAgICAgICAgICAgICAgICAgICAgIHRtIFsuLi8u
Li9jb3JlL2ZvcndhcmQuaDoyOTJdOiBtc2dfc2VuZF9idWZmZXIoKToNCiAgICAgICAgICAgICAg
ICAgICAgICB0Y3Bfc2VuZCBmYWlsZWQ8YnI+DQogICAgICAgICAgICAgICAgICAgICAgdG0gW3Rf
ZndkLmM6MTU4OF06IHRfc2VuZF9icmFuY2goKTogc2VuZGluZw0KICAgICAgICAgICAgICAgICAg
ICAgIHJlcXVlc3Qgb24gYnJhbmNoIDAgZmFpbGVkPC9kaXY+DQogICAgICAgICAgICAgICAgICAg
IDxkaXY+PGJyPg0KICAgICAgICAgICAgICAgICAgICA8L2Rpdj4NCiAgICAgICAgICAgICAgICAg
ICAgPGRpdj5Tb21lIG9mIHRob3NlIG1lc3NhZ2VzIGFyZSAxMDAlIHZhbGlkIGFzIGNsaWVudA0K
ICAgICAgICAgICAgICAgICAgICAgIGNhbiBnbyBhd2F5IG9yIHNvLiBTb21lIGFyZSBub3QsIGNh
dXNlIEknbSBzdXJlDQogICAgICAgICAgICAgICAgICAgICAgY2xpZW50IGlzIGFsaXZlIGFuZCBj
b25uZWN0ZWQuwqA8L2Rpdj4NCiAgICAgICAgICAgICAgICAgICAgPGRpdj48YnI+DQogICAgICAg
ICAgICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAgICAgICAgICAgICA8ZGl2PkJ1dCB0aGUgcHJv
YmxlbSBjb21lcyBsYXRlci4gQXQgc29tZSBtb21lbnQNCiAgICAgICAgICAgICAgICAgICAgICBw
cm94eSBLYW1haWxpbyBqdXN0IHN0b3BzIGFjY2VwdCBVRFAgdHJhZmZpYyBvbg0KICAgICAgICAg
ICAgICAgICAgICAgIHRoaXMgaW50ZXJmYWNlICh3aGVyZSBpdCBhbHNvIGFjY2VwdHMgYWxsDQog
ICAgICAgICAgICAgICAgICAgICAgTk9USUZZJ3MpLCBhdCB0aGUgc3RhcnQgb2YgdGhlICJzdG9w
cGluZw0KICAgICAgICAgICAgICAgICAgICAgIGFjY2VwdGluZyIgS2FtYWlsaW8gc2VuZHMgT1BU
SU9OUyB2aWEgRElTUEFUQ0hFUg0KICAgICAgICAgICAgICAgICAgICAgIGJ1dCBub3QgYWJsZSB0
byByZWNlaXZlIDIwMCBPSy4gPGJyPg0KICAgICAgICAgICAgICAgICAgICA8L2Rpdj4NCiAgICAg
ICAgICAgICAgICAgICAgPGRpdj48YnI+DQogICAgICAgICAgICAgICAgICAgIDwvZGl2Pg0KICAg
ICAgICAgICAgICAgICAgICA8ZGl2Pk92ZXIgVExTIG9uIHRoZSBzYW1lIGludGVyZmFjZSBhbGwg
aXMgb2suIE9uDQogICAgICAgICAgICAgICAgICAgICAgb3RoZXIgKGxvb3BiYWNrKSBpbnRlcmZh
Y2UgVURQIGlzIGJlaW5nIHByb2Nlc3NlZA0KICAgICAgICAgICAgICAgICAgICAgIGZpbmUsIHNv
IEkgZG9uJ3Qgc3VzcGVydCBzb21lIGxpbWl0IG9uIG9wZW4gZmlsZXMNCiAgICAgICAgICAgICAg
ICAgICAgICBoZXJlLjxicj4NCiAgICAgICAgICAgICAgICAgICAgPC9kaXY+DQogICAgICAgICAg
ICAgICAgICAgIDxkaXY+PGJyPg0KICAgICAgICAgICAgICAgICAgICA8L2Rpdj4NCiAgICAgICAg
ICAgICAgICAgICAgPGRpdj5Pbmx5IHJlc3RhcnQgb2YgS2FtYWlsaW8gcHJveHkgcHJvY2VzcyBo
ZWxwcyBpbg0KICAgICAgICAgICAgICAgICAgICAgIHRoaXMgY2FzZS4gPGJyPg0KICAgICAgICAg
ICAgICAgICAgICA8L2Rpdj4NCiAgICAgICAgICAgICAgICAgICAgPGRpdj48YnI+DQogICAgICAg
ICAgICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAgICAgICAgICAgICA8ZGl2PkkndmUgdHVuZWQg
bmV0LmNvcmUucm1lbV9tYXggYW5kDQogICAgICAgICAgICAgICAgICAgICAgbmV0LmNvcmUucm1l
bV9kZWZhdWx0IHRvIDI1IE1iLCBzbyBpbiB0aGVvcnkNCiAgICAgICAgICAgICAgICAgICAgICBi
dWZmZXIgc2hvdWxkIG5vdCBiZSB0aGUgY2FzZS48L2Rpdj4NCiAgICAgICAgICAgICAgICAgICAg
PGRpdj48YnI+DQogICAgICAgICAgICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAgICAgICAgICAg
ICA8ZGl2PklzIHRoZXJlIHNvbWUgaW50ZXJuYWwgImludGVyZmFjZSBidWZmZXIiIGluDQogICAg
ICAgICAgICAgICAgICAgICAgS2FtYWlsaW8gdGhhdCBpcyBub3QgZnJlZWQgdXBvbiBmYWlsdXJl
IHNlbmQgb3INCiAgICAgICAgICAgICAgICAgICAgICBtYXliZSBJJ3ZlIG1pc3NlZCBzb21ldGhp
Zz88YnI+DQogICAgICAgICAgICAgICAgICAgICAgPGJyPg0KICAgICAgICAgICAgICAgICAgICA8
L2Rpdj4NCiAgICAgICAgICAgICAgICAgICAgPGRpdj5LYW1haWxpbyA1LjYuNDwvZGl2Pg0KICAg
ICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4NCiAgICAgICAgICAgICAgICAgICAgPC9kaXY+DQog
ICAgICAgICAgICAgICAgICAgIDxkaXY+Zm9yaz15ZXM8YnI+DQogICAgICAgICAgICAgICAgICAg
ICAgY2hpbGRyZW49MTI8YnI+DQogICAgICAgICAgICAgICAgICAgICAgdGNwX2NoaWxkcmVuPTEy
PGJyPg0KICAgICAgICAgICAgICAgICAgICAgIDxicj4NCiAgICAgICAgICAgICAgICAgICAgICBl
bmFibGVfdGxzPXllczxicj4NCiAgICAgICAgICAgICAgICAgICAgICA8YnI+DQogICAgICAgICAg
ICAgICAgICAgICAgdGNwX2FjY2VwdF9ub19jbD15ZXM8YnI+DQogICAgICAgICAgICAgICAgICAg
ICAgdGNwX21heF9jb25uZWN0aW9ucz02MzUzNjxicj4NCiAgICAgICAgICAgICAgICAgICAgICB0
bHNfbWF4X2Nvbm5lY3Rpb25zPTYzNTM2PGJyPg0KICAgICAgICAgICAgICAgICAgICAgIHRjcF9h
Y2NlcHRfYWxpYXNlcz1ubzxicj4NCiAgICAgICAgICAgICAgICAgICAgICB0Y3BfYXN5bmM9eWVz
PGJyPg0KICAgICAgICAgICAgICAgICAgICAgIHRjcF9jb25uZWN0X3RpbWVvdXQ9MTA8YnI+DQog
ICAgICAgICAgICAgICAgICAgICAgdGNwX2Nvbm5fd3FfbWF4PTYzNTM2PGJyPg0KICAgICAgICAg
ICAgICAgICAgICAgIHRjcF9jcmxmX3Bpbmc9eWVzPGJyPg0KICAgICAgICAgICAgICAgICAgICAg
IHRjcF9kZWxheWVkX2Fjaz15ZXM8YnI+DQogICAgICAgICAgICAgICAgICAgICAgdGNwX2ZkX2Nh
Y2hlPXllczxicj4NCiAgICAgICAgICAgICAgICAgICAgICB0Y3Bfa2VlcGFsaXZlPXllczxicj4N
CiAgICAgICAgICAgICAgICAgICAgICB0Y3Bfa2VlcGNudD0zPGJyPg0KICAgICAgICAgICAgICAg
ICAgICAgIHRjcF9rZWVwaWRsZT0zMDxicj4NCiAgICAgICAgICAgICAgICAgICAgICB0Y3Bfa2Vl
cGludHZsPTEwPGJyPg0KICAgICAgICAgICAgICAgICAgICAgIHRjcF9saW5nZXIyPTMwPGJyPg0K
ICAgICAgICAgICAgICAgICAgICAgIHRjcF9yZF9idWZfc2l6ZT04MDAwMDxicj4NCiAgICAgICAg
ICAgICAgICAgICAgICB0Y3Bfc2VuZF90aW1lb3V0PTEwPGJyPg0KICAgICAgICAgICAgICAgICAg
ICAgIHRjcF93cV9ibGtfc2l6ZT0yMTAwPGJyPg0KICAgICAgICAgICAgICAgICAgICAgIHRjcF93
cV9tYXg9MTA0ODU3NjA8YnI+DQogICAgICAgICAgICAgICAgICAgICAgb3Blbl9maWxlc19saW1p
dD02MzUzNjwvZGl2Pg0KICAgICAgICAgICAgICAgICAgICA8ZGl2Pjxicj4NCiAgICAgICAgICAg
ICAgICAgICAgPC9kaXY+DQogICAgICAgICAgICAgICAgICAgIDxkaXY+U3lzY3RsPGJyPg0KICAg
ICAgICAgICAgICAgICAgICAgIDxicj4NCiAgICAgICAgICAgICAgICAgICAgICAjIFRvIGluY3Jl
YXNlIHRoZSBhbW91bnQgb2YgbWVtb3J5IGF2YWlsYWJsZSBmb3INCiAgICAgICAgICAgICAgICAg
ICAgICBzb2NrZXQgaW5wdXQvb3V0cHV0IHF1ZXVlczxicj4NCiAgICAgICAgICAgICAgICAgICAg
ICBuZXQuaXB2NC50Y3Bfcm1lbSA9IDQwOTYgMjUxNjU4MjQgMjUxNjU4MjQ8YnI+DQogICAgICAg
ICAgICAgICAgICAgICAgbmV0LmNvcmUucm1lbV9tYXggPSAyNTE2NTgyNDxicj4NCiAgICAgICAg
ICAgICAgICAgICAgICBuZXQuY29yZS5ybWVtX2RlZmF1bHQgPSAyNTE2NTgyNDxicj4NCiAgICAg
ICAgICAgICAgICAgICAgICBuZXQuaXB2NC50Y3Bfd21lbSA9IDQwOTYgNjU1MzYgMjUxNjU4MjQ8
YnI+DQogICAgICAgICAgICAgICAgICAgICAgbmV0LmNvcmUud21lbV9tYXggPSAyNTE2NTgyNDxi
cj4NCiAgICAgICAgICAgICAgICAgICAgICBuZXQuY29yZS53bWVtX2RlZmF1bHQgPSA2NTUzNjxi
cj4NCiAgICAgICAgICAgICAgICAgICAgICBuZXQuY29yZS5vcHRtZW1fbWF4ID0gMjUxNjU4MjQ8
YnI+DQogICAgICAgICAgICAgICAgICAgICAgPGJyPg0KICAgICAgICAgICAgICAgICAgICAgICMg
VG8gbGltaXQgdGhlIG1heGltdW0gbnVtYmVyIG9mIHJlcXVlc3RzIHF1ZXVlZA0KICAgICAgICAg
ICAgICAgICAgICAgIHRvIGEgbGlzdGVuIHNvY2tldDxicj4NCiAgICAgICAgICAgICAgICAgICAg
ICBuZXQuY29yZS5zb21heGNvbm4gPSAxMjg8YnI+DQogICAgICAgICAgICAgICAgICAgICAgPGJy
Pg0KICAgICAgICAgICAgICAgICAgICAgICMgVGVsbHMgVENQIHRvIGluc3RlYWQgbWFrZSBkZWNp
c2lvbnMgdGhhdCB3b3VsZA0KICAgICAgICAgICAgICAgICAgICAgIHByZWZlciBsb3dlciBsYXRl
bmN5Ljxicj4NCiAgICAgICAgICAgICAgICAgICAgICBuZXQuaXB2NC50Y3BfbG93X2xhdGVuY3k9
MTxicj4NCiAgICAgICAgICAgICAgICAgICAgICA8YnI+DQogICAgICAgICAgICAgICAgICAgICAg
IyBPcHRpb25hbCAoaXQgd2lsbCBpbmNyZWFzZSBwZXJmb3JtYW5jZSk8YnI+DQogICAgICAgICAg
ICAgICAgICAgICAgbmV0LmNvcmUubmV0ZGV2X21heF9iYWNrbG9nID0gMTAwMDxicj4NCiAgICAg
ICAgICAgICAgICAgICAgICBuZXQuaXB2NC50Y3BfbWF4X3N5bl9iYWNrbG9nID0gMTI4PGJyPg0K
ICAgICAgICAgICAgICAgICAgICAgIDxicj4NCiAgICAgICAgICAgICAgICAgICAgICAjIEZsdXNo
IHRoZSByb3V0aW5nIHRhYmxlIHRvIG1ha2UgY2hhbmdlcyBoYXBwZW4NCiAgICAgICAgICAgICAg
ICAgICAgICBpbnN0YW50bHkuPGJyPg0KICAgICAgICAgICAgICAgICAgICAgIG5ldC5pcHY0LnJv
dXRlLmZsdXNoPTE8YnI+DQogICAgICAgICAgICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAgICAg
ICAgICAgICA8ZGl2PjxzcGFuIGNsYXNzPSJnbWFpbF9zaWduYXR1cmVfcHJlZml4Ij4tLSA8L3Nw
YW4+PGJyPg0KICAgICAgICAgICAgICAgICAgICAgIDxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFp
bF9zaWduYXR1cmUiPg0KICAgICAgICAgICAgICAgICAgICAgICAgPGRpdiBkaXI9Imx0ciI+QmVz
dCByZWdhcmRzLA0KICAgICAgICAgICAgICAgICAgICAgICAgICA8ZGl2Pklob3IgKElnb3IpPGJy
Pg0KICAgICAgICAgICAgICAgICAgICAgICAgICA8L2Rpdj4NCiAgICAgICAgICAgICAgICAgICAg
ICAgIDwvZGl2Pg0KICAgICAgICAgICAgICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAgICAgICAg
ICAgICA8L2Rpdj4NCiAgICAgICAgICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAgICAgICAgIDwv
YmxvY2txdW90ZT4NCiAgICAgICAgICAgICAgPC9kaXY+DQogICAgICAgICAgICAgIDxiciBjbGVh
cj0iYWxsIj4NCiAgICAgICAgICAgICAgPGJyPg0KICAgICAgICAgICAgICA8c3BhbiBjbGFzcz0i
Z21haWxfc2lnbmF0dXJlX3ByZWZpeCI+LS0gPC9zcGFuPjxicj4NCiAgICAgICAgICAgICAgPGRp
diBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSI+DQogICAgICAgICAgICAgICAgPGRp
diBkaXI9Imx0ciI+QmVzdCByZWdhcmRzLA0KICAgICAgICAgICAgICAgICAgPGRpdj5JaG9yIChJ
Z29yKTxicj4NCiAgICAgICAgICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAgICAgICAgIDwvZGl2
Pg0KICAgICAgICAgICAgICA8L2Rpdj4NCiAgICAgICAgICAgICAgX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzxicj4NCiAgICAgICAgICAg
ICAgS2FtYWlsaW8gLSBVc2VycyBNYWlsaW5nIExpc3QgLSBOb24gQ29tbWVyY2lhbCBEaXNjdXNz
aW9uczxicj4NCiAgICAgICAgICAgICAgVG8gdW5zdWJzY3JpYmUgc2VuZCBhbiBlbWFpbCB0byA8
YQ0KICAgICAgICAgICAgICAgIGhyZWY9Im1haWx0bzpzci11c2Vycy1sZWF2ZUBsaXN0cy5rYW1h
aWxpby5vcmciDQogICAgICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2Vu
ZD0idHJ1ZSINCiAgICAgICAgICAgICAgICBjbGFzcz0ibW96LXR4dC1saW5rLWZyZWV0ZXh0Ij5z
ci11c2Vycy1sZWF2ZUBsaXN0cy5rYW1haWxpby5vcmc8L2E+PGJyPg0KICAgICAgICAgICAgICBJ
bXBvcnRhbnQ6IGtlZXAgdGhlIG1haWxpbmcgbGlzdCBpbiB0aGUgcmVjaXBpZW50cywgZG8gbm90
DQogICAgICAgICAgICAgIHJlcGx5IG9ubHkgdG8gdGhlIHNlbmRlciE8YnI+DQogICAgICAgICAg
ICAgIEVkaXQgbWFpbGluZyBsaXN0IG9wdGlvbnMgb3IgdW5zdWJzY3JpYmU6PGJyPg0KICAgICAg
ICAgICAgPC9ibG9ja3F1b3RlPg0KICAgICAgICAgIDwvZGl2Pg0KICAgICAgICAgIF9fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188YnI+DQog
ICAgICAgICAgS2FtYWlsaW8gLSBVc2VycyBNYWlsaW5nIExpc3QgLSBOb24gQ29tbWVyY2lhbCBE
aXNjdXNzaW9uczxicj4NCiAgICAgICAgICBUbyB1bnN1YnNjcmliZSBzZW5kIGFuIGVtYWlsIHRv
IDxhDQogICAgICAgICAgICBocmVmPSJtYWlsdG86c3ItdXNlcnMtbGVhdmVAbGlzdHMua2FtYWls
aW8ub3JnIg0KICAgICAgICAgICAgdGFyZ2V0PSJfYmxhbmsiIG1vei1kby1ub3Qtc2VuZD0idHJ1
ZSINCiAgICAgICAgICAgIGNsYXNzPSJtb3otdHh0LWxpbmstZnJlZXRleHQiPnNyLXVzZXJzLWxl
YXZlQGxpc3RzLmthbWFpbGlvLm9yZzwvYT48YnI+DQogICAgICAgICAgSW1wb3J0YW50OiBrZWVw
IHRoZSBtYWlsaW5nIGxpc3QgaW4gdGhlIHJlY2lwaWVudHMsIGRvIG5vdA0KICAgICAgICAgIHJl
cGx5IG9ubHkgdG8gdGhlIHNlbmRlciE8YnI+DQogICAgICAgICAgRWRpdCBtYWlsaW5nIGxpc3Qg
b3B0aW9ucyBvciB1bnN1YnNjcmliZTo8YnI+DQogICAgICAgIDwvYmxvY2txdW90ZT4NCiAgICAg
IDwvZGl2Pg0KICAgICAgPGJyIGNsZWFyPSJhbGwiPg0KICAgICAgPGJyPg0KICAgICAgPHNwYW4g
Y2xhc3M9ImdtYWlsX3NpZ25hdHVyZV9wcmVmaXgiPi0tIDwvc3Bhbj48YnI+DQogICAgICA8ZGl2
IGRpcj0ibHRyIiBjbGFzcz0iZ21haWxfc2lnbmF0dXJlIj4NCiAgICAgICAgPGRpdiBkaXI9Imx0
ciI+QmVzdCByZWdhcmRzLA0KICAgICAgICAgIDxkaXY+SWhvciAoSWdvcik8YnI+DQogICAgICAg
ICAgPC9kaXY+DQogICAgICAgIDwvZGl2Pg0KICAgICAgPC9kaXY+DQogICAgICA8YnI+DQogICAg
ICA8ZmllbGRzZXQgY2xhc3M9Im1vei1taW1lLWF0dGFjaG1lbnQtaGVhZGVyIj48L2ZpZWxkc2V0
Pg0KICAgICAgPHByZSBjbGFzcz0ibW96LXF1b3RlLXByZSIgd3JhcD0iIj5fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQpLYW1haWxpbyAt
IFVzZXJzIE1haWxpbmcgTGlzdCAtIE5vbiBDb21tZXJjaWFsIERpc2N1c3Npb25zDQpUbyB1bnN1
YnNjcmliZSBzZW5kIGFuIGVtYWlsIHRvIDxhIGNsYXNzPSJtb3otdHh0LWxpbmstYWJicmV2aWF0
ZWQiIGhyZWY9Im1haWx0bzpzci11c2Vycy1sZWF2ZUBsaXN0cy5rYW1haWxpby5vcmciPnNyLXVz
ZXJzLWxlYXZlQGxpc3RzLmthbWFpbGlvLm9yZzwvYT4NCkltcG9ydGFudDoga2VlcCB0aGUgbWFp
bGluZyBsaXN0IGluIHRoZSByZWNpcGllbnRzLCBkbyBub3QgcmVwbHkgb25seSB0byB0aGUgc2Vu
ZGVyIQ0KRWRpdCBtYWlsaW5nIGxpc3Qgb3B0aW9ucyBvciB1bnN1YnNjcmliZToNCjwvcHJlPg0K
ICAgIDwvYmxvY2txdW90ZT4NCiAgICA8cHJlIGNsYXNzPSJtb3otc2lnbmF0dXJlIiBjb2xzPSI3
MiI+LS0gDQpEYW5pZWwtQ29uc3RhbnRpbiBNaWVybGEgKEAgYXNpcHRvLmNvbSkNCnR3aXR0ZXIu
Y29tL21pY29uZGEgLS0gbGlua2VkaW4uY29tL2luL21pY29uZGENCkthbWFpbGlvIENvbnN1bHRh
bmN5IC0gVHJhaW5pbmcgU2VydmljZXMgLS0gYXNpcHRvLmNvbQ0KS2FtYWlsaW8gV29ybGQgQ29u
ZmVyZW5jZSAtIGthbWFpbGlvd29ybGQuY29tPC9wcmU+DQogIDwvYm9keT4NCjwvaHRtbD4NCg==

--===============0505140767==--


From bastian.triller@gmail.com Wed Aug 30 15:20:08 2023
From: Bastian Triller <bastian.triller@gmail.com>
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Re: Kamailio stops processing UDP on one interface
Date: Wed, 30 Aug 2023 17:19:43 +0200
Message-ID:
 <CALJ8HB65tA81g_NnanSogT132sFw7V27BcN_tOryj-cb+uZEfw@mail.gmail.com>
In-Reply-To:
 <CAJTkRNtpBDrSrwPiZpxZpEfFHjLaHAmh81pXF0bYodMoM3OHMg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="===============0371453079=="

--===============0371453079==
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Maybe it's not beneficial to have a receive buffer that high. If your
presence server is stateful, it will send retransmissions if packets get
dropped. But it will also send retransmissions if your proxy does not reply
fast enough. So your proxy may be damped by retransmissions. Did you check
SHM/active transactions on your proxy?

Regards,
Bastian

On Wed, Aug 30, 2023 at 3:56=E2=80=AFPM Ihor Olkhovskyi <igorolhovskiy(a)gmai=
l.com>
wrote:

> They are increasing, actually
>
> # ss -l -u -m src X.X.X.X/Y
> State       Recv-Q Send-Q
> Local Address:Port
>        Peer Address:Port
> UNCONN      25167616 0
> X.X.X.X:sip
>            *:*
>          skmem:(r25167616,rb25165824,t0,tb65536,f2304,w0,o0,bl0,d514894)
>
> Le mer. 30 ao=C3=BBt 2023 =C3=A0 15:04, Bastian Triller <bastian.triller(a)=
gmail.com>
> a =C3=A9crit :
>
>> Are drops increasing on that socket while it is happening?
>> ss -l src <local_interface> -u sport 5060 -m
>>
>> On Tue, Aug 29, 2023 at 3:26=E2=80=AFPM Ihor Olkhovskyi <igorolhovskiy(a)g=
mail.com>
>> wrote:
>>
>>> Just to add some info
>>>
>>> netstat -nlp
>>> Active Internet connections (only servers)
>>> Proto Recv-Q Send-Q Local Address           Foreign Address
>>> State       PID/Program name
>>> ...
>>> udp   25167616      0 <local_interface>:5060     0.0.0.0:*
>>>               211759/kamailio
>>> ...
>>>
>>> So I see a huge Receive Queue on UDP for Kamailio which is not clearing.
>>>
>>> Le mar. 29 ao=C3=BBt 2023 =C3=A0 14:29, Ihor Olkhovskyi <igorolhovskiy(a)=
gmail.com>
>>> a =C3=A9crit :
>>>
>>>> Hello,
>>>>
>>>> I've faced a bit strange issue, but a bit of preface. I have Kamailio
>>>> as a proxy (TLS/WS <-> UDP) and second Kamailio as a presence server. At
>>>> some point presence server accepts around 5K PUBLISH within 1 minute and
>>>> sending around the same amount of NOTIFY to proxy Kamailio.
>>>>
>>>> Proxy is "transforming" protocol to TLS, but at sime point I'm starting
>>>> to get these type of errors
>>>>
>>>> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
>>>> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
>>>> <script>: [RELAY] Relay to <sip:X.X.X.X:51571;transport=3Dtls> failed!
>>>> tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed
>>>> tm [t_fwd.c:1588]: t_send_branch(): sending request on branch 0 failed
>>>>
>>>> Some of those messages are 100% valid as client can go away or so. Some
>>>> are not, cause I'm sure client is alive and connected.
>>>>
>>>> But the problem comes later. At some moment proxy Kamailio just stops
>>>> accept UDP traffic on this interface (where it also accepts all NOTIFY's=
),
>>>> at the start of the "stopping accepting" Kamailio sends OPTIONS via
>>>> DISPATCHER but not able to receive 200 OK.
>>>>
>>>> Over TLS on the same interface all is ok. On other (loopback) interface
>>>> UDP is being processed fine, so I don't suspert some limit on open files
>>>> here.
>>>>
>>>> Only restart of Kamailio proxy process helps in this case.
>>>>
>>>> I've tuned net.core.rmem_max and net.core.rmem_default to 25 Mb, so in
>>>> theory buffer should not be the case.
>>>>
>>>> Is there some internal "interface buffer" in Kamailio that is not freed
>>>> upon failure send or maybe I've missed somethig?
>>>>
>>>> Kamailio 5.6.4
>>>>
>>>> fork=3Dyes
>>>> children=3D12
>>>> tcp_children=3D12
>>>>
>>>> enable_tls=3Dyes
>>>>
>>>> tcp_accept_no_cl=3Dyes
>>>> tcp_max_connections=3D63536
>>>> tls_max_connections=3D63536
>>>> tcp_accept_aliases=3Dno
>>>> tcp_async=3Dyes
>>>> tcp_connect_timeout=3D10
>>>> tcp_conn_wq_max=3D63536
>>>> tcp_crlf_ping=3Dyes
>>>> tcp_delayed_ack=3Dyes
>>>> tcp_fd_cache=3Dyes
>>>> tcp_keepalive=3Dyes
>>>> tcp_keepcnt=3D3
>>>> tcp_keepidle=3D30
>>>> tcp_keepintvl=3D10
>>>> tcp_linger2=3D30
>>>> tcp_rd_buf_size=3D80000
>>>> tcp_send_timeout=3D10
>>>> tcp_wq_blk_size=3D2100
>>>> tcp_wq_max=3D10485760
>>>> open_files_limit=3D63536
>>>>
>>>> Sysctl
>>>>
>>>> # To increase the amount of memory available for socket input/output
>>>> queues
>>>> net.ipv4.tcp_rmem =3D 4096 25165824 25165824
>>>> net.core.rmem_max =3D 25165824
>>>> net.core.rmem_default =3D 25165824
>>>> net.ipv4.tcp_wmem =3D 4096 65536 25165824
>>>> net.core.wmem_max =3D 25165824
>>>> net.core.wmem_default =3D 65536
>>>> net.core.optmem_max =3D 25165824
>>>>
>>>> # To limit the maximum number of requests queued to a listen socket
>>>> net.core.somaxconn =3D 128
>>>>
>>>> # Tells TCP to instead make decisions that would prefer lower latency.
>>>> net.ipv4.tcp_low_latency=3D1
>>>>
>>>> # Optional (it will increase performance)
>>>> net.core.netdev_max_backlog =3D 1000
>>>> net.ipv4.tcp_max_syn_backlog =3D 128
>>>>
>>>> # Flush the routing table to make changes happen instantly.
>>>> net.ipv4.route.flush=3D1
>>>> --
>>>> Best regards,
>>>> Ihor (Igor)
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Ihor (Igor)
>>> __________________________________________________________
>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
>>> Important: keep the mailing list in the recipients, do not reply only to
>>> the sender!
>>> Edit mailing list options or unsubscribe:
>>>
>> __________________________________________________________
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only to
>> the sender!
>> Edit mailing list options or unsubscribe:
>>
>
>
> --
> Best regards,
> Ihor (Igor)
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to
> the sender!
> Edit mailing list options or unsubscribe:
>



--===============0371453079==
Content-Type: text/html
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="attachment.html"
MIME-Version: 1.0

PGRpdiBkaXI9Imx0ciI+TWF5YmUgaXQmIzM5O3Mgbm90IGJlbmVmaWNpYWwgdG8gaGF2ZSBhIHJl
Y2VpdmUgYnVmZmVyIHRoYXQgaGlnaC4gSWYgeW91ciBwcmVzZW5jZSBzZXJ2ZXIgaXMgc3RhdGVm
dWwsIGl0IHdpbGwgc2VuZCByZXRyYW5zbWlzc2lvbnMgaWYgcGFja2V0cyBnZXQgZHJvcHBlZC4g
QnV0IGl0IHdpbGwgYWxzbyBzZW5kIHJldHJhbnNtaXNzaW9ucyBpZiB5b3VyIHByb3h5IGRvZXMg
bm90IHJlcGx5IGZhc3QgZW5vdWdoLiBTbyB5b3VyIHByb3h5IG1heSBiZSBkYW1wZWQgYnkgcmV0
cmFuc21pc3Npb25zLiBEaWQgeW91IGNoZWNrIFNITS9hY3RpdmUgdHJhbnNhY3Rpb25zIG9uIHlv
dXIgcHJveHk/PGRpdj48YnI+PC9kaXY+PGRpdj5SZWdhcmRzLDwvZGl2PjxkaXY+QmFzdGlhbjwv
ZGl2PjwvZGl2Pjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBkaXI9Imx0ciIgY2xh
c3M9ImdtYWlsX2F0dHIiPk9uIFdlZCwgQXVnIDMwLCAyMDIzIGF0IDM6NTbigK9QTSBJaG9yIE9s
a2hvdnNreWkgJmx0OzxhIGhyZWY9Im1haWx0bzppZ29yb2xob3Zza2l5QGdtYWlsLmNvbSI+aWdv
cm9saG92c2tpeUBnbWFpbC5jb208L2E+Jmd0OyB3cm90ZTo8YnI+PC9kaXY+PGJsb2NrcXVvdGUg
Y2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRl
ci1sZWZ0LXdpZHRoOjFweDtib3JkZXItbGVmdC1zdHlsZTpzb2xpZDtib3JkZXItbGVmdC1jb2xv
cjpyZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgZGlyPSJsdHIiPjxkaXY+
VGhleSBhcmUgaW5jcmVhc2luZywgYWN0dWFsbHk8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PiMg
c3MgLWwgLXUgLW0gc3JjIFguWC5YLlgvWTxicj5TdGF0ZSDCoCDCoCDCoCBSZWN2LVEgU2VuZC1R
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIExvY2FsIEFkZHJlc3M6UG9ydCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDCoCDC
oCDCoCDCoCDCoCDCoFBlZXIgQWRkcmVzczpQb3J0IMKgIMKgIMKgIMKgIMKgIMKgIMKgIMKgPGJy
PlVOQ09OTiDCoCDCoCDCoDI1MTY3NjE2IDAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgWC5YLlguWDpzaXAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAg
wqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAq
OiogwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqAgwqA8YnI+wqAgwqAgwqAgwqAgwqBza21lbToo
cjI1MTY3NjE2LHJiMjUxNjU4MjQsdDAsdGI2NTUzNixmMjMwNCx3MCxvMCxibDAsZDUxNDg5NCk8
L2Rpdj48L2Rpdj48YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiIGNs
YXNzPSJnbWFpbF9hdHRyIj5MZcKgbWVyLiAzMCBhb8O7dCAyMDIzIMOgwqAxNTowNCwgQmFzdGlh
biBUcmlsbGVyICZsdDs8YSBocmVmPSJtYWlsdG86YmFzdGlhbi50cmlsbGVyQGdtYWlsLmNvbSIg
dGFyZ2V0PSJfYmxhbmsiPmJhc3RpYW4udHJpbGxlckBnbWFpbC5jb208L2E+Jmd0OyBhIMOpY3Jp
dMKgOjxicj48L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJn
aW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQtd2lkdGg6MXB4O2JvcmRlci1sZWZ0LXN0
eWxlOnNvbGlkO2JvcmRlci1sZWZ0LWNvbG9yOnJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0
OjFleCI+PGRpdiBkaXI9Imx0ciI+QXJlIGRyb3BzIGluY3JlYXNpbmcgb24gdGhhdCBzb2NrZXQg
d2hpbGUgaXQgaXMgaGFwcGVuaW5nPzxkaXY+c3MgLWwgc3JjICZsdDs8c3BhbiBzdHlsZT0iY29s
b3I6cmdiKDAsMCwwKSI+bG9jYWxfaW50ZXJmYWNlJmd0Ozwvc3Bhbj7CoC11IHNwb3J0IDUwNjAg
LW08L2Rpdj48L2Rpdj48YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIi
IGNsYXNzPSJnbWFpbF9hdHRyIj5PbiBUdWUsIEF1ZyAyOSwgMjAyMyBhdCAzOjI24oCvUE0gSWhv
ciBPbGtob3Zza3lpICZsdDs8YSBocmVmPSJtYWlsdG86aWdvcm9saG92c2tpeUBnbWFpbC5jb20i
IHRhcmdldD0iX2JsYW5rIj5pZ29yb2xob3Zza2l5QGdtYWlsLmNvbTwvYT4mZ3Q7IHdyb3RlOjxi
cj48L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4
IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQtd2lkdGg6MXB4O2JvcmRlci1sZWZ0LXN0eWxlOnNv
bGlkO2JvcmRlci1sZWZ0LWNvbG9yOnJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+
PGRpdiBkaXI9Imx0ciI+PGRpdj5KdXN0IHRvIGFkZCBzb21lIGluZm88L2Rpdj48ZGl2Pjxicj48
L2Rpdj48ZGl2Pm5ldHN0YXQgLW5scDxicj5BY3RpdmUgSW50ZXJuZXQgY29ubmVjdGlvbnMgKG9u
bHkgc2VydmVycyk8YnI+UHJvdG8gUmVjdi1RIFNlbmQtUSBMb2NhbCBBZGRyZXNzIMKgIMKgIMKg
IMKgIMKgIEZvcmVpZ24gQWRkcmVzcyDCoCDCoCDCoCDCoCBTdGF0ZSDCoCDCoCDCoCBQSUQvUHJv
Z3JhbSBuYW1lIMKgIMKgPGJyPi4uLjxicj51ZHAgwqAgMjUxNjc2MTYgwqAgwqAgwqAwICZsdDts
b2NhbF9pbnRlcmZhY2UmZ3Q7OjUwNjAgwqAgwqAgMC4wLjAuMDoqIMKgIMKgIMKgIMKgIMKgIMKg
IMKgIMKgIMKgIMKgIMKgIMKgIMKgIDIxMTc1OS9rYW1haWxpbzxicj4uLi48YnI+PGJyPjwvZGl2
PjxkaXY+U28gSSBzZWUgYSBodWdlIFJlY2VpdmUgUXVldWUgb24gVURQIGZvciBLYW1haWxpbyB3
aGljaCBpcyBub3QgY2xlYXJpbmcuPGJyPjwvZGl2PjwvZGl2Pjxicj48ZGl2IGNsYXNzPSJnbWFp
bF9xdW90ZSI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX2F0dHIiPkxlwqBtYXIuIDI5IGFv
w7t0IDIwMjMgw6DCoDE0OjI5LCBJaG9yIE9sa2hvdnNreWkgJmx0OzxhIGhyZWY9Im1haWx0bzpp
Z29yb2xob3Zza2l5QGdtYWlsLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmlnb3JvbGhvdnNraXlAZ21h
aWwuY29tPC9hPiZndDsgYSDDqWNyaXTCoDo8YnI+PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9Imdt
YWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0LXdp
ZHRoOjFweDtib3JkZXItbGVmdC1zdHlsZTpzb2xpZDtib3JkZXItbGVmdC1jb2xvcjpyZ2IoMjA0
LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgZGlyPSJsdHIiPkhlbGxvLDxicj48ZGl2
Pjxicj48L2Rpdj48ZGl2PkkmIzM5O3ZlIGZhY2VkIGEgYml0IHN0cmFuZ2UgaXNzdWUsIGJ1dCBh
IGJpdCBvZiBwcmVmYWNlLiBJIGhhdmUgS2FtYWlsaW8gYXMgYSBwcm94eSAoVExTL1dTICZsdDst
Jmd0OyBVRFApIGFuZCBzZWNvbmQgS2FtYWlsaW8gYXMgYSBwcmVzZW5jZSBzZXJ2ZXIuIEF0IHNv
bWUgcG9pbnQgcHJlc2VuY2Ugc2VydmVyIGFjY2VwdHMgYXJvdW5kIDVLIFBVQkxJU0ggd2l0aGlu
IDEgbWludXRlIGFuZCBzZW5kaW5nIGFyb3VuZCB0aGUgc2FtZSBhbW91bnQgb2YgTk9USUZZIHRv
IHByb3h5IEthbWFpbGlvLjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+UHJveHkgaXMgJnF1b3Q7
dHJhbnNmb3JtaW5nJnF1b3Q7IHByb3RvY29sIHRvIFRMUywgYnV0IGF0IHNpbWUgcG9pbnQgSSYj
Mzk7bSBzdGFydGluZyB0byBnZXQgdGhlc2UgdHlwZSBvZiBlcnJvcnM8L2Rpdj48ZGl2Pjxicj48
L2Rpdj48ZGl2PnRtIFsuLi8uLi9jb3JlL2ZvcndhcmQuaDoyOTJdOiBtc2dfc2VuZF9idWZmZXIo
KTogdGNwX3NlbmQgZmFpbGVkPGJyPnRtIFt0X2Z3ZC5jOjE1ODhdOiB0X3NlbmRfYnJhbmNoKCk6
IHNlbmRpbmcgcmVxdWVzdCBvbiBicmFuY2ggMCBmYWlsZWQ8YnI+Jmx0O3NjcmlwdCZndDs6IFtS
RUxBWV0gUmVsYXkgdG8gJmx0O3NpcDpYLlguWC5YOjUxNTcxO3RyYW5zcG9ydD10bHMmZ3Q7IGZh
aWxlZCE8YnI+dG0gWy4uLy4uL2NvcmUvZm9yd2FyZC5oOjI5Ml06IG1zZ19zZW5kX2J1ZmZlcigp
OiB0Y3Bfc2VuZCBmYWlsZWQ8YnI+dG0gW3RfZndkLmM6MTU4OF06IHRfc2VuZF9icmFuY2goKTog
c2VuZGluZyByZXF1ZXN0IG9uIGJyYW5jaCAwIGZhaWxlZDwvZGl2PjxkaXY+PGJyPjwvZGl2Pjxk
aXY+U29tZSBvZiB0aG9zZSBtZXNzYWdlcyBhcmUgMTAwJSB2YWxpZCBhcyBjbGllbnQgY2FuIGdv
IGF3YXkgb3Igc28uIFNvbWUgYXJlIG5vdCwgY2F1c2UgSSYjMzk7bSBzdXJlIGNsaWVudCBpcyBh
bGl2ZSBhbmQgY29ubmVjdGVkLsKgPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5CdXQgdGhlIHBy
b2JsZW0gY29tZXMgbGF0ZXIuIEF0IHNvbWUgbW9tZW50IHByb3h5IEthbWFpbGlvIGp1c3Qgc3Rv
cHMgYWNjZXB0IFVEUCB0cmFmZmljIG9uIHRoaXMgaW50ZXJmYWNlICh3aGVyZSBpdCBhbHNvIGFj
Y2VwdHMgYWxsIE5PVElGWSYjMzk7cyksIGF0IHRoZSBzdGFydCBvZiB0aGUgJnF1b3Q7c3RvcHBp
bmcgYWNjZXB0aW5nJnF1b3Q7IEthbWFpbGlvIHNlbmRzIE9QVElPTlMgdmlhIERJU1BBVENIRVIg
YnV0IG5vdCBhYmxlIHRvIHJlY2VpdmUgMjAwIE9LLiA8YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+
PGRpdj5PdmVyIFRMUyBvbiB0aGUgc2FtZSBpbnRlcmZhY2UgYWxsIGlzIG9rLiBPbiBvdGhlciAo
bG9vcGJhY2spIGludGVyZmFjZSBVRFAgaXMgYmVpbmcgcHJvY2Vzc2VkIGZpbmUsIHNvIEkgZG9u
JiMzOTt0IHN1c3BlcnQgc29tZSBsaW1pdCBvbiBvcGVuIGZpbGVzIGhlcmUuPGJyPjwvZGl2Pjxk
aXY+PGJyPjwvZGl2PjxkaXY+T25seSByZXN0YXJ0IG9mIEthbWFpbGlvIHByb3h5IHByb2Nlc3Mg
aGVscHMgaW4gdGhpcyBjYXNlLiA8YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5JJiMzOTt2
ZSB0dW5lZCBuZXQuY29yZS5ybWVtX21heCBhbmQgbmV0LmNvcmUucm1lbV9kZWZhdWx0IHRvIDI1
IE1iLCBzbyBpbiB0aGVvcnkgYnVmZmVyIHNob3VsZCBub3QgYmUgdGhlIGNhc2UuPC9kaXY+PGRp
dj48YnI+PC9kaXY+PGRpdj5JcyB0aGVyZSBzb21lIGludGVybmFsICZxdW90O2ludGVyZmFjZSBi
dWZmZXImcXVvdDsgaW4gS2FtYWlsaW8gdGhhdCBpcyBub3QgZnJlZWQgdXBvbiBmYWlsdXJlIHNl
bmQgb3IgbWF5YmUgSSYjMzk7dmUgbWlzc2VkIHNvbWV0aGlnPzxicj48YnI+PC9kaXY+PGRpdj5L
YW1haWxpbyA1LjYuNDwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+Zm9yaz15ZXM8YnI+Y2hpbGRy
ZW49MTI8YnI+dGNwX2NoaWxkcmVuPTEyPGJyPjxicj5lbmFibGVfdGxzPXllczxicj48YnI+dGNw
X2FjY2VwdF9ub19jbD15ZXM8YnI+dGNwX21heF9jb25uZWN0aW9ucz02MzUzNjxicj50bHNfbWF4
X2Nvbm5lY3Rpb25zPTYzNTM2PGJyPnRjcF9hY2NlcHRfYWxpYXNlcz1ubzxicj50Y3BfYXN5bmM9
eWVzPGJyPnRjcF9jb25uZWN0X3RpbWVvdXQ9MTA8YnI+dGNwX2Nvbm5fd3FfbWF4PTYzNTM2PGJy
PnRjcF9jcmxmX3Bpbmc9eWVzPGJyPnRjcF9kZWxheWVkX2Fjaz15ZXM8YnI+dGNwX2ZkX2NhY2hl
PXllczxicj50Y3Bfa2VlcGFsaXZlPXllczxicj50Y3Bfa2VlcGNudD0zPGJyPnRjcF9rZWVwaWRs
ZT0zMDxicj50Y3Bfa2VlcGludHZsPTEwPGJyPnRjcF9saW5nZXIyPTMwPGJyPnRjcF9yZF9idWZf
c2l6ZT04MDAwMDxicj50Y3Bfc2VuZF90aW1lb3V0PTEwPGJyPnRjcF93cV9ibGtfc2l6ZT0yMTAw
PGJyPnRjcF93cV9tYXg9MTA0ODU3NjA8YnI+b3Blbl9maWxlc19saW1pdD02MzUzNjwvZGl2Pjxk
aXY+PGJyPjwvZGl2PjxkaXY+U3lzY3RsPGJyPjxicj4jIFRvIGluY3JlYXNlIHRoZSBhbW91bnQg
b2YgbWVtb3J5IGF2YWlsYWJsZSBmb3Igc29ja2V0IGlucHV0L291dHB1dCBxdWV1ZXM8YnI+bmV0
LmlwdjQudGNwX3JtZW0gPSA0MDk2IDI1MTY1ODI0IDI1MTY1ODI0PGJyPm5ldC5jb3JlLnJtZW1f
bWF4ID0gMjUxNjU4MjQ8YnI+bmV0LmNvcmUucm1lbV9kZWZhdWx0ID0gMjUxNjU4MjQ8YnI+bmV0
LmlwdjQudGNwX3dtZW0gPSA0MDk2IDY1NTM2IDI1MTY1ODI0PGJyPm5ldC5jb3JlLndtZW1fbWF4
ID0gMjUxNjU4MjQ8YnI+bmV0LmNvcmUud21lbV9kZWZhdWx0ID0gNjU1MzY8YnI+bmV0LmNvcmUu
b3B0bWVtX21heCA9IDI1MTY1ODI0PGJyPjxicj4jIFRvIGxpbWl0IHRoZSBtYXhpbXVtIG51bWJl
ciBvZiByZXF1ZXN0cyBxdWV1ZWQgdG8gYSBsaXN0ZW4gc29ja2V0PGJyPm5ldC5jb3JlLnNvbWF4
Y29ubiA9IDEyODxicj48YnI+IyBUZWxscyBUQ1AgdG8gaW5zdGVhZCBtYWtlIGRlY2lzaW9ucyB0
aGF0IHdvdWxkIHByZWZlciBsb3dlciBsYXRlbmN5Ljxicj5uZXQuaXB2NC50Y3BfbG93X2xhdGVu
Y3k9MTxicj48YnI+IyBPcHRpb25hbCAoaXQgd2lsbCBpbmNyZWFzZSBwZXJmb3JtYW5jZSk8YnI+
bmV0LmNvcmUubmV0ZGV2X21heF9iYWNrbG9nID0gMTAwMDxicj5uZXQuaXB2NC50Y3BfbWF4X3N5
bl9iYWNrbG9nID0gMTI4PGJyPjxicj4jIEZsdXNoIHRoZSByb3V0aW5nIHRhYmxlIHRvIG1ha2Ug
Y2hhbmdlcyBoYXBwZW4gaW5zdGFudGx5Ljxicj5uZXQuaXB2NC5yb3V0ZS5mbHVzaD0xPGJyPjwv
ZGl2PjxkaXY+PHNwYW4gY2xhc3M9ImdtYWlsX3NpZ25hdHVyZV9wcmVmaXgiPi0tIDwvc3Bhbj48
YnI+PGRpdiBkaXI9Imx0ciIgY2xhc3M9ImdtYWlsX3NpZ25hdHVyZSI+PGRpdiBkaXI9Imx0ciI+
QmVzdCByZWdhcmRzLDxkaXY+SWhvciAoSWdvcik8YnI+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+
PC9kaXY+CjwvYmxvY2txdW90ZT48L2Rpdj48YnIgY2xlYXI9ImFsbCI+PGJyPjxzcGFuIGNsYXNz
PSJnbWFpbF9zaWduYXR1cmVfcHJlZml4Ij4tLSA8L3NwYW4+PGJyPjxkaXYgZGlyPSJsdHIiIGNs
YXNzPSJnbWFpbF9zaWduYXR1cmUiPjxkaXYgZGlyPSJsdHIiPkJlc3QgcmVnYXJkcyw8ZGl2Pklo
b3IgKElnb3IpPGJyPjwvZGl2PjwvZGl2PjwvZGl2PgpfX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyPgpLYW1haWxpbyAtIFVzZXJzIE1h
aWxpbmcgTGlzdCAtIE5vbiBDb21tZXJjaWFsIERpc2N1c3Npb25zPGJyPgpUbyB1bnN1YnNjcmli
ZSBzZW5kIGFuIGVtYWlsIHRvIDxhIGhyZWY9Im1haWx0bzpzci11c2Vycy1sZWF2ZUBsaXN0cy5r
YW1haWxpby5vcmciIHRhcmdldD0iX2JsYW5rIj5zci11c2Vycy1sZWF2ZUBsaXN0cy5rYW1haWxp
by5vcmc8L2E+PGJyPgpJbXBvcnRhbnQ6IGtlZXAgdGhlIG1haWxpbmcgbGlzdCBpbiB0aGUgcmVj
aXBpZW50cywgZG8gbm90IHJlcGx5IG9ubHkgdG8gdGhlIHNlbmRlciE8YnI+CkVkaXQgbWFpbGlu
ZyBsaXN0IG9wdGlvbnMgb3IgdW5zdWJzY3JpYmU6PGJyPgo8L2Jsb2NrcXVvdGU+PC9kaXY+Cl9f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188
YnI+CkthbWFpbGlvIC0gVXNlcnMgTWFpbGluZyBMaXN0IC0gTm9uIENvbW1lcmNpYWwgRGlzY3Vz
c2lvbnM8YnI+ClRvIHVuc3Vic2NyaWJlIHNlbmQgYW4gZW1haWwgdG8gPGEgaHJlZj0ibWFpbHRv
OnNyLXVzZXJzLWxlYXZlQGxpc3RzLmthbWFpbGlvLm9yZyIgdGFyZ2V0PSJfYmxhbmsiPnNyLXVz
ZXJzLWxlYXZlQGxpc3RzLmthbWFpbGlvLm9yZzwvYT48YnI+CkltcG9ydGFudDoga2VlcCB0aGUg
bWFpbGluZyBsaXN0IGluIHRoZSByZWNpcGllbnRzLCBkbyBub3QgcmVwbHkgb25seSB0byB0aGUg
c2VuZGVyITxicj4KRWRpdCBtYWlsaW5nIGxpc3Qgb3B0aW9ucyBvciB1bnN1YnNjcmliZTo8YnI+
CjwvYmxvY2txdW90ZT48L2Rpdj48YnIgY2xlYXI9ImFsbCI+PGJyPjxzcGFuIGNsYXNzPSJnbWFp
bF9zaWduYXR1cmVfcHJlZml4Ij4tLSA8L3NwYW4+PGJyPjxkaXYgZGlyPSJsdHIiIGNsYXNzPSJn
bWFpbF9zaWduYXR1cmUiPjxkaXYgZGlyPSJsdHIiPkJlc3QgcmVnYXJkcyw8ZGl2Pklob3IgKEln
b3IpPGJyPjwvZGl2PjwvZGl2PjwvZGl2PgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fPGJyPgpLYW1haWxpbyAtIFVzZXJzIE1haWxpbmcg
TGlzdCAtIE5vbiBDb21tZXJjaWFsIERpc2N1c3Npb25zPGJyPgpUbyB1bnN1YnNjcmliZSBzZW5k
IGFuIGVtYWlsIHRvIDxhIGhyZWY9Im1haWx0bzpzci11c2Vycy1sZWF2ZUBsaXN0cy5rYW1haWxp
by5vcmciIHRhcmdldD0iX2JsYW5rIj5zci11c2Vycy1sZWF2ZUBsaXN0cy5rYW1haWxpby5vcmc8
L2E+PGJyPgpJbXBvcnRhbnQ6IGtlZXAgdGhlIG1haWxpbmcgbGlzdCBpbiB0aGUgcmVjaXBpZW50
cywgZG8gbm90IHJlcGx5IG9ubHkgdG8gdGhlIHNlbmRlciE8YnI+CkVkaXQgbWFpbGluZyBsaXN0
IG9wdGlvbnMgb3IgdW5zdWJzY3JpYmU6PGJyPgo8L2Jsb2NrcXVvdGU+PC9kaXY+Cg==

--===============0371453079==--