Hello,
I'm having strange behavior with the http_async_client module and https connections. Our kamailio config use REST interfaces to send push to mobile apps when calls are initiated. We also use a REST interface to register the client in our push database (and then be able to use the info to send a push). Our REST interface use TLS to cypher exchanged data between the SIP registrar and the Pusher service.
We have configured the tls connection using modparam:
# ------ http async client params ------- modparam("http_async_client", "workers", 16) modparam("http_async_client", "connection_timeout", 20000) # 20 seconds timeout on HTTP requests modparam("http_async_client", "hash_size", 2048) # update this value depending of the load #modparam("http_async_client", "curl_verbose", 1) # curl debug modparam("http_async_client", "tls_ca_path", "/etc/kamailio/ssl/ca")
What we have noticed is when there is 1 SIP Register requests triggering and HTTPS request all works fine, but as soon as we have several HTTPS request in //, the HTTPS request fails with curl error 77: curl: (77) Problem with the SSL CA cert (path? access rights?).
We tried to put the Root CA Path when building the request:
$http_req(all) = $null; # reset the parameters $http_req(method) = "POST"; $http_req(hdr) = "authorization: "+$hdr(authorization); $http_req(hdr) = "Content-Type: application/json"; $http_req(suspend) = 0; # don't suspend the transaction, continue routing script's execution $http_req(body) = "{ 'pushRegistration':{'callId':'" + $ci + "', 'fromUri':'" + $fU + "@" + $fd + "', 'contactLine':'" + $ct + "'}}"; $http_req(tls_ca_path) = "/etc/kamailio/ssl/ca/"; http_async_query("https://pusher:443/rest/push/register", "HTTP_REGISTER_REPLY");
But we have exactly the same behavior.
Does the http_async_client module supports muti-thread?
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Hi Giovanni, what's the related OS and libcurl versions please?
Regards, Giacomo
On 2 November 2017 at 09:54, gmele giovanni.mele@nagra.com wrote:
Hello,
I'm having strange behavior with the http_async_client module and https connections. Our kamailio config use REST interfaces to send push to mobile apps when calls are initiated. We also use a REST interface to register the client in our push database (and then be able to use the info to send a push). Our REST interface use TLS to cypher exchanged data between the SIP registrar and the Pusher service.
We have configured the tls connection using modparam:
# ------ http async client params ------- modparam("http_async_client", "workers", 16) modparam("http_async_client", "connection_timeout", 20000) # 20 seconds timeout on HTTP requests modparam("http_async_client", "hash_size", 2048) # update this value depending of the load #modparam("http_async_client", "curl_verbose", 1) # curl debug modparam("http_async_client", "tls_ca_path", "/etc/kamailio/ssl/ca")
What we have noticed is when there is 1 SIP Register requests triggering and HTTPS request all works fine, but as soon as we have several HTTPS request in //, the HTTPS request fails with curl error 77: curl: (77) Problem with the SSL CA cert (path? access rights?).
We tried to put the Root CA Path when building the request:
$http_req(all) = $null; # reset the parameters $http_req(method) = "POST"; $http_req(hdr) = "authorization: "+$hdr(authorization); $http_req(hdr) = "Content-Type: application/json"; $http_req(suspend) = 0; # don't suspend the transaction, continuerouting script's execution $http_req(body) = "{ 'pushRegistration':{'callId':'" + $ci + "', 'fromUri':'" + $fU + "@" + $fd + "', 'contactLine':'" + $ct + "'}}"; $http_req(tls_ca_path) = "/etc/kamailio/ssl/ca/"; http_async_query("https://pusher:443/rest/push/register", "HTTP_REGISTER_REPLY");
But we have exactly the same behavior.
Does the http_async_client module supports muti-thread?
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Giacomo,
I forgot to put these information ;-)
Kamailio : 5.0.2-3.1 OS: Linux CentOS 7.4.178 Curl : 7.29.0-42
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Thanks Giovanni, the good news is that I'm able to reproduce the issue (same CentOS and libcurl version, but kamailio 5.1.0-pre0).
I still don't understand the root cause though.
In my case under some light load in about 1 case every 20 I see something like this logged (debug level 3):
Nov 2 23:19:56 localhost /usr/local/sbin/kamailio[26767]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] warning: CURLOPT_CAPATH not a directory (/usr/local/etc/kamailio/ssl/ca/ca.pem) ... Nov 2 23:19:56 localhost /usr/local/sbin/kamailio[26767]: ERROR: http_async_client [http_multi.c:574]: check_multi_info(): handle 0x7fc351833080 returned error 77:
'/usr/local/etc/kamailio/ssl/ca/ca.pem' is the CA root file, while tls_ca_path is instead set to '/usr/local/etc/kamailio/ssl/ca'.
Can you see that curl warning in your case?
Regards, Giacomo
On 2 November 2017 at 11:30, gmele giovanni.mele@nagra.com wrote:
Hello Giacomo,
I forgot to put these information ;-)
Kamailio : 5.0.2-3.1 OS: Linux CentOS 7.4.178 Curl : 7.29.0-42
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Giacomo,
currently, I don't see these CURL warnings. Here are the logs I get using the verbose mode of Curl.
When it works:
Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] About to connect() to pusher-service port 8443 (#1) Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Trying 192.168.0.50... Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Connected to pusher-service (192.168.0.50) port 8443 (#1) *Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] CAfile: /etc/pki/tls/certs/ca-bundle.crt#012 CApath: /etc/kamailio/ssl/ca/* Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
When it doesn't work:
Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] About to connect() to pusher-service port 8443 (#0) Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Trying 192.168.0.50... Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Connected to pusher-service (192.168.0.50) port 8443 (#0) *Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Initializing NSS with certpath: sql:/etc/pki/nssdb* Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Closing connection 0 Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: ERROR: http_async_client [http_multi.c:570]: check_multi_info(): handle 0x1cf1b00 returned error 77:
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
OK Giovanni, this is likely to be an issue with the way CURLOPT_CAPATH is set. I'll submit a fix as soon as possible.
Giacomo
On 3 November 2017 at 09:49, gmele giovanni.mele@nagra.com wrote:
Hello Giacomo,
currently, I don't see these CURL warnings. Here are the logs I get using the verbose mode of Curl.
When it works:
Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] About to connect() to pusher-service port 8443 (#1) Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Trying 192.168.0.50... Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Connected to pusher-service (192.168.0.50) port 8443 (#1) *Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] CAfile: /etc/pki/tls/certs/ca-bundle.crt#012 CApath: /etc/kamailio/ssl/ca/* Nov 3 08:41:46 d-wn-sipregistrar-003 kamailio-registrar[18948]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
When it doesn't work:
Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] About to connect() to pusher-service port 8443 (#0) Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Trying 192.168.0.50... Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Connected to pusher-service (192.168.0.50) port 8443 (#0) *Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Initializing NSS with certpath: sql:/etc/pki/nssdb* Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Closing connection 0 Nov 3 08:43:44 d-wn-sipregistrar-003 kamailio-registrar[18949]: ERROR: http_async_client [http_multi.c:570]: check_multi_info(): handle 0x1cf1b00 returned error 77:
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Ok, thx!
BTW, do you know if it is possible to build the http_async_client rpm on the openSUSE download server?
The http_client is present, but not the http_async one....
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Hi Giovanni,
This commit attempts to fix the issue: https://github.com/kamailio/kamailio/commit/574a11d8c0c20d3d0c058726609df8ed... Would it be possible to try that branch or patch the module accordingly? I've tested it on CentOS 7.4.1708 with libcurl 7.56.1-1.0.cf.rhel7 (but I don't think the libcurl version has a role here).
For the rpms, I do see kamailio-http_async_client-5.0.4-25.el7.centos.x86_64.rpm in http://download.opensuse.org/repositories/home:/kamailio:/v5.0.x-rpms/CentOS... but I'm not sure this answers your question.
For debian, http_async_client is included in kamailio-extra-modules.
Regards, Giacomo
On 3 November 2017 at 13:28, gmele giovanni.mele@nagra.com wrote:
Ok, thx!
BTW, do you know if it is possible to build the http_async_client rpm on the openSUSE download server?
The http_client is present, but not the http_async one....
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Giacomo,
I patched my branch 5.0.2 with your modifications, but the result is the same: same problem occurs with heavy load and lot of http requests in parallel. Same error code 77.
I didn't upgrade the version of curl. I will maybe give it a try. Can you give me the link from where you downloaded the curl 7.56.1 rpm so I can use same version as yours?
Thx
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
OK Giovanni, then I'd need some more information to try and get to the root cause, like: - 5.0.2 git hash you've applied the patch on. - Rate of requests that seem to trigger this behaviour. - A sample .cfg file I can use to simulate your scenario (you posted one at the beginning but please see if you can share a full one). - Shared memory allocated, and number of http_async_client workers configured. (Does it show the same behaviour with a limited amount of workers, e.g. 2?) - Can you reproduce the problem with master? - Also if possible run with debug level 3 and curl verbosity on.
For the libcurl upgrade: https://mirror.city-fan.org/ftp/contrib/sysutils/Mirroring/curl-7.56.1-1.0.c... (+ libcurl and libcurl-devel).
Regards, Giacomo
On 6 November 2017 at 16:07, gmele giovanni.mele@nagra.com wrote:
Hello Giacomo,
I patched my branch 5.0.2 with your modifications, but the result is the same: same problem occurs with heavy load and lot of http requests in parallel. Same error code 77.
I didn't upgrade the version of curl. I will maybe give it a try. Can you give me the link from where you downloaded the curl 7.56.1 rpm so I can use same version as yours?
Thx
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Giacomo,
here are the requested info:
1) unfortunately, I cannot give you this information as we copied the source from git branch 5.0, August 18th and put it in our own source control (Perforce)
2)1-2 requests per seconds seems ok, but as soon rate increases, problem appears
3) I cannot give you the configuration file as we have some sensible information in it. But we have the following configuration: SIP Proxy -> SIP Registrar. The SIP registrar is using http_async_client when a REGISTER is recieved (to register PUSH information from mobile app). Usually, this works fine with 1-2 requests per seconds, but starts to fails as soon as rate increases. When an INVITE is receivedm we also call http_async_client in order to send a PUSH notification to the callee. I never saw this request work as usually, there is a REGISTER and right after an INVITE done, and it seems that this cause http_async_client to fail.
I understand this information may not help you :-(
4) I configured 16 workers, but also tried with one. Kamailio is configured with 8 childrens and SHM_MEMORY=2048. Maybe the problem is related to kamailio childrens? I will try with only one children...
5) Upgrading to latest version 5.x will take me some time. I will try to do this by the end of the week.
6) traces with debug level 3 and curl verbosity on are the same as the one I sent you: when all is ok, the CApath is /etc/kamailio/ssl/ca/, and when it doesn't work, it shows CApath: /etc/kamailio/ssl/ca/.
Let me try with children set to 1 to verify if it is a multi-thread problem with the module. I will try to send you more logs.
Thx for your support
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Hello,
I reduced the number of kamailio children to 1 and found something that may be interesting:
all requests are processed by 1 principal thread, but time to time, another thread handle them (why, I don't know as children=1). When this occurs, the https request ALWAYS fails with error 77. When the https requests are handled by the main thread, all is ok.....
Here are some traces (without curl verbose and debug). In the REGISTER, we call the http_async_client module...
Nov 7 14:39:02 d-wn-sipregistrar-003 kamailio-registrar[*3325*]: msgType=1 cSeq=30 method=REGISTER callId=F0JIro3OZ3 | NOTICE: SIP Request from sipSrc=sip:giovanni.mele@testrom2.com to sipDst=sip:giovanni.mele@testrom2.com Nov 7 14:39:02 d-wn-sipregistrar-003 kamailio-registrar[*3325*]: msgType=1 cSeq=30 method=REGISTER callId=F0JIro3OZ3 | NOTICE: Register device for user sipSrc=giovanni.mele@testrom2.com Nov 7 14:39:02 d-wn-sipregistrar-003 kamailio-registrar[*3332*]: ERROR: http_async_client [http_multi.c:570]: check_multi_info(): handle 0x224edc0 returned error 77: Nov 7 14:39:02 d-wn-sipregistrar-003 kamailio-registrar[*3332*]: ERROR: Curl errorCode= when trying to register device Nov 7 14:39:06 d-wn-sipregistrar-003 kamailio-registrar[*3325*]: msgType=1 cSeq=31 method=REGISTER callId=F0JIro3OZ3 | NOTICE: SIP Request from sipSrc=sip:giovanni.mele@testrom2.com to sipDst=sip:giovanni.mele@testrom2.com Nov 7 14:39:08 d-wn-sipregistrar-003 kamailio-registrar[*3325*]: msgType=1 cSeq=31 method=REGISTER callId=F0JIro3OZ3 | NOTICE: Register device for user sipSrc=giovanni.mele@testrom2.com
Hope it helps
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Hello Giacomo,
any news on this topic? Are you waiting something from me?
Thx
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Hi Giovanni,
This is what I've done, on the same CentOS 7.4.1708 (and libcurl 7.56.1-1.0.cf.rhel7) machine referred to in this thread.
- Checked out 071b85f66cabaa3a705a014b26b7c1eb31029b26 from branch 5.0 (which is the last one before Aug 18th). - Used the latest http_async_client module - make clean, make all, make install - Launched tests at 10 cps for several seconds with:
modparam("http_async_client", "tls_ca_path", "/usr/local/etc/kamailio/ssl/ca")
modparam("http_async_client", "curl_verbose", 1) ...
$http_req(suspend) = 0;
http_async_query("$var(target_url)", "HTTP_REPLY");
but I wasn't able to reproduce the issue you reported.
Please consider upgrading to latest 5.0 (or master, if possible) and test again.
Regards, Giacomo
On 9 November 2017 at 17:55, gmele giovanni.mele@nagra.com wrote:
Hello Giacomo,
any news on this topic? Are you waiting something from me?
Thx
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Giacomo,
thx for your investigation.
We will move to kamailio 5.1 in 2-3 weeks. I will resume the testing of the http_async_client after the migration on our lab. I hope this will fix the problem :-)
Do you think we alsoneed to migrate curl? Or could we stay with the official centos 7 version (curl-7.29.0-42.el7.x86_64)? I would prefer this solution as we need to install kamailio on production machines where we cannot always upgrade os packages...
Thx again
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Hello Giacomo,
I installed latest rpm found on http://download.opensuse.org/repositories/home:/kamailio:/v5.0.x-rpms/CentOS.... Build date is Nov 16.
I also installed curl version 7.52.1.
After the installation, I started to test the https connection and still get the same problem with curl error 77.
Question: do you know if the kamailio 5.0.4 rpms I downloaded from the above repo have your fix? Should I recompile locally the http_async_client?
BTW, in our kamailio config, we use both http_client and http_async_client, and both do https connections. Module http_async_client is loaded first.
When I gather logs, I can see that the wrong CA path is loaded:
INFO: http_async_client [http_multi.c:238]: debug_cb(): [cURL] Initializing NSS with certpath: sql:/etc/pki/nssdb
Maybe I could send you version of my kamailio config file (without some sensiitive data)? Do you have a private mail I could use?
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Hi,
I recompiled the http_async_client instead of using the one provided in the rpm repository: all is working fine!
I think we can close the topic!
Do do know where we can find rpm with the latest 5.0.4 release?
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Hi Giovanni, I'm glad it's fixed. The patch was pushed in 5.0 branch after the latest release (5.0.4, Oct 25th 2017), so it should be available as soon as 5.0.5 will be released (I think soon).
Best Regards, Giacomo
On 20 November 2017 at 15:12, gmele giovanni.mele@nagra.com wrote:
Hi,
I recompiled the http_async_client instead of using the one provided in the rpm repository: all is working fine!
I think we can close the topic!
Do do know where we can find rpm with the latest 5.0.4 release?
Regards
Giovanni
-- Sent from: http://sip-router.1086192.n5.nabble.com/Users-f3.html
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
Giacomo Vacca -
gmele