with latest master, i tried to add [group] to mysql db_urls and sqlops urls, eg.
modparam("mtree", "db_url", "mysql://user:pass@[sip_proxy]127.0.0.1/sip_proxy")
and in my.cfg the group was defined like this:
[sip_proxy] ssl-ca=/etc/mysql/ca-cert.pem ssl-cert=/etc/mysql/client-cert.pem ssl-key=/etc/mysql/client-key.pem
the result was that when i start kamailio, it starts to run out of mysql connections and dumps core no matter how high i set max_connections in my.cfg:
Jan 21 00:47:33 siika /usr/bin/sip-proxy[30612]: ERROR: db_mysql [km_my_con.c:114]: db_mysql_new_connection(): driver error: Too many connections Jan 21 00:47:33 siika /usr/bin/sip-proxy[30612]: ERROR: <core> [db.c:314]: db_do_init2(): could not add connection to the pool Jan 21 00:47:33 siika /usr/bin/sip-proxy[30612]: ERROR: mtree [mtree_mod.c:346]: mi_child_init(): failed to connect to database Jan 21 00:47:33 siika /usr/bin/sip-proxy[30612]: ERROR: <core> [sr_module.c:900]: init_mod_child(): Error while initializing module mtree (/usr/lib/sip-proxy/modules/mtree.so)
without the [group] kamailio starts fine and does not complain about too many connections.
is this a bug in group handling or something else?
-- juha
when [group] thing didn't work, i added
ssl-ca=/etc/mysql/cacert.pem
to [client] section of my.cfg that kamailio according to db_mysql/README is reading.
after that, kamailio started ok, but didn't use ssl for mysql queries.
what is it that i'm missing? has anyone succeeded in making kamailio to query mysql server over ssl?
-- juha
Juha Heinanen writes:
when [group] thing didn't work, i added
ssl-ca=/etc/mysql/cacert.pem
to [client] section of my.cfg that kamailio according to db_mysql/README is reading.
after that, kamailio started ok, but didn't use ssl for mysql queries.
what is it that i'm missing? has anyone succeeded in making kamailio to query mysql server over ssl?
based on zero responses, i guess the answer is "no". if so, that pretty much prevents using kamailio in an environment where mysql service is provided by a cloud service, such as amazon ec2.
should i put a note in db_mysql module README telling that we don't currently know, which [client] params of my.cfg the module supports?
-- juha
On 21/01/15 21:52, Juha Heinanen wrote:
Juha Heinanen writes:
when [group] thing didn't work, i added
ssl-ca=/etc/mysql/cacert.pem
to [client] section of my.cfg that kamailio according to db_mysql/README is reading.
after that, kamailio started ok, but didn't use ssl for mysql queries.
what is it that i'm missing? has anyone succeeded in making kamailio to query mysql server over ssl?
based on zero responses, i guess the answer is "no". if so, that pretty much prevents using kamailio in an environment where mysql service is provided by a cloud service, such as amazon ec2.
should i put a note in db_mysql module README telling that we don't currently know, which [client] params of my.cfg the module supports?
This doesn't have to do with kamailio at all, it is mysql client configuration in my.cfg.
Kamailio itself uses mysql_options() to set the group to be used: - http://dev.mysql.com/doc/refman/5.5/en/mysql-options.html
It is nothing else that kamailio can do. So whatever mysql supports for groups in my.cnf it is the mysql client library that deals with. Looking at above documentation and kamailio code, the order is right, so all seems fine in the code. If you cannot find something in mysql docs, it is a lack on their side.
Therefore I don't think the kamailio docs need to be updated with your remark -- again, that is about what mysql client library supports.
I would try to test with incremental number of children to see where the limits is reached, then try to identify where that values is set. Also, running mysql server in debug mode might help.
Cheers, Daniel
Daniel-Constantin Mierla writes:
This doesn't have to do with kamailio at all, it is mysql client configuration in my.cfg.
Kamailio itself uses mysql_options() to set the group to be used:
It is nothing else that kamailio can do. So whatever mysql supports for groups in my.cnf it is the mysql client library that deals with. Looking at above documentation and kamailio code, the order is right, so all seems fine in the code. If you cannot find something in mysql docs, it is a lack on their side.
Therefore I don't think the kamailio docs need to be updated with your remark -- again, that is about what mysql client library supports.
if i add line:
ssl-ca=/etc/mysql/ca-cert.pem
to [client] section of my.cfg and connect to the server with mysql cli command
mysql -u root -p
i get:
mysql> show status like 'ssl_cipher'; +---------------+--------------------+ | Variable_name | Value | +---------------+--------------------+ | Ssl_cipher | DHE-RSA-AES256-SHA | +---------------+--------------------+ 1 row in set (0.00 sec)
showing that ssl is used. without the ssl-ca line Value is empty showing that ssl is not used.
when i execute in my kamailio config:
sql_query("sip_proxy", "show status like 'ssl_cipher'", "result"); xlog("L_INFO", "**** got result <variable=$dbr(result=>[0, 0])/value=$dbr(result=>[0, 1])>\n"); sql_result_free("result");
i get:
Jan 22 00:05:53 siika /usr/bin/sip-proxy[6735]: INFO: **** got result <variable=Ssl_cipher/value=>
showing that ssl is not used even when ssl-ca line is in [client] section of my.cfg.
according to db_mysql README:
In addition to the given group, also the [client] section is read, in the order given in my.cnf.
if that is true, why is ssl not used for queries as specified in my.cfg?
-- juha
On 21/01/15 23:21, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
This doesn't have to do with kamailio at all, it is mysql client configuration in my.cfg.
Kamailio itself uses mysql_options() to set the group to be used:
It is nothing else that kamailio can do. So whatever mysql supports for groups in my.cnf it is the mysql client library that deals with. Looking at above documentation and kamailio code, the order is right, so all seems fine in the code. If you cannot find something in mysql docs, it is a lack on their side.
Therefore I don't think the kamailio docs need to be updated with your remark -- again, that is about what mysql client library supports.
if i add line:
ssl-ca=/etc/mysql/ca-cert.pem
to [client] section of my.cfg and connect to the server with mysql cli command
mysql -u root -p
i get:
mysql> show status like 'ssl_cipher'; +---------------+--------------------+ | Variable_name | Value | +---------------+--------------------+ | Ssl_cipher | DHE-RSA-AES256-SHA | +---------------+--------------------+ 1 row in set (0.00 sec)
showing that ssl is used. without the ssl-ca line Value is empty showing that ssl is not used.
when i execute in my kamailio config:
sql_query("sip_proxy", "show status like 'ssl_cipher'", "result"); xlog("L_INFO", "**** got result <variable=$dbr(result=>[0, 0])/value=$dbr(result=>[0, 1])>\n"); sql_result_free("result");i get:
Jan 22 00:05:53 siika /usr/bin/sip-proxy[6735]: INFO: **** got result <variable=Ssl_cipher/value=>
showing that ssl is not used even when ssl-ca line is in [client] section of my.cfg.
according to db_mysql README:
In addition to the given group, also the [client] section is read, in the order given in my.cnf.
if that is true, why is ssl not used for queries as specified in my.cfg?
A book indexed by google says that [client] section is also read along with the custom group. The example is shown with Perl, but I guess the same client library is behind. Same seems to be suggested in the perl docs:
- http://search.cpan.org/~capttofu/DBD-mysql-4.029/lib/DBD/mysql.pm#Class_Meth...
Now, those can be old, maybe newer versions of mysql client library behave differently.
Have you tried to put all wanted options to the group?
Daniel
Daniel-Constantin Mierla writes:
A book indexed by google says that [client] section is also read along with the custom group. The example is shown with Perl, but I guess the same client library is behind. Same seems to be suggested in the perl docs:
i gave up on using [group] because of kamailio didn't start. these tests are thus done just using the default [client] group.
Now, those can be old, maybe newer versions of mysql client library behave differently.
i would expect mysql cli client from debian wheezy mysql-client-5.5 package to use libmysqlclient18 package, i.e., the same one my kamailio is using.
Have you tried to put all wanted options to the group?
i have all wanted options in [client] group:
[client] port = 3306 socket = /var/run/mysqld/mysqld.sock ssl-ca=/etc/mysql/cacert.pem
according to mysql 5.5 ref man, ssl-ca option implies ssl option and those are enough for mysql cli client to use ssl.
-- juha
On 22/01/15 03:32, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
A book indexed by google says that [client] section is also read along with the custom group. The example is shown with Perl, but I guess the same client library is behind. Same seems to be suggested in the perl docs:
i gave up on using [group] because of kamailio didn't start. these tests are thus done just using the default [client] group.
Now, those can be old, maybe newer versions of mysql client library behave differently.
i would expect mysql cli client from debian wheezy mysql-client-5.5 package to use libmysqlclient18 package, i.e., the same one my kamailio is using.
I checked the dependencies of packages on wheezy and libmysqlclient is not listed for mysql-client, so the later is not using the lib.
Have you tried to put all wanted options to the group?
i have all wanted options in [client] group:
[client] port = 3306 socket = /var/run/mysqld/mysqld.sock ssl-ca=/etc/mysql/cacert.pem
according to mysql 5.5 ref man, ssl-ca option implies ssl option and those are enough for mysql cli client to use ssl.
What page did you read? That could be for mysql client cli tool, but appears it doesn't apply to libmysql client -- at the bottom of the next link, it says that a specific function has to be used in c code to set the certificates and enable connecting via tls:
- http://dev.mysql.com/doc/refman/5.5/en/using-ssl-connections.html
Daniel
Daniel-Constantin Mierla writes:
I checked the dependencies of packages on wheezy and libmysqlclient is not listed for mysql-client, so the later is not using the lib.
ok.
i have all wanted options in [client] group:
[client] port = 3306 socket = /var/run/mysqld/mysqld.sock ssl-ca=/etc/mysql/cacert.pem
according to mysql 5.5 ref man, ssl-ca option implies ssl option and those are enough for mysql cli client to use ssl.
What page did you read?
i don't know about page number but here is the text:
--ssl-ca=file_name
The path to a file in PEM format that contains a list of trusted SSL certificate authorities. This option implies --ssl.
That could be for mysql client cli tool, but appears it doesn't apply to libmysql client -- at the bottom of the next link, it says that a specific function has to be used in c code to set the certificates and enable connecting via tls:
here is the text:
The C API enables application programs to use SSL:
To establish a secure connection, use the mysql_ssl_set() C API function to set the appropriate certificate options before calling mysql_real_connect(). See Section 23.8.7.67, “mysql_ssl_set()”.
so if [client] has ssl-ca option, mysql_ssl_set() should be called?
-- juha
i tried again with [group] and my sip proxy started with ssl stuff in the group when i increase my.cfg max_connections from 200 to 400. i then checked and the number of connections to mysql server was 362! after that also queries were done over ssl.
now two problems remain:
- why ssl usage requires huge increase in number of db connections?
- why ssl is not used when i add ssl options to [client] instead of [group]?
-- juha
On 23 Jan 2015, at 03:58, Juha Heinanen jh@tutpro.com wrote:
- why ssl is not used when i add ssl options to [client] instead of
[group]?
That is indeed strange.
I've started labbing a bit with this. I did not get the client to connect with SSL unless I added ssl-cipher to the client like this
/usr/local/mysql/bin/mysql -h 127.0.0.1 -P 63306 --ssl --ssl-cipher=DHE-RSA-AES256-SHA:AES128-SHA << EOF SHOW STATUS LIKE 'Ssl_cipher'; EOF
The default for Mysql - even if you have the --ssl option is no ssl.
There's no way to really test the SSL configuration with common tools, since Mysql adds a few packets in front of the SSL negotiation. The server indicates SSL support and then the client starts SSL. http://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-P...
This opens up for a downgrade attack, much like STARTSSL in SMTP.
There's also no setting for version of SSL/TLS you want to use, only a CIPHER setting.
Mysql and/or MariaDB really needs to take this seriously and change their way of handling TLS. The current implementation indicates that they needed a check-box option, but it was a long time ago and haven't been updated since.
There is an overhead, which is propably why many people don't use it (apart from the weird configuration). This presentation has some numbers.
http://www.yassl.com/files/yassl_securing_mysql.pdf
We can move the MySQL TLS settings into the db_mysql module, it seems like there are API calls for it. If so, the db_mysql module could perform the query above and verify that SSL is active. Right now, I would put a query using SQLOPS like you showed in an earlier mail in htable::mod-init event route and disable the server if I got the wrong answer.
/O
Olle E. Johansson writes:
I've started labbing a bit with this. I did not get the client to connect with SSL unless I added ssl-cipher to the client like this
/usr/local/mysql/bin/mysql -h 127.0.0.1 -P 63306 --ssl --ssl-cipher=DHE-RSA-AES256-SHA:AES128-SHA << EOF SHOW STATUS LIKE 'Ssl_cipher'; EOF
i can connect to mysql server without any mysql cli options if i have
ssl-ca=/etc/mysql/ca-cert.pem
in [client] section of my.cfg:
# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1462 Server version: 5.5.40-0+wheezy1 (Debian)
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> SHOW STATUS LIKE 'Ssl_cipher'; +---------------+--------------------+ | Variable_name | Value | +---------------+--------------------+ | Ssl_cipher | DHE-RSA-AES256-SHA | +---------------+--------------------+ 1 row in set (0.00 sec)
did you try to connect to the server with ssl from kamailio using [group]?
-- juha
On 21 Jan 2015, at 21:52, Juha Heinanen jh@tutpro.com wrote:
Juha Heinanen writes:
when [group] thing didn't work, i added
ssl-ca=/etc/mysql/cacert.pem
to [client] section of my.cfg that kamailio according to db_mysql/README is reading.
after that, kamailio started ok, but didn't use ssl for mysql queries.
what is it that i'm missing? has anyone succeeded in making kamailio to query mysql server over ssl?
based on zero responses, i guess the answer is "no". if so, that pretty much prevents using kamailio in an environment where mysql service is provided by a cloud service, such as amazon ec2.
should i put a note in db_mysql module README telling that we don't currently know, which [client] params of my.cfg the module supports?
We've seen reports of issues with Postgresql with TLS too, I don't know what happened, but I think we need to focus on both and fix this.
There is a known geneal problem with libraries using OpenSSL - I don't know if this has been looked at in Kamailio, but we did a fix in Asterisk a while ago. If you have modules using libraries that use OpenSSL - like we have in Curl, Mysql, Postgres and possibly other modules - as well as our own use in the TLS module - there's a risk that OpenSSL gets initialized too many times and bad things happen. ("Bad things" need to be defined here).
I think Kevin did a library trick with the linker so that Asterisk catch these initialization calls first and use just one. Asterisk is multithreaded and Kamailio is multiprocess, so I don't know how this affects Kamailio or if we can get some inspiration by this fix.
Rambling a bit, but trying to point in some sort of general direction. :-)
I will put on my list to set up a lab with Mysql TLS connections and try.
/O
On Thu, Jan 22, 2015 at 2:47 AM, Olle E. Johansson oej@edvina.net wrote:
On 21 Jan 2015, at 21:52, Juha Heinanen jh@tutpro.com wrote:
Juha Heinanen writes:
when [group] thing didn't work, i added
ssl-ca=/etc/mysql/cacert.pem
to [client] section of my.cfg that kamailio according to db_mysql/README is reading.
after that, kamailio started ok, but didn't use ssl for mysql queries.
what is it that i'm missing? has anyone succeeded in making kamailio to query mysql server over ssl?
based on zero responses, i guess the answer is "no". if so, that pretty much prevents using kamailio in an environment where mysql service is provided by a cloud service, such as amazon ec2.
should i put a note in db_mysql module README telling that we don't currently know, which [client] params of my.cfg the module supports?
We've seen reports of issues with Postgresql with TLS too, I don't know what happened, but I think we need to focus on both and fix this.
There is a known geneal problem with libraries using OpenSSL - I don't know if this has been looked at in Kamailio, but we did a fix in Asterisk a while ago. If you have modules using libraries that use OpenSSL - like we have in Curl, Mysql, Postgres and possibly other modules - as well as our own use in the TLS module - there's a risk that OpenSSL gets initialized too many times and bad things happen. ("Bad things" need to be defined here).
I think Kevin did a library trick with the linker so that Asterisk catch these initialization calls first and use just one. Asterisk is multithreaded and Kamailio is multiprocess, so I don't know how this affects Kamailio or if we can get some inspiration by this fix.
Rambling a bit, but trying to point in some sort of general direction. :-)
I will put on my list to set up a lab with Mysql TLS connections and try.
Just chiming in to point out the magic module Olle is referring to:
http://svn.asterisk.org/svn/asterisk/trunk/main/libasteriskssl.c
For context, the peer review for the patch that fixed this issue is here:
https://reviewboard.asterisk.org/r/1006/
Although due to some issues in review board, part of the patch doesn't show up (hence the link to the actual source).
Matt
On 22/01/15 16:51, Matthew Jordan wrote:
On Thu, Jan 22, 2015 at 2:47 AM, Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net> wrote:
On 21 Jan 2015, at 21:52, Juha Heinanen <jh@tutpro.com <mailto:jh@tutpro.com>> wrote: > Juha Heinanen writes: > >> when [group] thing didn't work, i added >> >> ssl-ca=/etc/mysql/cacert.pem >> >> to [client] section of my.cfg that kamailio according to db_mysql/README >> is reading. >> >> after that, kamailio started ok, but didn't use ssl for mysql queries. >> >> what is it that i'm missing? has anyone succeeded in making kamailio to >> query mysql server over ssl? > > based on zero responses, i guess the answer is "no". if so, that pretty > much prevents using kamailio in an environment where mysql service is > provided by a cloud service, such as amazon ec2. > > should i put a note in db_mysql module README telling that we don't > currently know, which [client] params of my.cfg the module supports? We've seen reports of issues with Postgresql with TLS too, I don't know what happened, but I think we need to focus on both and fix this. There is a known geneal problem with libraries using OpenSSL - I don't know if this has been looked at in Kamailio, but we did a fix in Asterisk a while ago. If you have modules using libraries that use OpenSSL - like we have in Curl, Mysql, Postgres and possibly other modules - as well as our own use in the TLS module - there's a risk that OpenSSL gets initialized too many times and bad things happen. ("Bad things" need to be defined here). I think Kevin did a library trick with the linker so that Asterisk catch these initialization calls first and use just one. Asterisk is multithreaded and Kamailio is multiprocess, so I don't know how this affects Kamailio or if we can get some inspiration by this fix. Rambling a bit, but trying to point in some sort of general direction. :-) I will put on my list to set up a lab with Mysql TLS connections and try.Just chiming in to point out the magic module Olle is referring to:
http://svn.asterisk.org/svn/asterisk/trunk/main/libasteriskssl.c
For context, the peer review for the patch that fixed this issue is here:
https://reviewboard.asterisk.org/r/1006/
Although due to some issues in review board, part of the patch doesn't show up (hence the link to the actual source).
Interesting approach by temporarily replacing the libssl functions and call the original ones later.
In this case it wouldn't have worked because the postgress module is used to connect to database (e.g., some modules load records from database during module initialization), so libssl must be properly initialized to the phase of opening a tls connection.
I broke the initialization process in few phases that allows to get libssl initialized before any other module and seems to work, based on the feedback of who reported the issue.
MySQL issue was something different, related more to mysql client config.
Cheers, Daniel
I haven't used the group functionality at all, saw it that Jan added very long time ago (iirc), but didn't spend time to understand what implies.
However, if you get a core dump, we can look at the backtrace and see where it crashes.
Cheers, Daniel
On 20/01/15 23:52, Juha Heinanen wrote:
with latest master, i tried to add [group] to mysql db_urls and sqlops urls, eg.
modparam("mtree", "db_url", "mysql://user:pass@[sip_proxy]127.0.0.1/sip_proxy")
and in my.cfg the group was defined like this:
[sip_proxy] ssl-ca=/etc/mysql/ca-cert.pem ssl-cert=/etc/mysql/client-cert.pem ssl-key=/etc/mysql/client-key.pem
the result was that when i start kamailio, it starts to run out of mysql connections and dumps core no matter how high i set max_connections in my.cfg:
Jan 21 00:47:33 siika /usr/bin/sip-proxy[30612]: ERROR: db_mysql [km_my_con.c:114]: db_mysql_new_connection(): driver error: Too many connections Jan 21 00:47:33 siika /usr/bin/sip-proxy[30612]: ERROR: <core> [db.c:314]: db_do_init2(): could not add connection to the pool Jan 21 00:47:33 siika /usr/bin/sip-proxy[30612]: ERROR: mtree [mtree_mod.c:346]: mi_child_init(): failed to connect to database Jan 21 00:47:33 siika /usr/bin/sip-proxy[30612]: ERROR: <core> [sr_module.c:900]: init_mod_child(): Error while initializing module mtree (/usr/lib/sip-proxy/modules/mtree.so)
without the [group] kamailio starts fine and does not complain about too many connections.
is this a bug in group handling or something else?
-- juha
sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
Daniel-Constantin Mierla writes:
I haven't used the group functionality at all, saw it that Jan added very long time ago (iirc), but didn't spend time to understand what implies.
However, if you get a core dump, we can look at the backtrace and see where it crashes.
i don't think the crash is relevant here, because before the crash i got error messages about running out of connections.
-- juha
-
Daniel-Constantin Mierla -
Juha Heinanen -
Matthew Jordan -
Olle E. Johansson