Title
#fleet
m

Mark Noonan

02/15/2021, 10:05 PM
Hi guys, wondering if anyone has run into this while configuring tls for mysql.
Feb 15 21:50:44 <ec2> systemd[1]: Started Kolide Fleet.
Feb 15 21:50:44 <ec2> fleet[10720]: Using config file:  /etc/kolide/fleet.yml
Feb 15 21:50:44 <ec2> fleet[10720]: Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'true' is reserved
Feb 15 21:50:44 <ec2?Feb 15 21:50:44 osquery-service-><http://ab251.ec2.vzbuilders.com|ab251.ec2.vzbuilders.com> systemd[1]: Unit fleet.service entered failed state.
Feb 15 21:50:44 <ec2> systemd[1]: fleet.service failed.
It looks like it hates the key true, however from the documentation:
mysql_tls_config
The tls value in a MYSQL DSN. Can be true,false,skip-verify or the CN value of the certificate.
Default value: none
Environment variable: FLEET_MYSQL_TLS_CONFIG
Config file format:
 mysql:
 	tls_config: true
zwass

zwass

02/15/2021, 10:11 PM
Try
"true"
. It's looking for a string. I think the docs could make this more clear.
m

Mark Noonan

02/15/2021, 10:11 PM
Ohhhhhh
10:11 PM
🙂 will try that
zwass

zwass

02/15/2021, 10:12 PM
Could be nice to change this to on/off to avoid this situation.
m

Mark Noonan

02/15/2021, 10:13 PM
Here’s what I have
tls_config: "true"
10:14 PM
still getting
Feb 15 22:12:06 <ip> fleet: Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'true' is reserved
Jocelyn Bothe

Jocelyn Bothe

02/15/2021, 10:14 PM
same error for
"TRUE"
10:14 PM
and
'TRUE'
zwass

zwass

02/15/2021, 10:15 PM
Hmmm
Jocelyn Bothe

Jocelyn Bothe

02/15/2021, 10:15 PM
same results for any of
true
,
false
,
skip-verify
zwass

zwass

02/15/2021, 10:16 PM
Even
skip-verify
? Can you paste in a snippet from the config?
10:16 PM
Ideally the whole config with any secrets removed
m

Mark Noonan

02/15/2021, 10:16 PM
yup
Jocelyn Bothe

Jocelyn Bothe

02/15/2021, 10:18 PM
mysql:
  address:    [REDACTED].<http://us-east-1.rds.amazonaws.com:3306|us-east-1.rds.amazonaws.com:3306>
  database:   osq
  username:   osq
  password:   [REDACTED]
  tls_config: 'TRUE'
  tls_ca:     /etc/kolide/rds-combined-ca-bundle.pem
  tls_cert: [REDACTED].cert.pem
  tls_key:  [REDACTED].key.pem
  servername: [REDACTED].<http://us-east-1.rds.amazonaws.com|us-east-1.rds.amazonaws.com>
redis:
  address: 127.0.0.1:6379
  password: [REDACTED]
server:
  tls: false
logging:
  json: true
auth:
  jwt_key: [REDACTED]
session:
  duration: 12h
osquery:
  status_log_plugin: filesystem
  result_log_plugin: kinesis
filesystem:
  status_log_file: /var/log/osquery/status.log
kinesis:
  region: us-east-1
  sts_assume_role_arn: arn:aws:iam::734560425545:role/[REDACTED]
  result_stream: osquery-kinesis-stream-us-east-1
zwass

zwass

02/15/2021, 10:22 PM
How old a version of Fleet are you running? This config seems to be accepted by the current version as long as all of the
[REDACTED]
have quotes around them (
'[REDACTED]...'
)
Jocelyn Bothe

Jocelyn Bothe

02/15/2021, 10:22 PM
config worked fine without TLS
zwass

zwass

02/15/2021, 10:23 PM
At some point we replaced the yaml parsing library but I think that was a very long time ago.
Jocelyn Bothe

Jocelyn Bothe

02/15/2021, 10:23 PM
without using any quoters
10:23 PM
we're on 3.7.1
10:23 PM
which AFAIK is the latest, since the updates after were only for fleetctl
zwass

zwass

02/15/2021, 10:24 PM
Correct
10:24 PM
Is it possible you need quotes around the redacted portion of your mysql password?
Jocelyn Bothe

Jocelyn Bothe

02/15/2021, 10:25 PM
the error seems pretty specifically about TLS
10:25 PM
Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'skip-verify' is reserved
zwass

zwass

02/15/2021, 10:26 PM
Ah sorry yes you are correct
10:28 PM
Okay yeah the error is coming from Go's mysql driver.
10:39 PM
We upgraded the mysql driver recently... I'm trying to see if this was caused by that.
m

Mark Noonan

02/15/2021, 10:39 PM
Interesting, thanks
zwass

zwass

02/15/2021, 10:44 PM
So you are trying to configure mTLS here, correct?
10:45 PM
Can you try setting this to
custom
instead of true?
10:52 PM
I'm digging through this and it seems the
tls_config
option may have been misdocumented going all the way back to the initial implementation in 2017. Would be interested to confirm.
m

Mark Noonan

02/15/2021, 10:57 PM
will do
10:57 PM
That was it
10:58 PM
Thanks much, we were pounding our heads against it all afternoon 🙂
10:59 PM
we’re still running into oddness but it’s likely something on our side
10:59 PM
we’ll keep poking at it, have a great evening
zwass

zwass

02/16/2021, 1:15 AM
Glad it worked! I'm going to update the documentation.
Jocelyn Bothe

Jocelyn Bothe

02/16/2021, 3:28 PM
spoke too soon, using "custom" the error becomes
Error initializing datastore: remote error: tls: unknown certificate authority
3:28 PM
it returns this error for any value we've tried that isn't "true/false/skip-verify"
4:26 PM
we are able to verify a manual mysql connection is successful
mysql -vv -h [REDACTED].<http://us-east-1.rds.amazonaws.com|us-east-1.rds.amazonaws.com> --port=3306 --database=osq --user=osq --password=[REDACTED] --ssl-ca=/etc/kolide/rds-combined-ca-bundle.pem --ssl-mode=VERIFY_IDENTITY
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 173
Server version: 5.7.12 MySQL Community Server (GPL)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Reading history-file /root/.mysql_history
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> quit
Writing history-file /root/.mysql_history
Bye
zwass

zwass

02/16/2021, 4:34 PM
In your Fleet config you seem to be configuring client certificate auth, however this command seems not to be doing so. Is that intended?
Jocelyn Bothe

Jocelyn Bothe

02/16/2021, 5:03 PM
fleet won't start without the client cert/key
5:05 PM
Error initializing datastore: register TLS config for mysql: load mysql client cert and key: open : no such file or directory
5:06 PM
the CA should be the only thing required, but since Fleet wouldn't start without the client cert/key, I included them
5:09 PM
the manual mysql command is from the AWS documentation example of connecting to RDS using TLS
5:10 PM
I wanted to confirm we were able to establish a successful connection to the DB via TLS
zwass

zwass

02/16/2021, 5:37 PM
Will it connect over TLS if you leave all of the tls related fields blank?
Jocelyn Bothe

Jocelyn Bothe

02/16/2021, 5:37 PM
no
5:38 PM
{
  "mysql": "could not connect to db: Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON., sleeping 0s",
  "ts": "2021-02-16T17:38:43.182354467Z"
}
zwass

zwass

02/16/2021, 5:44 PM
I'll have to dig into this more later. Hoping someone else in the channel knows the proper way to do it currently. Asked in https://osquery.slack.com/archives/C01DXJL16D8/p1613497405140000