https://github.com/osquery/osquery logo
Powered by
#fleet
Title
# fleet
m

Mark Noonan

02/15/2021, 10:05 PM
Hi guys, wondering if anyone has run into this while configuring tls for mysql.
Copy code
Feb 15 21:50:44 <ec2> systemd[1]: Started Kolide Fleet.
Feb 15 21:50:44 <ec2> fleet[10720]: Using config file:  /etc/kolide/fleet.yml
Feb 15 21:50:44 <ec2> fleet[10720]: Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'true' is reserved
Feb 15 21:50:44 <ec2?Feb 15 21:50:44 osquery-service-><http://ab251.ec2.vzbuilders.com|ab251.ec2.vzbuilders.com> systemd[1]: Unit fleet.service entered failed state.
Feb 15 21:50:44 <ec2> systemd[1]: fleet.service failed.
It looks like it hates the key true, however from the documentation:
Copy code
mysql_tls_config
The tls value in a MYSQL DSN. Can be true,false,skip-verify or the CN value of the certificate.
Default value: none
Environment variable: FLEET_MYSQL_TLS_CONFIG
Config file format:
 mysql:
 	tls_config: true
z

zwass

02/15/2021, 10:11 PM
Try 
"true"
. It's looking for a string. I think the docs could make this more clear.
m

Mark Noonan

02/15/2021, 10:11 PM
Ohhhhhh
🙂 will try that
z

zwass

02/15/2021, 10:12 PM
Could be nice to change this to `on`/`off` to avoid this situation.
m

Mark Noonan

02/15/2021, 10:13 PM
Here’s what I have
Copy code
tls_config: "true"
still getting
Copy code
Feb 15 22:12:06 <ip> fleet: Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'true' is reserved
j

Jocelyn Bothe

02/15/2021, 10:14 PM
same error for
Copy code
"TRUE"
and
Copy code
'TRUE'
z

zwass

02/15/2021, 10:15 PM
Hmmm
j

Jocelyn Bothe

02/15/2021, 10:15 PM
same results for any of 
true
,
false
,
skip-verify
z

zwass

02/15/2021, 10:16 PM
Even 
skip-verify
? Can you paste in a snippet from the config?
Ideally the whole config with any secrets removed
m

Mark Noonan

02/15/2021, 10:16 PM
yup
j

Jocelyn Bothe

02/15/2021, 10:18 PM
Copy code
mysql:
  address:    [REDACTED].<http://us-east-1.rds.amazonaws.com:3306|us-east-1.rds.amazonaws.com:3306>
  database:   osq
  username:   osq
  password:   [REDACTED]
  tls_config: 'TRUE'
  tls_ca:     /etc/kolide/rds-combined-ca-bundle.pem
  tls_cert: [REDACTED].cert.pem
  tls_key:  [REDACTED].key.pem
  servername: [REDACTED].<http://us-east-1.rds.amazonaws.com|us-east-1.rds.amazonaws.com>
redis:
  address: 127.0.0.1:6379
  password: [REDACTED]
server:
  tls: false
logging:
  json: true
auth:
  jwt_key: [REDACTED]
session:
  duration: 12h
osquery:
  status_log_plugin: filesystem
  result_log_plugin: kinesis
filesystem:
  status_log_file: /var/log/osquery/status.log
kinesis:
  region: us-east-1
  sts_assume_role_arn: arn:aws:iam::734560425545:role/[REDACTED]
  result_stream: osquery-kinesis-stream-us-east-1
z

zwass

02/15/2021, 10:22 PM
How old a version of Fleet are you running? This config seems to be accepted by the current version as long as all of the 
[REDACTED]
have quotes around them (
'[REDACTED]...'
)
j

Jocelyn Bothe

02/15/2021, 10:22 PM
config worked fine without TLS
z

zwass

02/15/2021, 10:23 PM
At some point we replaced the yaml parsing library but I think that was a very long time ago.
j

Jocelyn Bothe

02/15/2021, 10:23 PM
without using any quoters
we're on 3.7.1
which AFAIK is the latest, since the updates after were only for fleetctl
z

zwass

02/15/2021, 10:24 PM
Correct
Is it possible you need quotes around the redacted portion of your mysql password?
j

Jocelyn Bothe

02/15/2021, 10:25 PM
the error seems pretty specifically about TLS
Copy code
Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'skip-verify' is reserved
z

zwass

02/15/2021, 10:26 PM
Ah sorry yes you are correct
Okay yeah the error is coming from Go's mysql driver.
We upgraded the mysql driver recently... I'm trying to see if this was caused by that.
m

Mark Noonan

02/15/2021, 10:39 PM
Interesting, thanks
z

zwass

02/15/2021, 10:44 PM
So you are trying to configure mTLS here, correct?
Can you try setting this to 
custom
instead of true?
I'm digging through this and it seems the 
tls_config
option may have been misdocumented going all the way back to the initial implementation in 2017. Would be interested to confirm.
m

Mark Noonan

02/15/2021, 10:57 PM
will do
That was it
Thanks much, we were pounding our heads against it all afternoon 🙂
we’re still running into oddness but it’s likely something on our side
we’ll keep poking at it, have a great evening
z

zwass

02/16/2021, 1:15 AM
Glad it worked! I'm going to update the documentation.
j

Jocelyn Bothe

02/16/2021, 3:28 PM
spoke too soon, using "custom" the error becomes
Copy code
Error initializing datastore: remote error: tls: unknown certificate authority
it returns this error for any value we've tried that isn't "true/false/skip-verify"
we are able to verify a manual mysql connection is successful
Copy code
mysql -vv -h [REDACTED].<http://us-east-1.rds.amazonaws.com|us-east-1.rds.amazonaws.com> --port=3306 --database=osq --user=osq --password=[REDACTED] --ssl-ca=/etc/kolide/rds-combined-ca-bundle.pem --ssl-mode=VERIFY_IDENTITY
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 173
Server version: 5.7.12 MySQL Community Server (GPL)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Reading history-file /root/.mysql_history
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> quit
Writing history-file /root/.mysql_history
Bye
z

zwass

02/16/2021, 4:34 PM
In your Fleet config you seem to be configuring client certificate auth, however this command seems not to be doing so. Is that intended?
j

Jocelyn Bothe

02/16/2021, 5:03 PM
fleet won't start without the client cert/key
Copy code
Error initializing datastore: register TLS config for mysql: load mysql client cert and key: open : no such file or directory
the CA should be the only thing required, but since Fleet wouldn't start without the client cert/key, I included them
the manual mysql command is from the AWS documentation example of connecting to RDS using TLS
I wanted to confirm we were able to establish a successful connection to the DB via TLS
z

zwass

02/16/2021, 5:37 PM
Will it connect over TLS if you leave all of the tls related fields blank?
j

Jocelyn Bothe

02/16/2021, 5:37 PM
no
Copy code
{
  "mysql": "could not connect to db: Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON., sleeping 0s",
  "ts": "2021-02-16T17:38:43.182354467Z"
}
z

zwass

02/16/2021, 5:44 PM
I'll have to dig into this more later. Hoping someone else in the channel knows the proper way to do it currently. Asked in https://osquery.slack.com/archives/C01DXJL16D8/p1613497405140000
11 Views
Powered by