https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi guys, wondering if anyone has run into this whi...
# fleet
m
Hi guys, wondering if anyone has run into this while configuring tls for mysql.
Copy code
Feb 15 21:50:44 <ec2> systemd[1]: Started Kolide Fleet.
Feb 15 21:50:44 <ec2> fleet[10720]: Using config file:  /etc/kolide/fleet.yml
Feb 15 21:50:44 <ec2> fleet[10720]: Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'true' is reserved
Feb 15 21:50:44 <ec2?Feb 15 21:50:44 osquery-service-><http://ab251.ec2.vzbuilders.com|ab251.ec2.vzbuilders.com> systemd[1]: Unit fleet.service entered failed state.
Feb 15 21:50:44 <ec2> systemd[1]: fleet.service failed.
It looks like it hates the key true, however from the documentation:
Copy code
mysql_tls_config
The tls value in a MYSQL DSN. Can be true,false,skip-verify or the CN value of the certificate.
Default value: none
Environment variable: FLEET_MYSQL_TLS_CONFIG
Config file format:
 mysql:
 	tls_config: true
z
Try 
"true"
. It's looking for a string. I think the docs could make this more clear.
m
Ohhhhhh
🙂 will try that
z
Could be nice to change this to `on`/`off` to avoid this situation.
m
Here’s what I have
Copy code
tls_config: "true"
still getting
Copy code
Feb 15 22:12:06 <ip> fleet: Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'true' is reserved
j
same error for
Copy code
"TRUE"
and
Copy code
'TRUE'
z
Hmmm
j
same results for any of 
true
,
false
,
skip-verify
z
Even 
skip-verify
? Can you paste in a snippet from the config?
Ideally the whole config with any secrets removed
m
yup
j
Copy code
mysql:
  address:    [REDACTED].<http://us-east-1.rds.amazonaws.com:3306|us-east-1.rds.amazonaws.com:3306>
  database:   osq
  username:   osq
  password:   [REDACTED]
  tls_config: 'TRUE'
  tls_ca:     /etc/kolide/rds-combined-ca-bundle.pem
  tls_cert: [REDACTED].cert.pem
  tls_key:  [REDACTED].key.pem
  servername: [REDACTED].<http://us-east-1.rds.amazonaws.com|us-east-1.rds.amazonaws.com>
redis:
  address: 127.0.0.1:6379
  password: [REDACTED]
server:
  tls: false
logging:
  json: true
auth:
  jwt_key: [REDACTED]
session:
  duration: 12h
osquery:
  status_log_plugin: filesystem
  result_log_plugin: kinesis
filesystem:
  status_log_file: /var/log/osquery/status.log
kinesis:
  region: us-east-1
  sts_assume_role_arn: arn:aws:iam::734560425545:role/[REDACTED]
  result_stream: osquery-kinesis-stream-us-east-1
z
How old a version of Fleet are you running? This config seems to be accepted by the current version as long as all of the 
[REDACTED]
have quotes around them (
'[REDACTED]...'
)
j
config worked fine without TLS
z
At some point we replaced the yaml parsing library but I think that was a very long time ago.
j
without using any quoters
we're on 3.7.1
which AFAIK is the latest, since the updates after were only for fleetctl
z
Correct
Is it possible you need quotes around the redacted portion of your mysql password?
j
the error seems pretty specifically about TLS
Copy code
Error initializing datastore: register TLS config for mysql: register mysql tls config: key 'skip-verify' is reserved
z
Ah sorry yes you are correct
Okay yeah the error is coming from Go's mysql driver.
We upgraded the mysql driver recently... I'm trying to see if this was caused by that.
m
Interesting, thanks
z
So you are trying to configure mTLS here, correct?
Can you try setting this to 
custom
instead of true?
I'm digging through this and it seems the 
tls_config
option may have been misdocumented going all the way back to the initial implementation in 2017. Would be interested to confirm.
m
will do
That was it
Thanks much, we were pounding our heads against it all afternoon 🙂
we’re still running into oddness but it’s likely something on our side
we’ll keep poking at it, have a great evening
z
Glad it worked! I'm going to update the documentation.
j
spoke too soon, using "custom" the error becomes
Copy code
Error initializing datastore: remote error: tls: unknown certificate authority
it returns this error for any value we've tried that isn't "true/false/skip-verify"
we are able to verify a manual mysql connection is successful
Copy code
mysql -vv -h [REDACTED].<http://us-east-1.rds.amazonaws.com|us-east-1.rds.amazonaws.com> --port=3306 --database=osq --user=osq --password=[REDACTED] --ssl-ca=/etc/kolide/rds-combined-ca-bundle.pem --ssl-mode=VERIFY_IDENTITY
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 173
Server version: 5.7.12 MySQL Community Server (GPL)

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Reading history-file /root/.mysql_history
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> quit
Writing history-file /root/.mysql_history
Bye
z
In your Fleet config you seem to be configuring client certificate auth, however this command seems not to be doing so. Is that intended?
j
fleet won't start without the client cert/key
Copy code
Error initializing datastore: register TLS config for mysql: load mysql client cert and key: open : no such file or directory
the CA should be the only thing required, but since Fleet wouldn't start without the client cert/key, I included them
the manual mysql command is from the AWS documentation example of connecting to RDS using TLS
I wanted to confirm we were able to establish a successful connection to the DB via TLS
z
Will it connect over TLS if you leave all of the tls related fields blank?
j
no
Copy code
{
  "mysql": "could not connect to db: Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON., sleeping 0s",
  "ts": "2021-02-16T17:38:43.182354467Z"
}
z
I'll have to dig into this more later. Hoping someone else in the channel knows the proper way to do it currently. Asked in https://osquery.slack.com/archives/C01DXJL16D8/p1613497405140000
13 Views
Open in Slack
PreviousNext
Powered by