Hey everyone, still struggling to troubleshoot why osquery is not using the os certs on Windows. Quick question, if we are terminating https at the load balancer, does the fleet server need the certs?

Hi Chad, it shouldn't be needed. Are you seeing any failures on the fleet side?

@Chad I'd guess because osquery is cross platform and it would require investment in code time to write code just for the windows cert store, probably time not well spent when you can just ship the CA cert bundled in with the cert, at same time as shipping osquery and this maintains a consistent experience across all platforms (including the documentation, no need to write doco for just the windows cert store etc).

@Tomas Touceda - I haven't looked yet as I am assuming the packets are not even getting that far (but will confirm in a bit). I am getting

Request error: certificate verify failed

error. Things I have tried: 1. I have tried both AWS ACM certs and LetsEncrypt certs 2. I have tried without specifying

--tls_server_certs

as well as providing the entire LE cert chain 3. I have confirmed that the certificate is valid and accessible with

curl

4. I have confirmed the CName matches the DNS name 5. I have confirmed there are not network issues as orbit with

--insecure

flag works I am using the osqueryd packaged with orbit, is this modified in anyway from the original binary?

@Chad if it helps, I have fleet using the same cert as osquery, I think it's required for auth (Fleet checks that the cert presented by osqueryd is issued by the same CA, this is a guess, but seems like that would be a thing). My front end is using an LE cert but I split traffic between front end and osquery so accessing Fleet WUI is using LE cert and osqueryd to fleet is using the self signed cert.

@Chad can you try

curl -vv --cacert fleet.pem <https://localhost:8080>

(replacing the cert path and URL as appropriate)?

@zwass Can confirm the cert matches:

curl -vv --cacert AWSCA.pem <https://devices.mydomain.com>
*   Trying xx.xx.xx.xx:443...
* TCP_NODELAY set
* Connected to <http://devices.mydomain.com|devices.mydomain.com> (xx.xx.xx.xx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: AWSCA.pem
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=<http://devices.mydomain.com|devices.mydomain.com>
*  start date: Aug 26 00:00:00 2021 GMT
*  expire date: Sep 24 23:59:59 2022 GMT
*  subjectAltName: host "<http://devices.mydomain.com|devices.mydomain.com>" matched cert's "<http://devices.mydomain.com|devices.mydomain.com>"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55fb02dd9860)

Managed to troubleshoot that osquery isn't using the computer cert store at all (for both mac and windows). So manually pointing osquery to the devices cert store resolved the issue. Not sure if this is just an issue on my end, but if not it would be helpful to update the docs to clarify this (in both osquery and orbit). Created a PR to add

--fleet-certificate

flag for the windows orbit agent and fixed a bug in building agents on linux.