Hi all - per my discussion with <@U03CZEZMK4J>, I'...
# fleet
Hi all - per my discussion with @roberto, I'm moving this convo back into the fleet channel for visibility. The issue is related to the error "Request error: certificate verify failed" - I'll put the details in the replies so it doesn't have a wall of text in here. 🙂
Thanks @Mike S. !
So the main issue is that when I attempt to enroll a client to the Fleet server, I get "Request error: certificate verify failed". Using the --insecure flag works when enrolling, but for production we wouldn't want to use that method of enrollment. Originally I was attempting to join a MacOS system to Fleet, and for an unknown reason it just started working after several weeks of this error occurring. The only change I'm aware of is an upgrade to Fleet as of a few days ago. However, when I moved onto a fresh install of Ubuntu to attempt an enrollment, the same issue occurs, so I think the upgrade can be ruled out as a resolution. These troubleshooting steps have been tried: • Verified that the certificate on the server and the certificate on the client are the same by checking the SHA256 hash. Both certs are in PEM format. • Verified that the server hostname matches the SAN in the certificate. • Ensured that the common name set in the TLS cert matches the FQDN of the server my flags file:
• Verified that the osquery.flags file on the client has the --tls_server_certs flag set and that the path to the cert is correct. • Verified that the cert itself is valid - I am able to access our web GUI without any certificate error being issued. • Downloaded certificate from server (using Advanced option under Add Hosts) and used that in the enrollment process. No change. • Attempted to verify the certificate with curl on host - output attached. • Disabled the proxy on the host to verify that there wasn't a MITM issue. No change. • Installed certificates to ca-certificates and updated the certificate list • Appended intermediate cert to entity cert and attempted to enroll using the new cert. • Prayed to Zeus to throw a lightning bolt at the people who created PKI. 😛 Let me know what other information I can provide! And I appreciate all of the amazing help provided so far!
• Verified date/time was correct on server and client.
• openssl s_client -connect <fqdn>:443 -showcerts - No errors found in the connection attempt. Root, intermediate and entity cert found in connection.
One question - Does it matter how the cert is generated in the enrollment process? I got this note from my cert admin when discussing potentially regenerating the certificate: I was incorrect, Digicert stores the
server platform
with the certificate request, so I can't just re-email w/ a new platform bundle, but I can duplicate the cert to rebundle. Attached all the platform options available to digicert. (the core cert info will be the same, just the file formats and location of various pieces of cert info will move around). Technically the same thing as running the existing cert through openssl Here's the platform info he's referring to:
So after writing all this, it looks like the Fleet package is working properly, so that's awesome! Hopefully the troubleshooting steps I put up there will be helpful for others! Part of the reason the current Ubuntu join isn't working is that I'm attempting to join the host in a different way than using the Fleet package. I'm attempting to test the use case of the user already having osquery installed, and I'm using osqueryd to perform the join. So I'm assuming there's an issue with the flags I am using. Sorry for all the updates, I'll try to quiet down now. 🙂
Thanks for all of the detailed information, This will absolutely help someone else down the road!! You're more than welcome to share your flagfile for the plain osquery setup if you'd like a hand troubleshooting that as well.
Mike this isn’t 100% relevant, but I found that the best way to handle this was to have Fleet using a ‘real’ certificate, then to delete the roots.pem and config references to the cert in the client. This makes the client use it’s own trust chain, so if your computer trusts the cert, it will ‘just work’.