Out of curiosity, is there a reason why Fleet does...
# fleet
a
Out of curiosity, is there a reason why Fleet doesn’t log the hostname and/or IP address to the service logs when an
osqueryd
client fails to enroll with the supplied enroll secret, despite the hostname being available in the
system_info
table that
osqueryd
sends with the enrollment request? For example, reviewing the logs for Fleet shows we have a few systems failing to enroll with a log line similar to:
Copy code
component=http user=unauthenticated method=POST uri=/api/v1/osquery/enroll hostIdentifier=[uuid-omitted] err="enroll failed: no matching secret found"
I’m putting the rest of this in the thread, since it’s slightly long. (🧵)
Obviously, trying to track down the host which generated the error is ... an issue, since aside from knowing which of our Fleet instances this error is from and re-pushing configuration to all hosts that should be connecting to it with updated configurations, the lack of a hostname and/or IP that attempted connecting makes figuring out which hosts failed to connect in a live environment... challenging, to say the least. Looking at the code for v4.36.0 (as a reference to tie this into the code): a) The
EnrollAgent
function in
/server/service/osquery.go
verifies the received enrollment secret using
svc.ds.VerifyEnrollSecret
b)
VerifyEnrollSecret
in
/server/datastore/mysql/app_configs.go
then issues a SQL
SELECT
to see if the passed in string matches a record in the
enroll_secrets
table. c) If it does not, an error is returned, passing back the string
no matching secret found
. d) Back in EnrollAgent, it appears as if the string I am seeing in my logs is built using
return "", newOsqueryErrorWithInvalidNode("enroll failed: " + err.Error())
on L129. I’m speculating that this sends an error back to the
osquery
node, but I don’t know from here how it makes its way into the syslog. Since I am seeing that
osquery
sends the
system_info
with the request, and it is used within the same function, would it not be possible for Fleet to include the hostname--or IP address, or both--so as to allow an administrator to pick out which specific hosts are failing to enroll? For instance, this could be done through shifting some of the existing code around, similar to the following:
Copy code
// the the device's uuid and serial from the system_info table provided with
// the osquery enrollment
var hardwareUUID, hardwareSerial string
if r, ok := hostDetails["system_info"]; ok {
	hostname = r["hostname"]
	hardwareUUID = r["uuid"]
	hardwareSerial = r["hardware_serial"]
}

secret, err := svc.ds.VerifyEnrollSecret(ctx, enrollSecret)
if err != nil {
	return "", newOsqueryErrorWithInvalidNode(fmt.Sprintf("enroll failed [%s]: %s", hostname, err.Error())
}
I’m not quite well versed in Golang, so I’m not (currently) able to trace the full path of how a request would enter via the routes to see where the IP address is readable from (or if there’s an
http.Request
object available in
/server/service/osquery.go
with which
extractIP()
from
/server/service/http_publicip.go
could be called from to extract the IP. But if there is/was a way to do it, the return line might instead read like (pseudocode, since I doubt it would work as-is):
Copy code
return "", newOsqueryErrorWithInvalidNode(fmt.Sprintf("enroll failed [%s @ %s]: %s", hostname, extractIP(request), err.Error())
This latter option would provide a Fleet instance administrator enough details in the service logs to track down the system(s) with an invalid enrollment secret so that the correct value can be pushed.
k
Hey @Alex Sullivan! This is a great feature request. I can bring this to the team to see if we can prioritize it for development. We've got some big initiatives going for new/improved features, so it may be a bit before we'd be able to implement the change.
a
Even if it takes a bit of time to add it in I’d be appreciative regardless, as having that information in the enrollment failure message would be super helpful in tracking down which of the nodes are failing to enroll. Thanks!
k
Makes sense to me!