Brandon
03/25/2021, 12:18 AMsave enroll failed: host identified by 1234123-1234-1234-1234-C3C04F373533 enrolling too often
Also, seeing
authentication error: missing node key
and
enroll failed: no matching secret found
and finally
failed to mark host seen: marking host seen: Error 1205: Lock wait timeout exceeded; try restarting transaction
zwass
--host_identifier=instance
in your osquery flagfile, or in Fleet 3.9.0 you can configure it within Fleet itself: https://github.com/fleetdm/fleet/blob/master/docs/2-Deployment/2-Configuration.md#osquery_host_identifierBrandon
03/25/2021, 12:46 AMScott Lampert
03/25/2021, 3:26 PMhost_identifier=instance
doesn’t help because the hosts already have the duplicated id stored in their osquery backing store and won’t regenerate a new one. Only new hosts that pick up that config change will have newly generated id’s.Brandon
03/25/2021, 3:52 PMzwass
host_identifier=instance
from the osquery options within Fleet?Scott Lampert
03/29/2021, 5:32 PMinstance
enabled in the flags or use ephemeral
in the flags.
The issue on the fleet side is that if you have a bunch of nodes trying to enroll with the same id already you really need to use the cooldown or the database will get thousand of lock contentions and fall over (we have 120,000+ nodes checking into fleet). If a large portion of those nodes are stuck with a non-unique id they never get to enroll since the rate of nodes trying to enroll will always trigger the cooldown. This means you can’t really count on any osquery config changes in fleet to be picked up related to uuid. This might not be an issue until a certain scale.zwass
instance_identifier
should be generated separately for any installation, regardless of existence/value of UUID.Scott Lampert
03/29/2021, 5:43 PMhost_identifier
settings on a few thousand hosts exhibiting the above behavior. We found that even ssh’ing into the host and re-running with --host_identifier=instance
fleet would still see the original duplicate hw uuid regardless of that setting. If we set it to ephemperal
it would also work correctly. If we deleted the backing store and restarted it with instance
it also would show up correctly.instance
would not seem to generate a new uuid in the osquery info.zwass
instance_id
is the column Fleet would use if you configure https://github.com/fleetdm/fleet/blob/master/docs/2-Deployment/2-Configuration.md#osquery_host_identifier. That should be unique per osquery database and if it's not that's a bug (please file an issue).Scott Lampert
03/29/2021, 6:12 PMinstance
. instance
stores the id in the backing store once it’s generated. Otherwise you would want ephemeral.instance uses an instance-unique UUID generated at process start, persisted in the backing store.
instance
will not generate a new uuidinstance
before osquery makes its db the first time.