Brandon03/25/2021, 12:18 AM
save enroll failed: host identified by 1234123-1234-1234-1234-C3C04F373533 enrolling too often
authentication error: missing node key
enroll failed: no matching secret found
failed to mark host seen: marking host seen: Error 1205: Lock wait timeout exceeded; try restarting transaction
in your osquery flagfile, or in Fleet 3.9.0 you can configure it within Fleet itself: https://github.com/fleetdm/fleet/blob/master/docs/2-Deployment/2-Configuration.md#osquery_host_identifier
Brandon03/25/2021, 12:46 AM
Scott Lampert03/25/2021, 3:26 PM
doesn’t help because the hosts already have the duplicated id stored in their osquery backing store and won’t regenerate a new one. Only new hosts that pick up that config change will have newly generated id’s.
Brandon03/25/2021, 3:52 PM
from the osquery options within Fleet?
Scott Lampert03/29/2021, 5:32 PM
enabled in the flags or use
in the flags.
The issue on the fleet side is that if you have a bunch of nodes trying to enroll with the same id already you really need to use the cooldown or the database will get thousand of lock contentions and fall over (we have 120,000+ nodes checking into fleet). If a large portion of those nodes are stuck with a non-unique id they never get to enroll since the rate of nodes trying to enroll will always trigger the cooldown. This means you can’t really count on any osquery config changes in fleet to be picked up related to uuid. This might not be an issue until a certain scale.
should be generated separately for any installation, regardless of existence/value of UUID.
Scott Lampert03/29/2021, 5:43 PM
settings on a few thousand hosts exhibiting the above behavior. We found that even ssh’ing into the host and re-running with
fleet would still see the original duplicate hw uuid regardless of that setting. If we set it to
it would also work correctly. If we deleted the backing store and restarted it with
it also would show up correctly.
would not seem to generate a new uuid in the osquery info.
is the column Fleet would use if you configure https://github.com/fleetdm/fleet/blob/master/docs/2-Deployment/2-Configuration.md#osquery_host_identifier. That should be unique per osquery database and if it's not that's a bug (please file an issue).
Scott Lampert03/29/2021, 6:12 PM
stores the id in the backing store once it’s generated. Otherwise you would want ephemeral.
instance uses an instance-unique UUID generated at process start, persisted in the backing store.
will not generate a new uuid
before osquery makes its db the first time.