Hi all does anyone have any idea how to track down what s ca

Question

Hi all does anyone have any idea how to track down what s causing a ghost device to show up in our hosts Even if it s deleted it seems to come back periodically There s no info associated with it that I can find and nothing obvious in the Fleet logs Thanks for any tips anyone can provide

Kathy Satterlee · Answer

Hi < brian> Are you using Fleet generated packages or plain osquery to enroll your hosts

brian · Answer

we re using Fleet generated packages

Kathy Satterlee · Answer

Are you specifying a host identifier in your agent options It sounds like this may be related to <https github com fleetdm fleet issues 9033>

brian · Answer

hmm we re not doing anything to specify a host identifier

Kathy Satterlee · Answer

What about in the Fleet configuration <https fleetdm com docs deploying configuration osquery host identifier>

brian · Answer

nope nothing there either for host identifier

Kathy Satterlee · Answer

Do you have access to the Fleet database I d like to see if anything comes back from this query ```SELECT FROM hosts WHERE orbit node key = NULL ```

brian · Answer

I could probably set that up tomorrow Fleet is running on ECS so I d have to launch a temporary EC2 instance to get to the RDS instance where the database is hosted

Kathy Satterlee · Answer

Sounds good I m betting that it s a similar situation and there s a host that didn t quite enroll properly and Orbit keeps trying to re enroll If we can track down what host that is next step will be to check out the Orbit <https fleetdm com docs using fleet orbit logs|logs> on the host to verify get additional information on what s happening Then likely remove both the fragment and the real host and see what happens

brian · Answer

+1 thanks for the help I ll let you know what I find tomorrow

brian · Answer

I think I found the troublesome host it looks like it has two entries in the hosts table one of them with a null orbit node key ```| 56 | 51b2b9d6 e102 49c3 9fd7 2f8cac64fb3d | 2023 01 30 19 33 49 | JJFxbCFU93oRY94xyzhz2l11WkIZ1tUI | localhost | 51b2b9d6 e102 49c3 9fd7 2f8cac64fb3d | 2023 01 30 19 33 49 | NULL | | 94 | 6c76c81a cb0d 4288 8201 06bc558dda5a | 2023 02 01 16 00 56 | IYKQaZsExKnuFXNv0rdBSUgEKyfktEi6 | | | 1970 01 02 00 00 00 | IYKQaZsExKnuFXNv0rdBSUgEKyfktEi6 |```

brian · Answer

brian · Answer

so doing some poking around it looks like this is probably related to <https github com fleetdm fleet issues 9132> since I m also seeing empty `orbit node key` and another host with `orbit node key` equal to `node key`

Kathy Satterlee · Answer

Absolutely Sorry for slow response today Are you able to share the for that host I d also like to see what happens if you Orbit on the host remove both the artifact and the host from Fleet and then re enroll it using a package created with the ` debug` flag for more verbose logging

brian · Answer

these are some of the debug lines we re seeing ```Feb 2 16 04 41 localhost orbit 58456 2023 02 02T16 04 41 05 00 DBG no update target=orbit Feb 2 16 04 41 localhost orbit 58456 2023 02 02T16 04 41 05 00 DBG no update target=osqueryd Feb 2 16 04 42 localhost orbit 58456 2023 02 02T16 04 42 05 00 DBG found expected target locally path= opt orbit bin osqueryd linux stable osqueryd target=osqueryd Feb 2 16 04 42 localhost orbit 58456 2023 02 02T16 04 42 05 00 DBG running single query SELECT uuid FROM system info Feb 2 16 04 42 localhost orbit 58456 2023 02 02T16 04 42 05 00 DBG UUID is 6e2c24c6 f205 4b39 8bf3 28fc825c2156 Feb 2 16 04 42 localhost orbit 58456 2023 02 02T16 04 42 05 00 INF initial flags update failed error= error getting flags from fleet unauthenticated or invalid token Feb 2 16 04 42 localhost orbit 58456 2023 02 02T16 04 42 05 00 DBG opt orbit extensions load not found nothing to update Feb 2 16 04 43 localhost orbit 58456 2023 02 02T16 04 43 05 00 DBG no update target=orbit Feb 2 16 04 43 localhost orbit 58456 2023 02 02T16 04 43 05 00 DBG no update target=osqueryd Feb 2 16 04 43 localhost orbit 58456 2023 02 02T16 04 43 05 00 INF token rotation is enabled Feb 2 16 04 43 localhost orbit 58456 2023 02 02T16 04 43 05 00 DBG starting flag updater Feb 2 16 04 43 localhost orbit 58456 2023 02 02T16 04 43 05 00 DBG starting extension runner Feb 2 16 04 43 localhost orbit 58456 2023 02 02T16 04 43 05 00 DBG start updater Feb 2 16 04 43 localhost orbit 58456 2023 02 02T16 04 43 05 00 DBG start osquery extension```

brian · Answer

that UUID `6e2c24c6 f205 4b39 8bf3 28fc825c2156` corresponds with the empty host s `osquery host id` value that ends up in the db

brian · Answer

that s after uninstalling and then reinstalling

brian · Answer

this is from the person whose laptop is getting registered twice So it appears that every time orbit is restarted this UUID is regenerated ```Feb 2 17 25 23 localhost orbit 74253 2023 02 02T17 25 23 05 00 DBG running single query SELECT uuid FROM system info Feb 2 17 25 23 localhost orbit 74253 2023 02 02T17 25 23 05 00 DBG UUID is 818ec869 0fc6 4522 8aa7 216e0f16f2b5``` If the unknown host is already registered in our Fleet instance then nothing happens If it s not registered then it gets registered

Ryan W. · Answer

This issue is also showing up in our environment and running the same set of SQL queries shows similar results

Ryan W. · Answer

I should mention we get this for about a dozen devices in an environment of maybe 75 total

Kathy Satterlee · Answer

Thanks for all of the information Can you both verify your Fleet version for me

Kathy Satterlee · Answer

If you are also on GitHub and can share your username feel free to DM I d like to tag you in the issue as well

Kathy Satterlee · Answer

You can also follow here <https github com fleetdm fleet issues 9132>

brian · Answer

Fleet 4 26 0 for us and osquery 5 7 0 it looks like GitHub username is `briandefiant`

Ryan W. · Answer

fleet 4 26 0 osquery 5 7 0

Ryan W. · Answer

github username is weetabix

Ryan W. · Answer

I can further report this did not happen in our first hasty test instance which used some of the same machines There isnt a decent enough overlap between the two groups though for me to suspect a left over config or something since it couldnt exist on at least a third of the suspect machine4s

Lucas Rodriguez · Answer

Hi folks If any of you have access to the hosts that are having this issue please take a look at <https github com fleetdm fleet issues 9132 issuecomment 1418943150>