Hi all does anyone have any idea how to track down what s ca osquery #fleet

Hi all, does anyone have any idea how to track dow...

brian

01/31/2023, 9:10 PM

Hi all, does anyone have any idea how to track down what’s causing a “ghost” device to show up in our hosts? Even if it’s deleted, it seems to come back periodically. There’s no info associated with it that I can find, and nothing obvious in the Fleet logs. Thanks for any tips anyone can provide.

Kathy Satterlee

01/31/2023, 9:45 PM

Hi @brian! Are you using Fleet generated packages or plain osquery to enroll your hosts?

brian

01/31/2023, 9:45 PM

we’re using Fleet-generated packages.

Kathy Satterlee

01/31/2023, 9:48 PM

Are you specifying a host identifier in your agent options? It sounds like this may be related to https://github.com/fleetdm/fleet/issues/9033

brian

01/31/2023, 9:50 PM

hmm. we’re not doing anything to specify a host identifier.

Kathy Satterlee

01/31/2023, 9:51 PM

What about in the Fleet configuration? https://fleetdm.com/docs/deploying/configuration#osquery-host-identifier

brian

01/31/2023, 9:56 PM

nope, nothing there either for host identifier.

Kathy Satterlee

01/31/2023, 10:05 PM

Do you have access to the Fleet database? I'd like to see if anything comes back from this query:

Copy code

SELECT * FROM hosts WHERE orbit_node_key = NULL;

brian

01/31/2023, 10:07 PM

I could probably set that up tomorrow (Fleet is running on ECS so I’d have to launch a temporary EC2 instance to get to the RDS instance where the database is hosted).

Kathy Satterlee

01/31/2023, 10:10 PM

Sounds good. I'm betting that it's a similar situation and there's a host that didn't quite enroll properly and Orbit keeps trying to re-enroll. If we can track down what host that is, next step will be to check out the Orbit logs on the host to verify/get additional information on what's happening. Then likely remove both the fragment and the real host and see what happens.

brian

01/31/2023, 10:17 PM

👍 thanks for the help! I’ll let you know what I find tomorrow.

brian

02/01/2023, 4:49 PM

I think I found the troublesome host, it looks like it has two entries in the hosts table, one of them with a null orbit_node_key:

Copy code

| 56 | 51b2b9d6-e102-49c3-9fd7-2f8cac64fb3d | 2023-01-30 19:33:49 | JJFxbCFU93oRY94xyzhz2l11WkIZ1tUI | localhost                | 51b2b9d6-e102-49c3-9fd7-2f8cac64fb3d | 2023-01-30 19:33:49 | NULL                             |
| 94 | 6c76c81a-cb0d-4288-8201-06bc558dda5a | 2023-02-01 16:00:56 | IYKQaZsExKnuFXNv0rdBSUgEKyfktEi6 |                          |                                      | 1970-01-02 00:00:00 | IYKQaZsExKnuFXNv0rdBSUgEKyfktEi6 |

brian

02/01/2023, 4:49 PM

maybe easier to read…

Copy code

| hostname                 | uuid                                 | last_enrolled_at    | orbit_node_key                   |
+--------------------------+--------------------------------------+---------------------+----------------------------------+
| localhost                | 51b2b9d6-e102-49c3-9fd7-2f8cac64fb3d | 2023-01-30 19:33:49 | NULL                             |
|                          |                                      | 1970-01-02 00:00:00 | IYKQaZsExKnuFXNv0rdBSUgEKyfktEi6 |

brian

02/02/2023, 8:24 PM

so doing some poking around, it looks like this is probably related to https://github.com/fleetdm/fleet/issues/9132, since I’m also seeing empty

orbit_node_key

and another host with

orbit_node_key

equal to

node_key

Kathy Satterlee

02/02/2023, 8:41 PM

Absolutely. Sorry for slow response today! Are you able to share the Orbit logs for that host? I'd also like to see what happens if you uninstall Orbit on the host, remove both the artifact and the host from Fleet, and then re-enroll it using a package created with the

--debug

flag for more verbose logging.

brian

02/02/2023, 9:15 PM

these are some of the debug lines we’re seeing:

Copy code

Feb  2 16:04:41 localhost orbit[58456]: 2023-02-02T16:04:41-05:00 DBG no update target=orbit
Feb  2 16:04:41 localhost orbit[58456]: 2023-02-02T16:04:41-05:00 DBG no update target=osqueryd
Feb  2 16:04:42 localhost orbit[58456]: 2023-02-02T16:04:42-05:00 DBG found expected target locally path=/opt/orbit/bin/osqueryd/linux/stable/osqueryd target=osqueryd
Feb  2 16:04:42 localhost orbit[58456]: 2023-02-02T16:04:42-05:00 DBG running single query (SELECT uuid FROM system_info)
Feb  2 16:04:42 localhost orbit[58456]: 2023-02-02T16:04:42-05:00 DBG UUID is 6e2c24c6-f205-4b39-8bf3-28fc825c2156
Feb  2 16:04:42 localhost orbit[58456]: 2023-02-02T16:04:42-05:00 INF initial flags update failed error="error getting flags from fleet: unauthenticated, or invalid token"
Feb  2 16:04:42 localhost orbit[58456]: 2023-02-02T16:04:42-05:00 DBG /opt/orbit/extensions.load not found, nothing to update
Feb  2 16:04:43 localhost orbit[58456]: 2023-02-02T16:04:43-05:00 DBG no update target=orbit
Feb  2 16:04:43 localhost orbit[58456]: 2023-02-02T16:04:43-05:00 DBG no update target=osqueryd
Feb  2 16:04:43 localhost orbit[58456]: 2023-02-02T16:04:43-05:00 INF token rotation is enabled
Feb  2 16:04:43 localhost orbit[58456]: 2023-02-02T16:04:43-05:00 DBG starting flag updater
Feb  2 16:04:43 localhost orbit[58456]: 2023-02-02T16:04:43-05:00 DBG starting extension runner
Feb  2 16:04:43 localhost orbit[58456]: 2023-02-02T16:04:43-05:00 DBG start updater
Feb  2 16:04:43 localhost orbit[58456]: 2023-02-02T16:04:43-05:00 DBG start osquery extension

brian

02/02/2023, 9:16 PM

that UUID

6e2c24c6-f205-4b39-8bf3-28fc825c2156

corresponds with the empty host’s

osquery_host_id

value that ends up in the db.

brian

02/02/2023, 9:19 PM

(that’s after uninstalling, and then reinstalling)

brian

02/02/2023, 10:57 PM

this is from the person whose laptop is getting registered twice: So it appears that every time orbit is restarted this UUID is regenerated

Copy code

Feb  2 17:25:23 localhost orbit[74253]: 2023-02-02T17:25:23-05:00 DBG running single query (SELECT uuid FROM system_info)
Feb  2 17:25:23 localhost orbit[74253]: 2023-02-02T17:25:23-05:00 DBG UUID is 818ec869-0fc6-4522-8aa7-216e0f16f2b5

If the unknown host is already registered in our Fleet instance, then nothing happens. If it’s not registered, then it gets registered.

Ryan W.

02/03/2023, 7:46 PM

This issue is also showing up in our environment, and running the same set of SQL queries shows similar results.

Ryan W.

02/03/2023, 7:51 PM

I should mention, we get this for about a dozen devices in an environment of maybe 75 total.

Kathy Satterlee

02/03/2023, 7:56 PM

Thanks for all of the information. Can you both verify your Fleet version for me?

Kathy Satterlee

02/03/2023, 7:58 PM

If you are also on GitHub and can share your username (feel free to DM), I'd like to tag you in the issue as well.

Kathy Satterlee

02/03/2023, 8:00 PM

You can also follow here: https://github.com/fleetdm/fleet/issues/9132

brian

02/03/2023, 8:03 PM

Fleet 4.26.0 for us. and osquery 5.7.0 it looks like. GitHub username is

briandefiant

Ryan W.

02/03/2023, 8:34 PM

fleet 4.26.0 , osquery 5.7.0

Ryan W.

02/03/2023, 8:34 PM

github username is weetabix

Ryan W.

02/03/2023, 8:37 PM

I can further report this did not happen in our first, hasty test instance, which used some of the same machines. There isnt a decent enough overlap between the two groups though, for me to suspect a left over config or something, since it couldnt exist on at least a third of the suspect machine4s.

Lucas Rodriguez

02/06/2023, 12:12 PM

Hi folks! If any of you have access to the hosts that are having this issue, please take a look at https://github.com/fleetdm/fleet/issues/9132#issuecomment-1418943150.

Open in Slack

Previous Next