https://github.com/osquery/osquery logo
Join Slack
Powered by
Hey all, hitting a missing data issue when using `...
# fleet
j
Hey all, hitting a missing data issue when using 
instance
over 
uuid
- Additional context: I'm shipping result logs to pubsub, where the hostIdentifer is correctly reporting the osquery instance id (osquery_host_id) It appears that when using a host identifier type of 
instance
, there's no way to correlate query outputs to hosts. 
/api/v1/fleet/hosts
doesn't provide osquery_host_id (I have scoured the code for a while and can't figure out why) - so unless there's some secret way to get that via the API, I guess I'll need to dump the DB to bigquery. Screenshots in 🧵
Hosts table (notice how the host id != uuid - this is good)
api response for host, both by listing all or just the single host (removed everything else, note that id, UUID and hostname are the only identifiers that return
Copy code
{
    "created_at": "2024-09-23T18:13:14Z",
    "updated_at": "2024-09-23T18:14:02Z",
    "software_updated_at": "2024-09-23T18:14:02Z",
    "id": 21,
    ...
    "hostname": "FWX02LW3Q9",
    "uuid": "61E1C9A5-8A18-5727-BDFF-13AF90C9EFAB",
    "platform": "darwin",
    "osquery_version": "5.13.1",
    "orbit_version": "1.33.0",
    ...
}
Finally, in Bigquery (via pubsub) we see the instance id correctly reported:
Basically, I'm dumping query results to BQ and need some way to JOIN the host so that I know what host sent that query result. I pull all hosts via the fleet API daily, but the osquery_host_id isn't available. Any ideas? I'd try to fix this myself, but I can't seem to locate the server code responsible for ultimately returning the API response for /hosts
r
Hi, James. Are you using 
instance
instead of 
uuid
, because your host is a virtual machine? Also, could you share one of the queries in question?
j
Yes - We've deployed fleet across a number of endpoints, with a large set being VMs - many which overlap UUIDs 😞 We use uuid for most endpoints, but are trying to use instance for the VM population. This isn't an issue with queries in particular, the 
hostIdentifier
in the screenshot above is tacked on by fleet, the full json object sent to pubsub for all scheduled query results looks like this:
Copy code
{
    "action": "added",
    "calendarTime": "Mon Sep 23 18:16:54 2024 UTC",
    "columns": {
        (Query results)
    },
    "counter": 0,
    "epoch": 0,
    "hostIdentifier": "d8caebae-b6e2-47d3-a5ae-27586da00f2c",
    "name": "pack_Global_SOME_QUERY_NAME",
    "numerics": false,
    "unixTime": 1727115414
}
That hostIdentifier above, it's the osquery_host_id, not the machine's UUID. When using 
instance
that osquery_host_id != uuid. The fleet API doesn't return 
osquery_host_id
, only 
uuid
(which is still collected and stored, even when using 
instance
- so I have no way to pair this result up with a hostname unless I hit the fleet db directly (which I'd really prefer not to do, since the API is pretty great and we're already using it to pull hosts)
If this still isn't super clear what's going on I can try for a diagram, this is probably an edge case, and I see the ability to even use instance is somewhat new
k
Are hostnames reliably unique in your environment? If so, you could add a 
decorator
and include the hostname in your osquery logs.
Then use that to reference the host in Fleet.
j
@Kathy Satterlee Hah! I was just about to try a pretty hacky version of this - this looks awesome I'll give it a go, thanks a ton!
Ah we already had that set, but I'm pretty sure the fleet pubsub logic ignores decorator fields.
k
They should definitely be included in the data Fleet is sending.
Do you have any overrides set in your agent configuration?
j
I do, but no decorator overrides (only platform specific ATCs)- I assumed overrides are additive and not a total replacement right? I'm going to lift the result logs off the endpoint to make sure the decorator is working, and also take a deeper look at the pubsub code here to figure out what objects should be looking like at each step [endpoint > fleet > pubsub]
Sweet, the decorators are on the host in the osquery results log, let me double check they aren't hitting pubsub
k
I assumed overrides are additive and not a total replacement right
They are actually a total replacement.
Hosts receive either the default options, or the overrides.
From the docs:
``` # Note configs in overrides take precedence over the default config defined
# under the config key above. Hosts receive overrides based on the platform
# returned by 
SELECT platform FROM os_version
. In this example, the base
# config would be used for Windows and CentOS hosts, while Mac and Ubuntu
# hosts would receive their respective overrides. Note, these overrides are
# NOT merged with the top level configuration.```
j
Yup, this all checks out. I've been using 
linux
instead of 
ubuntu
for ubuntu overrides, so upon first glance my mac logs had no decorations but my ubuntu machine did. I likely developed my logging transport logic off macos test data (and thus never noticed the decoration data available, even though I had set it!)
All is well once again. I may eventually PR an addition of the instance ID in the API resultset though, seems useful and is already part of the Host struct - but absolutely not a blocker anymore. Thanks a ton!
k
Awesome. I’m glad you’ve got a solid path forward!
6 Views
Open in Slack
PreviousNext
Powered by