I wonder if anyone can give some advice on how to debug queries that hang? We’re managing ~1000 hosts in a Fleet instance, and occasionally queries will run on most but hang on a small subset. In a test last week we targeted 127 hosts, and 125 returned, but 2 hung for a long time and we eventually cancelled. When a query is stuck like this what’s the best way to investigate the cause?

I usually start debugging queries by running them on my own device with osqueryi to determine if there is an issue which is causing them to timeout. Can you share the queries in question to Slack so we can take a look?

Looking at the launcher logs is a good first pass.

Thanks - I’ll try with

osqueryi

on the devices that didn’t return, to see if that helps shed any light.

What's in the osquery logs? Can't even tell if it got the query...

let me double check 🙂

How osquery logs is also os dependent. Results are presumablly going to fleet. But osquery logs might be going out via stdout/stderr.

yeah, sadly no journalctl on these hosts, they’re running 14.04 😞

You should be able to configure basic logging there too.

so at the moment it’s set to log only to Fleet - via the tls plugin, so I guess I can enable syslog on the clients too for debugging

I haven't looked at this recently. But those logs may show if the query was received.

okidoki

Really what I'm saying here is that you need logs to even start debugging. Did it hang? Was it never received? Was something dropped? Did something crash?