Title
#kolide
r

Ryan

06/29/2020, 10:03 AM
I wonder if anyone can give some advice on how to debug queries that hang? We’re managing ~1000 hosts in a Fleet instance, and occasionally queries will run on most but hang on a small subset. In a test last week we targeted 127 hosts, and 125 returned, but 2 hung for a long time and we eventually cancelled. When a query is stuck like this what’s the best way to investigate the cause?
f

fritz

06/29/2020, 12:15 PM
I usually start debugging queries by running them on my own device with osqueryi to determine if there is an issue which is causing them to timeout. Can you share the queries in question to Slack so we can take a look?
s

seph

06/29/2020, 12:23 PM
Looking at the launcher logs is a good first pass.
r

Ryan

06/29/2020, 1:14 PM
Thanks - I’ll try with
osqueryi
on the devices that didn’t return, to see if that helps shed any light.
1:14 PM
We’re not using Launcher so I don’t have any log from that to look at.
s

seph

06/29/2020, 1:30 PM
What's in the osquery logs? Can't even tell if it got the query...
r

Ryan

06/29/2020, 1:30 PM
let me double check 🙂
1:34 PM
so, the
/var/log/osquery/
is empty on the client
1:35 PM
the team member of mine who was actually exploring this isn’t online for me to get more information, but if there’s client-side logging I can enable I can roll that change out?
s

seph

06/29/2020, 1:36 PM
How osquery logs is also os dependent. Results are presumablly going to fleet. But osquery logs might be going out via stdout/stderr.
1:36 PM
Anyhow, not much to debug without logs.
r

Ryan

06/29/2020, 1:37 PM
yeah, sadly no journalctl on these hosts, they’re running 14.04 😞
s

seph

06/29/2020, 1:41 PM
You should be able to configure basic logging there too.
r

Ryan

06/29/2020, 1:44 PM
so at the moment it’s set to log only to Fleet - via the tls plugin, so I guess I can enable syslog on the clients too for debugging
s

seph

06/29/2020, 1:45 PM
I haven't looked at this recently. But those logs may show if the query was received.
r

Ryan

06/29/2020, 1:46 PM
okidoki
1:46 PM
thanks for the tips, I’ll try a couple of these later in the week and come back
s

seph

06/29/2020, 1:46 PM
Really what I'm saying here is that you need logs to even start debugging. Did it hang? Was it never received? Was something dropped? Did something crash?