https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi everyone - I’m trying to debug why a query pack...
# kolide
r
Hi everyone - I’m trying to debug why a query pack isn’t being scheduled on some hosts in Fleet. We have 1021 hosts online, I’ve scheduled a query pack with one query in it to run on all of them once per day. I’m only seeing 823 hosts returning results in 
/tmp/osquery_result
. I tried running the same query on-demand against some of the missing hosts and they worked fine, but if I run 
SELECT * FROM osquery_schedule
they return successfully, but with no records. Does anyone have any suggestions? Thanks 🙂
I’m going to try the tip here to add the 
SELECT * FROM time
query to a single pack and see if that works. https://github.com/kolide/fleet/blob/master/docs/infrastructure/faq.md#troubleshooting
👍 2
So, result of my test there, getting the same number of hosts returning data here, 823, and if I run an adhoc query of 
SELECT * FROM osquery_schedule
I get the same result. What could cause a query pack to fail to be scheduled by osquery on certain nodes?
CC @zwass and @Macear (hope you don’t mind, but I saw you added a ‘thumbs up’ to my original message 😄)
🙂 1
z
How are you targeting the pack? Does the query have any platform set? Shard?
r
I’ve set it to All Hosts, no platform or shard is set.
I did the targeting in the Pack itself though, not the Query as that seems to be only for running it on demand.
Query is this:
and this is the Pack:
m
@Ryan have the same behavior. Here is my issue https://github.com/kolide/fleet/issues/2260 Then I try to update pack, and logs from some servers went. But on some servers there are no scheduled queries so far. I did create new pack targeted only on problematic servers, will see
In my case restart of osqueryd helps, but it’s very uncomfortable. I tested on centos6 (osquery installed).
r
you mean restarting the daemon makes the scheduled packs work? does it need restarting periodically, or is this only needed after you schedule a new query?
m
For me it works after only one restart of the daemon. Strange
r
Yeah that is weird!
So you would create the query pack, then restart the daemon, and from then on it will work fine with no need for further restarts?
z
It would be good to run osquery with 
--verbose --tls_dump
on an effected host and see if the host is checking in for configs and Fleet is sending the correct packs to that host. What you describe with the restart of osqueryd has me suspecting perhaps an osquery bug.
👍 1
r
Sure thing, what’s the best way to get osquery to log something locally to try this?
without breaking the logging back to fleet I mean
z
--verbose --tls_dump
in the osquery flags
then it will log requests and responses to stderr
r
aha
ok great
I will give this a try tomorrow 👍
Ok so an update from my experimentation on this issue - with 
--verbose
and 
--tls_dump
in place I don’t see any particular errors, but nor do I see anything that mentions scheduled queries. I can see the distributed queries used for our labels coming in on the affected host, but it doesn’t appear to be receiving any packs to run on a schedule. I tried upgrading it from 
4.3.0
to 
4.4.0
and fully restarting it, but no joy sadly. I’m running Fleet version 
2.6.0
. Any other suggestions will be greatly appreciated :)
m
@Ryan check the following option: --pack_refresh_interval=3600 And also check the “shard” parameter of the queries in your pack. Drop it’s value if it’s set Maybe this will help
r
The pack refresh option isn’t set at all in my flags file.
I can add it explicitly if that’ll help?
Just verified, the shard parameter is not set.
m
Honestly, have no idea what should fix this issue. You can add this flag or also you may try to check your configuration by this command: osqueryd —flagfile <path_to_flags> --config_dump
Ideally, your queries should be in output included. Otherwise I advise you to create new issue
r
Ok, thanks, I’ll give this a try too 🙂
I’ll let it run overnight with the additional flag above and come back tomorrow. Thanks for the help so far!
z
Do you see the config requests and responses? That interval is configured by 
config_refresh
. You'd want to look for whether the returned config includes the expected packs.
r
I’ll check that tomorrow as I’m wrapping up now, thanks for the tip!
m
👍 1
r
Cool thanks @Macear
2 Views
Open in Slack
PreviousNext
Powered by