https://github.com/osquery/osquery logo
Powered by
Title
s

Stephan

01/14/2022, 2:10 PM
I'm testing fleet 4.6.1 and I'm not seeing any of the performance data coming back from the osquery client for each of the scheduled queries. What's a good way to troubleshoot that?
l

Lucas Rodriguez

01/14/2022, 4:29 PM
The scheduled queries and pack stats are aggregated every one hour, are they still not showing up?
s

Stephan

01/14/2022, 4:40 PM
Yeah it's been weeks and nothing shows
l

Lucas Rodriguez

01/14/2022, 4:41 PM
OK, what MySQL database are you using? (and version)
s

Stephan

01/14/2022, 4:43 PM
mysql-community-server               5.7.32-1debian9
l

Lucas Rodriguez

01/14/2022, 4:43 PM
OK, do you have access to fleet's logs?
s

Stephan

01/14/2022, 4:46 PM
Yup
All flowing into a splunk instance
l

Lucas Rodriguez

01/14/2022, 4:47 PM
OK, could you look for any 
level=error
logs? (possibly with 
aggregating
)
Can you see the stats of scheduled queries for individual hosts? (If you visit the host and go to the schedule section)
s

Stephan

01/14/2022, 5:29 PM
Ah looks like I shot myself in the foot. Need to wait for our DB team to make a change before I can get back into our staging environment. Will report back in a bit.
I've checked individual hosts and I also don't see the metrics there @Lucas Rodriguez. I was looking at the logs and I don't think they are flowing the way I need. This is running in kube and only stdout logs are forwarded to Splunk. Looking at https://fleetdm.com/docs/deploying/configuration#logging-fleet-server-logging it doesn't give any options about where the server logs go. Do you happen to know if they go to stdout or the file system?
l

Lucas Rodriguez

01/14/2022, 10:50 PM
By default, 
fleet serve
logs go to 
stderr
actually.
s

Stephan

01/14/2022, 10:52 PM
Perfect, just found the logs in splunk. Searched for 
aggregating
and 
level=error
and the only entry I was able to find is 
enroll failed: timestamp: 2022-01-14T22:40:37Z: no matching secret found
Example from one of the hosts
If I look at the osquery logs of the host there are a bunch of entries with 
Executing scheduled query pack...
and the results of that query are in Splunk
So even the 
Has not run
is confusing since these queries are running. But it seems that metadata never makes it back to fleet
l

Lucas Rodriguez

01/14/2022, 11:05 PM
Try running the following live query on one of the hosts: 
SELECT *, (SELECT value from osquery_flags where name = 'pack_delimiter') AS delimiter FROM osquery_schedule
(Which is what fleet sends to get the stats from the hosts.)
Go to host > Query > Create custom query (and then just run the query, do not save it)
That's what we call "live" query.
If it sends back data, then all is working there and you may need to run 
fleet serve
with 
--logging_debug
to continue troubleshooting.
s

Stephan

01/14/2022, 11:10 PM
yeah I get a bunch of records back. Most of the data is 0 right now. I suspect that's because I restart osqueryd a few minutes back but 2 rows/queries show something for 
average_memory
and 
last_executed
. For example, 
pack/osquery/monitoring-4d172597684da71da8335fe755447e63f25e3107/info-4d172597684da71da8335fe755447e63f25e3107
but when I view that in the UI
I have 
FLEET_LOGGING_DEBUG
set to 
true
in an env variable in kube. Is that considered the same as the 
logging_debug
CLI flag?
l

Lucas Rodriguez

01/14/2022, 11:12 PM
What about 
executions
what value do they return?
s

Stephan

01/14/2022, 11:13 PM
shows 
1
for those rows
l

Lucas Rodriguez

01/14/2022, 11:15 PM
Yes, 
FLEET_LOGGING_DEBUG
is the same as 
--logging_debug
Now you should look for log lines that have 
level=debug
and that have 
query
or 
pack
👀 1
s

Stephan

01/14/2022, 11:17 PM
Also ran a 
fleetctl debug connection
in one of the pods which came back clean as well as 
fleetctl debug errors
l

Lucas Rodriguez

01/14/2022, 11:18 PM
I have to go (EoD here 🙂, will let the team know about this thread.
s

Stephan

01/14/2022, 11:20 PM
Over the last hour got 4 events for 
level=debug "query" OR "pack"
. Either 
SELECT *, (SELECT value from osquery_flags where name = 'pack_delimiter') AS delimiter FROM osquery_schedule
or 
live_query
related
No problem, thanks for the help and enjoy the evening
I believe this has something to do with our query and pack names. I have a test pack called 
users/root-4d172597684da71da8335fe755447e63f25e3107
and a test query called 
users/root/suid_bin-4d172597684da71da8335fe755447e63f25e3107
. Note that in osquery we have 
pack_delimiter
set to 
/
. If I remove the 
/
in the pack and query name performance data starts working. Is it not possible to use the 
pack_delimiter
in the names?
l

Lucas Rodriguez

01/17/2022, 12:45 PM
Hi Stephan! Thanks for all this information. You got that right, it looks like Fleet doesn't currently handle 
pack_delimiter
in the pack and query names. Creating a Github issue right away.
And here it is: https://github.com/fleetdm/fleet/issues/3717
s

Stephan

01/17/2022, 10:42 PM
Thank you. We'll go ahead and come up with a different naming convention
l

Lucas Rodriguez

01/18/2022, 12:51 AM
Thank you for finding the issue! We'll post any updates on the issue.
3 Views
#fleetJoin Slack
Powered by