Title
#fleet
s

Stephan

01/14/2022, 2:10 PM
I'm testing fleet 4.6.1 and I'm not seeing any of the performance data coming back from the osquery client for each of the scheduled queries. What's a good way to troubleshoot that?
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 4:29 PM
The scheduled queries and pack stats are aggregated every one hour, are they still not showing up?
s

Stephan

01/14/2022, 4:40 PM
Yeah it's been weeks and nothing shows
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 4:41 PM
OK, what MySQL database are you using? (and version)
s

Stephan

01/14/2022, 4:43 PM
mysql-community-server               5.7.32-1debian9
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 4:43 PM
OK, do you have access to fleet's logs?
s

Stephan

01/14/2022, 4:46 PM
Yup
4:46 PM
All flowing into a splunk instance
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 4:47 PM
OK, could you look for any
level=error
logs? (possibly with
aggregating
)
4:48 PM
Can you see the stats of scheduled queries for individual hosts? (If you visit the host and go to the schedule section)
s

Stephan

01/14/2022, 5:29 PM
Ah looks like I shot myself in the foot. Need to wait for our DB team to make a change before I can get back into our staging environment. Will report back in a bit.
10:48 PM
I've checked individual hosts and I also don't see the metrics there @Lucas Rodriguez. I was looking at the logs and I don't think they are flowing the way I need. This is running in kube and only stdout logs are forwarded to Splunk. Looking at https://fleetdm.com/docs/deploying/configuration#logging-fleet-server-logging it doesn't give any options about where the server logs go. Do you happen to know if they go to stdout or the file system?
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 10:50 PM
By default,
fleet serve
logs go to
stderr
actually.
s

Stephan

01/14/2022, 10:52 PM
Perfect, just found the logs in splunk. Searched for
aggregating
and
level=error
and the only entry I was able to find is
enroll failed: timestamp: 2022-01-14T22:40:37Z: no matching secret found
10:53 PM
Example from one of the hosts
10:55 PM
If I look at the osquery logs of the host there are a bunch of entries with
Executing scheduled query pack...
and the results of that query are in Splunk
10:56 PM
So even the
Has not run
is confusing since these queries are running. But it seems that metadata never makes it back to fleet
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 11:05 PM
Try running the following live query on one of the hosts:
SELECT *, (SELECT value from osquery_flags where name = 'pack_delimiter') AS delimiter FROM osquery_schedule
(Which is what fleet sends to get the stats from the hosts.)
11:06 PM
Go to host > Query > Create custom query (and then just run the query, do not save it)
11:06 PM
That's what we call "live" query.
11:07 PM
If it sends back data, then all is working there and you may need to run
fleet serve
with
--logging_debug
to continue troubleshooting.
s

Stephan

01/14/2022, 11:10 PM
yeah I get a bunch of records back. Most of the data is 0 right now. I suspect that's because I restart osqueryd a few minutes back but 2 rows/queries show something for
average_memory
and
last_executed
. For example,
pack/osquery/monitoring-4d172597684da71da8335fe755447e63f25e3107/info-4d172597684da71da8335fe755447e63f25e3107
but when I view that in the UI
11:12 PM
I have
FLEET_LOGGING_DEBUG
set to
true
in an env variable in kube. Is that considered the same as the
logging_debug
CLI flag?
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 11:12 PM
What about
executions
what value do they return?
s

Stephan

01/14/2022, 11:13 PM
shows
1
for those rows
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 11:15 PM
Yes,
FLEET_LOGGING_DEBUG
is the same as
--logging_debug
11:16 PM
Now you should look for log lines that have
level=debug
and that have
query
or
pack
👀 1
s

Stephan

01/14/2022, 11:17 PM
Also ran a
fleetctl debug connection
in one of the pods which came back clean as well as
fleetctl debug errors
Lucas Rodriguez

Lucas Rodriguez

01/14/2022, 11:18 PM
I have to go (EoD here 🙂, will let the team know about this thread.
s

Stephan

01/14/2022, 11:20 PM
Over the last hour got 4 events for
level=debug "query" OR "pack"
. Either
SELECT *, (SELECT value from osquery_flags where name = 'pack_delimiter') AS delimiter FROM osquery_schedule
or
live_query
related
11:20 PM
No problem, thanks for the help and enjoy the evening
8:39 PM
I believe this has something to do with our query and pack names. I have a test pack called
users/root-4d172597684da71da8335fe755447e63f25e3107
and a test query called
users/root/suid_bin-4d172597684da71da8335fe755447e63f25e3107
. Note that in osquery we have
pack_delimiter
set to
/
. If I remove the
/
in the pack and query name performance data starts working. Is it not possible to use the
pack_delimiter
in the names?
Lucas Rodriguez

Lucas Rodriguez

01/17/2022, 12:45 PM
Hi Stephan! Thanks for all this information. You got that right, it looks like Fleet doesn't currently handle
pack_delimiter
in the pack and query names. Creating a Github issue right away.
s

Stephan

01/17/2022, 10:42 PM
Thank you. We'll go ahead and come up with a different naming convention
Lucas Rodriguez

Lucas Rodriguez

01/18/2022, 12:51 AM
Thank you for finding the issue! We'll post any updates on the issue.