I m testing fleet 4 6 1 and I m not seeing any of the perfor osquery #fleet

I'm testing fleet 4.6.1 and I'm not seeing any of ...

Stephan

01/14/2022, 2:10 PM

I'm testing fleet 4.6.1 and I'm not seeing any of the performance data coming back from the osquery client for each of the scheduled queries. What's a good way to troubleshoot that?

Lucas Rodriguez

01/14/2022, 4:29 PM

The scheduled queries and pack stats are aggregated every one hour, are they still not showing up?

Stephan

01/14/2022, 4:40 PM

Yeah it's been weeks and nothing shows

Lucas Rodriguez

01/14/2022, 4:41 PM

OK, what MySQL database are you using? (and version)

Stephan

01/14/2022, 4:43 PM

Copy code

mysql-community-server               5.7.32-1debian9

Lucas Rodriguez

01/14/2022, 4:43 PM

OK, do you have access to fleet's logs?

Stephan

01/14/2022, 4:46 PM

Yup

Stephan

01/14/2022, 4:46 PM

All flowing into a splunk instance

Lucas Rodriguez

01/14/2022, 4:47 PM

OK, could you look for any

level=error

logs? (possibly with

aggregating

)

Lucas Rodriguez

01/14/2022, 4:48 PM

Can you see the stats of scheduled queries for individual hosts? (If you visit the host and go to the schedule section)

Stephan

01/14/2022, 5:29 PM

Ah looks like I shot myself in the foot. Need to wait for our DB team to make a change before I can get back into our staging environment. Will report back in a bit.

Stephan

01/14/2022, 10:48 PM

I've checked individual hosts and I also don't see the metrics there @Lucas Rodriguez. I was looking at the logs and I don't think they are flowing the way I need. This is running in kube and only stdout logs are forwarded to Splunk. Looking at https://fleetdm.com/docs/deploying/configuration#logging-fleet-server-logging it doesn't give any options about where the server logs go. Do you happen to know if they go to stdout or the file system?

Lucas Rodriguez

01/14/2022, 10:50 PM

By default,

fleet serve

logs go to

stderr

actually.

Stephan

01/14/2022, 10:52 PM

Perfect, just found the logs in splunk. Searched for

aggregating

and

level=error

and the only entry I was able to find is

enroll failed: timestamp: 2022-01-14T22:40:37Z: no matching secret found

Stephan

01/14/2022, 10:53 PM

Example from one of the hosts

Stephan

01/14/2022, 10:55 PM

If I look at the osquery logs of the host there are a bunch of entries with

Executing scheduled query pack...

and the results of that query are in Splunk

Stephan

01/14/2022, 10:56 PM

So even the

Has not run

is confusing since these queries are running. But it seems that metadata never makes it back to fleet

Lucas Rodriguez

01/14/2022, 11:05 PM

Try running the following live query on one of the hosts:

SELECT *, (SELECT value from osquery_flags where name = 'pack_delimiter') AS delimiter FROM osquery_schedule

(Which is what fleet sends to get the stats from the hosts.)

Lucas Rodriguez

01/14/2022, 11:06 PM

Go to host > Query > Create custom query (and then just run the query, do not save it)

Lucas Rodriguez

01/14/2022, 11:06 PM

That's what we call "live" query.

Lucas Rodriguez

01/14/2022, 11:07 PM

If it sends back data, then all is working there and you may need to run

fleet serve

with

--logging_debug

to continue troubleshooting.

Stephan

01/14/2022, 11:10 PM

yeah I get a bunch of records back. Most of the data is 0 right now. I suspect that's because I restart osqueryd a few minutes back but 2 rows/queries show something for

average_memory

and

last_executed

. For example,

pack/osquery/monitoring-4d172597684da71da8335fe755447e63f25e3107/info-4d172597684da71da8335fe755447e63f25e3107

but when I view that in the UI

Stephan

01/14/2022, 11:12 PM

I have

FLEET_LOGGING_DEBUG

set to

true

in an env variable in kube. Is that considered the same as the

logging_debug

CLI flag?

Lucas Rodriguez

01/14/2022, 11:12 PM

What about

executions

what value do they return?

Stephan

01/14/2022, 11:13 PM

shows

for those rows

Lucas Rodriguez

01/14/2022, 11:15 PM

Yes,

FLEET_LOGGING_DEBUG

is the same as

--logging_debug

Lucas Rodriguez

01/14/2022, 11:16 PM

Now you should look for log lines that have

level=debug

and that have

query

pack

👀 1

Stephan

01/14/2022, 11:17 PM

Also ran a

fleetctl debug connection

in one of the pods which came back clean as well as

fleetctl debug errors

Lucas Rodriguez

01/14/2022, 11:18 PM

I have to go (EoD here 🙂, will let the team know about this thread.

Stephan

01/14/2022, 11:20 PM

Over the last hour got 4 events for

level=debug "query" OR "pack"

. Either

SELECT *, (SELECT value from osquery_flags where name = 'pack_delimiter') AS delimiter FROM osquery_schedule

live_query

Stephan

01/14/2022, 11:20 PM

No problem, thanks for the help and enjoy the evening

Stephan

01/15/2022, 8:39 PM

I believe this has something to do with our query and pack names. I have a test pack called

users/root-4d172597684da71da8335fe755447e63f25e3107

and a test query called

users/root/suid_bin-4d172597684da71da8335fe755447e63f25e3107

. Note that in osquery we have

pack_delimiter

set to

. If I remove the

in the pack and query name performance data starts working. Is it not possible to use the

pack_delimiter

in the names?

Lucas Rodriguez

01/17/2022, 12:45 PM

Hi Stephan! Thanks for all this information. You got that right, it looks like Fleet doesn't currently handle

pack_delimiter

in the pack and query names. Creating a Github issue right away.

Lucas Rodriguez

01/17/2022, 1:17 PM

And here it is: https://github.com/fleetdm/fleet/issues/3717

Stephan

01/17/2022, 10:42 PM

Thank you. We'll go ahead and come up with a different naming convention

Lucas Rodriguez

01/18/2022, 12:51 AM

Thank you for finding the issue! We'll post any updates on the issue.

5 Views

Open in Slack

Previous Next