https://github.com/osquery/osquery logo
Join Slack
Powered by
I've just setup a new fleet installation and added...
# fleet
s
I've just setup a new fleet installation and added my own Ubuntu 23.04 laptop via a package I created with fleetctl package. In the host screen, after a while this popup appears: 
We're having trouble fetching fresh vitals for this host. Please try again later.
Logs on the server show:
Copy code
error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || error in query ingestion || getting app config: selecting app config: context canceled
ingestion-err
ingest detail query: selecting app config: context canceled
(this is slightly mangled/reformatted) I read somewhere that websocket support is necessary, and I think I've enabled that. The Users and Software tabs on my host in fleet are both empty. Since this is the first time I've set it up I'm not sure if something's broken, but it seems like it. Any pointers?
Using fleet 4.36.0 (just upgraded), and osquery 5.8.2.
guess I should update fleetctl...
l
Hi @Scott Cruzen! That kind of error server is usually caused by issues with the connection from Fleet to MySQL. A few questions: Could you check the MySQL instance? Are you using MySQL read replicas? How many hosts have you enrolled?
s
I have only 1 host (my laptop)
but after sleeping on it, I think I have an idea, I'll reply with my findings shortly
Ok, I tried increasing the iops of the underlying mysql volumes, but it doesn't seem like it made any difference. I notice that the ingest requests seem to time out after about 3 seconds (this seems consistent), but in retrospect that seems more like a config issue (I don't think a fetch of a single host is really producing a huge amount of IO load on mysql). The next thing I'm going to look into is the size of the redis instance... I'm curious how the ingest flow looks. I assume things hit redis first (like a write thru cache)?
l
OK.
Copy code
getting app config: selecting app config: context canceled
This seems to indicate the issue is when reading the app config from MySQL
Fleet is usable in the UI? Like can you create queries, policies, etc.?
s
yes, live query works fine
(and I can save queries and policies)
l
OK, could you check for issues in the device running the agent? On ubuntu you can do 
grep orbit /var/log/syslog
to get the logs of the agent.
s
I don't see anything particularly interesting in the orbit logs. Some "Event publisher not enabled" for BPFEventPublisher, auditeventpublisher, inotify, syslog. some "INF calling flags update" and some "INF periodic check of token failed, initiating rotation error='missing or invalid license'"
messages like this:
Copy code
E0818 15:23:03.944536 276151 mounts.cpp:65] getmntent_r failed with errno 22
have started to appear on my client.
I tried replacing my first mysql db with a fresh one in RDS, just in case the defaults set by mysql-operator were somehow worse or possibly mysql-router was causing some issue. Anyway, with this new RDS DB the results are identical.
k
Hey @Scott Cruzen I suspect that the errors for app config and token rotation may be related. Do you manage your app config with a yaml file?
s
Everything is in k8s, so yeah, it's all manifests.
k
Are you able to try applying that config with 
fleetctl apply
? I suspect there may be a deprecated field in there somewhere that's gumming up the works and I'd like to see what happens
s
apply what exactly? I guess fleet is currently running entirely with defaults aside from env variables.
k
Alright, I misunderstood you there. let's try this:
Copy code
> fleetctl get config > fleet_config.yml

> fleetctl apply -f fleet_config.yml
I'm trying to see if that results in an error.
s
ok
I hadn't logged in using fleetctl before, but get config and apply worked fine.
k
That's a good sign. Have you got a sec to hop on Zoom to pull up a few things?
s
sure
k
I'll DM you a link.
11 Views
Open in Slack
PreviousNext
Powered by