Hey Team, I am seeing 0 online devices whereas everything was fine until few hours ago. I dont see any err in logs or any error that its not able to communicate with fleet server. Also for alot of devices it still shows last fetched a minute ago and thats keeps updating. Not able to run any query though

The second part is interesting, so for some specific hosts, you can see a recent

updated_at

key returned from the API but status is still

offline

? Are you able to refetch that host's information and get new update host information?

@Rachel Perkins nope refetch dosent work nor does the query

Is Netskope a recent addition to your environment? If you're using a proxy server, do you see any logs there that indicate that connections are being refused and why?

nah! netkope is old one @Kathy Satterlee its been there for many months and was there when we deployed fleet.

I know you said you didn’t see any error in the logs. Which logs have you checked? There could be useful information in the logs for your Load balancer, Fleet server or the host’s osquery logs.

i am checking the logs on host machine:

Okay. Anything happening in the Fleet logs to either indicate an error in that end or confirm that traffic is making it through to Fleet?

@Kathy Satterlee which logs do i check like the path of the fleet logs? i’ll have to login to fleet server for this right? FYI i am running it on ECS so not sure how to login

The Fleet logs will likely be in Cloudwatch

Hey @Kathy Satterlee i got these logs and seems to be auth issues. missing node keys & no matching secret found.

So sorry, @Ojas. I missed that update!!

i had this error on all the hosts @Kathy Satterlee. All hosts stopped reporting back all of sudden. I updated fleet to latest and then i can see most hosts online now. But still i see many hosts which i know are running but on fleet they are offline. i dont see any identifiers in agent options so it should be whatever default value is

Yes, I just wanted to make sure that the errors in the logs were all related to the same host. If you were using something that could be duplicated as the identifier, that might explain the other issue. If not, the errors in the Fleet log are likely unrelated. Can you run

fleetctl get config --include-server-config

and share the results? You can either redact sensitive information or send it to me via DM.

I am working on updating the whole agents and things after that i’ll fetch this and share with you 🙂