https://github.com/osquery/osquery logo
Title
o

Ojas

01/31/2023, 12:08 PM
Hey Team, I am seeing 0 online devices whereas everything was fine until few hours ago. I dont see any err in logs or any error that its not able to communicate with fleet server. Also for alot of devices it still shows last fetched a minute ago and thats keeps updating. Not able to run any query though
r

Rachel Perkins

01/31/2023, 3:37 PM
The second part is interesting, so for some specific hosts, you can see a recent
updated_at
key returned from the API but status is still
offline
? Are you able to refetch that host's information and get new update host information?
o

Ojas

01/31/2023, 4:32 PM
@Rachel Perkins nope refetch dosent work nor does the query
dose this log mean that netskope is not allowing the connection? it says provider rejected
k

Kathy Satterlee

01/31/2023, 5:34 PM
Is Netskope a recent addition to your environment? If you're using a proxy server, do you see any logs there that indicate that connections are being refused and why?
o

Ojas

02/01/2023, 4:24 AM
nah! netkope is old one @Kathy Satterlee its been there for many months and was there when we deployed fleet.
@Kathy Satterlee @Rachel Perkins this is becoming a p1 issue for us as our crucial part of vuln management is dependent on fleet side. Any thing i should do to debug this?
k

Kathy Satterlee

02/01/2023, 3:18 PM
I know you said you didn’t see any error in the logs. Which logs have you checked? There could be useful information in the logs for your Load balancer, Fleet server or the host’s osquery logs.
o

Ojas

02/01/2023, 4:43 PM
i am checking the logs on host machine:
service is running and everything seems fine, i dont see any error logs in the machine
k

Kathy Satterlee

02/01/2023, 5:10 PM
Okay. Anything happening in the Fleet logs to either indicate an error in that end or confirm that traffic is making it through to Fleet?
o

Ojas

02/01/2023, 7:13 PM
@Kathy Satterlee which logs do i check like the path of the fleet logs? i’ll have to login to fleet server for this right? FYI i am running it on ECS so not sure how to login
k

Kathy Satterlee

02/01/2023, 7:17 PM
The Fleet logs will likely be in Cloudwatch
o

Ojas

02/02/2023, 10:10 AM
Hey @Kathy Satterlee i got these logs and seems to be auth issues. missing node keys & no matching secret found.
@Kathy Satterlee any idea on how to fix this? I dont see any other error logs in there
k

Kathy Satterlee

02/08/2023, 4:27 PM
So sorry, @Ojas. I missed that update!!
It sounds like all of those errors are related to one host, with an invalid enroll secret. Just to be sure though, what are you using as your identifier?
o

Ojas

02/08/2023, 6:52 PM
i had this error on all the hosts @Kathy Satterlee. All hosts stopped reporting back all of sudden. I updated fleet to latest and then i can see most hosts online now. But still i see many hosts which i know are running but on fleet they are offline. i dont see any identifiers in agent options so it should be whatever default value is
k

Kathy Satterlee

02/08/2023, 7:03 PM
Yes, I just wanted to make sure that the errors in the logs were all related to the same host. If you were using something that could be duplicated as the identifier, that might explain the other issue. If not, the errors in the Fleet log are likely unrelated. Can you run
fleetctl get config --include-server-config
and share the results? You can either redact sensitive information or send it to me via DM.
o

Ojas

02/09/2023, 11:09 AM
I am working on updating the whole agents and things after that i’ll fetch this and share with you 🙂