Manish
04/06/2022, 6:09 AMTomas Touceda
04/06/2022, 12:38 PMLucas Rodriguez
04/06/2022, 1:16 PMC:\Windows\system32\config\systemprofile\AppData\Local\FleetDM\Orbit\Logs\orbit-osquery.log
orbit.exe
to orbit.exe.old
, then a second rename (2) from the new downloaded update tmpFile
file to orbit.exe
.
AFAICS,
• if rename (1) fails and the rename is not atomic (we should check if that could happen) then orbit.exe
could be missing...
• if rename (2) fails there should be a orbit.exe.old
in the bin/orbit/windows/stable
directory.
Any particular file system setup on these hosts? Do all hosts failed the same way? (missing orbit.exe
)Manish
04/06/2022, 5:30 PMbin/orbit/windows
itself was missing and in the other orbit.exe
was missing from the bin/orbit/windows/stable
folder as seen in the picture. AFAIK, most of the machines are offline due to same issue (unable to start the service, cannot find the file specified
) I am not sure but in some systems they have network mounted drives, but I hope not C drive, could that be an issue? I do not have direct access to the machines, but I can check things if required via some other person. If you could guide me with most probable causes, I will check them off the list to get to the root cause. I can of course uninstall Fleet osquery via puppet in all machines and re-install them via puppet to fix the problem for time being but that might not work if same problem crops up again.Lucas Rodriguez
04/08/2022, 11:27 AMHi Lucas, I didn't see any error that seemed related to orbit, most errors seemed related to the queries as seen in the above screenshot. (will check once more).OK, let us know if you find any ERR logs, it would help us troubleshoot.
There are about 1000 windows nodes enrolled and about 400 are offline nowIs there any way you can inspect Windows Event logs (Maybe the file deletion events show up there.)
I am not sure but in some systems they have network mounted drives, but I hope not C drive, could that be an issue?We don't know. Something worth checking though.
I can of course uninstall Fleet osquery via puppet in all machines and re-install them via puppet to fix the problem for time being but that might not work if same problem crops up again.I would suggest doing this on a couple of hosts and start monitoring for the issue. PS: I've searched our issues and didn't find anything related. Please keep us posted.
Manish
04/11/2022, 11:47 AMLucas Rodriguez
04/11/2022, 1:13 PMadding an exception in the kaspersky?For now yes. But I'll be creating an issue to investigate why this is happening. I'll link it here so that you can add all the information you have there.
zwass
Manish
04/12/2022, 6:08 PMzwass