Title
#fleet
z

zhong

01/11/2022, 6:29 PM
Hi folks, looking for some help with connectivity to fleet. I currently have a few RHEL hosts that are up on fleet. Some are connected and show that they're online, but others will periodically show as offline. I've tried deleting them from fleet and reinstalling my 'fleet-osquery' package to see if it fixes the issue and it seems like it does but it is not permanent as those same hosts will show offline a few hours later. I've connected these hosts to a domain controller so I edited
/etc/resolv.conf
with the domain nameserver IP and edited
/etc/NetworkManager/NetworkManager.conf
so that the changes are on
/resolv.conf
stay the same even after reboot. I've also disabled
SELINUX
because it was not letting the hosts connect. Currently am stumped on what could be blocking the connection or why the hosts go offline. Any help on what could be the issue?
zwass

zwass

01/11/2022, 6:32 PM
Are you able to SSH onto one of the effected hosts when it goes offline? Can you
curl
the Fleet server at that time?
z

zhong

01/11/2022, 6:34 PM
i can still SSH into them and when i
curl
 the Fleet server it returns HTML
zwass

zwass

01/11/2022, 6:36 PM
Anything in
/var/log/osquery
?
Lucas Rodriguez

Lucas Rodriguez

01/11/2022, 6:36 PM
Also try checking syslog messages in the host (
/var/log/messages
IIRC
z

zhong

01/11/2022, 6:38 PM
/var/log/osquery
is empty and i do not see
messages
in
/var/log
zwass

zwass

01/11/2022, 6:39 PM
systemctl status orbit.service
?
z

zhong

01/11/2022, 6:41 PM
shows that it is active and running
zwass

zwass

01/11/2022, 6:43 PM
Can you edit
/usr/lib/systemd/system/orbit.service
to add
--debug
to the
orbit
command and then reload+restart the service?
z

zhong

01/11/2022, 6:47 PM
sorry, still a bit new to fleet/osquery, where in orbit.servicewould i add
--debug
?
zwass

zwass

01/11/2022, 6:53 PM
Can you show the contents of that file?
6:54 PM
(I don't have a Linux box up at the moment to reference easily)
z

zhong

01/11/2022, 6:55 PM
here are the contents, i appreciate all the help so far 😄
zwass

zwass

01/11/2022, 7:03 PM
Add
--debug
at the end of the
ExecStart
line please.
7:03 PM
Then
sudo systemctl daemon-reload && sudo systemctl restart orbit.service
z

zhong

01/11/2022, 7:06 PM
doing that brought the host back online
7:09 PM
mind if i ask what adding
--debug
did for orbit?
zwass

zwass

01/11/2022, 7:09 PM
Can you check that logs are more verbose in
systemctl status orbit.service
?
7:09 PM
It just turned on more verbose logging. It was probably restarting the orbit/osquery process that brought it back online.
7:10 PM
If we have more verbose logs now, hopefully we can determine what the issue is when it goes offline again.
7:10 PM
I suspect
systemctl restart orbit.service
without any other changes would have temporarily "fixed" it because that would have restarted the processes.
z

zhong

01/11/2022, 7:10 PM
ah i see, will those logs be in
/var/log/orbit
?
zwass

zwass

01/11/2022, 7:14 PM
Yes I think so.
z

zhong

01/11/2022, 7:16 PM
awesome, I will be keeping an eye on the host and see when it goes offline again and update here. Thank you for the help!
zwass

zwass

01/11/2022, 7:25 PM
Thank you!
z

zhong

01/12/2022, 2:57 PM
None of the hosts that had the logging turned on have gone offline since yesterday 🤦‍♂️. Still waiting on them! Anything else I could check that could be the issue in the meantime?
zwass

zwass

01/12/2022, 4:41 PM
I am not sure what else we could check with them currently working as expected. Let us know if they go bad again.
z

zhong

01/12/2022, 5:35 PM
will do!