Title
#fleet
m

MarkMurdock

12/16/2021, 7:10 PM
I've got a situation where some Windows hosts (typically the same hosts) are frequently offline. They are registering fine, and if I restart osqueryd (5.0.1) they show online for a while but eventually go offline. I can confirm that the osqueryd process is still running on those hosts, and that they are still periodically communicating with the FleetDM server (4.6.2) on port 8080/TLS. For example, here is a tcpdump output showing the server is checking in every 15 seconds, despite showing offline in the FleetDM UI:
user@server:/usr/local/build# tcpdump -n -i ens160 -s 1514  host 10.126.3.32
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 1514 bytes
19:08:01.249229 IP 10.126.3.32.56300 > 10.126.3.149.8080: Flags [.], seq 668286310:668286311, ack 3209453802, win 2052, length 1: HTTP
19:08:01.249250 IP 10.126.3.149.8080 > 10.126.3.32.56300: Flags [.], ack 1, win 501, options [nop,nop,sack 1 {0:1}], length 0
19:08:16.260289 IP 10.126.3.32.56300 > 10.126.3.149.8080: Flags [.], seq 0:1, ack 1, win 2052, length 1: HTTP
19:08:16.260308 IP 10.126.3.149.8080 > 10.126.3.32.56300: Flags [.], ack 1, win 501, options [nop,nop,sack 1 {0:1}], length 0
19:08:31.269602 IP 10.126.3.32.56300 > 10.126.3.149.8080: Flags [.], seq 0:1, ack 1, win 2052, length 1: HTTP
19:08:31.269621 IP 10.126.3.149.8080 > 10.126.3.32.56300: Flags [.], ack 1, win 501, options [nop,nop,sack 1 {0:1}], length 0
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
Benjamin Edwards

Benjamin Edwards

12/16/2021, 8:26 PM
is this only windows hosts? are you running osqueryd (standalone) or
fleet osquery
(via fleetctl package) unfortunately on windows services don't get logged well. If you can capture any logs that would be helpful. I usually capture logs by running it manually via Powershell:
"C:\Program Files\Orbit\bin\orbit\orbit.exe" --root-dir "C:\Program Files\Orbit\." --log-file "C:\Program Files\Orbit\orbit-log.txt" --fleet-url "<https://dogfood.fleetdm.com>"  --enroll-secret-path "C:\Program Files\Orbit\secret.txt"  --update-url "<https://tuf.fleetctl.com>"  --orbit-channel "stable" --osqueryd-channel "stable"
for example, and then it'll log to stdout
m

MarkMurdock

12/16/2021, 8:34 PM
I'm running it using the Kolide Launcher, which makes logging even a little more challenging, I believe.
8:35 PM
And yes this would only be Windows hosts.
8:36 PM
But I could probably arrange running osqueryd manually (not through Launcher or a service) on one of those systems and obtain the logs. It's probably a step I'm going to have to take to see if that gives me any more information.
Benjamin Edwards

Benjamin Edwards

12/16/2021, 8:49 PM
yeah that would be the most helpful.
m

MarkMurdock

12/17/2021, 12:57 AM
So I ran osqueryd from this host and captured a log file (attached). During the entire time this was running, the host never appeared Online in FleetDM. But, from the log, it appears that it is communicating with the FleetDM server successfully.
12:58 AM
This was a relatively short run, but I ran a longer run and it did not show Online either. Something unusual is definitely going on here.