Hey friends I have a bunch of devices that enroll perfectly

Question

Hey friends I have a bunch of devices that enroll perfectly fine in Fleet but somehow appear offline right after the enrolment and I can run no queries on them How do I go about debugging it thinking face

pvirani · Answer

Ran `sudo systemctl status osqueryd` ```osqueryd service The osquery Daemon Loaded loaded usr lib systemd system osqueryd service disabled vendor preset enabled Active active running since Mon 2022 02 07 20 46 41 UTC 22h ago Process 11556 ExecStartPre= bin sh c if f $LOCAL PIDFILE then mv $LOCAL PIDFILE $PIDFILE fi code=exited status=0 SUCCESS Process 11553 ExecStartPre= bin sh c if f $FLAG FILE then touch $FLAG FILE fi code=exited status=0 SUCCESS Main PID 11559 osqueryd Tasks 17 limit 4915 CGroup system slice osqueryd service 11559 opt osquery bin osqueryd flagfile etc osquery osquery flags config path etc osquery osquery conf 11579 opt osquery bin osqueryd```

pvirani · Answer

Checked Fleet logs which say ```level=debug ts=2022 02 08T19 05 53 182008489Z component=http user=user example com method=POST uri= api v1 fleet targets took=12 652937ms level=debug ts=2022 02 08T19 05 55 530378807Z component=http user=user example com method=POST uri= api v1 fleet queries run took=26 777372ms sql= select from os version query id=73 numHosts=7 level=debug ts=2022 02 08T19 05 58 246882886Z component=http user=user example com method=POST uri= api v1 fleet queries run took=30 401335ms sql= select from os version query id=73 numHosts=7 level=debug ts=2022 02 08T19 07 04 954014079Z component=http method=POST uri= api v1 osquery config took=5 646714ms ip addr=10 80 200 223 62612 x for ip addr=34 221 186 187 level=debug ts=2022 02 08T19 07 22 864051857Z component=http user=user example com method=POST uri= api v1 fleet queries run took=26 078575ms sql= select from os version query id=73 numHosts=7 level=debug ts=2022 02 08T19 08 09 788593277Z component=http method=POST uri= api v1 osquery config took=5 616807ms ip addr=10 80 211 37 55436 x for ip addr=52 33 96 133 level=debug ts=2022 02 08T19 08 56 019325305Z component=http method=POST uri= api v1 osquery config took=4 331758ms ip addr=10 80 211 37 57986 x for ip addr=44 228 242 168```

Tomas Touceda · Answer

hi there online offline is determined by the last seen time do you have any debug logs of osqueryd running on the hosts to rule out connection issues

pvirani · Answer

I to have the status as pasted above but there are no osqueryd logs inside ` var log osquery `

Tomas Touceda · Answer

hm could you share the contents of etc osquery osquery conf

pvirani · Answer

I have `osquery flags` but no `osquery conf`

zwass · Answer

It looks like the command your service is running is ` opt osquery bin osqueryd flagfile etc osquery osquery flags config path etc osquery osquery conf` Is that intended

pvirani · Answer

it s not I don t know where that ` config path etc osquery osquery conf` flag is coming from

zwass · Answer

Probably this is the default systemd unit that the osquery package installs

pvirani · Answer

ooooh

pvirani · Answer

yes that would be Here s a better question in that case what s the cleanest way to install and kick off osquery to enroll it in Fleet

pvirani · Answer

I installed osquery like this ```export OSQUERY KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B sudo apt key adv keyserver recv keys $OSQUERY KEY sudo add apt repository deb arch=amd64 deb main sudo apt get update sudo apt get install osquery``` Then ran the following commands ```echo lt my enrollment secret gt gt etc osquery osquery secret Fill osquery flags default with stuff echo config plugin=tls enroll secret path= etc osquery osquery secret enroll tls endpoint= api v1 osquery enroll config tls endpoint= api v1 osquery config tls hostname=fleetdm segment build config refresh=300 config tls accelerated refresh=300 config tls max attempts=9999 gt etc osquery osquery flags default ln s etc osquery osquery flags default etc osquery osquery flags sudo systemctl start osqueryd```

zwass · Answer

You probably want to also write the systemd unit to include the appropriate command to start up osquery

pvirani · Answer

This is my default systemd file what all should I change Just remove the `config path` ```$ cat osqueryd service Unit Description=The osquery Daemon After=network service syslog service Service TimeoutStartSec=0 EnvironmentFile= etc default osqueryd ExecStartPre= bin sh c if f $FLAG FILE then touch $FLAG FILE fi ExecStartPre= bin sh c if f $LOCAL PIDFILE then mv $LOCAL PIDFILE $PIDFILE fi ExecStart= opt osquery bin osqueryd flagfile $FLAG FILE config path $CONFIG FILE Restart=on failure KillMode=control group KillSignal=SIGTERM TimeoutStopSec=15 CPUQuota=20 Install WantedBy=multi user target```

zwass · Answer

Yes And make sure `$FLAG FILE` is set appropriately in ` etc default osqueryd` Or just set the path explicitly in the `ExecStart`

pvirani · Answer

yep `$FLAG FILE` is correctly set Removed the ` config path $CONFIG FILE` still doesn t work disappointed

zwass · Answer

Can you try running the command `sudo osqueryd flagfile replace with path verbose tls dump` The logs should help us see what s wrong

pvirani · Answer

```$ sudo osqueryd flagfile etc osquery osquery flags verbose tls dump I0209 06 22 05 103773 20100 init cpp 357 osquery initialized version=5 1 0 I0209 06 22 05 128333 20100 system cpp 348 Found stale process for osqueryd 17563 I0209 06 22 05 128521 20100 system cpp 380 Writing osqueryd pid 20100 to var run osqueryd pidfile I0209 06 22 05 128660 20100 extensions cpp 453 Could not autoload extensions Cannot open file for reading etc osquery extensions load I0209 06 22 05 128785 20100 dispatcher cpp 78 Adding new service WatcherRunner 0x55f14f2505e8 to thread 140381271172864 0x55f14f24c5e0 in process 20100 I0209 06 22 05 129822 20101 watcher cpp 656 osqueryd watcher 20100 executing worker 20102 I0209 06 22 05 147351 20102 init cpp 354 osquery worker initialized watcher=20100 I0209 06 22 05 147812 20102 dispatcher cpp 78 Adding new service WatcherWatcherRunner 0x55d1ad67d488 to thread 139705997043456 0x55d1ad67d530 in process 20102 I0209 06 22 05 147922 20102 rocksdb cpp 132 Opening RocksDB handle var osquery osquery db I0209 06 22 05 177213 20102 dispatcher cpp 78 Adding new service ExtensionWatcher 0x55d1ad780e48 to thread 139705871152896 0x55d1ad77f650 in process 20102 I0209 06 22 05 177352 20102 dispatcher cpp 78 Adding new service ExtensionRunnerCore 0x55d1ad70af68 to thread 139705879545600 0x55d1ad6857d0 in process 20102 I0209 06 22 05 177487 20102 auto constructed tables cpp 97 Removing stale ATC entries I0209 06 22 05 177799 20102 dispatcher cpp 78 Adding new service ConfigRefreshRunner 0x55d1ad67e4d8 to thread 139705887938304 0x55d1ad705810 in process 20102 I0209 06 22 05 178025 20102 tls cpp 255 TLS HTTPS POST request to URI node key Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK I0209 06 22 05 179087 20202 interface cpp 299 Extension manager service starting var osquery osquery em decorators load SELECT uuid AS host uuid FROM system info SELECT hostname AS hostname FROM system info options disable distributed false distributed interval 10 distributed plugin tls distributed tls max attempts 3 logger plugin tls logger tls endpoint api v1 osquery log logger tls period 10 pack delimiter packs Global queries I0209 06 22 05 277238 20102 smbios tables cpp 105 Reading SMBIOS from sysfs DMI node I0209 06 22 05 278728 20102 smbios tables cpp 105 Reading SMBIOS from sysfs DMI node I0209 06 22 05 279028 20102 dispatcher cpp 78 Adding new service TLSLogForwarder 0x55d1ad979cb8 to thread 139705942390528 0x55d1ad97a0c0 in process 20102 I0209 06 22 05 279947 20102 eventfactory cpp 156 Event publisher not enabled BPFEventPublisher Publisher disabled via configuration I0209 06 22 05 280010 20102 eventfactory cpp 156 Event publisher not enabled auditeventpublisher Publisher disabled via configuration I0209 06 22 05 280045 20102 eventfactory cpp 156 Event publisher not enabled inotify Publisher disabled via configuration I0209 06 22 05 280078 20102 eventfactory cpp 156 Event publisher not enabled syslog Publisher disabled via configuration I0209 06 22 05 280200 20102 events cpp 70 Skipping subscriber apparmor events Subscriber disabled via configuration I0209 06 22 05 280308 20102 events cpp 70 Skipping subscriber process file events Subscriber disabled via configuration I0209 06 22 05 280354 20102 events cpp 70 Skipping subscriber seccomp events Seccomp subscriber disabled via configuration I0209 06 22 05 280397 20102 events cpp 70 Skipping subscriber selinux events Subscriber disabled via configuration I0209 06 22 05 280439 20102 events cpp 70 Skipping subscriber socket events Subscriber disabled via configuration I0209 06 22 05 280640 20102 dispatcher cpp 78 Adding new service DistributedRunner 0x55d1ad99c518 to thread 139705925605120 0x55d1ad8f7f30 in process 20102 I0209 06 22 05 280720 20102 dispatcher cpp 78 Adding new service SchedulerRunner 0x55d1ad9429f8 to thread 139705917212416 0x55d1ad9a0c00 in process 20102 I0209 06 22 05 281080 20206 tls cpp 255 TLS HTTPS POST request to URI node key Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK I0209 06 22 05 281682 20205 eventfactory cpp 390 Starting event publisher run loop udev I0209 06 22 05 282253 20204 tls cpp 255 TLS HTTPS POST request to URI data hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename init cpp line 354 message osquery worker initialized watcher=20100 version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename smbios tables cpp line 105 message Reading SMBIOS from sysfs DMI node version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename smbios tables cpp line 105 message Reading SMBIOS from sysfs DMI node version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename dispatcher cpp line 78 message Adding new service TLSLogForwarder 0x55d1ad979cb8 to thread 139705942390528 0x55d1ad97a0c0 in process 20102 version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename dispatcher cpp line 78 message Adding new service WatcherWatcherRunner 0x55d1ad67d488 to thread 139705997043456 0x55d1ad67d530 in process 20102 version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename rocksdb cpp line 132 message Opening RocksDB handle var osquery osquery db version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename dispatcher cpp line 78 message Adding new service ExtensionWatcher 0x55d1ad780e48 to thread 139705871152896 0x55d1ad77f650 in process 20102 version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename dispatcher cpp line 78 message Adding new service ExtensionRunnerCore 0x55d1ad70af68 to thread 139705879545600 0x55d1ad6857d0 in process 20102 version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename auto constructed tables cpp line 97 message Removing stale ATC entries version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename dispatcher cpp line 78 message Adding new service ConfigRefreshRunner 0x55d1ad67e4d8 to thread 139705887938304 0x55d1ad705810 in process 20102 version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename tls cpp line 255 message TLS HTTPS POST request to URI version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal hostIdentifier ip 10 0 0 180 calendarTime Wed Feb 9 06 22 05 2022 UTC unixTime 1644387725 severity 0 filename interface cpp line 299 message Extension manager service starting var osquery osquery em version 5 1 0 decorations host uuid ec24182c 6d1d bb81 7fc0 4fc58d16baf2 hostname ip 10 0 0 180 us west 2 compute internal log type status node key Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK < DOCTYPE html> lt html gt lt head gt lt meta charset= UTF 8 gt lt meta name= viewport content= width=device width initial scale=1 gt lt meta name= robots content= noindex gt lt link rel= stylesheet type= text css href= assets bundle 975339cff640dfa0dd9f css gt lt link rel= shortcut icon href= assets favicon ico gt lt title gt Fleet for osquery lt title gt lt script type= text javascript gt var urlPrefix = lt script gt lt head gt lt body gt lt div id= app gt lt div gt lt script async defer src= assets bundle 09a7bc98deb21b6cf4d9 js onload= this parentElement removeChild this gt lt script gt lt script gt document addEventListener touchstart function false lt script gt lt body gt lt html gt I0209 06 22 06 358170 20206 tls cpp 255 TLS HTTPS POST request to URI node key Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK < DOCTYPE html> lt html gt lt head gt lt meta charset= UTF 8 gt lt meta name= viewport content= width=device width initial scale=1 gt lt meta name= robots content= noindex gt lt link rel= stylesheet type= text css href= assets bundle 975339cff640dfa0dd9f css gt lt link rel= shortcut icon href= assets favicon ico gt lt title gt Fleet for osquery lt title gt lt script type= text javascript gt var urlPrefix = lt script gt lt head gt lt body gt lt div id= app gt lt div gt lt script async defer src= assets bundle 09a7bc98deb21b6cf4d9 js onload= this parentElement removeChild this gt lt script gt lt script gt document addEventListener touchstart function false lt script gt lt body gt lt html gt I0209 06 22 10 428839 20206 tls cpp 255 TLS HTTPS POST request to URI node key Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK < DOCTYPE html> lt html gt lt head gt lt meta charset= UTF 8 gt lt meta name= viewport content= width=device width initial scale=1 gt lt meta name= robots content= noindex gt lt link rel= stylesheet type= text css href= assets bundle 975339cff640dfa0dd9f css gt lt link rel= shortcut icon href= assets favicon ico gt lt title gt Fleet for osquery lt title gt lt script type= text javascript gt var urlPrefix = lt script gt lt head gt lt body gt lt div id= app gt lt div gt lt script async defer src= assets bundle 09a7bc98deb21b6cf4d9 js onload= this parentElement removeChild this gt lt script gt lt script gt document addEventListener touchstart function false lt script gt lt body gt lt html gt ```

pvirani · Answer

Everything starting on the `TLS HTTPS POST request to URI <https fleetdm segment build>` line just keeps repeating on and on and on

pvirani · Answer

Also ran this without the verbose flag ```$ sudo osqueryd flagfile etc osquery osquery flags tls dump I0209 06 26 04 775446 25884 eventfactory cpp 156 Event publisher not enabled BPFEventPublisher Publisher disabled via configuration I0209 06 26 04 775571 25884 eventfactory cpp 156 Event publisher not enabled auditeventpublisher Publisher disabled via configuration I0209 06 26 04 775632 25884 eventfactory cpp 156 Event publisher not enabled inotify Publisher disabled via configuration I0209 06 26 04 775835 25884 eventfactory cpp 156 Event publisher not enabled syslog Publisher disabled via configuration```

zwass · Answer

Ah what s in your flagfile I bet you are missing some of the paths

pvirani · Answer

```$ cat osquery flags config plugin=tls enroll secret path= etc osquery osquery secret enroll tls endpoint= api v1 osquery enroll config tls endpoint= api v1 osquery config tls hostname=fleetdm segment build config refresh=300 config tls accelerated refresh=300 config tls max attempts=9999```

zwass · Answer

Ah yes that s the problem

zwass · Answer

You re missing some ``` distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block```

pvirani · Answer

dang Adding em now

pvirani · Answer

For the future reference where how would I have found out about these paths without bothering y all pray skin tone 4

zwass · Answer

pvirani · Answer

Thanks a lot always so helpful pray skin tone 4