hi all, i recently ran a couple major version upgrades of Fleet to 4.35.2. we have some linux cloud hosts that were previously running older versions of osquery (3.x), have upgraded to 5.9.1 via

yum

(both

osqueryd --version

and

osqueryi --version

report 5.9.1), but they are still reporting as 3.3 in Fleet. we also had mac hosts running 3.x, but they upgraded and report to Fleet just fine. I’ve exhausted a few routes of troubleshooting, tried

osqueryctl stop

and

osqueryctl start

several times, checked the flags file, confirmed there are no duplicate processes running, but can’t figure out why Fleet is not reporting the right version or where the disconnect is. wondering anyone has ideas for where to look next or any insight into a major version osquery upgrade on linux.

Are these hosts showing up as online in Fleet?

when i search for them directly by hostname in the live query menu, they show up as online, but the live query does not work. they do not show up in the hosts management page

Can you share the osquery status logs as well as the flags set on one of those hosts? It sounds like there may have been a change to the distributed endpoints since your previous version.

$ cat <http://osqueryd.INFO|osqueryd.INFO>
Log file created at: 2023/09/06 18:54:32
Running on machine: <http://jthiel.vm.dev.etsycloud.com|jthiel.vm.dev.etsycloud.com>
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0906 18:54:32.056210 28101 eventfactory.cpp:156] Event publisher not enabled: BPFEventPublisher: Publisher disabled via configuration
I0906 18:54:32.056285 28101 eventfactory.cpp:156] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration
I0906 18:54:32.056293 28101 eventfactory.cpp:156] Event publisher not enabled: inotify: Publisher disabled via configuration
I0906 18:54:32.056300 28101 eventfactory.cpp:156] Event publisher not enabled: syslog: Publisher disabled via configuration

flags:

$ cat /etc/osquery/osquery.flags
# Use this comment to trigger restart - 1
--config_plugin=tls,filesystem
--config_tls_endpoint=/api/v1/osquery/config
--config_tls_refresh=10
--enroll_tls_endpoint=/api/v1/osquery/enroll
--enroll_secret_path=<redacted>
--host_identifier=hostname
--tls_server_certs=/var/osquery/certs/certs.pem
--tls_hostname=<redacted>

Thanks! It looks like you need to update the flags a bit. There are a couple of flags you need for distributed read and write, and the config and enroll endpoints have changed. Here's an example:

--enroll_secret_path=/etc/osquery/enroll_secret 
 --tls_server_certs=/etc/osquery/fleet.crt
 --tls_hostname=<http://fleet.example.com|fleet.example.com> 
 --host_identifier=uuid 
 --enroll_tls_endpoint=/api/osquery/enroll 
 --config_plugin=tls 
 --config_tls_endpoint=/api/osquery/config 
 --config_refresh=10 
 --disable_distributed=false
 --distributed_plugin=tls 
 --distributed_interval=10 
 --distributed_tls_max_attempts=3 
 --distributed_tls_read_endpoint=/api/osquery/distributed/read 
 --distributed_tls_write_endpoint=/api/osquery/distributed/write 
 --logger_plugin=tls 
 --logger_tls_endpoint=/api/osquery/log 
 --logger_tls_period=10

aha! thank you! i will try updating the flags

osqueryctl

is generally used to run osquery when you're testing out new configs, but it's a valid way to start things up so I don't see any likely issues with it. That being said, I'd generally run it as a system service just because it's more the default.

hey @Kathy Satterlee sorry for ghosting this thread, i took a long weekend! and i appreciate your help immensely. i tried restarting osqueryd with that updated flag file, but now it won’t start up at all

Sep 11 17:32:55 osqueryd[3922]: ERROR: illegal value '10 ' specified for uint64 flag 'config_refresh'
Sep 11 17:32:55 osqueryd[3922]: ERROR: illegal value '10 ' specified for uint64 flag 'distributed_interval'
Sep 11 17:32:55 osqueryd[3922]: ERROR: illegal value '3 ' specified for uint64 flag 'distributed_tls_max_attempts'
Sep 11 17:32:55 osqueryd[3922]: ERROR: illegal value '10# Use this comment to trigger restart - 1' specified for uint64 flag 'logger_tls_period'