Hey friends I have a bunch of devices that enroll perfectly osquery #fleet

Hey friends! I have a bunch of devices that enroll...

pvirani

02/08/2022, 7:11 PM

Hey friends! I have a bunch of devices that enroll perfectly fine in Fleet but somehow appear offline right after the enrolment and I can run no queries on them. How do I go about debugging it? 🤔

pvirani

02/08/2022, 7:15 PM

Ran

sudo systemctl status osqueryd

Copy code

osqueryd.service - The osquery Daemon
   Loaded: loaded (/usr/lib/systemd/system/osqueryd.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2022-02-07 20:46:41 UTC; 22h ago
  Process: 11556 ExecStartPre=/bin/sh -c if [ -f $LOCAL_PIDFILE ]; then mv $LOCAL_PIDFILE $PIDFILE; fi (code=exited, status=0/SUCCESS)
  Process: 11553 ExecStartPre=/bin/sh -c if [ ! -f $FLAG_FILE ]; then touch $FLAG_FILE; fi (code=exited, status=0/SUCCESS)
 Main PID: 11559 (osqueryd)
    Tasks: 17 (limit: 4915)
   CGroup: /system.slice/osqueryd.service
           ├─11559 /opt/osquery/bin/osqueryd --flagfile /etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf
           └─11579 /opt/osquery/bin/osqueryd

pvirani

02/08/2022, 7:18 PM

Checked Fleet logs which say

Copy code

level=debug ts=2022-02-08T19:05:53.182008489Z component=http user=user@example.com method=POST uri=/api/v1/fleet/targets took=12.652937ms
level=debug ts=2022-02-08T19:05:55.530378807Z component=http user=user@example.com method=POST uri=/api/v1/fleet/queries/run took=26.777372ms sql="select * from os_version;" query_id=73 numHosts=7
level=debug ts=2022-02-08T19:05:58.246882886Z component=http user=user@example.com method=POST uri=/api/v1/fleet/queries/run took=30.401335ms sql="select * from os_version;" query_id=73 numHosts=7
level=debug ts=2022-02-08T19:07:04.954014079Z component=http method=POST uri=/api/v1/osquery/config took=5.646714ms ip_addr=10.80.200.223:62612 x_for_ip_addr=34.221.186.187
level=debug ts=2022-02-08T19:07:22.864051857Z component=http user=user@example.com method=POST uri=/api/v1/fleet/queries/run took=26.078575ms sql="select * from os_version;" query_id=73 numHosts=7
level=debug ts=2022-02-08T19:08:09.788593277Z component=http method=POST uri=/api/v1/osquery/config took=5.616807ms ip_addr=10.80.211.37:55436 x_for_ip_addr=52.33.96.133
level=debug ts=2022-02-08T19:08:56.019325305Z component=http method=POST uri=/api/v1/osquery/config took=4.331758ms ip_addr=10.80.211.37:57986 x_for_ip_addr=44.228.242.168

Tomas Touceda

02/08/2022, 7:42 PM

hi there! online/offline is determined by the last seen time, do you have any debug logs of osqueryd running on the hosts to rule out connection issues?

pvirani

02/08/2022, 8:27 PM

I to have the status as pasted above but there are no osqueryd logs inside

/var/log/osquery/

Tomas Touceda

02/08/2022, 8:33 PM

hm, could you share the contents of /etc/osquery/osquery.conf ?

pvirani

02/08/2022, 8:44 PM

I have

osquery.flags

but no

osquery.conf

zwass

02/08/2022, 8:48 PM

It looks like the command your service is running is

/opt/osquery/bin/osqueryd --flagfile /etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf

-- Is that intended?

pvirani

02/08/2022, 8:57 PM

it's not ... I don't know where that

--config_path /etc/osquery/osquery.conf

flag is coming from

zwass

02/08/2022, 9:12 PM

Probably this is the default systemd unit that the osquery package installs?

pvirani

02/08/2022, 9:14 PM

ooooh

pvirani

02/08/2022, 9:16 PM

yes that would be. Here's a better question in that case ... what's the cleanest way to install and kick off osquery to enroll it in Fleet?

pvirani

02/08/2022, 9:19 PM

I installed osquery like this:

Copy code

export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
sudo apt-key adv --keyserver <hkp://keyserver.ubuntu.com:80> --recv-keys $OSQUERY_KEY
sudo add-apt-repository 'deb [arch=amd64] <https://pkg.osquery.io/deb> deb main'
sudo apt-get update
sudo apt-get install osquery

Then ran the following commands

Copy code

echo "<my-enrollment-secret>" > /etc/osquery/osquery.secret
# Fill osquery.flags.default with stuff
echo "--config_plugin=tls
--enroll_secret_path=/etc/osquery/osquery.secret
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_tls_endpoint=/api/v1/osquery/config
--tls_hostname=fleetdm.segment.build
--config_refresh=300
--config_tls_accelerated_refresh=300
--config_tls_max_attempts=9999" > /etc/osquery/osquery.flags.default
ln -s /etc/osquery/osquery.flags.default /etc/osquery/osquery.flags
sudo systemctl start osqueryd

zwass

02/08/2022, 9:20 PM

You probably want to also write the systemd unit to include the appropriate command to start up osquery.

pvirani

02/08/2022, 9:23 PM

This is my default systemd file what all should I change? Just remove the

config-path

Copy code

$ cat osqueryd.service 
[Unit]
Description=The osquery Daemon
After=network.service syslog.service

[Service]
TimeoutStartSec=0
EnvironmentFile=/etc/default/osqueryd
ExecStartPre=/bin/sh -c "if [ ! -f $FLAG_FILE ]; then touch $FLAG_FILE; fi"
ExecStartPre=/bin/sh -c "if [ -f $LOCAL_PIDFILE ]; then mv $LOCAL_PIDFILE $PIDFILE; fi"
ExecStart=/opt/osquery/bin/osqueryd \
  --flagfile $FLAG_FILE \
  --config_path $CONFIG_FILE
Restart=on-failure
KillMode=control-group
KillSignal=SIGTERM
TimeoutStopSec=15
CPUQuota=20%

[Install]
WantedBy=multi-user.target

zwass

02/08/2022, 11:37 PM

Yes. And make sure

$FLAG_FILE

is set appropriately in

/etc/default/osqueryd

. Or just set the path explicitly in the

ExecStart

pvirani

02/09/2022, 12:04 AM

yep

$FLAG_FILE

is correctly set . Removed the

--config_path $CONFIG_FILE

... still doesn't work 😞

zwass

02/09/2022, 12:09 AM

Can you try running the command

sudo osqueryd --flagfile /replace/with/path --verbose --tls_dump

? The logs should help us see what's wrong.

pvirani

02/09/2022, 4:33 PM

Copy code

$ sudo osqueryd --flagfile /etc/osquery/osquery.flags --verbose --tls_dump
I0209 06:22:05.103773 20100 init.cpp:357] osquery initialized [version=5.1.0]
I0209 06:22:05.128333 20100 system.cpp:348] Found stale process for osqueryd (17563)
I0209 06:22:05.128521 20100 system.cpp:380] Writing osqueryd pid (20100) to /var/run/osqueryd.pidfile
I0209 06:22:05.128660 20100 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load
I0209 06:22:05.128785 20100 dispatcher.cpp:78] Adding new service: WatcherRunner (0x55f14f2505e8) to thread: 140381271172864 (0x55f14f24c5e0) in process 20100
I0209 06:22:05.129822 20101 watcher.cpp:656] osqueryd watcher (20100) executing worker (20102)
I0209 06:22:05.147351 20102 init.cpp:354] osquery worker initialized [watcher=20100]
I0209 06:22:05.147812 20102 dispatcher.cpp:78] Adding new service: WatcherWatcherRunner (0x55d1ad67d488) to thread: 139705997043456 (0x55d1ad67d530) in process 20102
I0209 06:22:05.147922 20102 rocksdb.cpp:132] Opening RocksDB handle: /var/osquery/osquery.db
I0209 06:22:05.177213 20102 dispatcher.cpp:78] Adding new service: ExtensionWatcher (0x55d1ad780e48) to thread: 139705871152896 (0x55d1ad77f650) in process 20102
I0209 06:22:05.177352 20102 dispatcher.cpp:78] Adding new service: ExtensionRunnerCore (0x55d1ad70af68) to thread: 139705879545600 (0x55d1ad6857d0) in process 20102
I0209 06:22:05.177487 20102 auto_constructed_tables.cpp:97] Removing stale ATC entries
I0209 06:22:05.177799 20102 dispatcher.cpp:78] Adding new service: ConfigRefreshRunner (0x55d1ad67e4d8) to thread: 139705887938304 (0x55d1ad705810) in process 20102
I0209 06:22:05.178025 20102 tls.cpp:255] TLS/HTTPS POST request to URI: <https://fleetdm.segment.build/api/v1/osquery/config>
{"node_key":"Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK"}
I0209 06:22:05.179087 20202 interface.cpp:299] Extension manager service starting: /var/osquery/osquery.em
{
  "decorators": {
    "load": [
      "SELECT uuid AS host_uuid FROM system_info;",
      "SELECT hostname AS hostname FROM system_info;"
    ]
  },
  "options": {
    "disable_distributed": false,
    "distributed_interval": 10,
    "distributed_plugin": "tls",
    "distributed_tls_max_attempts": 3,
    "logger_plugin": "tls",
    "logger_tls_endpoint": "/api/v1/osquery/log",
    "logger_tls_period": 10,
    "pack_delimiter": "/"
  },
  "packs": {
    "Global": {
      "queries": {}
    }
  }
}

I0209 06:22:05.277238 20102 smbios_tables.cpp:105] Reading SMBIOS from sysfs DMI node
I0209 06:22:05.278728 20102 smbios_tables.cpp:105] Reading SMBIOS from sysfs DMI node
I0209 06:22:05.279028 20102 dispatcher.cpp:78] Adding new service: TLSLogForwarder (0x55d1ad979cb8) to thread: 139705942390528 (0x55d1ad97a0c0) in process 20102
I0209 06:22:05.279947 20102 eventfactory.cpp:156] Event publisher not enabled: BPFEventPublisher: Publisher disabled via configuration
I0209 06:22:05.280010 20102 eventfactory.cpp:156] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration
I0209 06:22:05.280045 20102 eventfactory.cpp:156] Event publisher not enabled: inotify: Publisher disabled via configuration
I0209 06:22:05.280078 20102 eventfactory.cpp:156] Event publisher not enabled: syslog: Publisher disabled via configuration
I0209 06:22:05.280200 20102 events.cpp:70] Skipping subscriber: apparmor_events: Subscriber disabled via configuration
I0209 06:22:05.280308 20102 events.cpp:70] Skipping subscriber: process_file_events: Subscriber disabled via configuration
I0209 06:22:05.280354 20102 events.cpp:70] Skipping subscriber: seccomp_events: Seccomp subscriber disabled via configuration
I0209 06:22:05.280397 20102 events.cpp:70] Skipping subscriber: selinux_events: Subscriber disabled via configuration
I0209 06:22:05.280439 20102 events.cpp:70] Skipping subscriber: socket_events: Subscriber disabled via configuration
I0209 06:22:05.280640 20102 dispatcher.cpp:78] Adding new service: DistributedRunner (0x55d1ad99c518) to thread: 139705925605120 (0x55d1ad8f7f30) in process 20102
I0209 06:22:05.280720 20102 dispatcher.cpp:78] Adding new service: SchedulerRunner (0x55d1ad9429f8) to thread: 139705917212416 (0x55d1ad9a0c00) in process 20102
I0209 06:22:05.281080 20206 tls.cpp:255] TLS/HTTPS POST request to URI: <https://fleetdm.segment.build>
{"node_key":"Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK"}
I0209 06:22:05.281682 20205 eventfactory.cpp:390] Starting event publisher run loop: udev
I0209 06:22:05.282253 20204 tls.cpp:255] TLS/HTTPS POST request to URI: <https://fleetdm.segment.build/api/v1/osquery/log>
{"data":[{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"init.cpp","line":"354","message":"osquery worker initialized [watcher=20100]","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"smbios_tables.cpp","line":"105","message":"Reading SMBIOS from sysfs DMI node","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"smbios_tables.cpp","line":"105","message":"Reading SMBIOS from sysfs DMI node","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"dispatcher.cpp","line":"78","message":"Adding new service: TLSLogForwarder (0x55d1ad979cb8) to thread: 139705942390528 (0x55d1ad97a0c0) in process 20102","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"dispatcher.cpp","line":"78","message":"Adding new service: WatcherWatcherRunner (0x55d1ad67d488) to thread: 139705997043456 (0x55d1ad67d530) in process 20102","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"rocksdb.cpp","line":"132","message":"Opening RocksDB handle: /var/osquery/osquery.db","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"dispatcher.cpp","line":"78","message":"Adding new service: ExtensionWatcher (0x55d1ad780e48) to thread: 139705871152896 (0x55d1ad77f650) in process 20102","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"dispatcher.cpp","line":"78","message":"Adding new service: ExtensionRunnerCore (0x55d1ad70af68) to thread: 139705879545600 (0x55d1ad6857d0) in process 20102","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"auto_constructed_tables.cpp","line":"97","message":"Removing stale ATC entries","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"dispatcher.cpp","line":"78","message":"Adding new service: ConfigRefreshRunner (0x55d1ad67e4d8) to thread: 139705887938304 (0x55d1ad705810) in process 20102","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"tls.cpp","line":"255","message":"TLS/HTTPS POST request to URI: <https://fleetdm.segment.build/api/v1/osquery/config>","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}},{"hostIdentifier":"ip-10-0-0-180","calendarTime":"Wed Feb  9 06:22:05 2022 UTC","unixTime":"1644387725","severity":"0","filename":"interface.cpp","line":"299","message":"Extension manager service starting: /var/osquery/osquery.em","version":"5.1.0","decorations":{"host_uuid":"ec24182c-6d1d-bb81-7fc0-4fc58d16baf2","hostname":"ip-10-0-0-180.us-west-2.compute.internal"}}],"log_type":"status","node_key":"Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK"}
{}

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <meta name="robots" content="noindex">

    <link rel="stylesheet" type="text/css" href="/assets/bundle-975339cff640dfa0dd9f.css">
    <link rel="shortcut icon" href="/assets/favicon.ico">

    <title>Fleet for osquery</title>
    <script type="text/javascript">
        var urlPrefix = "";
    </script>
  </head>
  <body>
    <div id="app"></div>
    <script async defer src="/assets/bundle-09a7bc98deb21b6cf4d9.js" onload="this.parentElement.removeChild(this)"></script>
    
    <script>document.addEventListener("touchstart", function() {},false);</script>
    
  </body>
</html>

I0209 06:22:06.358170 20206 tls.cpp:255] TLS/HTTPS POST request to URI: <https://fleetdm.segment.build>
{"node_key":"Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK"}
<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <meta name="robots" content="noindex">

    <link rel="stylesheet" type="text/css" href="/assets/bundle-975339cff640dfa0dd9f.css">
    <link rel="shortcut icon" href="/assets/favicon.ico">

    <title>Fleet for osquery</title>
    <script type="text/javascript">
        var urlPrefix = "";
    </script>
  </head>
  <body>
    <div id="app"></div>
    <script async defer src="/assets/bundle-09a7bc98deb21b6cf4d9.js" onload="this.parentElement.removeChild(this)"></script>
    
    <script>document.addEventListener("touchstart", function() {},false);</script>
    
  </body>
</html>

I0209 06:22:10.428839 20206 tls.cpp:255] TLS/HTTPS POST request to URI: <https://fleetdm.segment.build>
{"node_key":"Oh4i+3w+HseyT1ygMJbRsjKh04ie00TK"}
<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <meta name="robots" content="noindex">

    <link rel="stylesheet" type="text/css" href="/assets/bundle-975339cff640dfa0dd9f.css">
    <link rel="shortcut icon" href="/assets/favicon.ico">

    <title>Fleet for osquery</title>
    <script type="text/javascript">
        var urlPrefix = "";
    </script>
  </head>
  <body>
    <div id="app"></div>
    <script async defer src="/assets/bundle-09a7bc98deb21b6cf4d9.js" onload="this.parentElement.removeChild(this)"></script>
    
    <script>document.addEventListener("touchstart", function() {},false);</script>
    
  </body>
</html>

pvirani

02/09/2022, 4:34 PM

Everything starting on the

TLS/HTTPS POST request to URI: <https://fleetdm.segment.build>

line just keeps repeating on and on and on

pvirani

02/09/2022, 4:35 PM

Also ran this without the --verbose flag

Copy code

$ sudo osqueryd --flagfile /etc/osquery/osquery.flags --tls_dump
I0209 06:26:04.775446 25884 eventfactory.cpp:156] Event publisher not enabled: BPFEventPublisher: Publisher disabled via configuration
I0209 06:26:04.775571 25884 eventfactory.cpp:156] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration
I0209 06:26:04.775632 25884 eventfactory.cpp:156] Event publisher not enabled: inotify: Publisher disabled via configuration
I0209 06:26:04.775835 25884 eventfactory.cpp:156] Event publisher not enabled: syslog: Publisher disabled via configuration

zwass

02/09/2022, 4:36 PM

Ah, what's in your flagfile? I bet you are missing some of the paths.

pvirani

02/09/2022, 4:37 PM

Copy code

$ cat osquery.flags
--config_plugin=tls
--enroll_secret_path=/etc/osquery/osquery.secret
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_tls_endpoint=/api/v1/osquery/config
--tls_hostname=fleetdm.segment.build
--config_refresh=300
--config_tls_accelerated_refresh=300
--config_tls_max_attempts=9999

zwass

02/09/2022, 4:38 PM

Ah yes that's the problem

zwass

02/09/2022, 4:40 PM

You're missing some:

Copy code

--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block

pvirani

02/09/2022, 4:46 PM

dang! Adding em now

pvirani

02/09/2022, 4:47 PM

For the future reference, where/how would I have found out about these paths without bothering y'all? 🙏🏽

zwass

02/09/2022, 4:48 PM

https://fleetdm.com/docs/using-fleet/adding-hosts#launching-osqueryd

pvirani

02/09/2022, 4:51 PM

Thanks a lot! always so helpful 🙏🏽

🍻 1

8 Views

Open in Slack

Previous Next