https://github.com/osquery/osquery logo
Title
p

pvirani

01/11/2023, 8:04 PM
Hello helpful folks! I'm running into a super weird issue and not sure where the breakdown maybe. I have a test host enrolled into Fleet. ā€¢ The scheduled queries are running perfectly fine on the host but the results don't show up in
/var/log/osquery/osqueryd.results.log
on the host. see the example log line from it below
{
  "name": "pack/osquery_monitoring/schedule",
  "hostIdentifier": "ip-172-31-55-24",
  "calendarTime": "Wed Jan 11 19:46:37 2023 UTC",
  "unixTime": 1673466397,
  "epoch": 0,
  "counter": 0,
  "numerics": false,
  "decorations": {
    "host_uuid": "ec2fb435-c38c-9e86-d043-106a4c7ec832",
    "hostname": "ip-172-31-55-24.us-west-2.compute.internal"
  },
  "columns": {
    "average_memory": "0",
    "avg_system_time": "",
    "avg_user_time": "",
    "executions": "0",
    "interval": "86400",
    "last_executed": "0",
    "name": "pack/it_compliance/iptables",
    "output_size": "0",
    "wall_time": "0"
  },
  "action": "added"
}
ā€¢ On the Fleet Webserver
cat osquery_status -f
shows the last query execution to have happened in August 2022 šŸ™€
{
  "hostIdentifier": "ip-172-31-23-127",
  "calendarTime": "Fri Aug 26 18:54:26 2022 UTC",
  "unixTime": "1661540066",
  "severity": "2",
  "filename": "scheduler.cpp",
  "line": "118",
  "message": "Error executing scheduled query pack/osx_attacks/OSX_Mughthesec: no such table: launchd",
  "version": "5.2.2",
  "decorations": {
    "host_uuid": "EC2C455A-F709-B975-ADA7-C33DE9F7EABA",
    "hostname": "ip-172-31-23-127.us-west-2.compute.internal"
  }
}
Where do I even begin debugging?
k

Kathy Satterlee

01/11/2023, 8:28 PM
Hey @pvirani! Can you verify the logger plugin by running this query against that test host;
SELECT value FROM osquery_flags WHERE name = "logger_plugin";
p

pvirani

01/11/2023, 8:33 PM
hostname value ip-172-31-55-24.us-west-2.compute.internal filesystem
^ @Kathy Satterlee
that also sounds wrong because when I launched the webserver I'd spun it up with the following flags
- name: FLEET_OSQUERY_RESULT_LOG_PLUGIN
              value: "firehose"
            - name: FLEET_FIREHOSE_REGION
              value: "us-west-2"
            - name: FLEET_FIREHOSE_RESULT_STREAM
              value: "fleetdm-results-stream-prod"
k

Kathy Satterlee

01/11/2023, 8:43 PM
Are you using Orbit or vanilla osquery?
p

pvirani

01/11/2023, 8:45 PM
Vanilla
This is how I'm spinning osquery up on my hosts
#!/usr/bin/env bash

# DUO
# changed due to : <https://help.duo.com/s/article/5503?language=en_US>
# curl -sSL <https://duo.com/APT-GPG-KEY-DUO> | apt-key add -
curl -sSL <https://duo.com/DUO-GPG-PUBLIC-KEY.asc> | apt-key add -

# OSQUERY
apt-key adv --keyserver <http://keyserver.ubuntu.com|keyserver.ubuntu.com> --recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
apt-key adv --keyserver <http://keyserver.ubuntu.com|keyserver.ubuntu.com> --recv-keys 33EE313BAD9589B7
add-apt-repository "deb [arch=amd64] <https://osquery-packages.s3.amazonaws.com/deb> deb main"
apt install osquery

# OSQUERY
rm -f /etc/osquery/osquery.secret
rm -f /etc/osquery/osquery.flags
rm -f /etc/osquery/osquery.flags.default

# Fill osquery.secret with stuff
# Note, this is not actually sensitive and does not need to be any more secret
echo "enrollment_secret" > /etc/osquery/osquery.secret

# Fill osquery.flags.default with stuff
echo "--config_plugin=tls
--enroll_secret_path=/etc/osquery/osquery.secret
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_tls_endpoint=/api/v1/osquery/config
--tls_hostname=<http://fleetdm.segment.com|fleetdm.segment.com>
--config_refresh=300
--config_tls_accelerated_refresh=300
--config_tls_max_attempts=9999
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block" > /etc/osquery/osquery.flags.default

ln -s /etc/osquery/osquery.flags.default /etc/osquery/osquery.flags
z

zwass

01/11/2023, 8:51 PM
Looks like your osquery is configured to log to the local filesystem and not the Fleet server?
p

pvirani

01/11/2023, 8:52 PM
How do I specify it should log to the Fleet server? This is the result of
sudo systemctl start osqueryd
so looks like I gotta specify it in the flagfile?
osqueryd.service - The osquery Daemon
   Loaded: loaded (/usr/lib/systemd/system/osqueryd.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2023-01-11 18:18:46 UTC; 2h 29min ago
  Process: 29213 ExecStartPre=/bin/sh -c if [ -f $LOCAL_PIDFILE ]; then mv $LOCAL_PIDFILE $PIDFILE; fi (code=exited, status=0/SUCCESS)
  Process: 29206 ExecStartPre=/bin/sh -c if [ ! -f $FLAG_FILE ]; then touch $FLAG_FILE; fi (code=exited, status=0/SUCCESS)
 Main PID: 29215 (osqueryd)
    Tasks: 15 (limit: 4561)
   CGroup: /system.slice/osqueryd.service
           ā”œā”€29215 /opt/osquery/bin/osqueryd --flagfile /etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf
           ā””ā”€29646 /opt/osquery/bin/osqueryd

Jan 11 19:38:54 ip-172-31-55-24 osqueryd[29215]: W0111 19:38:54.303018 29680 options.cpp:106] The CLI only flag --logger_plugin set via config file will be ignored, please use a flagfile or pass it to the process at startup
Jan 11 19:38:54 ip-172-31-55-24 osqueryd[29215]: I0111 19:38:54.356914 29680 eventfactory.cpp:352] The minimum events expiration timeout for hardware_events has been adjusted: 21660
Jan 11 19:38:54 ip-172-31-55-24 osqueryd[29215]: I0111 19:38:54.356961 29680 config.cpp:908] Calling configure for logger filesystem
Jan 11 19:43:54 ip-172-31-55-24 osqueryd[29215]: W0111 19:43:54.380683 29680 options.cpp:106] The CLI only flag --logger_plugin set via config file will be ignored, please use a flagfile or pass it to the process at startup
Jan 11 19:43:54 ip-172-31-55-24 osqueryd[29215]: I0111 19:43:54.455612 29680 eventfactory.cpp:352] The minimum events expiration timeout for hardware_events has been adjusted: 21660
Jan 11 19:43:54 ip-172-31-55-24 osqueryd[29215]: I0111 19:43:54.455659 29680 config.cpp:908] Calling configure for logger filesystem
Jan 11 19:46:37 ip-172-31-55-24 osqueryd[29215]: I0111 19:46:37.928295 29684 query.cpp:102] Storing initial results for new scheduled query: pack/osquery_monitoring/schedule
Jan 11 20:17:12 ip-172-31-55-24 osqueryd[29215]: I0111 20:17:12.569401 29684 query.cpp:102] Storing initial results for new scheduled query: pack/hardware_monitoring/pci_devices
Jan 11 20:19:34 ip-172-31-55-24 osqueryd[29215]: I0111 20:19:34.316332 29684 query.cpp:102] Storing initial results for new scheduled query: pack/incident_response/open_sockets
Jan 11 20:27:22 ip-172-31-55-24 osqueryd[29215]: I0111 20:27:22.376417 29684 query.cpp:102] Storing initial results for new scheduled query: pack/it_compliance/usb_devices
k

Kathy Satterlee

01/11/2023, 8:52 PM
--logger_plugin=tls 
 --logger_tls_endpoint=/api/v1/osquery/log 
 --logger_tls_period=10
It looks like you're missing those flags in osquery that will send logs to the Fleet server. Once they hit Fleet, the
FLEET_OSQUERY_RESULT_LOG_PLUGIN
comes in to play.
p

pvirani

01/11/2023, 8:54 PM
adding em back in
the image will be ready in 22 mins. I'll report back with the status then. Thanks folks! Definitely pinning https://osquery.readthedocs.io/en/stable/installation/cli-flags/ to my brower so I don't have to distrub y'all with stupid questions again šŸ™‡ā€ā™€ļøšŸ»
k

Kathy Satterlee

01/11/2023, 9:01 PM
Always happy to help! And you never know... this might help someone else down the road as well. Thanks for reaching out.
Just let me know if that doesn't do the trick.
p

pvirani

01/11/2023, 9:03 PM
Oh for sure! I've benefitted so much from keyword search results yielding other people's posts šŸ˜„
it worked! Thanks a bunch šŸ™‚
k

Kathy Satterlee

01/11/2023, 11:44 PM
ā˜ļø šŸš€
c

clong

01/12/2023, 6:35 PM
thanks for this thread, it just help me out too šŸ™‚