https://github.com/osquery/osquery logo
Join Slack
Powered by
Hello helpful folks! I'm running into a super weir...
# fleet
p
Hello helpful folks! I'm running into a super weird issue and not sure where the breakdown maybe. I have a test host enrolled into Fleet. ā€¢ The scheduled queries are running perfectly fine on the host but the results don't show up in 
/var/log/osquery/osqueryd.results.log
on the host. see the example log line from it below
Copy code
{
  "name": "pack/osquery_monitoring/schedule",
  "hostIdentifier": "ip-172-31-55-24",
  "calendarTime": "Wed Jan 11 19:46:37 2023 UTC",
  "unixTime": 1673466397,
  "epoch": 0,
  "counter": 0,
  "numerics": false,
  "decorations": {
    "host_uuid": "ec2fb435-c38c-9e86-d043-106a4c7ec832",
    "hostname": "ip-172-31-55-24.us-west-2.compute.internal"
  },
  "columns": {
    "average_memory": "0",
    "avg_system_time": "",
    "avg_user_time": "",
    "executions": "0",
    "interval": "86400",
    "last_executed": "0",
    "name": "pack/it_compliance/iptables",
    "output_size": "0",
    "wall_time": "0"
  },
  "action": "added"
}
ā€¢ On the Fleet Webserver 
cat osquery_status -f
shows the last query execution to have happened in August 2022 šŸ™€
Copy code
{
  "hostIdentifier": "ip-172-31-23-127",
  "calendarTime": "Fri Aug 26 18:54:26 2022 UTC",
  "unixTime": "1661540066",
  "severity": "2",
  "filename": "scheduler.cpp",
  "line": "118",
  "message": "Error executing scheduled query pack/osx_attacks/OSX_Mughthesec: no such table: launchd",
  "version": "5.2.2",
  "decorations": {
    "host_uuid": "EC2C455A-F709-B975-ADA7-C33DE9F7EABA",
    "hostname": "ip-172-31-23-127.us-west-2.compute.internal"
  }
}
Where do I even begin debugging?
k
Hey @pvirani! Can you verify the logger plugin by running this query against that test host;
Copy code
SELECT value FROM osquery_flags WHERE name = "logger_plugin";
p
hostname value ip-172-31-55-24.us-west-2.compute.internal filesystem
^ @Kathy Satterlee
that also sounds wrong because when I launched the webserver I'd spun it up with the following flags
Copy code
- name: FLEET_OSQUERY_RESULT_LOG_PLUGIN
              value: "firehose"
            - name: FLEET_FIREHOSE_REGION
              value: "us-west-2"
            - name: FLEET_FIREHOSE_RESULT_STREAM
              value: "fleetdm-results-stream-prod"
k
Are you using Orbit or vanilla osquery?
p
Vanilla
This is how I'm spinning osquery up on my hosts
Copy code
#!/usr/bin/env bash

# DUO
# changed due to : <https://help.duo.com/s/article/5503?language=en_US>
# curl -sSL <https://duo.com/APT-GPG-KEY-DUO> | apt-key add -
curl -sSL <https://duo.com/DUO-GPG-PUBLIC-KEY.asc> | apt-key add -

# OSQUERY
apt-key adv --keyserver <http://keyserver.ubuntu.com|keyserver.ubuntu.com> --recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
apt-key adv --keyserver <http://keyserver.ubuntu.com|keyserver.ubuntu.com> --recv-keys 33EE313BAD9589B7
add-apt-repository "deb [arch=amd64] <https://osquery-packages.s3.amazonaws.com/deb> deb main"
apt install osquery

# OSQUERY
rm -f /etc/osquery/osquery.secret
rm -f /etc/osquery/osquery.flags
rm -f /etc/osquery/osquery.flags.default

# Fill osquery.secret with stuff
# Note, this is not actually sensitive and does not need to be any more secret
echo "enrollment_secret" > /etc/osquery/osquery.secret

# Fill osquery.flags.default with stuff
echo "--config_plugin=tls
--enroll_secret_path=/etc/osquery/osquery.secret
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_tls_endpoint=/api/v1/osquery/config
--tls_hostname=<http://fleetdm.segment.com|fleetdm.segment.com>
--config_refresh=300
--config_tls_accelerated_refresh=300
--config_tls_max_attempts=9999
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block" > /etc/osquery/osquery.flags.default

ln -s /etc/osquery/osquery.flags.default /etc/osquery/osquery.flags
z
Looks like your osquery is configured to log to the local filesystem and not the Fleet server?
p
How do I specify it should log to the Fleet server? This is the result of 
sudo systemctl start osqueryd
so looks like I gotta specify it in the flagfile?
Copy code
osqueryd.service - The osquery Daemon
   Loaded: loaded (/usr/lib/systemd/system/osqueryd.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2023-01-11 18:18:46 UTC; 2h 29min ago
  Process: 29213 ExecStartPre=/bin/sh -c if [ -f $LOCAL_PIDFILE ]; then mv $LOCAL_PIDFILE $PIDFILE; fi (code=exited, status=0/SUCCESS)
  Process: 29206 ExecStartPre=/bin/sh -c if [ ! -f $FLAG_FILE ]; then touch $FLAG_FILE; fi (code=exited, status=0/SUCCESS)
 Main PID: 29215 (osqueryd)
    Tasks: 15 (limit: 4561)
   CGroup: /system.slice/osqueryd.service
           ā”œā”€29215 /opt/osquery/bin/osqueryd --flagfile /etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf
           ā””ā”€29646 /opt/osquery/bin/osqueryd

Jan 11 19:38:54 ip-172-31-55-24 osqueryd[29215]: W0111 19:38:54.303018 29680 options.cpp:106] The CLI only flag --logger_plugin set via config file will be ignored, please use a flagfile or pass it to the process at startup
Jan 11 19:38:54 ip-172-31-55-24 osqueryd[29215]: I0111 19:38:54.356914 29680 eventfactory.cpp:352] The minimum events expiration timeout for hardware_events has been adjusted: 21660
Jan 11 19:38:54 ip-172-31-55-24 osqueryd[29215]: I0111 19:38:54.356961 29680 config.cpp:908] Calling configure for logger filesystem
Jan 11 19:43:54 ip-172-31-55-24 osqueryd[29215]: W0111 19:43:54.380683 29680 options.cpp:106] The CLI only flag --logger_plugin set via config file will be ignored, please use a flagfile or pass it to the process at startup
Jan 11 19:43:54 ip-172-31-55-24 osqueryd[29215]: I0111 19:43:54.455612 29680 eventfactory.cpp:352] The minimum events expiration timeout for hardware_events has been adjusted: 21660
Jan 11 19:43:54 ip-172-31-55-24 osqueryd[29215]: I0111 19:43:54.455659 29680 config.cpp:908] Calling configure for logger filesystem
Jan 11 19:46:37 ip-172-31-55-24 osqueryd[29215]: I0111 19:46:37.928295 29684 query.cpp:102] Storing initial results for new scheduled query: pack/osquery_monitoring/schedule
Jan 11 20:17:12 ip-172-31-55-24 osqueryd[29215]: I0111 20:17:12.569401 29684 query.cpp:102] Storing initial results for new scheduled query: pack/hardware_monitoring/pci_devices
Jan 11 20:19:34 ip-172-31-55-24 osqueryd[29215]: I0111 20:19:34.316332 29684 query.cpp:102] Storing initial results for new scheduled query: pack/incident_response/open_sockets
Jan 11 20:27:22 ip-172-31-55-24 osqueryd[29215]: I0111 20:27:22.376417 29684 query.cpp:102] Storing initial results for new scheduled query: pack/it_compliance/usb_devices
k
Copy code
--logger_plugin=tls 
 --logger_tls_endpoint=/api/v1/osquery/log 
 --logger_tls_period=10
It looks like you're missing those flags in osquery that will send logs to the Fleet server. Once they hit Fleet, the 
FLEET_OSQUERY_RESULT_LOG_PLUGIN
comes in to play.
p
adding em back in
the image will be ready in 22 mins. I'll report back with the status then. Thanks folks! Definitely pinning https://osquery.readthedocs.io/en/stable/installation/cli-flags/ to my brower so I don't have to distrub y'all with stupid questions again šŸ™‡šŸ»ā€ā™€ļø
k
Always happy to help! And you never know... this might help someone else down the road as well. Thanks for reaching out.
Just let me know if that doesn't do the trick.
p
Oh for sure! I've benefitted so much from keyword search results yielding other people's posts šŸ˜„
it worked! Thanks a bunch šŸ™‚
k
ā˜ļø šŸš€
c
thanks for this thread, it just help me out too šŸ™‚
3 Views
Open in Slack
PreviousNext
Powered by