https://github.com/osquery/osquery logo
Join Slack
Powered by
I’m running into an issue with Kolide Fleet, where...
# kolide
d
I’m running into an issue with Kolide Fleet, where our scheduled queries on Macs aren’t populating the result.log file on the Kolide Fleet server. We can see that scheduled queries are executing on the hosts, and they are receiving their confs from Fleet. Might be a big ask for some troubleshooting assistance, but kind of stuck on what’s going on here
z
d
so i ran through that earlier… set up a scheduled query with select * from time, made it a snapshot, verified it had run on host querying the scheduled_queries table, but empty result.log
Copy code
--disable_audit=false
--audit_persist=true
--audit_allow_config=true
--disable_events=false
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_tls_refresh=10
--database_path=/var/osquery/osquery.db
--disable_distributed=false
--disable_logging=true
--distributed_interval=10
--distributed_plugin=tls
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--enroll_secret_path=/var/osquery/enroll
--enroll_tls_endpoint=/api/v1/osquery/enroll
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10
--tls_hostname=<http://kolide.company.com|kolide.company.com>
our flagfile
z
Add 
--tls_dump
and see what the config looks like that osquery receives. Then you'll also see if osquery is sending logs back to the server.
d
Copy code
effectively seeing this: I0518 18:57:55.433888 49946624 config.cpp:1140] Refreshing configuration state
I0518 18:57:55.435231 49946624 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/config>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
  "decorators": {
    "load": [
      "SELECT uuid AS host_uuid FROM system_info;",
      "SELECT hostname AS hostname FROM system_info;"
    ]
  },
  "file_paths": {
    "binaries": [
      "/usr/bin/%%",
      "/usr/sbin/%%",
      "/bin/%%",
      "/sbin/%%",
      "/usr/local/bin/%%",
      "/usr/local/sbin/%%"
    ],
    "configuration": [
      "/etc/group",
      "/etc/passwd",
      "/etc/gshadow",
      "/etc/shadow",
      "/etc/security/%%",
      "/etc/nsswitch.conf",
      "/etc/pam.d/%%",
      "/etc/issue%",
      "/etc/hosts%",
      "/etc/ssh/%%",
      "/etc/sysconfig/network",
      "/etc/sysconfig/network-scripts/%%",
      "/etc/localtime",
      "/etc/selinux/%%",
      "/etc/sudoers",
      "/etc/sudoers.d/%%",
      "/etc/cron%/%%",
      "/etc/crontab",
      "/etc/fstab",
      "/etc/hostname",
      "/etc/ld.so.conf",
      "/etc/ld.so.conf.d/%%",
      "/etc/rc%/%%"
    ]
  },
  "options": {
    "disable_distributed": false,
    "distributed_interval": 10,
    "distributed_plugin": "tls",
    "distributed_tls_max_attempts": 3,
    "distributed_tls_read_endpoint": "/api/v1/osquery/distributed/read",
    "distributed_tls_write_endpoint": "/api/v1/osquery/distributed/write",
    "enable_monitor": true,
    "events_expiry": 300,
    "events_max": 50000,
    "logger_path": "/var/log/osquery",
    "logger_tls_endpoint": "/api/v1/osquery/log",
    "logger_tls_period": 10,
    "pack_delimiter": "/",
    "pack_refresh_interval": 60,
    "schedule_default_interval": 3597,
    "schedule_splay_percent": 10,
    "utc": true,
    "worker_threads": 2
  },
  "packs": {
    "test": {
      "queries": {
        "Scheduled Query Test": {
          "query": "SELECT * FROM time;",
          "interval": 600,
          "platform": "darwin",
          "snapshot": true,
          "removed": false
        }
      }
    }
  }
}

I0518 18:58:03.631273 53166080 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/distributed/read>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
  "queries": {}
}

I0518 18:58:05.708802 49946624 config.cpp:1140] Refreshing configuration state
Copy code
I0518 19:00:01.741070 53702656 scheduler.cpp:96] Executing scheduled query pack/test/Scheduled Query Test: SELECT * FROM time;
I0518 19:00:01.745529 53702656 database.cpp:140] Resetting the database plugin: rocksdb
I0518 19:00:01.747506 53702656 rocksdb.cpp:131] Opening RocksDB handle: /var/osquery/osquery.db
I0518 19:00:06.776628 53166080 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/distributed/read>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
  "queries": {}
}

I0518 19:00:08.775307 49946624 config.cpp:1140] Refreshing configuration state
z
Maybe try running the query every 10 seconds. Hard to debug if it only runs every 10 mins.
Then look for the request in which osquery tries to write the logs.
d
got it. no difference. same output repeated. and nothing happening in the 
status.log
on the fleet server
it’s very possible this is a vpn issue as all our macs are remote and we haven’t exposed our fleet instance externally yet. but wanted to rule out or identify an issue with our configuration first
though i dont know if that makes sense if everything else is working except for results being recorded on the fleet server. i appreciate you looking into this though
z
Would not be a VPN issue. Does osquery ever make a request to api/v1/osquery/log?
d
No, not seeing that
just api/v1/osquery/distributed/read and api/v1/osquery/config
z
Something is probably configuring osquery to log somewhere besides TLS
Or setting a really high interval. Perhaps something you edited out in the configs you shared above?
d
does something need to be configured in fleet, e.g., 
fleetctl get options
has a separate config than the flagfile pushed to each mac endpoint
z
Options set there can potentially override the flags in the flagfile.
d
so looking through issues i see your reply here: https://github.com/kolide/fleet/issues/1915#issuecomment-590968565, this makes me think that i’m missing something in fleet’s osquery config.
Copy code
apiVersion: v1
kind: options
spec:
  config:
    decorators:
      load:
      - SELECT uuid AS host_uuid FROM system_info;
      - SELECT hostname AS hostname FROM system_info;
    file_paths:
      binaries:
      - /usr/bin/%%
      - /usr/sbin/%%
      - /bin/%%
      - /sbin/%%
      - /usr/local/bin/%%
      - /usr/local/sbin/%%
      configuration:
      - /etc/group
      - /etc/passwd
      - /etc/gshadow
      - /etc/shadow
      - /etc/security/%%
      - /etc/nsswitch.conf
      - /etc/pam.d/%%
      - /etc/issue%
      - /etc/hosts%
      - /etc/ssh/%%
      - /etc/sysconfig/network
      - /etc/sysconfig/network-scripts/%%
      - /etc/localtime
      - /etc/selinux/%%
      - /etc/sudoers
      - /etc/sudoers.d/%%
      - /etc/cron%/%%
      - /etc/crontab
      - /etc/fstab
      - /etc/hostname
      - /etc/ld.so.conf
      - /etc/ld.so.conf.d/%%
      - /etc/rc%/%%
    options:
      disable_distributed: false
      distributed_interval: 10
      distributed_plugin: tls
      distributed_tls_max_attempts: 3
      distributed_tls_read_endpoint: /api/v1/osquery/distributed/read
      distributed_tls_write_endpoint: /api/v1/osquery/distributed/write
      enable_monitor: true
      events_expiry: 300
      events_max: 50000
      logger_path: /var/log/osquery
      logger_tls_endpoint: /api/v1/osquery/log
      logger_tls_period: 10
      pack_delimiter: /
      pack_refresh_interval: 60
      schedule_default_interval: 3597
      schedule_splay_percent: 10
      utc: true
      worker_threads: 2
  overrides: {}
z
Just noticed 
--disable_logging=true
in your flagfile
d
i just noticed that as well
haha, all fixed
i had come back on this morning to let you know i found the issue. good eye
z
Nice! Glad it's sorted.
d
got everything flowing into the SIEM now. thanks a lot for the help (and for osquery and fleet!)
🍻 1
z
Glad to hear it. Feel free to reach out if you ever need help or would like to work together 🙂
5 Views
Open in Slack
PreviousNext
Powered by