https://github.com/osquery/osquery logo
Powered by
Title
d

doggles

05/18/2020, 10:36 AM
I’m running into an issue with Kolide Fleet, where our scheduled queries on Macs aren’t populating the result.log file on the Kolide Fleet server. We can see that scheduled queries are executing on the hosts, and they are receiving their confs from Fleet. Might be a big ask for some troubleshooting assistance, but kind of stuck on what’s going on here
z

zwass

05/18/2020, 3:43 PM
Check out https://github.com/kolide/fleet/blob/master/docs/infrastructure/faq.md#scheduled-queries
d

doggles

05/18/2020, 4:39 PM
so i ran through that earlier… set up a scheduled query with select * from time, made it a snapshot, verified it had run on host querying the scheduled_queries table, but empty result.log
--disable_audit=false
--audit_persist=true
--audit_allow_config=true
--disable_events=false
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_tls_refresh=10
--database_path=/var/osquery/osquery.db
--disable_distributed=false
--disable_logging=true
--distributed_interval=10
--distributed_plugin=tls
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--enroll_secret_path=/var/osquery/enroll
--enroll_tls_endpoint=/api/v1/osquery/enroll
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10
--tls_hostname=<http://kolide.company.com|kolide.company.com>
our flagfile
z

zwass

05/18/2020, 4:40 PM
Add 
--tls_dump
and see what the config looks like that osquery receives. Then you'll also see if osquery is sending logs back to the server.
d

doggles

05/18/2020, 4:59 PM
effectively seeing this: I0518 18:57:55.433888 49946624 config.cpp:1140] Refreshing configuration state
I0518 18:57:55.435231 49946624 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/config>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
  "decorators": {
    "load": [
      "SELECT uuid AS host_uuid FROM system_info;",
      "SELECT hostname AS hostname FROM system_info;"
    ]
  },
  "file_paths": {
    "binaries": [
      "/usr/bin/%%",
      "/usr/sbin/%%",
      "/bin/%%",
      "/sbin/%%",
      "/usr/local/bin/%%",
      "/usr/local/sbin/%%"
    ],
    "configuration": [
      "/etc/group",
      "/etc/passwd",
      "/etc/gshadow",
      "/etc/shadow",
      "/etc/security/%%",
      "/etc/nsswitch.conf",
      "/etc/pam.d/%%",
      "/etc/issue%",
      "/etc/hosts%",
      "/etc/ssh/%%",
      "/etc/sysconfig/network",
      "/etc/sysconfig/network-scripts/%%",
      "/etc/localtime",
      "/etc/selinux/%%",
      "/etc/sudoers",
      "/etc/sudoers.d/%%",
      "/etc/cron%/%%",
      "/etc/crontab",
      "/etc/fstab",
      "/etc/hostname",
      "/etc/ld.so.conf",
      "/etc/ld.so.conf.d/%%",
      "/etc/rc%/%%"
    ]
  },
  "options": {
    "disable_distributed": false,
    "distributed_interval": 10,
    "distributed_plugin": "tls",
    "distributed_tls_max_attempts": 3,
    "distributed_tls_read_endpoint": "/api/v1/osquery/distributed/read",
    "distributed_tls_write_endpoint": "/api/v1/osquery/distributed/write",
    "enable_monitor": true,
    "events_expiry": 300,
    "events_max": 50000,
    "logger_path": "/var/log/osquery",
    "logger_tls_endpoint": "/api/v1/osquery/log",
    "logger_tls_period": 10,
    "pack_delimiter": "/",
    "pack_refresh_interval": 60,
    "schedule_default_interval": 3597,
    "schedule_splay_percent": 10,
    "utc": true,
    "worker_threads": 2
  },
  "packs": {
    "test": {
      "queries": {
        "Scheduled Query Test": {
          "query": "SELECT * FROM time;",
          "interval": 600,
          "platform": "darwin",
          "snapshot": true,
          "removed": false
        }
      }
    }
  }
}

I0518 18:58:03.631273 53166080 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/distributed/read>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
  "queries": {}
}

I0518 18:58:05.708802 49946624 config.cpp:1140] Refreshing configuration state
I0518 19:00:01.741070 53702656 scheduler.cpp:96] Executing scheduled query pack/test/Scheduled Query Test: SELECT * FROM time;
I0518 19:00:01.745529 53702656 database.cpp:140] Resetting the database plugin: rocksdb
I0518 19:00:01.747506 53702656 rocksdb.cpp:131] Opening RocksDB handle: /var/osquery/osquery.db
I0518 19:00:06.776628 53166080 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/distributed/read>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
  "queries": {}
}

I0518 19:00:08.775307 49946624 config.cpp:1140] Refreshing configuration state
z

zwass

05/18/2020, 5:01 PM
Maybe try running the query every 10 seconds. Hard to debug if it only runs every 10 mins.
Then look for the request in which osquery tries to write the logs.
d

doggles

05/18/2020, 5:06 PM
got it. no difference. same output repeated. and nothing happening in the 
status.log
on the fleet server
it’s very possible this is a vpn issue as all our macs are remote and we haven’t exposed our fleet instance externally yet. but wanted to rule out or identify an issue with our configuration first
though i dont know if that makes sense if everything else is working except for results being recorded on the fleet server. i appreciate you looking into this though
z

zwass

05/18/2020, 5:14 PM
Would not be a VPN issue. Does osquery ever make a request to api/v1/osquery/log?
d

doggles

05/18/2020, 5:22 PM
No, not seeing that
just api/v1/osquery/distributed/read and api/v1/osquery/config
z

zwass

05/18/2020, 5:30 PM
Something is probably configuring osquery to log somewhere besides TLS
Or setting a really high interval. Perhaps something you edited out in the configs you shared above?
d

doggles

05/18/2020, 5:33 PM
does something need to be configured in fleet, e.g., 
fleetctl get options
has a separate config than the flagfile pushed to each mac endpoint
z

zwass

05/18/2020, 5:36 PM
Options set there can potentially override the flags in the flagfile.
d

doggles

05/18/2020, 6:13 PM
so looking through issues i see your reply here: https://github.com/kolide/fleet/issues/1915#issuecomment-590968565, this makes me think that i’m missing something in fleet’s osquery config.
apiVersion: v1
kind: options
spec:
  config:
    decorators:
      load:
      - SELECT uuid AS host_uuid FROM system_info;
      - SELECT hostname AS hostname FROM system_info;
    file_paths:
      binaries:
      - /usr/bin/%%
      - /usr/sbin/%%
      - /bin/%%
      - /sbin/%%
      - /usr/local/bin/%%
      - /usr/local/sbin/%%
      configuration:
      - /etc/group
      - /etc/passwd
      - /etc/gshadow
      - /etc/shadow
      - /etc/security/%%
      - /etc/nsswitch.conf
      - /etc/pam.d/%%
      - /etc/issue%
      - /etc/hosts%
      - /etc/ssh/%%
      - /etc/sysconfig/network
      - /etc/sysconfig/network-scripts/%%
      - /etc/localtime
      - /etc/selinux/%%
      - /etc/sudoers
      - /etc/sudoers.d/%%
      - /etc/cron%/%%
      - /etc/crontab
      - /etc/fstab
      - /etc/hostname
      - /etc/ld.so.conf
      - /etc/ld.so.conf.d/%%
      - /etc/rc%/%%
    options:
      disable_distributed: false
      distributed_interval: 10
      distributed_plugin: tls
      distributed_tls_max_attempts: 3
      distributed_tls_read_endpoint: /api/v1/osquery/distributed/read
      distributed_tls_write_endpoint: /api/v1/osquery/distributed/write
      enable_monitor: true
      events_expiry: 300
      events_max: 50000
      logger_path: /var/log/osquery
      logger_tls_endpoint: /api/v1/osquery/log
      logger_tls_period: 10
      pack_delimiter: /
      pack_refresh_interval: 60
      schedule_default_interval: 3597
      schedule_splay_percent: 10
      utc: true
      worker_threads: 2
  overrides: {}
z

zwass

05/18/2020, 8:42 PM
Just noticed 
--disable_logging=true
in your flagfile
d

doggles

05/19/2020, 8:45 AM
i just noticed that as well
haha, all fixed
i had come back on this morning to let you know i found the issue. good eye
z

zwass

05/19/2020, 4:04 PM
Nice! Glad it's sorted.
d

doggles

05/19/2020, 8:39 PM
got everything flowing into the SIEM now. thanks a lot for the help (and for osquery and fleet!)
🍻 1
z

zwass

05/19/2020, 8:40 PM
Glad to hear it. Feel free to reach out if you ever need help or would like to work together 🙂
5 Views
#kolideJoin Slack
Powered by