Channels
#kolide
Title
# kolide
d
doggles
05/18/2020, 10:36 AMI’m running into an issue with Kolide Fleet, where our scheduled queries on Macs aren’t populating the result.log file on the Kolide Fleet server. We can see that scheduled queries are executing on the hosts, and they are receiving their confs from Fleet. Might be a big ask for some troubleshooting assistance, but kind of stuck on what’s going on here
z
zwass
05/18/2020, 3:43 PM
d
doggles
05/18/2020, 4:39 PMso i ran through that earlier… set up a scheduled query with select * from time, made it a snapshot, verified it had run on host querying the scheduled_queries table, but empty result.log
Copy code
--disable_audit=false
--audit_persist=true
--audit_allow_config=true
--disable_events=false
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_tls_refresh=10
--database_path=/var/osquery/osquery.db
--disable_distributed=false
--disable_logging=true
--distributed_interval=10
--distributed_plugin=tls
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--enroll_secret_path=/var/osquery/enroll
--enroll_tls_endpoint=/api/v1/osquery/enroll
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10
--tls_hostname=<http://kolide.company.com|kolide.company.com>z
zwass
05/18/2020, 4:40 PM
Add
--tls_dumpd
doggles
05/18/2020, 4:59 PM Copy code
effectively seeing this: I0518 18:57:55.433888 49946624 config.cpp:1140] Refreshing configuration state
I0518 18:57:55.435231 49946624 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/config>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
"decorators": {
"load": [
"SELECT uuid AS host_uuid FROM system_info;",
"SELECT hostname AS hostname FROM system_info;"
]
},
"file_paths": {
"binaries": [
"/usr/bin/%%",
"/usr/sbin/%%",
"/bin/%%",
"/sbin/%%",
"/usr/local/bin/%%",
"/usr/local/sbin/%%"
],
"configuration": [
"/etc/group",
"/etc/passwd",
"/etc/gshadow",
"/etc/shadow",
"/etc/security/%%",
"/etc/nsswitch.conf",
"/etc/pam.d/%%",
"/etc/issue%",
"/etc/hosts%",
"/etc/ssh/%%",
"/etc/sysconfig/network",
"/etc/sysconfig/network-scripts/%%",
"/etc/localtime",
"/etc/selinux/%%",
"/etc/sudoers",
"/etc/sudoers.d/%%",
"/etc/cron%/%%",
"/etc/crontab",
"/etc/fstab",
"/etc/hostname",
"/etc/ld.so.conf",
"/etc/ld.so.conf.d/%%",
"/etc/rc%/%%"
]
},
"options": {
"disable_distributed": false,
"distributed_interval": 10,
"distributed_plugin": "tls",
"distributed_tls_max_attempts": 3,
"distributed_tls_read_endpoint": "/api/v1/osquery/distributed/read",
"distributed_tls_write_endpoint": "/api/v1/osquery/distributed/write",
"enable_monitor": true,
"events_expiry": 300,
"events_max": 50000,
"logger_path": "/var/log/osquery",
"logger_tls_endpoint": "/api/v1/osquery/log",
"logger_tls_period": 10,
"pack_delimiter": "/",
"pack_refresh_interval": 60,
"schedule_default_interval": 3597,
"schedule_splay_percent": 10,
"utc": true,
"worker_threads": 2
},
"packs": {
"test": {
"queries": {
"Scheduled Query Test": {
"query": "SELECT * FROM time;",
"interval": 600,
"platform": "darwin",
"snapshot": true,
"removed": false
}
}
}
}
}
I0518 18:58:03.631273 53166080 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/distributed/read>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
"queries": {}
}
I0518 18:58:05.708802 49946624 config.cpp:1140] Refreshing configuration state Copy code
I0518 19:00:01.741070 53702656 scheduler.cpp:96] Executing scheduled query pack/test/Scheduled Query Test: SELECT * FROM time;
I0518 19:00:01.745529 53702656 database.cpp:140] Resetting the database plugin: rocksdb
I0518 19:00:01.747506 53702656 rocksdb.cpp:131] Opening RocksDB handle: /var/osquery/osquery.db
I0518 19:00:06.776628 53166080 tls.cpp:253] TLS/HTTPS POST request to URI: <https://kolide.company.com/api/v1/osquery/distributed/read>
{"node_key":"O3e2Eynt1+bqCcxmBEjko09yNdiLg0SN"}
{
"queries": {}
}
I0518 19:00:08.775307 49946624 config.cpp:1140] Refreshing configuration statez
zwass
05/18/2020, 5:01 PM
Maybe try running the query every 10 seconds. Hard to debug if it only runs every 10 mins.
Then look for the request in which osquery tries to write the logs.
d
doggles
05/18/2020, 5:06 PMgot it. no difference. same output repeated. and nothing happening in the
status.logit’s very possible this is a vpn issue as all our macs are remote and we haven’t exposed our fleet instance externally yet. but wanted to rule out or identify an issue with our configuration first
though i dont know if that makes sense if everything else is working except for results being recorded on the fleet server. i appreciate you looking into this though
z
zwass
05/18/2020, 5:14 PM
Would not be a VPN issue. Does osquery ever make a request to api/v1/osquery/log?
d
doggles
05/18/2020, 5:22 PMNo, not seeing that
just api/v1/osquery/distributed/read and api/v1/osquery/config
z
zwass
05/18/2020, 5:30 PM
Something is probably configuring osquery to log somewhere besides TLS
Or setting a really high interval. Perhaps something you edited out in the configs you shared above?
d
doggles
05/18/2020, 5:33 PMdoes something need to be configured in fleet, e.g.,
fleetctl get optionsz
zwass
05/18/2020, 5:36 PM
Options set there can potentially override the flags in the flagfile.
d
doggles
05/18/2020, 6:13 PMso looking through issues i see your reply here: https://github.com/kolide/fleet/issues/1915#issuecomment-590968565, this makes me think that i’m missing something in fleet’s osquery config.
Copy code
apiVersion: v1
kind: options
spec:
config:
decorators:
load:
- SELECT uuid AS host_uuid FROM system_info;
- SELECT hostname AS hostname FROM system_info;
file_paths:
binaries:
- /usr/bin/%%
- /usr/sbin/%%
- /bin/%%
- /sbin/%%
- /usr/local/bin/%%
- /usr/local/sbin/%%
configuration:
- /etc/group
- /etc/passwd
- /etc/gshadow
- /etc/shadow
- /etc/security/%%
- /etc/nsswitch.conf
- /etc/pam.d/%%
- /etc/issue%
- /etc/hosts%
- /etc/ssh/%%
- /etc/sysconfig/network
- /etc/sysconfig/network-scripts/%%
- /etc/localtime
- /etc/selinux/%%
- /etc/sudoers
- /etc/sudoers.d/%%
- /etc/cron%/%%
- /etc/crontab
- /etc/fstab
- /etc/hostname
- /etc/ld.so.conf
- /etc/ld.so.conf.d/%%
- /etc/rc%/%%
options:
disable_distributed: false
distributed_interval: 10
distributed_plugin: tls
distributed_tls_max_attempts: 3
distributed_tls_read_endpoint: /api/v1/osquery/distributed/read
distributed_tls_write_endpoint: /api/v1/osquery/distributed/write
enable_monitor: true
events_expiry: 300
events_max: 50000
logger_path: /var/log/osquery
logger_tls_endpoint: /api/v1/osquery/log
logger_tls_period: 10
pack_delimiter: /
pack_refresh_interval: 60
schedule_default_interval: 3597
schedule_splay_percent: 10
utc: true
worker_threads: 2
overrides: {}z
zwass
05/18/2020, 8:42 PM
Just noticed
--disable_logging=trued
doggles
05/19/2020, 8:45 AMi just noticed that as well
haha, all fixed
i had come back on this morning to let you know i found the issue. good eye
z
zwass
05/19/2020, 4:04 PM
Nice! Glad it's sorted.
d
doggles
05/19/2020, 8:39 PMgot everything flowing into the SIEM now. thanks a lot for the help (and for osquery and fleet!)
🍻 1
z
zwass
05/19/2020, 8:40 PM
Glad to hear it. Feel free to reach out if you ever need help or would like to work together 🙂
5 Views