Title
#kolide
z

Zach Zeid

08/20/2020, 6:52 PM
Is there a difference in how fleet manages scheduled packs already on endpoints vs. ones that are created through the UI? I see results from the packs already existing on the endpoints, but I don't see anything from scheduled packs I create through the UI.
sundsta

sundsta

08/20/2020, 8:08 PM
Do you have
--config_tls_endpoint
set?
8:08 PM
That option is what tells osquery to obtain new configurations over TLS
z

Zach Zeid

08/20/2020, 8:56 PM
yeah I do
"config_tls_endpoint": "/api/v1/osquery/config",
zwass

zwass

08/20/2020, 8:58 PM
If osquery is reading configs from disk it's not reading configs from Fleet
8:58 PM
You need to set
config_plugin=tls
z

Zach Zeid

08/20/2020, 8:58 PM
I do have that set
8:59 PM
"config_plugin": "tls",
9:01 PM
{
  "options": {
    "config_refresh": "600",
    "logger_plugin": "tls",
    "enroll_secret_path": "foobar",
    "tls_server_certs": "/foobaz",
    "tls_hostname": "<http://dev.fleet.example.org|dev.fleet.example.org>",
    "enroll_tls_endpoint": "/api/v1/osquery/enroll",
    "config_plugin": "tls",
    "config_tls_endpoint": "/api/v1/osquery/config",
    "config_refresh": "10",
    "disable_distributed": "false",
    "distributed_plugin": "tls",
    "distributed_interval": "10",
    "distributed_tls_max_attempts": "3",
    "distributed_tls_read_endpoint": "/api/v1/osquery/distributed/read",
    "distributed_tls_write_endpoint": "/api/v1/osquery/distributed/write",
    "logger_tls_endpoint": "/api/v1/osquery/log",
    "logger_tls_period": "10",
    "schedule_splay_percent": "10",
    "pidfile": "/var/run/osquery.pidfile",
    "events_expiry": "3600",
    "database_path": "/var/osquery/osquery.db",
    "disable_tables": "",
    "read_max": "100000",
    "events_max": "100000",
    "host_identifier": "uuid",
    "logger_min_status": 0,
    "disable_logging": false,
    "utc": "true"
  },
zwass

zwass

08/20/2020, 9:19 PM
That is being sent from Fleet?
z

Zach Zeid

08/20/2020, 9:22 PM
I manually edited this..
9:40 PM
I see I had
KOLIDE_SERVER_TLS=false
which may have something to do with the issue.
zwass

zwass

08/20/2020, 9:41 PM
I would add
--verbose --tls_dump
to the osquery flags so you can see (1) whether osquery is requesting config from Fleet and (2) what Fleet is returning
z

Zach Zeid

08/20/2020, 9:49 PM
the host does get enrolled because it does show up in the dashboard, but I don't see any logs being generated at all.
9:52 PM
when I do set
KOLIDE_SERVER_TLS=true
I get
`    "terminated": "tls: failed to find any PEM data in key input",
even thought I'm passing
server_key
and
server_cert
as envvars
10:28 PM
I guess I'm curious why the host will enroll with fleet, but can't pull down configs. I know I'm missing something, but I can't figure out what exactly I'm missing.
10:29 PM
Running an ad-hoc query also works, so 🤷
zwass

zwass

08/20/2020, 10:36 PM
You'll need to run osquery in verbose mode and see the network comms
z

Zach Zeid

08/20/2020, 10:40 PM
huh,
I0820 18:40:36.960268 28460 glog_logger.cpp:44] Could not get RPM header flag.
over and over again.
zwass

zwass

08/20/2020, 10:44 PM
Seems unrelated
10:44 PM
You'll need to look at the tls_dump logs
z

Zach Zeid

08/20/2020, 10:47 PM
ah interesting! I had both
--verbose
and
--tls_dump
set in
osquery.flags
but it wasn't actually doing anything. running
osqueryd
with those flags shows something.
10:48 PM
I0820 18:46:38.206319 28547 tls.cpp:253] TLS/HTTPS POST request to URI: <https://fleet.boop.org/api/v1/osquery/distributed/read>
{"node_key":"foobar"}
{
  "queries": {}
}
10:51 PM
This is interesting! I'm definitely seeing the queries from the packs being sent up to fleet, but I'm assuming where
"queries":
is
{}
is the opposite, meaning that it's getting a config from fleet that's empty?
zwass

zwass

08/20/2020, 10:54 PM
No, this is a live query request
10:54 PM
Need to find the request to
api/v1/osquery/config
10:55 PM
If there is none, you haven't configured osquery to retrieve configs from Fleet
z

Zach Zeid

08/20/2020, 10:57 PM
ok, then that's where I'm confused, I'm sure this is a mostly correct config
"options": {
		"config_refresh": "600",
		"logger_plugin": "tls",
		"tls_server_certs": "/etc/osquery/certs/somecert.crt",
		"tls_client_key": "/etc/osquery/certs/somekey.key",
		"tls_hostname": "<http://fleet.beep.boop.org|fleet.beep.boop.org>",
		"enroll_tls_endpoint": "/api/v1/osquery/enroll",
		"config_plugin": "tls",
		"config_tls_endpoint": "/api/v1/osquery/config",
		"config_refresh": "10",
		"disable_distributed": "false",
		"distributed_plugin": "tls",
		"distributed_interval": "10",
		"distributed_tls_max_attempts": "3",
		"distributed_tls_read_endpoint": "/api/v1/osquery/distributed/read",
		"distributed_tls_write_endpoint": "/api/v1/osquery/distributed/write",
		"logger_tls_endpoint": "/api/v1/osquery/log",
		"logger_tls_period": "10",
		"schedule_splay_percent": "10",
		"pidfile": "/var/run/osquery.pidfile",
		"events_expiry": "3600",
		"database_path": "/var/osquery/osquery.db",
		"disable_tables": "",
		"read_max": "100000",
		"events_max": "100000",
		"host_identifier": "uuid",
		"logger_min_status": 0,
		"disable_logging": false,
		"utc": "true"
	},
10:57 PM
is it not configured to pull from fleet?
10:58 PM
this is on the endpoint, so should this config be living on kolide fleet?
zwass

zwass

08/20/2020, 11:01 PM
I'm not sure that you can load a config from disk first and then osquery will switch to TLS. Typically folks hook everything up with a flagfile and serve the config from TLS.
z

Zach Zeid

08/20/2020, 11:04 PM
🤔 so more something like this?
cat /etc/osquery/osquery.flags 
Ansible managed
--config_plugin=filesystem, tls
--config_path=/etc/osquery/osquery.conf
--config_refresh=600
--watchdog_level=1
--tls_dump
--verbose
--extensions_socket=/var/osquery/osquery.em
--pidfile=/var/run/osquery.pidfile
zwass

zwass

08/20/2020, 11:10 PM
Yes, but you have an invalid value for config_plugin and you need to set the rest of the flags for connecting to fleet.
11:10 PM
Please check out the docs
z

Zach Zeid

08/20/2020, 11:21 PM
I'll doubly check them, this got me a lot further. Thank you!
z

Zach Zeid

08/21/2020, 12:02 PM
I was able to get this working, thanks for your help!