Title
#kolide
n

nyanshak

12/08/2019, 7:15 PM
@Seren / @seph re: disabling distributed queries ... moving to thread so it's easier to follow this conversation ...
7:16 PM
Isn't that sort of possible through options? e.g., in fleet, you could set
disable_distributed
to
true
7:16 PM
There are still some drawbacks, since I think you can't exactly disable someone from running them, but they just won't get picked up by the host, I think?
s

seph

12/09/2019, 2:43 AM
As an osquery option? I don’t know if fleet ever uses distributed queries for internal purposes
s

Seren

12/09/2019, 3:07 AM
I can reconfirm, but if I run osqueryd (not launcher) with distributed queries disabled, the host information doesn’t seem to get completely populated. I’ll check on Monday if running launcher with distributed queries disabled causes problems (I’ve been doing testing mostly by running osqueryd directly rather than launcher so I need to make sure I’m remembering correctly). With osqueryd there were 3 fields in the hosts table which weren’t being populated.
n

nyanshak

12/09/2019, 3:43 PM
🤷 Fleet does use distributed queries for internal purposes. You can ""safely"" ** disable distributed queries via osquery options / set them in fleet with a few caveats... 1. You won't get all the host info 2. Queries entered into distributed query panel still seem to get saved for the host, because I don't think fleet really tracks this. So if you re-enable distributed queries and you've run queries against all hosts, then you'll see those hosts get and push data for the queries.... 3. The UI / support for disabling is not that great, and it still will totally look like you can run distributed queries against hosts even though they have it disabled. See: https://github.com/kolide/fleet/issues/2140
zwass

zwass

12/09/2019, 5:31 PM
The above understanding looks correct to me. There isn't (currently) a clean way to disable distributed queries while maintaining consistency in the UI.
s

seph

12/09/2019, 5:41 PM
I’m curious about the use case. Why disable them?
s

Seren

12/09/2019, 5:49 PM
Thanks for the info. I'm ok with any UI issues. The only thing that is a blocker is that query-packs aren't being sent to the to the hosts. There were 3 fields in the hosts table that weren't being populated correctly (primary_ip_id, config_tls_refresh, uptime), but I've updated osquery to 4.11 so I want to reconfirm that everything's the same now when a) using launcher and b) having started with a fresh Fleet db.
zwass

zwass

12/09/2019, 5:52 PM
The host details and labels are both populated by distributed queries, so what you can do will be quite limited. For a hack, you could go into the code and disable the endpoint that actually creates a query. I think this would allow the details and labels to populate while preventing a user from creating a new query.
s

Seren

12/09/2019, 5:55 PM
@seph We're rolling out osquery for servers and workstations. Servers are pretty straight-forward and distributed-queries are great. Workstations are a bit more sensitive since in our org is small everyone is used to freedom and privacy (users are technical and have admin on their machines). The compromise to running osquery on the workstations is that we only run pre-approved query-packs and disable the ability to randomly snoop on what's going on (nevermind that we could schedule snooping queries every minute, we're trying to keep everyone calm and happy).
s

seph

12/09/2019, 5:59 PM
I was wondering about that underlying intent — It’s a weird balance. Y’all are aware that you can just schedule things, but it’s seen as a higher bar than the LQ side.
5:59 PM
It’s hard to find the right balance there.
s

Seren

12/09/2019, 5:59 PM
@zwass Interesting idea. I may have to do that, though at that point users are trusting we're running a modified server, so just a promise not to run distributed queries would probably be sufficient. I was hoping there could be something client-side that would make it obvious distributed queries were disabled, but that may not be possible. It's ok in this case for the UI (labels and status) to be borked, as long as the query packs are sent to the hosts and the logs can be received from clients. I'll get back in a minute and see if query-pack assignment still doesn't work
s

seph

12/09/2019, 5:59 PM
We’ve done some work around that in our SaaS (which is fairly different than fleet). Blacklists might also get you somewhere
zwass

zwass

12/09/2019, 6:01 PM
I think query pack assignment will no longer work because it uses labels and they will not be populated.
s

Seren

12/09/2019, 6:01 PM
Interesting, I'll have to look at blacklists, though the users would still have to trust that we're not running the queries, no? Btw, what is "LQ"?
zwass

zwass

12/09/2019, 6:02 PM
LQ - Live Query
s

Seren

12/09/2019, 6:03 PM
Ahh, thanks. Yeah, that seemed to be the behavior I saw, though it seemed like labels were being populated:
6:05 PM
This was with osquery 4.02: With osqueryd (LQ disabled):
osqueryd --host_identifier=$(scutil --get ComputerName) --verbose --allow_unsafe --tls_hostname=<http://fleet.globus.org:443|fleet.globus.org:443> --tls_server_certs=/etc/osquery/globus_osquery.crt --enroll_secret_path=/etc/osquery/globus_osquery_secret  --enroll_tls_endpoint=/api/v1/osquery/enroll  --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60  --disable_distributed=false --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --logger_tls_period=60 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
Table fields that differ:
MySQL [kolide]> select id, platform, primary_ip_id, config_tls_refresh, uptime from hosts;
+----+----------+---------------+--------------------+--------+
| id | platform | primary_ip_id | config_tls_refresh | uptime |
+----+----------+---------------+--------------------+--------+
| 11 | darwin   |          NULL |                  0 |      0 |
+----+----------+---------------+--------------------+--------+
1 row in set (0.00 sec)
With launcher (LQ disabled):
build/launcher --hostname <http://testfleet.globus.org:443|testfleet.globus.org:443> --enroll_secret 'xEXAMPLExEXAMPLE' --root_pem=/etc/osquery/globus_osquery.crt --osqueryd_path=/usr/local/bin/osqueryd --root_directory ~/.osquery --debug
Fields that differ:
MySQL [kolide]> select id, platform, primary_ip_id, config_tls_refresh, uptime from hosts;
+----+----------+---------------+--------------------+-----------------+
| id | platform | primary_ip_id | config_tls_refresh | uptime          |
+----+----------+---------------+--------------------+-----------------+
| 12 | darwin   |           109 |                300 | 917500000000000 |
+----+----------+---------------+--------------------+-----------------+
6:18 PM
Ok, retested now. Running launcher as well as osqueryd failed to produce hosts that can be assigned packs (as zwass predicted). The osqueryd host at least seems to have some fields populated, but the launcher host hangs (at least in the UI):
6:18 PM
6:20 PM
The db reflects this:
| id | osquery_host_id                      | created_at          | updated_at          | deleted_at | deleted | detail_update_time  | node_key                         | host_name                     | uuid                                 | platform | osquery_version | os_version       | build   | platform_like | code_name | uptime | physical_memory | cpu_type | cpu_subtype           | cpu_brand                                        | cpu_physical_cores | cpu_logical_cores | hardware_vendor | hardware_model | hardware_version | hardware_serial | computer_name       | primary_ip_id | seen_time           | distributed_interval | logger_tls_period | config_tls_refresh |


| 25 | ip-192-168-86-26.ec2.internal        | 2019-12-09 18:13:22 | 2019-12-09 18:17:24 | NULL       |       0 | 1970-01-02 00:00:00 | 1k17AggNWIxQgBTvfRyBs0mLYQHKHsgo | ip-192-168-86-26.ec2.internal | A61DC7D2-0C6D-56B9-92A5-REDACTED | darwin   | 4.1.1           | Mac OS X 10.14.6 | 18G1012 | darwin        |           |      0 |     17179869184 | x86_64h  | Intel x86-64h Haswell | Intel(R) Core(TM) i7-6567U CPU @ 3.30GHz         |                  2 |                 4 | Apple Inc.      | MacBookPro13,2 | 1.0              | redacted-serial    | mylaptopname-current |          NULL | 2019-12-09 18:17:25 |                   10 |                10 |                  0 |


| 26 | d3bc7341-f65e-4df5-a9a6-f26c7fe42c1e | 2019-12-09 18:13:47 | 2019-12-09 18:14:47 | NULL       |       0 | 1970-01-02 00:00:00 | Ux8TA4D2yF9bnNcNFIzCOFTZg2KI0uwx |                               |                                      |          |                 |                  |         |               |           |      0 |               0 |          |                       |                                                  |                  0 |                 0 |                 |                |                  |                 |                     |          NULL | 2019-12-09 18:14:48 |                   10 |                10 |                  0 |
6:20 PM
(sorry for the text flood in a thread column)
6:22 PM
Anyway, it seems like trying to disable LQ for the client is a no-go, so I'll either need to distribute query-packs and configs, and use Fleet (at least for workstations) as a centralized log collector, or see if promising not to use LQ is sufficient.
zwass

zwass

12/09/2019, 6:24 PM
Yeah, using Fleet just to receive logs from such hosts seems like the best solution here IMO. This also makes it easier for users to see the full osquery configuration (which would be more tricky to do if the query packs come from Fleet).
s

Seren

12/09/2019, 6:26 PM
Thanks for your (as seph's) help!