https://github.com/osquery/osquery logo
Powered by
Title
n

Nico

07/03/2019, 8:35 PM
hey 👋 I have fleet setup with my hosts reporting to it. I'm trying to get the firehose output to work correctly. So far I get the status log in there, but not the results. They are both set to save to the same firehose stream, could that be the issue? I have a pack enabled with 
select * from time
running every 30s but I don't see any results from that. I can run manual queries and see the result in the UI fine, but nothing in firehose. When I do that I see 
Executing distributed query: kolide_distributed_query_45: SELECT * FROM time
sent to firehose but no actual result. Here are the fleet env vars I'm using (excluding mysql and redis). I did a manual write to the stream with the same creds with awscli successfully (The role has permission to describe and put record batch). 
export KOLIDE_LOGGING_DEBUG=true
export KOLIDE_FIREHOSE_REGION=${KOLIDE_CONFIG_FIREHOSE_REGION}
export KOLIDE_FIREHOSE_RESULT_STREAM=${KOLIDE_CONFIG_FIREHOSE_RESULT_STREAM}
export KOLIDE_FIREHOSE_STATUS_STREAM=${KOLIDE_CONFIG_FIREHOSE_STATUS_STREAM}
export KOLIDE_OSQUERY_RESULT_LOG_PLUGIN=firehose
export KOLIDE_OSQUERY_STATUS_LOG_PLUGIN=firehose
I'm using firehose with elasticsearch and s3 backup, and there is no failed delivery files in s3. What is the best way to debug this issue? I don't see any errors in the fleet output itself. Thanks!
sorry had posted in general by mistake
z

zwass

07/03/2019, 8:49 PM
First, live query results will never go to the logger. You'll only see them through the UI/fleetctl.
My guess is that pack is not targeted appropriately to the host. Can you take a look at how it is targeted?
Also, take a look at the logs on the host to see whether it is every running the scheduled query.
n

Nico

07/03/2019, 8:50 PM
the pack is targeted to all hosts
let me check the logs on a host
the only thing I see in there is the distributed query read and log. 
--tls-hostname XXX
--logger_plugin=tls
--config_plugin=tls
--verbose
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_tls_endpoint=/api/v1/osquery/config
--enroll_secret_env=OSQUERY_ENROLL_SECRET
my flags
do I need a special one to enable packs?
oh, I think I just realised, I need the launcher for the packs right? the osqueryd doesn't have the ability to fetch new config right?
s

sundsta

07/03/2019, 9:16 PM
You don't need Launcher. The osquery_status logs should at least show you if it is connecting to Fleet and if it is executing the scheduled queries
n

Nico

07/03/2019, 9:19 PM
I do see 
Jul 03 21:17:46 ip-172-23-6-159 osquery.sh[1019]: I0703 21:17:46.221714  1263 tls.cpp:240] TLS/HTTPS POST request to URI:
for distributed read and log
no other log
s

sundsta

07/03/2019, 9:21 PM
Then it is not a target for that pack. You're looking for logs like 
{"hostIdentifier":"FB2C3611-EA6B-C10B-3611-3C4CF096B325","calendarTime":"Wed Jul  3 21:17:52 2019 UTC","unixTime":"1562188672","severity":"0","filename":"scheduler.cpp","line":"100","message":"Executing scheduled query pack/events/user_events: SELECT * FROM user_events \nWHERE time < DATE('now') - 300","version":"3.3.2","decorations":{"host_uuid":"FB2C3611-EA6B-C10B-3611-3C4CF096B325","hostname":"REDACTED.local"}}
(we log in JSON, so your formatting will be a bit different)
n

Nico

07/03/2019, 9:21 PM
right sorry I do have those
they are forwarded to firehose
s

sundsta

07/03/2019, 9:22 PM
OK, so it's working?
n

Nico

07/03/2019, 9:22 PM
things like opening rocksdb etc
but nothing about scheduled
s

sundsta

07/03/2019, 9:22 PM
OK, then that host isn't targeted for the pack or the pack contains no queries.
n

Nico

07/03/2019, 9:23 PM
z

zwass

07/03/2019, 10:05 PM
This pack looks targeted correctly
Can you run osqueryd on the host with —verbose —tls_dump and see what config it receives?
n

Nico

07/04/2019, 8:18 AM
The issue was I didn't have config-refresh set on osqueryd, I do get my results now. Thanks @zwass @sundsta 🙂
👍 1
5 Views
#kolideJoin Slack
Powered by