Hi all - I'm revisiting an old issue with configuring firehose to send logs from FleetDM to Splunk. I have the firehose configured in my fleet.yml file, but nothing is sending to the firehose. I don't see any errors related to this in my /var/log/syslog file (I see other errors related to Fleet so I'm pretty sure that's where the errors would output). I do have some automations configured, so in theory we should be seeing some data hitting that firehose and moving on to Splunk. Not sure where to proceed from here... fleet.yml (snippet): osquery: osquery_status_log_plugin: firehose osquery_result_log_plugin: firehose firehose: region: REDACT access_key_id: REDACT secret_access_key: REDACT status_stream: osquery_status result_stream: osquery_result audit_stream: fleet_audit Agent options (from Fleet GUI) config: options: pack_delimiter: / distributed_plugin: tls disable_distributed: false logger_tls_endpoint: /api/v1/osquery/log distributed_interval: 30 distributed_tls_max_attempts: 3 decorators: load: - SELECT uuid AS host_uuid FROM system_info; - SELECT hostname AS hostname FROM system_info; command_line_flags: verbose: true config_plugin: tls disable_audit: false logger_plugin: tls config_refresh: 300 disable_events: false enable_file_events: true watchdog_memory_limit: 1024 audit_allow_process_events: true enable_ntfs_event_publisher: true enable_windows_events_publisher: true enable_windows_events_subscriber: true

@Mike S. Can you run

SELECT * FROM osquery_flags;

and show us the results? Can you also send us the fleetd logs?

DM'ing!

@Mike S. We see some queries coming from packs. How are you managing those? Do you have automations scheduled for each of your scheduled queries? If not, then the logs will not be forwarded.

@Rebecca Cowart We do have automations scheduled for many of these queries. For the queries coming from packs, we just manage those through the Fleet UI. Fleet version is 4.64 One thing I just noticed is that the destination is set to the filesystem, which might be the issue.

@Mike S. Perfect. Update that destination, and let's see if that solves your issue!

Would I set that in the agent options, or in Fleet.yml?

@Mike S. I would check the Fleet UI first and ensure your destination is set to your firehose. Is that where you just saw it incorrectly set to filesystem?

I noticed it when I double-checked to see if automations were enabled:

@Mike S. How are you passing your config file to Fleet? It looks as though your config file is not being applied. Have you restarted Fleet since configuring your Firehose?

We were doing it through agent options, but I believe that was sending everything, not just the automatied queries, and we didn't want that. So we have that configured in fleet.yml.

@Mike S. How do you have Fleet deployed? We are trying to determine how you are passing that yaml file to Fleet.

Sorry, i saw you asked that earlier but missed it. Let me give it a reboot since I can't be sure that has happened.

@Mike S. Are you using our Terraform? It sounds as if you're using AWS. Typically, in cloud-based deployment, you would be setting environment variables, rather than using a config file.

Nah, we built this manually before we had Terraform setup. We are using AWS.

@Mike S. In this case, we would expect this all to be set using environment variables. Is that config file something you created just for this, or did it already exist?

It was created as part of the Fleet server configuration - I pulled the info from here: https://fleetdm.com/docs/configuration/fleet-server-configuration

@Mike S. I would recommend configuring these as environment variables.

as long as you are launching the fleet server with

serve --config config.yml

(sub name of your config file) it should work fine. Make sure you aren't confusing the fleet server yaml configuration with agent configuration. The firehose settings are only relevant for the server configuration. If you are still seeing filesystem as the backend (its the default value) then your configuration isn't being supplied correctly or the server needs to be restarted to pick up the new changes.

Yep, that's how it's being served - /usr/bin/fleetdm/fleet serve --config /usr/bin/fleetdm/fleet.yml

the server config should look something like:

server:
  tls: false
logging:
  debug: true
osquery:
  result_log_plugin: firehose
firehose:
  region: ca-central-1
  result_stream: osquery_result

(omitted some of the other keys for brevity)

I do see a difference in mine: osquery: osquery_status_log_plugin: firehose osquery_result_log_plugin: firehose Should that just be result_log_plugin and status_log_plugin?

yeah those aren't valid keys.

@Mike S. Ah! And I checked over your config values so thoroughly, too! My eyes deceived me!

No worries! That’s been sitting there for who knows how long and I missed it too!