I am having trouble sending fleet data to my s3 bucket. I have the config setup in the terraform but i am not getting anything in my s3 bucket.

module "osquery-carve" {
  # The carve bucket also stores software.
  source = "github.com/fleetdm/fleet//terraform/addons/osquery-carve?ref=tf-mod-addon-osquery-carve-v1.1.0"
  osquery_carve_s3_bucket = {
    name = local.osquery_carve_bucket_name
  }   
} 

module "firehose-logging" {
  source = "github.com/fleetdm/fleet//terraform/addons/logging-destination-firehose?ref=tf-mod-addon-logging-destination-firehose-v1.1.0"
  osquery_results_s3_bucket = {
    name = local.osquery_results_bucket_name
  }
  osquery_status_s3_bucket = {
    name = local.osquery_status_bucket_name
  }
}

If you were able to apply without errors, the buckets were created, and the Fleet service is running, it certainly sounds like the setup is correct! Are you seeing any errors in Fleet around logging?

I am not seeing any errors in around logging, I have cloudwatch turned on. I set up queries but its been like 12 hours and nothing gets dropped into the S3 buckets

Did you enable automations for your queries?

yes I did, not sure if it should say FileSystem though

Do you also have the buckets added to your locals?

yes

I see that this isn't explicitly included in the example (I'll take a look at that shortly), but are you passing the outputs from the firehose logging module to Fleet?

I am not sure, how would i check that?

That would be in the fleet_config. I see that he example is passing the iam policies, but not the environment variables. Try:

extra_environment_variables = concat (
  local.fleet_environment_variables
  module.firehose-logging.fleet_extra_environment_variables
)

seems like the concat threw me an error, going to try with a merge

I think I missed a comma in there too, sorry about that!

seems like that worked! THANK YOU SOO MUCH!

Awesome! I'm glad I could help.

Is there any documentation on sending OSquery to SPlunk?

There isn't a direct method, you'd forward from Firehose to Splunk. There's a Terraform module for that: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream#splunk-destination

do you happen to know how to break up the snapshot in so it could parse in Splunk? looks like the query returned too many items

There's some osquery configuration for logging that you can tweak., compression may also be an option if that's supported by Splunk.

@Alex Guerrero did you ever figure out how to properly parse results in Splunk? We're running into a similar issue