Does this config look good for sending logs to kinesis data stream ? i am not seeing the logs there nor do i see any err in orbit err file :? config: spec: host_settings: enable_software_inventory: true vulnerability_settings: databases_path: /tmp/ kinesis: region: eu-west-2 access_key_id: ********** result_stream: osquery_result status_stream: osquery_status result_log_file: /var/log/osquery/result.log secret_access_key: ********* sts_assume_role_arn: >- ********** options: logger_plugin: ‘firehose,aws_firehose’ disable_carver: false pack_delimiter: / proxy_hostname: ********** carver_block_size: 2097152 logger_tls_period: 10 distributed_plugin: tls disable_distributed: false logger_tls_endpoint: /api/osquery/log distributed_interval: 10 carver_start_endpoint: /api/v1/osquery/carve/begin carver_disable_function: false carver_continue_endpoint: /api/v1/osquery/carve/block distributed_tls_max_attempts: 3 osquery: osquery_result_log_plugin: kinesis osquery_status_log_plugin: kinesis decorators: load: - SELECT uuid AS host_uuid FROM system_info; - SELECT hostname AS hostname FROM system_info; overrides: {}

@Ojas How are you applying these options? Based on the structure, I think it might be in the UI.

i set them in global agent option @Kathy Satterlee

The settings for

kinesis

and

osquery

are server settings and need to be configured in the Fleet binary using command line flags, environmental variables or a config file.

oh it wont work from global agent settings then, i’ll have to redeploy all the agents agains 😞

You can configure your logging destination in the

options

section, there are just some settings you're missing there. Check out the example here: https://osquery.readthedocs.io/en/stable/deployment/aws-logging/

its always confusing for me to get the flags right coz osquery has bit different and fleet global agent has differnet. i am still unclear that this options section do i need it while building the agent or in the fleet ui in global agent options? { “options”: { “host_identifier”: “hostname”, “schedule_splay_percent”: 10, “logger_plugin”: “aws_kinesis,aws_firehose”, “aws_kinesis_stream”: “foo_stream”, “aws_firehose_stream”: “bar_delivery_stream”, “aws_access_key_id”: “ACCESS_KEY”, “aws_secret_access_key”: “SECRET_KEY”, “aws_region”: “us-east-1" }, as per this i have all of it in the kinesis section :

That documentation is for the server settings. If you wanted to set this in

agent_options

rather than as a server setting, you would add the necessary osquery flags (listed in the osquery docs listed above) to

options

in the Fleet UI.

okay and after that config where do i see the logs if it still dosnet report to kinesis? config: spec: host_settings: enable_software_inventory: true vulnerability_settings: databases_path: /tmp/ options: host_identifier: hostname schedule_splay_percent: 10 logger_plugin: aws_kinesis,aws_firehose aws_kinesis_stream: fleetosquery**** aws_firehose_stream: fleetosquery*** aws_access_key_id: ********* aws_secret_access_key: ********* aws_region: eu-west-2 disable_carver: false pack_delimiter: / proxy_hostname: ** carver_block_size: 2097152 logger_tls_period: 10 distributed_plugin: tls disable_distributed: false logger_tls_endpoint: /api/osquery/log distributed_interval: 10 carver_start_endpoint: /api/v1/osquery/carve/begin carver_disable_function: false carver_continue_endpoint: /api/v1/osquery/carve/block distributed_tls_max_attempts: 3 schedule: time: query: SELECT * FROM time; interval: 2 removed: false osquery: osquery_result_log_plugin: kinesis osquery_status_log_plugin: kinesis decorators: load: - SELECT uuid AS host_uuid FROM system_info; - SELECT hostname AS hostname FROM system_info; overrides: {} this is my config now and still dosent report to aws

Just to make sure I can give you the best information possible, I want to ask again: How are you applying these options? In the Fleet UI? Using

fleetctl apply

? By passing a config file when starting the server? The format for those is all a little different.

in fleet ui

Thank you! In that case, the structure should look a bit like this:

config:
  options:
    host_identifier: hostname
    schedule_splay_percent: 10
    logger_plugin: aws_kinesis,aws_firehose
    aws_kinesis_stream: fleetosquery****
    aws_firehose_stream: fleetosquery***
    aws_access_key_id: *********
    aws_secret_access_key: *********
    aws_region: eu-west-2
    disable_carver: false
    pack_delimiter: /
    proxy_hostname: **
    carver_block_size: 2097152
    logger_tls_period: 10
    distributed_plugin: tls
    disable_distributed: false
    distributed_interval: 10
    carver_start_endpoint: /api/v1/osquery/carve/begin
    carver_disable_function: false
    carver_continue_endpoint: /api/v1/osquery/carve/block
    distributed_tls_max_attempts: 3
  decorators:
    load:
      - SELECT uuid AS host_uuid FROM system_info;
      - SELECT hostname AS hostname FROM system_info;
  schedule:
      time:
        query: SELECT * FROM time;
        interval: 2
        removed: false   
overrides: {}

The properties you're setting here are all part of

agent_options

. You can use

agent_options.config

to set up the osquery configuration for your hosts. You don't have to include

agent_options

itself because that's the section you're editing. I hope that helps! Try applying those options and enrolling a new host. If things don't start passing through, check AWS and the host logs to see if there's any interesting data there.

@Kathy Satterlee i didnt quite understand, so my config is right then?