Title
#fleet
o

Ojas

09/14/2022, 9:15 AM
Does this config look good for sending logs to kinesis data stream ? i am not seeing the logs there nor do i see any err in orbit err file :? config: spec: host_settings: enable_software_inventory: true vulnerability_settings: databases_path: /tmp/ kinesis: region: eu-west-2 access_key_id: ********** result_stream: osquery_result status_stream: osquery_status result_log_file: /var/log/osquery/result.log secret_access_key: ********* sts_assume_role_arn: >- ********** options: logger_plugin: ‘firehose,aws_firehose’ disable_carver: false pack_delimiter: / proxy_hostname: ********** carver_block_size: 2097152 logger_tls_period: 10 distributed_plugin: tls disable_distributed: false logger_tls_endpoint: /api/osquery/log distributed_interval: 10 carver_start_endpoint: /api/v1/osquery/carve/begin carver_disable_function: false carver_continue_endpoint: /api/v1/osquery/carve/block distributed_tls_max_attempts: 3 osquery: osquery_result_log_plugin: kinesis osquery_status_log_plugin: kinesis decorators: load: - SELECT uuid AS host_uuid FROM system_info; - SELECT hostname AS hostname FROM system_info; overrides: {}
Kathy Satterlee

Kathy Satterlee

09/14/2022, 3:34 PM
@Ojas How are you applying these options? Based on the structure, I think it might be in the UI.
o

Ojas

09/15/2022, 2:28 AM
i set them in global agent option @Kathy Satterlee
Kathy Satterlee

Kathy Satterlee

09/15/2022, 2:40 PM
The settings for
kinesis
and
osquery
are server settings and need to be configured in the Fleet binary using command line flags, environmental variables or a config file.
o

Ojas

09/15/2022, 2:44 PM
oh it wont work from global agent settings then, i’ll have to redeploy all the agents agains 😞
Kathy Satterlee

Kathy Satterlee

09/15/2022, 3:10 PM
You can configure your logging destination in the
options
section, there are just some settings you're missing there. Check out the example here: https://osquery.readthedocs.io/en/stable/deployment/aws-logging/
o

Ojas

09/15/2022, 6:54 PM
its always confusing for me to get the flags right coz osquery has bit different and fleet global agent has differnet. i am still unclear that this options section do i need it while building the agent or in the fleet ui in global agent options? { “options”: { “host_identifier”: “hostname”, “schedule_splay_percent”: 10, “logger_plugin”: “aws_kinesis,aws_firehose”, “aws_kinesis_stream”: “foo_stream”, “aws_firehose_stream”: “bar_delivery_stream”, “aws_access_key_id”: “ACCESS_KEY”, “aws_secret_access_key”: “SECRET_KEY”, “aws_region”: “us-east-1" }, as per this i have all of it in the kinesis section :
Kathy Satterlee

Kathy Satterlee

09/15/2022, 7:12 PM
That documentation is for the server settings. If you wanted to set this in
agent_options
rather than as a server setting, you would add the necessary osquery flags (listed in the osquery docs listed above) to
options
in the Fleet UI.
o

Ojas

09/16/2022, 5:50 AM
okay and after that config where do i see the logs if it still dosnet report to kinesis? config: spec: host_settings: enable_software_inventory: true vulnerability_settings: databases_path: /tmp/ options: host_identifier: hostname schedule_splay_percent: 10 logger_plugin: aws_kinesis,aws_firehose aws_kinesis_stream: fleetosquery**** aws_firehose_stream: fleetosquery*** aws_access_key_id: ********* aws_secret_access_key: ********* aws_region: eu-west-2 disable_carver: false pack_delimiter: / proxy_hostname: ** carver_block_size: 2097152 logger_tls_period: 10 distributed_plugin: tls disable_distributed: false logger_tls_endpoint: /api/osquery/log distributed_interval: 10 carver_start_endpoint: /api/v1/osquery/carve/begin carver_disable_function: false carver_continue_endpoint: /api/v1/osquery/carve/block distributed_tls_max_attempts: 3 schedule: time: query: SELECT * FROM time; interval: 2 removed: false osquery: osquery_result_log_plugin: kinesis osquery_status_log_plugin: kinesis decorators: load: - SELECT uuid AS host_uuid FROM system_info; - SELECT hostname AS hostname FROM system_info; overrides: {} this is my config now and still dosent report to aws
7:22 AM
@Kathy Satterlee sorry to bug you time n again but still now working and i dont see any err or anything 😞
Kathy Satterlee

Kathy Satterlee

09/16/2022, 4:09 PM
Just to make sure I can give you the best information possible, I want to ask again: How are you applying these options? In the Fleet UI? Using
fleetctl apply
? By passing a config file when starting the server? The format for those is all a little different.
o

Ojas

09/16/2022, 4:17 PM
in fleet ui
Kathy Satterlee

Kathy Satterlee

09/16/2022, 4:35 PM
Thank you! In that case, the structure should look a bit like this:
config:
  options:
    host_identifier: hostname
    schedule_splay_percent: 10
    logger_plugin: aws_kinesis,aws_firehose
    aws_kinesis_stream: fleetosquery****
    aws_firehose_stream: fleetosquery***
    aws_access_key_id: *********
    aws_secret_access_key: *********
    aws_region: eu-west-2
    disable_carver: false
    pack_delimiter: /
    proxy_hostname: **
    carver_block_size: 2097152
    logger_tls_period: 10
    distributed_plugin: tls
    disable_distributed: false
    distributed_interval: 10
    carver_start_endpoint: /api/v1/osquery/carve/begin
    carver_disable_function: false
    carver_continue_endpoint: /api/v1/osquery/carve/block
    distributed_tls_max_attempts: 3
  decorators:
    load:
      - SELECT uuid AS host_uuid FROM system_info;
      - SELECT hostname AS hostname FROM system_info;
  schedule:
      time:
        query: SELECT * FROM time;
        interval: 2
        removed: false   
overrides: {}
The properties you're setting here are all part of
agent_options
. You can use
agent_options.config
to set up the osquery configuration for your hosts. You don't have to include
agent_options
itself because that's the section you're editing. I hope that helps! Try applying those options and enrolling a new host. If things don't start passing through, check AWS and the host logs to see if there's any interesting data there.
o

Ojas

09/16/2022, 7:19 PM
@Kathy Satterlee i didnt quite understand, so my config is right then?
7:20 PM
this looks like my config only but its not pushing logs. i did check osquery.results file and it has data in there but .err etc didnt show anything for sending logs to aws so i dont have any logs for why its not pushing to aws