im attempting to use an amazonlinux instance in aws with osquery on it to send osquery logs to elasticsearch. it doesnt look like i can write directly so I setup a firehose to write to elasticsearch and am attempting to write osquery logs to the firehose. are there any additional flags I can use for troubleshooting? I have a role attached to the instance that gives full list and write ability to the firehose but im still not seeing any activity on it.

Do you have any queries scheduled? If you run osquery in

--verbose

mode do you see any error output? You can try enabling filesystem logging as well by using

--logger_plugin=aws_firehose,filesystem

and see if you get logs that way.

oh "aws_firehose" is a different flag than i read in the documentation. ill try that

Yeah the name of the logger plugin is

aws_firehose

{"name":"listeningPorts","hostIdentifier":"ip-172-31-24-196.ec2.internal","calendarTime":"Mon Jul 1 132306 2019 UTC","unixTime":1561987386,"epoch":0,"counter":0,"columns":{"address":"::","family":"10","fd":"10","net_namespace":"4026531993","path":"","pid":"2576","port":"626","protocol":"17","socket":"1573"},"action":"added"}

Have you read https://osquery.readthedocs.io/en/stable/deployment/aws-logging/?

yes i read https://osquery.readthedocs.io/en/stable/deployment/aws-logging/? and used it as my basis for the config

I don't see any obvious issue with your config... There's no verbose logging indicating attempts/errors in Firehose logging?

i think making the change ""logger_plugin": "aws_firehose,filesystem"," might have fixed it. im seeing log activity to firehose. waiting to see when it writes to elasticsearch.

Nice

best view ive had in 2 days

Awesome, I don't have any experience with Firehose -> Elastic, but I'm sure there's plenty of info about that on the internet.

ive got that part down pat 🙂 just wanted to announce success based on what logs are ingested or if they are borked in formatting