im attempting to use an amazonlinux instance in aw...
# aws
im attempting to use an amazonlinux instance in aws with osquery on it to send osquery logs to elasticsearch. it doesnt look like i can write directly so I setup a firehose to write to elasticsearch and am attempting to write osquery logs to the firehose. are there any additional flags I can use for troubleshooting? I have a role attached to the instance that gives full list and write ability to the firehose but im still not seeing any activity on it.
Do you have any queries scheduled? If you run osquery in
mode do you see any error output? You can try enabling filesystem logging as well by using
and see if you get logs that way.
Or consider using
oh "aws_firehose" is a different flag than i read in the documentation. ill try that
so I am getting log output to the local filesystem in /var/log/results. but it is the sending to firehose that I am having issue with. I will try some additional flags. I already did with the verbose flag and didnt find the messages too helpful
Yeah the name of the logger plugin is
{"name":"listeningPorts","hostIdentifier":"ip-172-31-24-196.ec2.internal","calendarTime":"Mon Jul 1 132306 2019 UTC","unixTime":1561987386,"epoch":0,"counter":0,"columns":{"address":"::","family":"10","fd":"10","net_namespace":"4026531993","path":"","pid":"2576","port":"626","protocol":"17","socket":"1573"},"action":"added"}
thats what im getting locally
i have a role attached to the instance, do i really need to use akid and key?
Looks like your queries are generating results
yes i read and used it as my basis for the config
this is my current config in full
I don't see any obvious issue with your config... There's no verbose logging indicating attempts/errors in Firehose logging?
i think making the change ""logger_plugin": "aws_firehose,filesystem"," might have fixed it. im seeing log activity to firehose. waiting to see when it writes to elasticsearch.
best view ive had in 2 days
thank you for your help! i will reply to the thread in a bit if logging made it to elasticsearch like i wanted it to
Awesome, I don't have any experience with Firehose -> Elastic, but I'm sure there's plenty of info about that on the internet.
ive got that part down pat 🙂 just wanted to announce success based on what logs are ingested or if they are borked in formatting
confirmed that I am now ingesting osquery into firehose and writing to aws elasticsearch service. thanks again for the help!
🎉 1