https://github.com/osquery/osquery logo
Title
s

sonofagl1tch

07/01/2019, 5:15 PM
im attempting to use an amazonlinux instance in aws with osquery on it to send osquery logs to elasticsearch. it doesnt look like i can write directly so I setup a firehose to write to elasticsearch and am attempting to write osquery logs to the firehose. are there any additional flags I can use for troubleshooting? I have a role attached to the instance that gives full list and write ability to the firehose but im still not seeing any activity on it.
z

zwass

07/01/2019, 5:52 PM
Do you have any queries scheduled? If you run osquery in
--verbose
mode do you see any error output? You can try enabling filesystem logging as well by using
--logger_plugin=aws_firehose,filesystem
and see if you get logs that way.
Or consider using
--logtostderr
s

sonofagl1tch

07/01/2019, 6:10 PM
oh "aws_firehose" is a different flag than i read in the documentation. ill try that
so I am getting log output to the local filesystem in /var/log/results. but it is the sending to firehose that I am having issue with. I will try some additional flags. I already did with the verbose flag and didnt find the messages too helpful
z

zwass

07/01/2019, 6:15 PM
Yeah the name of the logger plugin is
aws_firehose
s

sonofagl1tch

07/01/2019, 6:15 PM
{"name":"listeningPorts","hostIdentifier":"ip-172-31-24-196.ec2.internal","calendarTime":"Mon Jul 1 13:23:06 2019 UTC","unixTime":1561987386,"epoch":0,"counter":0,"columns":{"address":"::","family":"10","fd":"10","net_namespace":"4026531993","path":"","pid":"2576","port":"626","protocol":"17","socket":"1573"},"action":"added"}
thats what im getting locally
i have a role attached to the instance, do i really need to use akid and key?
Looks like your queries are generating results
s

sonofagl1tch

07/01/2019, 6:18 PM
yes i read https://osquery.readthedocs.io/en/stable/deployment/aws-logging/? and used it as my basis for the config
this is my current config in full
z

zwass

07/01/2019, 6:23 PM
I don't see any obvious issue with your config... There's no verbose logging indicating attempts/errors in Firehose logging?
s

sonofagl1tch

07/01/2019, 6:26 PM
i think making the change ""logger_plugin": "aws_firehose,filesystem"," might have fixed it. im seeing log activity to firehose. waiting to see when it writes to elasticsearch.
z

zwass

07/01/2019, 6:27 PM
Nice
s

sonofagl1tch

07/01/2019, 6:28 PM
best view ive had in 2 days
thank you for your help! i will reply to the thread in a bit if logging made it to elasticsearch like i wanted it to
z

zwass

07/01/2019, 6:31 PM
Awesome, I don't have any experience with Firehose -> Elastic, but I'm sure there's plenty of info about that on the internet.
s

sonofagl1tch

07/01/2019, 6:32 PM
ive got that part down pat 🙂 just wanted to announce success based on what logs are ingested or if they are borked in formatting
confirmed that I am now ingesting osquery into firehose and writing to aws elasticsearch service. thanks again for the help!
🎉 1