Title
#kolide
zwass

zwass

10/06/2020, 9:42 PM
Hey folks, how are you generating alerts with your osquery logs? Would it be useful to have this functionality directly in Fleet?
z

Zach Zeid

10/07/2020, 12:05 PM
on this. We have a lambda function that pulls the results out of ECS cloudwatch logs and sends them to Streamalert, but would love to have that integration, or even to be able to set up alerts on
missing
or
offline
hosts
sundsta

sundsta

10/07/2020, 3:14 PM
Currently our logs are sent to our central logging platform via GCP PubSub where we have some alerts. We’re looking at building out a more flexible alerting pipeline by adding additional subscribers to the PubSub topic, but if that was built into Fleet that would save us a lot of time 🙂
z

Zach Zeid

10/07/2020, 3:15 PM
Just being able to have more ways to get the results from fleet to somewhere else would be choice.
Dan Achin

Dan Achin

10/08/2020, 8:53 PM
Yes, I think that would he helpful to have. We'll be using Splunk to generate our alerts, but having the flexibility to do that also from Fleet would be helpful, especially if we decide to somehow limit what we send to Splunk due to cost.
zwass

zwass

10/08/2020, 9:41 PM
Thanks for the responses! Where would you want alerts to go? Email? Webhook? Others?
sundsta

sundsta

10/08/2020, 9:52 PM
Both of those, along with a way to template the alert message. The webhook should also support headers, so authorization tokens, etc. can be included
z

Zach Zeid

10/08/2020, 10:01 PM
to the above.
sundsta

sundsta

10/08/2020, 10:03 PM
If you’re feeling really nice, pre-defined webhooks with the required fields clearly laid out for Slack, Teams, and others are nice as it makes it harder to mess up the configuration 🙂 But of course, if there’s a generic webhook feature, the community can pretty easily add those as needed
zwass

zwass

10/08/2020, 10:10 PM
This is great, please keep the ideas coming.
Ayan

Ayan

10/08/2020, 10:17 PM
In our environment we collect logs via fleet which are sent over to elasticsearch via filebeat and from there we generate alerts on it using elastalert
10:17 PM
For alerts we use Slack Webhooks
sundsta

sundsta

10/08/2020, 11:18 PM
@zwass If you’re up for it, the alerting engine could use OPA’s Rego language (https://www.openpolicyagent.org/docs/latest/policy-language/). I’ve used it pretty extensively now for writing Kubernetes policies and while the language has a small learning curve, it is a real pleasure to use for making decisions on structured data. It wouldn’t be hard to adapt for this case, and OPA has some nice features such as built-in ability to make HTTP requests to remote endpoints and/or sync other data into its cache to be used in the decision making, allowing for much more complex policies (aka alerts in this context) outside of relying solely on the data returned from osquery on the endpoint
Dan Achin

Dan Achin

10/08/2020, 11:35 PM
+ 1 to OPA
zwass

zwass

10/08/2020, 11:36 PM
At a glance Rego looks hard to learn for someone used to writing SQL statements. I had been looking at https://github.com/antonmedv/expr which seems much more intuitive.
11:36 PM
I'd be curious to hear more about what draws each of you to OPA/Rego.
defensivedepth

defensivedepth

11/10/2020, 2:14 PM
@zwass We send the osquery logs to elasticsearch, and then use the Sigma toolkit to generate an ElastAlert rule (https://github.com/Neo23x0/sigma)