Mystery Incorporated

Mystery Incorporated

10/26/2020, 4:14 AM
Hiya, What is the best way to not let osquery result log get too big? Sitting there recording event log in about 12 hours has used 574MB lol

Prateek Kumar Nischal

10/26/2020, 7:38 AM
This may be of interest: https://github.com/osquery/osquery/issues/302 I don’t think osquery implements log rotation natively. You might want to use a platform native log rotation mechanism. eg: logrotate for linux


10/26/2020, 2:41 PM
Generally this would be the responsibility of whatever is ingesting the logs. Many deployments have osquery push the logs off local disk directly, bypassing this completely.
Mystery Incorporated

Mystery Incorporated

10/26/2020, 10:58 PM
Thanks yes I was thinking msybe filebeat or logstash or something as i want to push to elastic but was thinking what would happen in a situation where the device was offline for a prolonged time and couldn't send logs back to elastic?
Gonzalo Saad

Gonzalo Saad

10/26/2020, 11:52 PM
If it helps: We use kinesis for this the logs and we observed that given an impossibility of publishing (for example wrong creds or absence of internet), it caches the results until it can publish again. When back online all of the results come at once, but got reports of osquery taking higher resources than expected in the devices.
11:52 PM
Also I really recommend the Kinesis + Lambda for processing the logs. It scales as hell and it's pretty cheap. We're processing 16M records per day at a cost of ~85 USD a month (only counting computing power, not storage services).