Hello everyone. I'm wondering if there is a fleet...
# fleet
Hello everyone. I'm wondering if there is a fleet setting that controls the max size of the osquery_results and osquery_status log files. We have run into an issue where our Fleet servers have stopped logging and these files are all at exactly 500M in size. It seems like the logs are filling faster than the rotation happens (which I think is just once per day). We stopped fleetd, backed up the results and then zeroed out the original file and started fleetd. This resulted in new data being written to osquery_results, and it filled up the 500M within several seconds. I'm not sure if that was brand new data coming in from clients or somehow queued data. How can we configure Fleet to write larger logs or rotate more frequently?
Hmm, it seems like it should auto-rotate at 500MB or 28 days according to https://github.com/kolide/fleet/blob/master/docs/infrastructure/configuring-the-fleet-binary.md Perhaps we can't write new files....looking at that.
if anyone is viewing this - can you shed some light on the differences between the osquery and filesystem stanzas?
node_key_size: 24
status_log_plugin: filesystem
result_log_plugin: filesystem
status_log_file: /var/log/osquery/osquery_status.log
result_log_file: /var/log/osquery/osquery_results.log
status_log_file: /var/log/osquery/osquery_status.log
result_log_file: /var/log/osquery/osquery_results.log
enable_log_rotation: true
enable_log_compression: true
is deprecated in the
block as the configuration for the filesystem logging was moved to the
block when we introduced the concept of logging plugins.
Probably the log fills up quickly after Fleet is restarted because the clients have been buffering logs while Fleet was offline.
yes...that's what I was thinking about the filling. I've definitely narrowed it down to not being able to write a new logfile. Need to figure out if any of our puppet code changed permissions. Last week we configured the fleet servers themselves to register with fleet and i'm wondering if we someone changed permissions on /var/log/osquery
it was permissions. We run fleetd as a specific user, but osquerys is running as root. our puppet code to configure osqueryd on the fleet servers as managed clients of fleet must have changed the permissions on /var/log/osquery to root root, hence breaking the ability for fleet to log. thanks for the feedback @zwass
🍻 1