Hello everyone. I'm wondering if there is a fleet...
# fleetd
Dan Achin
02/18/2021, 6:27 PMHello everyone. I'm wondering if there is a fleet setting that controls the max size of the osquery_results and osquery_status log files. We have run into an issue where our Fleet servers have stopped logging and these files are all at exactly 500M in size. It seems like the logs are filling faster than the rotation happens (which I think is just once per day). We stopped fleetd, backed up the results and then zeroed out the original file and started fleetd. This resulted in new data being written to osquery_results, and it filled up the 500M within several seconds. I'm not sure if that was brand new data coming in from clients or somehow queued data.
How can we configure Fleet to write larger logs or rotate more frequently?
Dan Achin
02/18/2021, 7:35 PMHmm, it seems like it should auto-rotate at 500MB or 28 days according to https://github.com/kolide/fleet/blob/master/docs/infrastructure/configuring-the-fleet-binary.md
Perhaps we can't write new files....looking at that.
Dan Achin
02/18/2021, 7:38 PMif anyone is viewing this - can you shed some light on the differences between the osquery and filesystem stanzas?
osquery:node_key_size: 24status_log_plugin: filesystemresult_log_plugin: filesystemstatus_log_file: /var/log/osquery/osquery_status.logresult_log_file: /var/log/osquery/osquery_results.logfilesystem:status_log_file: /var/log/osquery/osquery_status.logresult_log_file: /var/log/osquery/osquery_results.logenable_log_rotation: trueenable_log_compression: truez
zwass
02/18/2021, 7:47 PM
status_log_fileosqueryfilesystemzwass
02/18/2021, 7:47 PM
You can disable log rotation if you like with https://github.com/fleetdm/fleet/blob/master/docs/2-Deployment/2-Configuration.md#filesystem_enable_log_rotation.
👍 1
zwass
02/18/2021, 7:48 PM
Probably the log fills up quickly after Fleet is restarted because the clients have been buffering logs while Fleet was offline.
d
Dan Achin
02/18/2021, 7:59 PMyes...that's what I was thinking about the filling. I've definitely narrowed it down to not being able to write a new logfile. Need to figure out if any of our puppet code changed permissions. Last week we configured the fleet servers themselves to register with fleet and i'm wondering if we someone changed permissions on /var/log/osquery
Dan Achin
02/18/2021, 8:08 PMit was permissions. We run fleetd as a specific user, but osquerys is running as root. our puppet code to configure osqueryd on the fleet servers as managed clients of fleet must have changed the permissions on /var/log/osquery to root root, hence breaking the ability for fleet to log. thanks for the feedback @zwass
🍻 1
7 Views