Channels
android_tests
apple-silicon
arm-architecture
auditing-warroom
aws
carving
code-review
community-feeds
core
darkbytes
doorman
ebpf
eclecticiq-polylogyx-extension
extensions
file-carving
fim
fleet
fleet-dev
fleetosquery
foundation
fuzzing
general
golang
goquery
infrastructure
jobs
kolide
linen-dev
linux
macos
officehours
osctrl
plugins
process-auditing
querycon
queryhub
random
selfgroup
sql
tls
uptycs
vendor-feeds
website
windows
zeek
zentral
zercurity
Title
d
Dan Achin
02/18/2021, 6:27 PMHello everyone. I'm wondering if there is a fleet setting that controls the max size of the osquery_results and osquery_status log files. We have run into an issue where our Fleet servers have stopped logging and these files are all at exactly 500M in size. It seems like the logs are filling faster than the rotation happens (which I think is just once per day). We stopped fleetd, backed up the results and then zeroed out the original file and started fleetd. This resulted in new data being written to osquery_results, and it filled up the 500M within several seconds. I'm not sure if that was brand new data coming in from clients or somehow queued data.
How can we configure Fleet to write larger logs or rotate more frequently?
Hmm, it seems like it should auto-rotate at 500MB or 28 days according to https://github.com/kolide/fleet/blob/master/docs/infrastructure/configuring-the-fleet-binary.md
Perhaps we can't write new files....looking at that.
if anyone is viewing this - can you shed some light on the differences between the osquery and filesystem stanzas?
osquery:node_key_size: 24status_log_plugin: filesystemresult_log_plugin: filesystemstatus_log_file: /var/log/osquery/osquery_status.logresult_log_file: /var/log/osquery/osquery_results.logfilesystem:status_log_file: /var/log/osquery/osquery_status.logresult_log_file: /var/log/osquery/osquery_results.logenable_log_rotation: trueenable_log_compression: truez
zwass
02/18/2021, 7:47 PMstatus_log_fileosqueryfilesystemYou can disable log rotation if you like with https://github.com/fleetdm/fleet/blob/master/docs/2-Deployment/2-Configuration.md#filesystem_enable_log_rotation.
👍 1
Probably the log fills up quickly after Fleet is restarted because the clients have been buffering logs while Fleet was offline.
d
Dan Achin
02/18/2021, 7:59 PMyes...that's what I was thinking about the filling. I've definitely narrowed it down to not being able to write a new logfile. Need to figure out if any of our puppet code changed permissions. Last week we configured the fleet servers themselves to register with fleet and i'm wondering if we someone changed permissions on /var/log/osquery
it was permissions. We run fleetd as a specific user, but osquerys is running as root. our puppet code to configure osqueryd on the fleet servers as managed clients of fleet must have changed the permissions on /var/log/osquery to root root, hence breaking the ability for fleet to log. thanks for the feedback @zwass
🍻 1