https://github.com/osquery/osquery logo
#kolide
Title
# kolide
m

marpaia

07/06/2018, 4:11 PM
is it consistently reproducible given some conditions?
s

stefanmaerz

07/06/2018, 4:47 PM
Its been doing it ever since I stood up fleet on OpenShift. I've observed it with a variety of queries.
I'm load balancing two fleet pods and both are writing to one single persistent volume claim. I'm wondering if this behavior is related to that. I've scaled down to 1 fleet pod to test this theory.
m

marpaia

07/06/2018, 4:57 PM
only on openshift?
s

stefanmaerz

07/06/2018, 5:08 PM
Correct -- I don't recall this behavior back when I was running the fleet server on a VM.
I also just turned log retention off. It seems to make logging more finicky. Should have some good data here pretty soon once my queries run.
okay logs are starting to look good now. That raises a question about architecting load balanced fleet: should I make my fleet instances log to different files? right now they are sharing a file on a shared persistent volume claim.
Well, now I can't reproduce it. Everything is working shrugs
if I can figure out a way to reliably reproduce, I'll file a github issue
Nope. Have reproduced. It is the multiple pods writing to one file. Not a bug with fleet, just me being dumb.
Thanks for the help though!
z

zwass

07/10/2018, 12:44 AM
Just following up on this... I think it's best to have each Fleet instance logging to its own file (because they don't have any coordination for writes to a shared file).
👍 1
s

stefanmaerz

07/10/2018, 2:02 PM
Yep, that's my plan right now @zwass. Doing a little bit of openshift magic to make it happen, then will configure my splunkforwarder pod to merge the discrete log files back together transparently into a single index.
👍 1
2 Views