Title
#kolide
marpaia

marpaia

07/06/2018, 4:11 PM
is it consistently reproducible given some conditions?
stefanmaerz

stefanmaerz

07/06/2018, 4:47 PM
Its been doing it ever since I stood up fleet on OpenShift. I've observed it with a variety of queries.
4:48 PM
I'm load balancing two fleet pods and both are writing to one single persistent volume claim. I'm wondering if this behavior is related to that. I've scaled down to 1 fleet pod to test this theory.
marpaia

marpaia

07/06/2018, 4:57 PM
only on openshift?
stefanmaerz

stefanmaerz

07/06/2018, 5:08 PM
Correct -- I don't recall this behavior back when I was running the fleet server on a VM.
5:09 PM
I also just turned log retention off. It seems to make logging more finicky. Should have some good data here pretty soon once my queries run.
5:18 PM
okay logs are starting to look good now. That raises a question about architecting load balanced fleet: should I make my fleet instances log to different files? right now they are sharing a file on a shared persistent volume claim.
5:35 PM
Well, now I can't reproduce it. Everything is working shrugs
5:36 PM
if I can figure out a way to reliably reproduce, I'll file a github issue
5:39 PM
Nope. Have reproduced. It is the multiple pods writing to one file. Not a bug with fleet, just me being dumb.
5:39 PM
Thanks for the help though!
zwass

zwass

07/10/2018, 12:44 AM
Just following up on this... I think it's best to have each Fleet instance logging to its own file (because they don't have any coordination for writes to a shared file).
stefanmaerz

stefanmaerz

07/10/2018, 2:02 PM
Yep, that's my plan right now @zwass. Doing a little bit of openshift magic to make it happen, then will configure my splunkforwarder pod to merge the discrete log files back together transparently into a single index.