is it consistently reproducible given some conditions?
07/06/2018, 4:47 PM
Its been doing it ever since I stood up fleet on OpenShift. I've observed it with a variety of queries.
I'm load balancing two fleet pods and both are writing to one single persistent volume claim. I'm wondering if this behavior is related to that. I've scaled down to 1 fleet pod to test this theory.
07/06/2018, 4:57 PM
only on openshift?
07/06/2018, 5:08 PM
Correct -- I don't recall this behavior back when I was running the fleet server on a VM.
I also just turned log retention off. It seems to make logging more finicky. Should have some good data here pretty soon once my queries run.
okay logs are starting to look good now. That raises a question about architecting load balanced fleet: should I make my fleet instances log to different files? right now they are sharing a file on a shared persistent volume claim.
Well, now I can't reproduce it. Everything is working shrugs
if I can figure out a way to reliably reproduce, I'll file a github issue
Nope. Have reproduced. It is the multiple pods writing to one file. Not a bug with fleet, just me being dumb.
Thanks for the help though!
07/10/2018, 12:44 AM
Just following up on this... I think it's best to have each Fleet instance logging to its own file (because they don't have any coordination for writes to a shared file).
Yep, that's my plan right now @zwass. Doing a little bit of openshift magic to make it happen, then will configure my splunkforwarder pod to merge the discrete log files back together transparently into a single index.