is it consistently reproducible given some conditi...
# kolidem
marpaia
07/06/2018, 4:11 PMis it consistently reproducible given some conditions?
s
stefanmaerz
07/06/2018, 4:47 PMIts been doing it ever since I stood up fleet on OpenShift. I've observed it with a variety of queries.
stefanmaerz
07/06/2018, 4:48 PMI'm load balancing two fleet pods and both are writing to one single persistent volume claim. I'm wondering if this behavior is related to that. I've scaled down to 1 fleet pod to test this theory.
m
marpaia
07/06/2018, 4:57 PMonly on openshift?
s
stefanmaerz
07/06/2018, 5:08 PMCorrect -- I don't recall this behavior back when I was running the fleet server on a VM.
stefanmaerz
07/06/2018, 5:09 PMI also just turned log retention off. It seems to make logging more finicky. Should have some good data here pretty soon once my queries run.
stefanmaerz
07/06/2018, 5:18 PMokay logs are starting to look good now. That raises a question about architecting load balanced fleet: should I make my fleet instances log to different files? right now they are sharing a file on a shared persistent volume claim.
stefanmaerz
07/06/2018, 5:35 PMWell, now I can't reproduce it. Everything is working shrugs
stefanmaerz
07/06/2018, 5:36 PMif I can figure out a way to reliably reproduce, I'll file a github issue
stefanmaerz
07/06/2018, 5:39 PMNope. Have reproduced. It is the multiple pods writing to one file. Not a bug with fleet, just me being dumb.
stefanmaerz
07/06/2018, 5:39 PMThanks for the help though!
z
zwass
07/10/2018, 12:44 AM
Just following up on this... I think it's best to have each Fleet instance logging to its own file (because they don't have any coordination for writes to a shared file).
👍 1
s
stefanmaerz
07/10/2018, 2:02 PMYep, that's my plan right now @zwass. Doing a little bit of openshift magic to make it happen, then will configure my splunkforwarder pod to merge the discrete log files back together transparently into a single index.
👍 1
2 Views