Fleet seems to be logging a lot of lines like this during st

Question

Fleet seems to be logging a lot of lines like this during startup does anyone know what they mean ```level=error ts=2022 02 08T18 18 27 469599841Z op=QueriesForHost err= load active queries EOF ```

Tomas Touceda · Answer

that part reaches out to redis to check for live queries that might be running

Tomas Touceda · Answer

is redis running properly

Ryan · Answer

ah right so EOF could mean a problem connecting to Redis

Ryan · Answer

I have suspicions that it isn t able to connect properly

Ryan · Answer

thanks for the pointer thumbsup

Tomas Touceda · Answer

yup it connects to redis and runs a SMEMBERS if at some point something just chops off the connection that s the error that could happen

Ryan · Answer

right oh

Ryan · Answer

i ll double check the setup

Ryan · Answer

Still debugging but I may have to continue tomorrow

Ryan · Answer

Incidentally we re seeing this logged a lot when executing distributed queries ```method=POST uri= api v1 osquery distributed write took=1 86989925s ip addr= lt ip gt 45002 x for ip addr= ingestion err= campaign waiting for listener please retry err= timestamp 2022 02 08T18 57 28Z error in query ingestion```

Ryan · Answer

I ll pick this up again tomorrow

Tomas Touceda · Answer

certainly looks like something around redis misbehaving

Ryan · Answer

Hi < Tomas Touceda> I m still trying to debug what s going on here the Redis instance itself seems fine but we re still seeing all sorts of weird behaviour in Fleet as a result Is there a quick way to verify the connection from Fleet to Redis is ok

Ryan · Answer

I m going to verify it using `tlsconnect go`

Ryan · Answer

```2022 02 09 10 50 12 pool created successfully 2022 02 09 10 50 13 command result NOAUTH Authentication required NOAUTH Authentication required ``` Looks good the AUTH value is set in the Fleet yml doesn t seem to be possible to test that with the `tlsconnect go` tool though

Ryan · Answer

```ingestion err= writing results PUBLISH failed to channel results 49 EOF err= timestamp 2022 02 09T10 57 52Z error in query ingestion ```

Ryan · Answer

the weird thing is Fleet is logging all of these errors but Live Queries do work as do scheduled query packs so I have no idea what the issue is thinking face

Ryan · Answer

this is using a GCP Memorystore instance with TLS and AUTH enabled in case anyone can reproduce this

Tomas Touceda · Answer

let me consult with the team to see if anybody else has any other ideas

Ryan · Answer

thanks slightly smiling face

Ryan · Answer

There are some constraints on Memorystore compared to normal Redis but I don t think any of them should affect us <https cloud google com memorystore docs redis product constraints>

Benjamin Edwards · Answer

Are you using Standard or Basic tier

Ryan · Answer

Basic tier for this particular instance I think

Ryan · Answer

Yeah it s basic tier

Ryan · Answer

I m pretty sure the issues being logged started when switching to TLS though we used the same Basic Memorystore without TLS previously and it was fine I think

Ryan · Answer

Stupidly we changed several things at once though including upgrading to v4 9 1

Benjamin Edwards · Answer

There are other limitations for basic tier but not sure yet how they might be affecting your setup <https cloud google com memorystore docs redis product constraints basic tier limitations|https cloud google com memorystore docs redis product constraints basic tier limitations>

Ryan · Answer

Yeah those are acceptable I think it s basically maintenance windows they will upgrade but no maintenance has been performed so far

Ryan · Answer

looks like our scheduled queries continued to work overnight by the way

Ryan · Answer

seems like Live Queries are working still too

Ryan · Answer

actually no they re not working 100

Ryan · Answer

hanging at Online 1276 hosts 1047 results

Ryan · Answer

so yeah I suspect it isn t happy with the TLS feature for some reason

Ryan · Answer

Update for you both < Benjamin Edwards> and < Tomas Touceda> running another Fleet instance with Memorystore Redis when TLS is disabled is working absolutely fine so we re suspecting there s something not quite right when TLS is enabled but it s not outright failing just not working correctly The only thing I can think of that isn t quite right is GCP Memorystore signs the certificate using the IP address of the instance not the DNS name we specify in `address` But we do set the `tls server name` config option to the IP to fix that error so it should be fine It s a Redis 6 instance too is that significant

Ryan · Answer

I m tempted to switch off TLS to see if everything works properly again what do you think

Ryan · Answer

Thanks for your time on this so far slightly smiling face

Tomas Touceda · Answer

if the certificate had an issue then it would fail all the time Temporary failures feels more related to something else Eg unreliable infra on the GCP side or increased CPU usage for the TLS layer on the fleet instances

Tomas Touceda · Answer

what does the CPU memory usage look like for the fleet instances now compared to when TLS was on

Ryan · Answer

Basically the same it s hardly doing any work

Ryan · Answer

~6 CPU 3MB out of 1GB capacity

Ryan · Answer

client activity

Ryan · Answer

this for the past 6 hours ^

Ryan · Answer

For the time being I ve turned off TLS on our Redis Memorystore instance Interestingly even with the non TLS setup I see a lot of this logged when executing a distributed query gt fleet 9704 level=error ts=2022 02 11T16 29 51 534788035Z component=http method=POST uri= api v1 osquery distributed write took=196 493333ms ip addr= lt ip gt 51318 x for ip addr= ingestion err= campaign waiting for listener please retry err= timestamp 2022 02 11T16 29 51Z error in query ingestion However the distributed query executes just fine and I get results back from all hosts so I m not really sure what s going on but I m going to leave it in this configuration for a while and see how it goes

Ryan · Answer

We have 1274 hosts in this Fleet setup FYI