still having issues with redis even after upgrading to 4 2 4

Question

still having issues with redis even after upgrading to 4 2 4 today We re now seeing the error ```err= scan keys dial tcp 10 10 24 224 6380 i o timeout msg= failed to migrate live query redis keys ```

Tomas Touceda · Answer

we changed how some things behave in 4 2 4 and improved them further in 4 3 0 around redis clusters It might be worth a try

Tomas Touceda · Answer

that said can you connect directly to that node with redis cli

Jocelyn Bothe · Answer

yup redis cli works fine

Jocelyn Bothe · Answer

but it shows no keys

Jocelyn Bothe · Answer

<http osq cluster east nl4nlg clustercfg use1 cache amazonaws com 6380|osq cluster east nl4nlg clustercfg use1 cache amazonaws com 6380> gt scan 0 1 0 2 empty array

Tomas Touceda · Answer

does it take longer than 5s to connect

Tomas Touceda · Answer

it s ok if there are no keys

Jocelyn Bothe · Answer

nope it was definitely less than 5 seconds

Tomas Touceda · Answer

and this is from the same host that is running fleet serve correct

Jocelyn Bothe · Answer

yes

Jocelyn Bothe · Answer

if I enable debug logging will it have more info about that redis timeout

Jocelyn Bothe · Answer

```Sep 08 19 44 34 systemd 1 Started Kolide Fleet Sep 08 19 44 34 fleet 15741 Using config file etc kolide fleet yml Sep 08 19 44 51 fleet 15741 level=info ts=2021 09 08T19 44 51 921971702Z err= scan keys dial tcp 10 10 24 226 6380 i o timeout msg= failed to migrate live query redis keys ```

Jocelyn Bothe · Answer

there is more than 5 seconds between fleet reading the config yaml and getting the redis error on startup

Tomas Touceda · Answer

yeah something is preventing it to connect fast enough or at all

Jocelyn Bothe · Answer

okay interesting when I restarted in debug it didn t get the error so it seems intermittent

Jocelyn Bothe · Answer

is that 5s timeout going to be configurable in 4 3

Tomas Touceda · Answer

are the metrics for the redis cluster ok

Tomas Touceda · Answer

I m looking into the timeout to see what we can provide

Jocelyn Bothe · Answer

yeah the cluster is basically doing nothing right now

Tomas Touceda · Answer

is it in the same region as the fleet instance

Jocelyn Bothe · Answer

it is a global cluster but yes the instance I m testing on is in the same region

Jocelyn Bothe · Answer

as the primary redis

Tomas Touceda · Answer

gotcha thank you for answering my million questions

Jocelyn Bothe · Answer

thanks for helping me troubleshoot

Tomas Touceda · Answer

we ll make the timeout configurable

Jocelyn Bothe · Answer

+1

Jocelyn Bothe · Answer

and if you re taking requests a configurable retry for that connection would be lovely slightly smiling face

Tomas Touceda · Answer

always welcoming requests

Tomas Touceda · Answer

will look into retries the issue is at the connection level key migrations are just one point that uses that

Tomas Touceda · Answer

retries will probably not make it to 4 3 0 could you create a feature request issue as for the timeout

Jocelyn Bothe · Answer

done

mikermcneil · Answer

fyi here s the link to < Jocelyn Bothe> s GitHub issue Thanks Jocelyn