still having issues with redis, even after upgradi...
# fleetj
Jocelyn Bothe
09/08/2021, 6:37 PMstill having issues with redis, even after upgrading to 4.2.4 today. We're now seeing the error
Copy code
err="scan keys: dial tcp 10.10.24.224:6380: i/o timeout" msg="failed to migrate live query redis keys"t
Tomas Touceda
09/08/2021, 7:42 PMwe changed how some things behave in 4.2.4 and improved them further in 4.3.0 around redis clusters. It might be worth a try
Tomas Touceda
09/08/2021, 7:42 PMthat said, can you connect directly to that node with redis-cli?
j
Jocelyn Bothe
09/08/2021, 7:42 PMyup, redis-cli works fine
Jocelyn Bothe
09/08/2021, 7:42 PMbut it shows no keys
Jocelyn Bothe
09/08/2021, 7:43 PMosq-cluster-east.nl4nlg.clustercfg.use1.cache.amazonaws.com:6380> scan 0
1) "0"
2) (empty array)
t
Tomas Touceda
09/08/2021, 7:43 PMdoes it take longer than 5s to connect?
Tomas Touceda
09/08/2021, 7:43 PMit's ok if there are no keys
j
Jocelyn Bothe
09/08/2021, 7:44 PMnope, it was definitely less than 5 seconds
t
Tomas Touceda
09/08/2021, 7:45 PMand this is from the same host that is running fleet serve, correct?
j
Jocelyn Bothe
09/08/2021, 7:45 PMyes
Jocelyn Bothe
09/08/2021, 7:47 PMif I enable debug logging, will it have more info about that redis timeout?
Jocelyn Bothe
09/08/2021, 7:47 PM Copy code
Sep 08 19:44:34 <http://osquery-service-vab147.ec2.vzbuilders.com|osquery-service-vab147.ec2.vzbuilders.com> systemd[1]: Started Kolide Fleet.
Sep 08 19:44:34 <http://osquery-service-vab147.ec2.vzbuilders.com|osquery-service-vab147.ec2.vzbuilders.com> fleet[15741]: Using config file: /etc/kolide/fleet.yml
Sep 08 19:44:51 <http://osquery-service-vab147.ec2.vzbuilders.com|osquery-service-vab147.ec2.vzbuilders.com> fleet[15741]: level=info ts=2021-09-08T19:44:51.921971702Z err="scan keys: dial tcp 10.10.24.226:6380: i/o timeout" msg="failed to migrate live query redis keys"Jocelyn Bothe
09/08/2021, 7:47 PMthere is more than 5 seconds between fleet reading the config yaml and getting the redis error on startup
t
Tomas Touceda
09/08/2021, 7:48 PMyeah, something is preventing it to connect fast enough, or at all
j
Jocelyn Bothe
09/08/2021, 7:50 PMokay, interesting, when I restarted in debug it didn't get the error, so it seems intermittent
Jocelyn Bothe
09/08/2021, 7:51 PMis that 5s timeout going to be configurable in 4.3?
t
Tomas Touceda
09/08/2021, 7:51 PMare the metrics for the redis cluster ok?
Tomas Touceda
09/08/2021, 7:51 PMI'm looking into the timeout to see what we can provide
j
Jocelyn Bothe
09/08/2021, 7:51 PMyeah, the cluster is basically doing nothing right now
t
Tomas Touceda
09/08/2021, 7:52 PMis it in the same region as the fleet instance?
j
Jocelyn Bothe
09/08/2021, 7:53 PMit is a global cluster, but yes, the instance I'm testing on is in the same region
Jocelyn Bothe
09/08/2021, 7:53 PMas the primary redis
t
Tomas Touceda
09/08/2021, 7:53 PMgotcha, thank you for answering my million questions
j
Jocelyn Bothe
09/08/2021, 7:53 PMthanks for helping me troubleshoot
t
Tomas Touceda
09/08/2021, 7:53 PMwe'll make the timeout configurable
j
Jocelyn Bothe
09/08/2021, 7:53 PM👍
Jocelyn Bothe
09/08/2021, 7:58 PMand if you're taking requests, a configurable retry for that connection would be lovely 🙂
t
Tomas Touceda
09/08/2021, 8:00 PMalways welcoming requests!
Tomas Touceda
09/08/2021, 8:02 PMwill look into retries, the issue is at the connection level, key migrations are just one point that uses that
Tomas Touceda
09/08/2021, 8:13 PMretries will probably not make it to 4.3.0, could you create a feature request issue: https://github.com/fleetdm/fleet/issues/new?assignees=&labels=idea&template=feature-request.md&title= ? as for the timeout, https://github.com/fleetdm/fleet/pull/1968
j
Jocelyn Bothe
09/08/2021, 8:19 PMdone!
ty 1
m
mikermcneil
09/09/2021, 8:08 PMfyi here's the link to @Jocelyn Bothe's GitHub issue: https://github.com/fleetdm/fleet/issues/1969
(Thanks Jocelyn!)
👍 1
2 Views