Channels
android_tests
apple-silicon
arm-architecture
auditing-warroom
aws
carving
code-review
community-feeds
core
darkbytes
doorman
ebpf
eclecticiq-polylogyx-extension
extensions
file-carving
fim
fleet
fleet-dev
fleetosquery
foundation
fuzzing
general
golang
goquery
infrastructure
jobs
kolide
linen-dev
linux
macos
officehours
osctrl
plugins
process-auditing
querycon
queryhub
random
selfgroup
sql
tls
uptycs
vendor-feeds
website
windows
zeek
zentral
zercurity
Title
c
crimsonknave
04/09/2021, 5:53 PMI'm seeing the following errors in my logs
"err":"retrieve live queries: receive sql: redigo: nil returned"fleetctllogger_tls_periodz
zwass
04/09/2021, 5:59 PMIs 5 minutes also your distributed_interval?
c
crimsonknave
04/09/2021, 6:06 PMNo, that's 30 seconds
But, I haven't run a live query in over an hour and I'm still seeing these messages come in
Well, they're less regular in 5 minute increments now, but that could be due to more than one query stuck like this
From what I can tell the redis is empty (but redis is a bit of a mystery to me)
z
zwass
04/09/2021, 6:16 PMkeys '*'c
crimsonknave
04/09/2021, 6:17 PMI'm using redis commander and there aren't any entries in the GUI. I got an error when I tried to run that in the commandline bit. Let me dig a bit more.
The tree view should show keys, but it's empty. Which is what I recall when I had to go in and clean up the live queries before the cleanup logic was implemented.
Also, we just updated to 3.10.1 from 3.1 (I think, it was old).
I added a test key and can see it in the tree, so I'm pretty sure it's empty aside from that key.
z
zwass
04/09/2021, 6:25 PMOkay thanks for all that info. I'm going to see if we can do some better cleanup for this in the next release.
c
crimsonknave
04/09/2021, 6:40 PMThanks! Two quick questions, are these sort of errors something I should be worried about? Are the expected if I run or cancel a query?
z
zwass
04/09/2021, 6:41 PMI don't think you need to worry about them. We're being overly noisy about cleaning up older queries.
But it is strange that you still see them even though Redis is empty... You shouldn't be able to hit that code path if Redis is empty.
c
crimsonknave
04/09/2021, 6:47 PMAnything else I can do to debug right now?
z
zwass
04/09/2021, 6:49 PMIs it still happening even after verifying Redis empty?
c
crimsonknave
04/09/2021, 6:49 PMYup, I also redeployed our fleet (in kubernetes)
z
zwass
04/09/2021, 6:49 PMAre you seeing those errors in the Fleet server logs or in the logs that osquery clients are writing to Fleet?
Can you paste a full log line?
c
crimsonknave
04/09/2021, 6:50 PMThat was from Fleet, one sec. I may have misspoken.
I am still seeing them. 9k in the last 15 minutes.
{
"component": "service",
"err": "retrieve live queries: receive sql: redigo: nil returned",
"ip_addr": "10.125.6.0:19173",
"level": "info",
"method": "GetDistributedQueries",
"took": "12.15416ms",
"ts": "2021-04-09T18:50:47.401563414Z",
"x_for_ip_addr": "10.127.50.32"
}z
zwass
04/09/2021, 6:54 PMIs it possible your Redis UI is connected to a different DB than Fleet? Can you verify by running a live query and seeing that some keys appear?
c
crimsonknave
04/09/2021, 6:56 PMRunning one now. I see
livequerysqlCanceled it and those went away
z
zwass
04/09/2021, 7:06 PMAnd does live query actually work? I'm just looking at the code and it seems like it should not be possible to hit that line if there are no keys at all.
c
crimsonknave
04/09/2021, 7:08 PMYup, last time I let it run for a while I got
⠓ 59% responded (100% online) | 7169/12159 targeted hosts (7169/7205 online)And a bunch of data flowed in
There are keys when the query runs, when I cancel it they get cleaned up.