Channels
android_tests
apple-silicon
arm-architecture
auditing-warroom
aws
carving
code-review
community-feeds
core
darkbytes
doorman
ebpf
eclecticiq-polylogyx-extension
extensions
file-carving
fim
fleet
fleet-dev
fleetosquery
foundation
fuzzing
general
golang
goquery
infrastructure
jobs
kolide
linen-dev
linux
macos
officehours
osctrl
plugins
process-auditing
querycon
queryhub
random
selfgroup
sql
tls
uptycs
vendor-feeds
website
windows
zeek
zentral
zercurity
Title
b
BJF
08/18/2020, 2:00 PMWhere are Fleet servers expected to store their connection information: MySQL or Redis?
s
sundsta
08/18/2020, 5:47 PMThe distributed (live queries) state lives in redis
b
BJF
08/18/2020, 5:48 PMThank you. How would one query that?
s
sundsta
08/18/2020, 5:54 PMYou would use a Redis client, there are a few. The connection settings would be the same as what you have configured Fleet with, but you would of course need to be on a network that could access the Redis instance
b
BJF
08/18/2020, 5:56 PMI have access to redis-sli on the redis server itself.
What I would like to check initially would be the connection status of osquery clients.
CLIENT LIST does list the connections from both Fleet servers.
s
sundsta
08/18/2020, 5:58 PMThe osquery agents talk to Fleet, not redis. They would never be listed there.
b
BJF
08/18/2020, 5:58 PMWhat I am trying to resolve is why osquery agents connected to Fleet server #2 appear offline to Fleet server #1, but not the other way around.
Thank you. Would I find connection information in MySQL? If so, how might I locate that information?
s
sundsta
08/19/2020, 6:00 PMI don’t think “active connections” are stored in MySQL, but there is a
last_seenb
BJF
08/19/2020, 6:01 PMInteresting. I will take a look into that. Thanks!
In MySQL I found something interesting: the seen_time field in the hosts table does not get updated when osquery clients are not logged into fleet server #2 although they are updated when connected to fleet server #2.
I suspect some permissions issue is in place.
s
sundsta
08/19/2020, 6:25 PMWhoops, yeah the column is
seen_timelast_seenb
BJF
08/19/2020, 6:25 PM🙂
Interesting, only one fleet server at a time can write to kolide/hosts/seen_time
s
sundsta
08/19/2020, 6:52 PMThat seems… odd and unexpected. I’d be interested what the logs say if you turn on debug logging
b
BJF
08/19/2020, 6:52 PMWhen Fleet server #1 is up, only it can write, Fleet server #2 can only write when Fleet server #1 is down
I will look into that.
The storage method is InnoDB. Is that relevant?
I found root cause of the issue: the time of the two Fleet servers were not synchronized.
👍 2
Once I ensured that chronyd was functioning properly, the issue was resolved.
One server's time was two minutes behind the other so client connection times to it always appeared out-of-date. Synchronizing the time resolved this.