Where are Fleet servers expected to store their co...
# kolideb
BJF
08/18/2020, 2:00 PMWhere are Fleet servers expected to store their connection information: MySQL or Redis?
s
sundsta
08/18/2020, 5:47 PMThe distributed (live queries) state lives in redis
b
BJF
08/18/2020, 5:48 PMThank you. How would one query that?
s
sundsta
08/18/2020, 5:54 PMYou would use a Redis client, there are a few. The connection settings would be the same as what you have configured Fleet with, but you would of course need to be on a network that could access the Redis instance
b
BJF
08/18/2020, 5:56 PMI have access to redis-sli on the redis server itself.
What I would like to check initially would be the connection status of osquery clients.
BJF
08/18/2020, 5:57 PMCLIENT LIST does list the connections from both Fleet servers.
s
sundsta
08/18/2020, 5:58 PMThe osquery agents talk to Fleet, not redis. They would never be listed there.
b
BJF
08/18/2020, 5:58 PMWhat I am trying to resolve is why osquery agents connected to Fleet server #2 appear offline to Fleet server #1, but not the other way around.
BJF
08/19/2020, 5:48 PMThank you. Would I find connection information in MySQL? If so, how might I locate that information?
s
sundsta
08/19/2020, 6:00 PMI don’t think “active connections” are stored in MySQL, but there is a
last_seenb
BJF
08/19/2020, 6:01 PMInteresting. I will take a look into that. Thanks!
BJF
08/19/2020, 6:13 PMIn MySQL I found something interesting: the seen_time field in the hosts table does not get updated when osquery clients are not logged into fleet server #2 although they are updated when connected to fleet server #2.
I suspect some permissions issue is in place.
s
sundsta
08/19/2020, 6:25 PMWhoops, yeah the column is
seen_timelast_seenb
BJF
08/19/2020, 6:25 PM🙂
BJF
08/19/2020, 6:40 PMInteresting, only one fleet server at a time can write to kolide/hosts/seen_time
s
sundsta
08/19/2020, 6:52 PMThat seems… odd and unexpected. I’d be interested what the logs say if you turn on debug logging
b
BJF
08/19/2020, 6:52 PMWhen Fleet server #1 is up, only it can write, Fleet server #2 can only write when Fleet server #1 is down
BJF
08/19/2020, 6:58 PMI will look into that.
BJF
08/19/2020, 7:00 PMThe storage method is InnoDB. Is that relevant?
BJF
08/20/2020, 2:01 PMI found root cause of the issue: the time of the two Fleet servers were not synchronized.
👍 2
BJF
08/20/2020, 2:30 PMOnce I ensured that chronyd was functioning properly, the issue was resolved.
One server's time was two minutes behind the other so client connection times to it always appeared out-of-date. Synchronizing the time resolved this.
2 Views