Title
#kolide
b

BJF

08/18/2020, 2:00 PM
Where are Fleet servers expected to store their connection information: MySQL or Redis?
sundsta

sundsta

08/18/2020, 5:47 PM
The distributed (live queries) state lives in redis
b

BJF

08/18/2020, 5:48 PM
Thank you. How would one query that?
sundsta

sundsta

08/18/2020, 5:54 PM
You would use a Redis client, there are a few. The connection settings would be the same as what you have configured Fleet with, but you would of course need to be on a network that could access the Redis instance
b

BJF

08/18/2020, 5:56 PM
I have access to redis-sli on the redis server itself. What I would like to check initially would be the connection status of osquery clients.
5:57 PM
CLIENT LIST does list the connections from both Fleet servers.
sundsta

sundsta

08/18/2020, 5:58 PM
The osquery agents talk to Fleet, not redis. They would never be listed there.
b

BJF

08/18/2020, 5:58 PM
What I am trying to resolve is why osquery agents connected to Fleet server #2 appear offline to Fleet server #1, but not the other way around.
5:48 PM
Thank you. Would I find connection information in MySQL? If so, how might I locate that information?
sundsta

sundsta

08/19/2020, 6:00 PM
I don’t think “active connections” are stored in MySQL, but there is a
last_seen
column. I would guess that is what is used to show in the UI if a host is online or not, but I’m not certain.
b

BJF

08/19/2020, 6:01 PM
Interesting. I will take a look into that. Thanks!
6:13 PM
In MySQL I found something interesting: the seen_time field in the hosts table does not get updated when osquery clients are not logged into fleet server #2 although they are updated when connected to fleet server #2. I suspect some permissions issue is in place.
sundsta

sundsta

08/19/2020, 6:25 PM
Whoops, yeah the column is
seen_time
, not
last_seen
. But it sounds like you’re on the right track
b

BJF

08/19/2020, 6:25 PM
🙂
6:40 PM
Interesting, only one fleet server at a time can write to kolide/hosts/seen_time
sundsta

sundsta

08/19/2020, 6:52 PM
That seems… odd and unexpected. I’d be interested what the logs say if you turn on debug logging
b

BJF

08/19/2020, 6:52 PM
When Fleet server #1 is up, only it can write, Fleet server #2 can only write when Fleet server #1 is down
6:58 PM
I will look into that.
7:00 PM
The storage method is InnoDB. Is that relevant?
2:01 PM
I found root cause of the issue: the time of the two Fleet servers were not synchronized.
2:30 PM
Once I ensured that chronyd was functioning properly, the issue was resolved. One server's time was two minutes behind the other so client connection times to it always appeared out-of-date. Synchronizing the time resolved this.