Title
#fleet
r

Ryan

07/29/2021, 9:49 AM
After upgrading to v4 we're seeing a lot of these logs in the Fleet journal:
fleet[14715]: level=info ts=2021-07-29T09:48:04.444973864Z component=service method=AuthenticateHost ip_addr=10.9.0.64:49157 x_for_ip_addr= err="authentication error: missing node key" took=20.913ยตs
Several nodes appear to be offline. The enrol secret appears to be correct, so I'm not quite sure what has happened here. Does anyone have any suggestions? Thanks.
Rachel Perkins

Rachel Perkins

07/29/2021, 7:17 PM
Hmm, can you use
--verbose --tls_dump
as documented in this FAQ question to get more details? https://github.com/fleetdm/fleet/blob/main/docs/2-Deploying/FAQ.md#why-arent-my-osquery-agents-connecting-to-fleet
r

Ryan

07/30/2021, 10:19 AM
I'll give that a try shortly. Weirdly enough over night the problem seems to have stopped, and we're now seeing this being logged intermittently instead:
Jul 30 10:17:58 de-kolide-fleet-01 fleet[14715]: 2021/07/30 10:17:58 http: TLS handshake error from 10.230.34.204:40996: local error: tls: bad record MAC
Jul 30 10:17:58 de-kolide-fleet-01 fleet[14715]: 2021/07/30 10:17:58 http: TLS handshake error from 10.230.34.204:40998: local error: tls: bad record MAC
Jul 30 10:17:59 de-kolide-fleet-01 fleet[14715]: 2021/07/30 10:17:59 http: TLS handshake error from 10.230.34.204:41000: local error: tls: bad record MAC
Jul 30 10:17:59 de-kolide-fleet-01 fleet[14715]: 2021/07/30 10:17:59 http: TLS handshake error from 10.230.34.204:41002: local error: tls: bad record MAC
Jul 30 10:18:03 de-kolide-fleet-01 fleet[14715]: 2021/07/30 10:18:03 http: TLS handshake error from 10.230.34.204:41008: local error: tls: bad record MAC
Jul 30 10:18:03 de-kolide-fleet-01 fleet[14715]: 2021/07/30 10:18:03 http: TLS handshake error from 10.230.34.204:41010: local error: tls: bad record MAC
10:19 AM
but all the hosts appear to be online and responding to queries, so I'm not sure about this
10:20 AM
i'll try running verbose and tls_dump flags on that node
10:23 AM
ok - false alarm, this is an old CentOS 6 node I decommissioned, it's not been terminated yet, so false alarm, that's cleared all the errors ๐Ÿ™‚
10:24 AM
I'm still not sure why the
authentication error: missing node key
error fixed itself over night, but happy to chalk that up to upgrade fun and games ๐Ÿ™‚
Rachel Perkins

Rachel Perkins

07/30/2021, 9:14 PM
Glad you got it sorted! Enjoy your weekend!