Hi all. Yesterday we upgraded from 4.32 to 4.38.0 and my fleet servers are getting bogged down hard with memory usage. I am seeing this flooding /var/log/messages:

Oct  6 10:23:10 osq-fleet01 fleet: {"component":"http","err":"authentication error: invalid node key: YXTtp4zxXQUlbjtvSkLRj3qWhSMs1TY7","level":"info","path":"/api/v1/osquery/distributed/read","ts":"2023-10-06T14:23:10.089866852Z","uuid":"f17d3370-20c5-4986-a7bc-75829fa434ee"}

is there a cause for this? I looked through logs pre upgrade and am not seeing this appear

Hey @Jonathan Ober! Do you have access to the Fleet Database? I'd like to see if there's a specific host or set of hosts that might be responsible for these errors. When a host gets an invalid node key response, it should immediately try to re-enroll in the Fleet server. Running this query in the Fleet database will return the 50 most recently enrolled hosts. If you run this a few times and see that the

last_erolled_time

keeps updating for a host/hosts, that should help us narrow down where those errors are coming from.

select hostname, node_key, last_enrolled_at, hostname, uuid, computer_name from hosts order by last_enrolled_at desc limit 50;

thanks. seems like the last_enrolled_at has been steadily updating for the past couple of minutes. Before than i see some pretty big gaps. like a few this morning then nothing at all between 9:30PM and 8AM

It looks like there might be something odd going on with those 9 hosts. Do they share anything in common?

not that i can see. It's a mix of linux and mac clients. whenever i run the query again it shows different hosts

Can you grab the osquery status logs for a couple of those hosts? I'd be interested to see what's going on there.

just a systemctl status osqueryd?

Are you running fleetd packages or plain osquery on your hosts?

just plain osquery

Good deal. The osquery status logs on the hosts may have some additional information about that failed attempt.

yeah all im seeing is the failed enrollment request a bunch of times. fwiw the machine seems to be communicating? I can go to fleet, view it and run basic queries on it

On one of those hosts, would it be possible to add these osquery flags:

--verbose
--tls_dump

Then let things run for a but and check out the osquery logs at

/var/log/osquery/

?

Tried this on a RHEL 8 system running osquery 5.5.1. I added those lines to /etc/osquery/osquery.flags and restarted osqueryd service. osqueryd.INFO is pretty much endlessly posting these few messages:

I1009 16:06:41.822561 872296 tls.cpp:255] TLS/HTTPS POST request to URI: <https://myosquery.url:8080/api/v1/osquery/enroll>

I1009 16:06:42.678791 872296 tls.cpp:255] TLS/HTTPS POST request to URI: <https://myosquery.url.edu:8080/api/v1/osquery/distributed/read>

(actual URL replaced with myosquery.url) I also made a test ubuntu 22.04 system this morning and it has yet to report into fleet. when i check the service's status i don't see any failed enrollment token messages either in the systemd status command or the osquery.INFO log. when the VM get provisioned, it currently goes out with 5.5.1 so i updated to 5.9 and that didnt make a difference. i saw this morning it was seemingly running queries so not really sure why it isnt appearing i can try this on other linux systems if you'd like and see if i get different results. we have mac systems that are appearing in that query but i dont have immediate access to them right now to try anything

Are there any osqueryd.ERROR logs?

i see this message a ton of times in the RHEL host:

E1009 14:59:33.011750 509038 glog_logger.cpp:26] deb_packages: Failed to open the dpkg database: InvalidAdminDirPath (admindir='/var/lib/dpkg')