https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi all, I was wondering if there are any known iss...
# fleet
j
Hi all, I was wondering if there are any known issues with rhel9? rhel9 hosts are not reporting into fleet and the osqueryd service is giving me errors after restarting it. im gonna reply inline with some troubleshooting ive tried so far so i dont post a huge wall of text. looking for any insight on this as im running out of thing to try 🙂
• this seems to be isolated to rhel9. i have rhel7 and rhel8 machines reporting in with no issues. fwiw ubuntu as well • the service has been restarted to no accord • the version we are using is 5.5.1. i updated to the latest version 5.8.2 and ran a yum update -y but that didn't make a difference • I've scoured old messages on here to see if anyone else has reported this but i can't find anything on rhel9. i did see issues with the cert being the root cause of this error but I am somewhat perplexed that it is only rhel9 doing this. • The osquery.flags file seems to be correct. it is referencing the server URL and not an IP . i don't see any differences between machines that are working • the machine from this screenshot is from a brand new virtual machine but there are physical machines that are behaving the same way. • I am uncertain as to how long this has been going on. Fleet server was recently updated but i dont think this is the culprit because there are machines that should have been reporting in before the upgrade that are not in there at all. i would expect them to have at least shown up and give an offline status
any insight is appreciated. thanks in advance
k
No node key returned
typically indicates that the enroll secret is being rejected. Adding the 
--tls-dump
flag to osquery will add the json being sent to the logs to verify. If you could share the flagfile, I can also take a look at that. Feel free to redact any sensitive info or send that to me over DM. If you DM, I may share a snippet here but will make sure sensitive data is removed.
I see that you're pulling the enroll secret from file (as recommended). Can you check the contents with 
cat /etc/osquery/enroll_secret
to make sure that's pulling as expected?
j
yeah, it matches. contents are the same and permission is 600
k
👍 Then taking a look at the actual data being sent back and forth would be the next step.
j
will --tls-dump just give me that one line of output or will it report to any other logs? im just seeing this
eh and stuff in /var/log/messages. systemctl status gives the same
k
Sorry, could have been clearer! Add that to your osquery flagfile and start as usual.
j
oh gotcha
k
Then you should get the same logs as before, just including the body of the request and response.
Yeah, it’s odd if it’s only specific hosts showing this error given that it’s a MySQL error. What have you got set for MySQL max connections?
j
according to my.cnf, 
max_connections = 230
k
What do you have set for max open and max idle connections in Fleet?
b
FLEET_MYSQL_MAX_IDLE_CONNS=40 FLEET_MYSQL_MAX_OPEN_CONNS=75 This is the setting per fleet node, and we have 3 nodes. (I work with Jonathan) 🙂
k
Thanks! This is likely directly related to your offline hosts as well. How many Fleet instances do y'all have running?
b
We have 3 instances, 2 are behind our load balancer for client check ins.
k
Try bumping the FLEET_MYSQL_MAX_OPEN_CONNS down to 10. We can always bump that back up if needed.
b
On all 3 instances?
k
👍
b
Will do.
k
Between all of your instances, you're likely exceeding that 
230
total connections. 
10
may be a bit lower than you need, but better to start conservatively and adjust upwards as needed.
Let's do the same with idle connections as well.
b
Both values changed on all 3 nodes. I am still seeing the errors, however I am seeing queries coming in from the server to the end point.
We are not seeing too many connection errors, we are still seeing invalid node keys. I am seeing queries being sent to the RHEL 9 box I am testing with.
k
Invalid node keys are expected to pop up from time to time when the device tokens rotate.
Which I'd expect to see a lot after a connection issue.
As long as you aren't seeing the same UUID repeating for a longish period, I wouldn't be concerned about those.
b
That’s good, we’ll keep checking on that. I am worried that I am not seeing the RHEL9 box register though.
k
As far as those RHEL machine goes, I'd like to go back to the osquery logs to see if the responses have changed at all.
There may be a secondary issue that was hidden by this one.
b
That is my feeling - hopefully we can keep on peeling back the layers.
k
In fact, those machines attempting to enroll repeatedly may be part of what drove up the number of connections depending on how many boxes we're working with there.
I see that we're getting a UUID in the request, so that's a good thing.
j
could this be an issue with redhat 9.2? i see some hosts in our environment that are on 9.1 but nothing on 9.2
b
That is what I was thinking too - we were almost DDOS’ing ourselves with the failed enrollments. That is why I would like to see if we can get to the bottom of it.
k
Possibly, but it's all idle speculation until we get at those logs again 🙂
b
Which logs in particular?
k
The osquery logs for the host that @Jonathan Ober has access to and enabled the tls dump on.
j
im re-running that now. gonna let it run for a few minutes
k
If there are enrollment errors, they should pop up pretty fast.
j
im just stopping osqueryd service and then running 
sudo osqueryd --flagfile /etc/osquery/osquery.flags --tls_dump --verbose > osquery.log 2>&1
if you want me to do it a differnt way let me know
k
Works for me.
Looking back at the log you shared earlier, you've got your server URL and enroll secret exposed. Might want to edit that out JIC.
You're also welcome to DM me the next batch of logs. If I pull anything over here, I'll sanitize first.
j
i cant edit, only delete on here but will remove
k
I'm downloading an ISO to test just in case the logs aren't super helpful.
j
cool. just DMed you
k
👀
What version of Fleet are y'all working with?
b
4.32.0
k
Seeing something interesting in the logs: It enrolls successfully:
Copy code
I0714 13:14:38.634207 21948 tls.cpp:263] TLS/HTTPS POST request to URI: {{server_url}}/api/v1/osquery/enroll
{[...]}
{
  "node_key": "IsyEcUEkirMcCZmoNkFtU1uzPimOWDJN"
}
Then immediately attempts to connect to the server and gets the invalid node key:
Copy code
I0714 13:14:39.276793 21948 tls.cpp:263] TLS/HTTPS POST request to URI: {{server_url}}:8080/api/v1/osquery/distributed/read
{"node_key":"IsyEcUEkirMcCZmoNkFtU1uzPimOWDJN"}
{
  "error": "authentication error: invalid node key: IsyEcUEkirMcCZmoNkFtU1uzPimOWDJN",
  "node_invalid": true,
  "uuid": "[...]"
}
So that explains some of the Fleet errors. Are you seeing any devices added to Fleet? Any errors in those logs other than the invalid node key errors?
b
Yes, we are able to enroll other OS’s successfully.
k
Sorry, are you seeing this specific device enrolled in Fleet?
Is this a physical box, or VM?
b
VM, and no we are not seeing it show up in the GUI at all.
k
Ah. Do you have the ability to query the Fleet database?
b
Yes.
for a few minutes - i have to jump to a meeting shortly.
k
I suspect they all have the same UUID.
b
How would we find that out?
k
Double-checking schema. One sec.
Copy code
SELECT id, hostname, primary_ip, orbit_node_key FROM hosts WHERE uuid="d9570f42-be62-6214-a6b6-7f80ec4c3d2f";
I'm betting that if you run that multiple times, you'll see data change in between runs.
b
yup
k
Since you're using 
UUID
as your host identifier, Fleet sees all of these as the same host.
b
So much for the Unique in UUID…
k
The existing host is updated each time a new machine enrolls.
b
Yeah, that makes sense. What do you suggest we do in this scenario?
And before I forget, thank you for helping us get to a root cause Kathy!
k
Glad we got there.
b
Would you suggest we change the setting in fleet to
Copy code
instance
?
k
Yes. I'm fairly certain that we have a related ticket, working to dig that up for you.
Ah, it looks like that ticket was to implement a solution for hosts enrolled with the 
fleetd
package rather than plain osquery. For y'all, changing the host identifier should do the trick.
b
Cool, we are using vanilla osquery.
k
TL;DR for anyone that has a similar issue: 1. There was an unrelated database issue that we resolved. 2. The root issue is that these are VMs with duplicated UUIDS. That was determined by: a. Adding 
--tls-dump
to the osquery logs b. In those logs, the host was enrolling and then immediately getting an error response that the node key was invalid c. Querying the Fleet database showed that a host with that UUID (Pulled from the body of the enrollment request in the logs) was enrolled and that the data was continuously updating 
SELECT id, hostname, primary_ip, orbit_node_key FROM hosts WHERE uuid="{{UUID}}";
3. This can be resolved by changing the host identifier to 
instance
rather than 
uuid
j
where is that uuid coming from? i dont see it when i run blkid
k
Are the machines cloned from a base image?
j
from a template on vsphere yeah
just thrown off that value isnt there. im sure that uuid is from the template
k
Does the template include osquery? It could be that the UUID is already set in the osquery database.
j
ooooh good question
ill have to find out
k
You could try just deleting the osquery database on the host we've been working with.
If that does the trick, we know the answer 🙂
j
no errors now (yay) but not repoting into fleet yet (boo). maybe im just inpatient haha
k
If it hasn't checked in as of yet, I'd head back to the osquery logs to see what's what there.
5 Views
Open in Slack
PreviousNext
Powered by