https://github.com/osquery/osquery logo
#fleet
Title
# fleet
j

Jincheng Yin

01/17/2023, 4:16 PM
Hi team, do you know how long does fleet update the Software Inventory? Right now, it's been 7 days since last update. And it seems it will update the inventory when I update fleet server to a new version.
k

Kathy Satterlee

01/17/2023, 4:27 PM
Hi @Jincheng YinWhat version of Fleet are you running? The default interval for vulnerability and software updates is 1 hour.
j

Jincheng Yin

01/17/2023, 4:52 PM
@Kathy Satterlee 4.25
k

Kathy Satterlee

01/17/2023, 5:09 PM
Are your hosts otherwise checking in properly? Any errors in the Fleet logs?
j

Jincheng Yin

01/17/2023, 6:02 PM
Let me double check the logs. it seems it only updated the inventory when restarts the pods or update fleet server
@Kathy Satterlee
image.png
k

Kathy Satterlee

01/17/2023, 8:56 PM
Can you verify that the path set for
database_path
is present and that the fleet user has read and write privileges?
j

Jincheng Yin

01/18/2023, 3:35 PM
There is /tmp/ folder, but it's empty. Checking the privileges.
@Kathy Satterlee we have these settings in the fleet yaml file
@Kathy Satterlee We got debug logs. And it's showing
unable to acquire lock
k

Kathy Satterlee

01/30/2023, 4:21 PM
Is the “unable to acquire lock” error showing up consistently, or was that a one-off? How many Fleet instances do you have running?
j

Jincheng Yin

01/30/2023, 4:34 PM
@Kathy Satterlee We have 3 fleet instances, it's showing up consistently. When it's not showing up, we can get the new Software Inventory update, last time is 3 days ago.
We had another error is
syncing NVD CVE feed
when we pass the acquire lock, it should relate to our Firewall rules.
k

Kathy Satterlee

01/30/2023, 4:57 PM
Can you show me that exact error?
j

Jincheng Yin

01/30/2023, 4:59 PM
image.png,image.png
k

Kathy Satterlee

01/30/2023, 5:03 PM
Got it. Fleet should continue processing even if the databases can't sync. Just to clarify:
When it's not showing up, we can get the new Software Inventory update, last time is 3 days ago.
Does that mean you last saw that error 3 days ago and software is currently updating, or that the error is still coming up and you haven't seen a software update in 3 days?
j

Jincheng Yin

01/30/2023, 5:04 PM
the error(unable to acquire lock) is still coming up and I haven't seen a software update in 3 days
k

Kathy Satterlee

01/30/2023, 5:06 PM
Thanks. just wanted to make sure. Do you have access to the Fleet database? I'd like to see what shows up in the `locks`table:
Copy code
select * from locks where name = 'vulnerabilities';
And are you using the default value for current_instance_checks ?
Last question for now... how much memory do you have allocated to each Fleet instance?
j

Jincheng Yin

01/30/2023, 5:12 PM
We did not change the current_instance_checks Only add the vul_path
Can't access the MySQL database 🫠
Do you think restart DB can help?
k

Kathy Satterlee

01/30/2023, 5:21 PM
This is the suggested resource allocation in Kubernetes from the documentation:
Copy code
resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "2048Mi" # vulnerability processing
            cpu: "500m"
That memory limit is likely the culprit.
j

Jincheng Yin

01/30/2023, 5:22 PM
Thanks! We will have a try.
k

Kathy Satterlee

01/30/2023, 5:23 PM
Awesome! Let me know if that doesn't do the trick.
j

Jincheng Yin

02/02/2023, 10:45 AM
@Kathy Satterlee Sorry, last time check the wrong file. We have this settings in our deployment file, I think it should be OK
VulnAutomationEnabled: ""
This is the last log we seen in the container which running the
Vuln check
5mins later, it start ticking again
Last time when it's working, the jobID is
cron_sync_host_software
Now the jobID is
cron_vulnerabilities
Hi @Kathy Satterlee , do you have any suggestion? Basically the log shows the Vul check started, and then nothing happened. Last time it works is 7 days ago, when it works, we have more logs about software things. (Error logs as we don't have CVE files in the container)
k

Kathy Satterlee

02/09/2023, 2:41 PM
Vulnerabilities processing requires a minimum of 4GB of memory (it looks like the recommendation in that example I shared is even too low, sorry about the oversight!). That’s still the most likely culprit.
j

Jincheng Yin

02/13/2023, 10:43 AM
@Kathy Satterlee Thanks. It's working now. It's Memory issue and caused by the GKE autopilot modified resource limit to match resource request.
2 Views