https://github.com/osquery/osquery logo
Title
r

Rafa

08/03/2022, 9:10 PM
Hi people! We are currently running fleet with more then 3k hosts and growing but we find something odd. Even though software_inventory is set to true it seems that the vulnerabilities are not being checked. And when I look at the logs I see:
cron=vulnerabilities cron=vulnerabilities databases-path=/tmp/vulndbs
but I manually entered in this directory in each node and it is empty. This folder has read and write permissions. Is there another env that I have to set? Version 4.18
k

Kathy Satterlee

08/03/2022, 9:13 PM
Hey, @Rafa. That database path would be on the server side. Are you seeing any errors about vulnerabilities, or do you just have none showing up in the API/UI?
r

Rafa

08/03/2022, 9:14 PM
Hello! I am seeing some vulnerabilities on the API/UI but I think should be more (because we run a lot of old systems). And the folder where it should download the db is empty...
And there are only software vulns, nothing about OS vulns
k

Kathy Satterlee

08/03/2022, 9:21 PM
The folder existing is a good sign, but I would definitely expect there to be some data there especially since you are seeing some vulnerabilities. That info has to be coming from somewhere, unless it just got cleared from
/tmp
.
You'd see some errors in the logs if the cron job was failing. Can you try setting a new database path to see what happens?
r

Rafa

08/03/2022, 9:24 PM
Done!
level=error ts=2022-08-03T21:13:00.733144068Z component=crons cron=cleanups cron=leader err="sending statistics" details="error posting to <https://fleetdm.com/api/v1/webhooks/receive-usage-analytics>: 400. "
8/3/2022 6:13:17 PM level=error ts=2022-08-03T21:13:17.696003379Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: Карта: fts5: syntax error near \"\""
8/3/2022 6:13:17 PM level=error ts=2022-08-03T21:13:17.696117432Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: . .: fts5: syntax error near \"\""
8/3/2022 6:13:17 PM level=error ts=2022-08-03T21:13:17.696203752Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: . . .: fts5: syntax error near \"\""
8/3/2022 6:13:17 PM level=error ts=2022-08-03T21:13:17.696288523Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: Неведомый Космос: fts5: syntax error near \"\""
8/3/2022 6:13:17 PM level=error ts=2022-08-03T21:13:17.696368434Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: Рай: fts5: syntax error near \"\""
k

Kathy Satterlee

08/03/2022, 9:37 PM
Weird question maybe, but how much RAM is Fleet working with?
r

Rafa

08/03/2022, 9:37 PM
2 gb
k

Kathy Satterlee

08/03/2022, 9:39 PM
That could be the culprit. Vulnerability scanning needs 4GB and we start seeing weird errors like this when memory starts to get eaten.
r

Rafa

08/03/2022, 9:39 PM
hmmm I will change this now. one minute
k

Kathy Satterlee

08/03/2022, 9:39 PM
🤞
r

Rafa

08/03/2022, 9:46 PM
I noticed something odd:
level=debug ts=2022-08-03T21:44:00.440381147Z component=crons cron=vulnerabilities cron=vulnerabilities msg="Not the leader. Skipping..."
8/3/2022 6:44:00 PM
I am running with one node only. It should be the leader right?
b

Benjamin Edwards

08/03/2022, 9:54 PM
The lock, database row, from the last vuln processing attempt might still be live. It will eventually expire and the next iteration should acquire leader status.
r

Rafa

08/03/2022, 9:55 PM
Cool! I will wait and let you know. Thanks a lot for the attention and for this amazing software!
b

Benjamin Edwards

08/03/2022, 9:57 PM
Also how are you running fleet? Container?
r

Rafa

08/03/2022, 9:57 PM
Yeap!
First 3 pods, but I reduced to 1 just to solve this leader problem
b

Benjamin Edwards

08/03/2022, 9:58 PM
Ok just checking thanks
r

Rafa

08/04/2022, 1:54 PM
Hello!
How are you?
Looking at the deploy I saw that it downloaded the vulns:
And in the logs I see: fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.549502862Z component=crons cron=vulnerabilities cron=vulnerabilities cpe-processing-3=start fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.54967825Z component=crons cron=vulnerabilities cron=vulnerabilities cpe-processing-1=start fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.549706165Z component=crons cron=vulnerabilities cron=vulnerabilities cpe-processing-0=start fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.549715001Z component=crons cron=vulnerabilities cron=vulnerabilities cpe-processing-2=start fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.678175379Z component=crons cron=vulnerabilities cron=vulnerabilities pushingcpes=done fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.67854018Z component=crons cron=vulnerabilities cron=vulnerabilities cpe-processing-1=done fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.678925991Z component=crons cron=vulnerabilities cron=vulnerabilities cpe-processing-0=done fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.679066333Z component=crons cron=vulnerabilities cron=vulnerabilities cpe-processing-2=done fleet-85f4d45756-cprgm fleet level=debug ts=2022-08-04T13:21:50.680472866Z component=crons cron=vulnerabilities cron=vulnerabilities cpe-processing-3=done
k

Kathy Satterlee

08/04/2022, 2:56 PM
Looking good!
How's the software looking now?
And doing well this morning, how's your day going?
r

Rafa

08/04/2022, 3:06 PM
Good! I am fine too 😃
About the software, it still seems to do not have scanned all vulnerabilities. Is there a way to run the scan in my host added to fleet with fleetctl? I already downloaded the vulns with
fleetctl vulnerability-data-stream --dir .
The odd part is that I am still getting these errors: fleet-6ff77cf554-4nfvp fleet level=error ts=2022-08-04T17:04:44.892674563Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: Карта: fts5: syntax error near \"\"" fleet-6ff77cf554-4nfvp fleet level=error ts=2022-08-04T17:04:44.892780759Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: . .: fts5: syntax error near \"\"" fleet-6ff77cf554-4nfvp fleet level=error ts=2022-08-04T17:04:44.892888206Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: . . .: fts5: syntax error near \"\"" fleet-6ff77cf554-4nfvp fleet level=error ts=2022-08-04T17:04:44.893001757Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: Неведомый Космос: fts5: syntax error near \"\"" fleet-6ff77cf554-4nfvp fleet level=error ts=2022-08-04T17:04:44.893094859Z component=crons cron=vulnerabilities cron=vulnerabilities software->cpe="error translating to CPE, skipping..." err="getting cpes for: Рай: fts5: syntax error near \"\""
b

Benjamin Edwards

08/04/2022, 5:27 PM
are you running the fleet containers from dockerhub or did you build your own?
r

Rafa

08/04/2022, 6:45 PM
From dockerhub
m

Michal Nicpon

08/04/2022, 10:01 PM
I was able to reproduce the error and created a github issue https://github.com/fleetdm/fleet/issues/7067 Note that this only affects software with non-ascii names.
r

Rafa

08/04/2022, 10:37 PM
Thanks! But @Michal Nicpon would this break the vulnerabilities scan as a hole or only for that software?
m

Michal Nicpon

08/04/2022, 10:38 PM
Only for that software. We should probably change this to a warning
r

Rafa

08/04/2022, 10:39 PM
Great! Thanks!
Hi!
How are you? Sorry for the delay but I was trying to find evidence of the problem.
So
This is my host
0 vulnerable software
But after running a scan in my machine:
And looking inside the folder with the downloaded database by fleet:
And
CVE-2022-1652
is in the
fleet_oval_ubuntu_2004-2022_08_05.json
.
So this vulnerability should be in the board, right?
k

Kathy Satterlee

08/05/2022, 4:26 PM
I believe that's expected behavior since it's an OS vulnerability rather than installed software, but I'm going to double check just to make sure.
And verified. Fleet doesn't scan for OS vulnerabilities and is only looking at additional software at this point.
You could set up a policy to check if the OS is up to date. It's generally a good way to make sure things are as secure as possible :)
r

Rafa

08/05/2022, 4:40 PM
Cool! doing that now! thanks a lot!