https://github.com/osquery/osquery logo
Title
b

benbass

02/09/2022, 8:10 PM
I am looking to grab the vulnerability data (NIST feeds and the NVD DB) and host them on an internal server. I see that the NIST data needs to be the JSON files from /feeds/json/cve/1.1/ for the JSON files, however I am not sure what files should be pulled from https://github.com/fleetdm/nvd/releases. Is Fleet expecting the sqlite.gz, the zip file or the tar.gz? Or is it looking for the decompressed DB itself?
b

Benjamin Edwards

02/09/2022, 8:31 PM
https://fleetdm.com/docs/deploying/configuration#cpe-database-url might be helpful.
It expects a file in the same format as can be found in https://github.com/fleetdm/nvd/releases.
Assuming that means the sqlite.gz file
b

benbass

02/09/2022, 8:32 PM
That is what I wasn’t sure of, as the files that are there are all 3 types i listed, with none specifically the zip file.
If the case that it does need the DB, I can see if I can have that pulled down when it is updated and then decompressed.
t

Tomas Touceda

02/09/2022, 8:33 PM
there's
fleetctl vulnerability-data-stream
that is meant to download everything needed, then you can configure it to skip the data sync, if you place those files in the databases path, it'll pick them up from there
b

Benjamin Edwards

02/09/2022, 8:33 PM
yeah its not super clear. The zip and tar.gz files are the source code of this repo.
oh right thanks @Tomas Touceda forgot about fleetctl command
👍 1
b

benbass

02/09/2022, 8:35 PM
Interesting. I’ll have to play with it - I want to build an automated process and am looking to make sure I understand all of the pieces.
t

Tomas Touceda

02/09/2022, 8:40 PM
this is meant to fit a CI/CD pipeline, in case you want to run the files through other checks before moving them to a server
b

benbass

02/09/2022, 8:41 PM
My goal is to have the pulling of the DB and the NIST data into an artifactory repo, then have my fleet servers pull from there.
I want to wrap my head around what/how fleet does things before reaching out to our artifactory admins.
OK, I see what’s up.
fleetctl vulnerability-data-stream
pulls that data locally onto your admin system. And FYI, it does pull the straight cpe.sqlite db.
t

Tomas Touceda

02/09/2022, 8:46 PM
yes, and the CVE data streams, everything that fleet does when left to its own devices for vulnerability processing
b

benbass

02/09/2022, 8:47 PM
Cool. That gives me a much clearer idea of what I need. Thank you!
t

Tomas Touceda

02/09/2022, 8:47 PM
you'll lose the differential downloading that you get when you let fleet download things, but it's not a ton of data, so I would bet it's fine
b

benbass

02/09/2022, 8:48 PM
Eh, I am not worried about that.
z

zwass

02/10/2022, 4:16 AM
@koo can you please help turn this discussion into Fleet documentation?
b

benbass

02/10/2022, 2:14 PM
As part of the documentation, can you add the specific links that need to be mirrored? I know you outline the NIST ones, however without the url for the NVD DB, it will take some digging to figure out exactly what needs to be hosted/mirrored if fleetctl is not being invoked.
t

Tomas Touceda

02/10/2022, 2:17 PM
what do you think would be needed from
fleetctl vulnerability-data-stream
to cover that need?
b

benbass

02/10/2022, 2:18 PM
I guess what url is being used to pull down the NVD db. I don’t want to use wireshark and see what is being pulled today, and then have it break when the next version is released. I need to let the artifactory admins know what files/urls need to be synchronized.
t

Tomas Touceda

02/10/2022, 2:22 PM
so what I'm hearing is: the fleetctl command that downloads the data is not going to be used, you need to know what URLs to pull from. Is this understanding correct?
b

benbass

02/10/2022, 2:22 PM
That is correct.
t

Tomas Touceda

02/10/2022, 2:22 PM
good to know
b

benbass

02/10/2022, 2:23 PM
I can use fleetctl if I was spinning up a server, however I am trying to integrate with an existing cache/proxy service we have.
I also noticed when using the
fleetctl vulnerability-data-stream
with
--debug true
I did not see any urls sent to std out as I was hoping for and expecting.
t

Tomas Touceda

02/10/2022, 2:32 PM
yeah, we could look into adding that