Hey guys, we recently use fleet carve feature try to pull a file from all of our hosts, there are two issues we found: 1. In one carve request for each host, there are multiple carve_metadata records generated in database in a different time (most of hosts). Looks like multiple carve requests from osquery agent sent to fleet. 2. Even though there are multiple carve_metadata records, part of hosts are still not able to upload data to database. (All their carve_metadata.max_block value is -1, as I understand that means no block created for the carve_metadata) Issue 2 is more important, because we are missing part of hosts upload file via carve. Anyone has insight on why this happened?

Greetings @Jian Zheng, What version of Fleet would you be running? Also, how large are the files and how many endpoints are you attempting to carve from?

@Grant Bilstad Our fleet version is: 4.20.1 The file is not too big only few line, It's less than one block size. There are many hosts we try to carve from. Is there limit on number of hosts to run carve in one query?

Not aware of any limit for numbers from a single query. Asked more to get an idea of infrastructure and your use case. Carving being a more advanced feature of osquery, is used mostly in incident response and live queries.

great than 10k.

Would you be using our fleetd package? or standalone osquery? If a few lines could maybe leverage file_lines instead.

@Grant Bilstad I don't think that work for our case. Right now, I'm trying to figure out what should be the issue on carve, so in future we use carve again, we won't see the issue again. Instead of using an alternative way. Any idea why this issue happens on carve?