Hi all, I'm having troubles with multipart uploads of carves to MinIO backend. I'm using Fleet version 4.60.1, deployed in Kubernetes. My carver settings in the flagfile are:

--disable_carver=false
--carver_disable_function=false
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block
--carver_block_size=8000000

When I carve file smaller than the blocksize, it works just fine, as expected. But when I try to carve a larger file, I just don't get it in MinIO. I tried to carve a file, take the carve guid and check it's status with

SELECT * FROM carves WHERE carve_guid="<guid>";

and the status says "SUCCESS". I tried to get the carve info with Fleet API, and this is the JSON:

{
  "id": 36,
  "created_at": "2024-12-24T09:30:02Z",
  "host_id": 8,
  "name": "<carve_name>",
  "block_count": 8,
  "block_size": 5242880,
  "carve_size": 41497600,
  "carve_id": "<carve_id>",
  "request_id": "<request_id>",
  "session_id": "<session_id>",
  "expired": false,
  "error": null,
  "max_block": -1
}

I have no logs about the carves in Fleet's pod. Can anyone please help? 🙏

@Ortal Kombat I'm sorry that you're still having issues with file carving. What version of osquery are you running?

Hi, my osquery agents version is 5.13.1

@Ortal Kombat Could you try upgrading your osquery to the latest stable version, which is 5.14.1? That may not fix the issue, but we are continuing to look into it.

I'll try that, but I had the following versions of osquery: 4.8.0 5.11.0 5.13.1 And it didn't work in any of these version, so I suspect it's maybe something I did wrong, but can't figure out what. Are there known problems when Fleet is deployed on k8s?

@Ortal Kombat Can you unset the carve settings you have for Fleet, which will allow the carve results to be send to MySQL, rather than MinIO? Then we diagnose the issue further. Here is the setting that applies: https://fleetdm.com/docs/configuration/fleet-server-configuration#s-3-carves-bucket

I tried that, and now when I request carves with the REST API, I the field "error" says "block_id does not match expected block (0): 5"

@Ortal Kombat Can you try enabling verbose logging and tls logging in osquery? This could possibly shed some light. https://osquery.readthedocs.io/en/stable/deployment/debugging/#:~:text=or%20code%20improvements.-,Runnin[…]20in%20verbose%20mode,-This%20is%20pretty https://osquery.readthedocs.io/en/stable/deployment/debugging/#:~:text=Inspecting%20TLS/HTTPS%20body%20request%20and%20responses

@Rebecca Cowart Thanks a lot! I had this problem for months now, and with your help I finally solved it (I didn't even imagine I could log the TLS communication in the client). In those logs I searched for POST requests to Fleet's carve block uploading URL, and noticed that the response for any such request is "413 request entity too large", and is received from my NGINX ingress. After a quick search on the internet, I found out how to increase the max request size on NGINX, and now I'm finally able to carve large files with multiple blocks. Thanks for the help!😁