Hi. Suddenly my Fleet GitOps PRs are failing with ...
# fleet
j
Hi. Suddenly my Fleet GitOps PRs are failing with
Error: error deleting EULA: getting eula metadata: GET /api/latest/fleet/setup_experience/eula/metadata received status 400 unknown
I am running on a self-hosted instance, on the latest version (4.71.0). I do not have end user auth enabled, and no EULA. So, GitOps is trying to delete and look up metadata for an EULA that is not there. Anyone else seeing this? Any breaking changes around this that I have missed recently?
👀 1
m
Hello @Jarle Kittilsen! This is definitely interesting, let me take a look and see if we're tracking any known bugs for this issue. Once I get an idea of whats going on, I'll be back with next steps.
l
Hi @Jarle Kittilsen! Any chance you can get Fleet server logs at the time GitOps is executed? (To understand why it's returning a 400 response on the GET instead of a 404)
j
I will share it with you in a DM. I can't see anything that looks connected in the logs though. If I do an API call from my laptop using `fleetctl api`for the same endpoint, I do get the expected 404, and there is a log entry...
l
Thanks! Forgot to mention to enable debugging
--logging_debug
/
FLEET_LOGGING_DEBUG=true
Next thing we will try (discussed via DM) is to run
fleetctl gitops
with
--debug
(which prints all requests and responses)
j
So this is what I am getting from that:
Copy code
GET <https://fleet.core.deepinsight.io/api/latest/fleet/setup_experience/eula/metadata>
{}
GET <https://fleet.core.deepinsight.io/api/latest/fleet/setup_experience/eula/metadata> 400 Bad Request (14ms)
Read body error: stream error: stream ID 15; PROTOCOL_ERROR; received from peerError: error deleting EULA: getting eula metadata: GET /api/latest/fleet/setup_experience/eula/metadata: do request: Get "<https://fleet.core.deepinsight.io/api/latest/fleet/setup_experience/eula/metadata>": stream error: stream ID 15; PROTOCOL_ERROR; received from peer
Could be related to our kubernetes reverse proxy/gateway...will investigate that...what is strange is: If I try
fleetctl api /api/latest/fleet/setup_experience/eula/metadata
i get 404 If I try to add a test EULA, I still get 404 on that endpoint, but if I do v1 instead of latest,
fleetctl api /api/v1/fleet/setup_experience/eula/metadata
I get:
Copy code
{
  "name": "Test EULA.pdf",
  "bytes": null,
  "sha256": "rUsvlGPk1dnm1qJ1bHs1ik1e2oAZBAEqRhKGbvGZSlI=",
  "token": "3af02481-2d56-4c23-920a-32d4b9a56fe0",
  "created_at": "2025-08-07T09:05:27Z"
}
So that works just fine...
l
Weird indeed. How about
fleetctl api --debug /api/latest/fleet/setup_experience/eula/metadata
j
That revealed something fun, the fleetctl api command adds /api/v1/fleet/:
Copy code
GET <https://fleet.core.deepinsight.io/api/v1/fleet/api/latest/fleet/setup_experience/eula/metadata>

GET <https://fleet.core.deepinsight.io/api/v1/fleet/api/latest/fleet/setup_experience/eula/metadata> 404 Not Found (30ms)
404 page not found
fleetctl api --debug /setup_experience/eula/metadata
gives:
Copy code
GET <https://fleet.core.deepinsight.io/api/v1/fleet/setup_experience/eula/metadata>

GET <https://fleet.core.deepinsight.io/api/v1/fleet/setup_experience/eula/metadata> 404 Not Found (74ms)
{
  "message": "Resource Not Found",
  "errors": [
    {
      "name": "base",
      "reason": "MDMEULA was not found in the datastore"
    }
  ]
}
Error: Got non 2XX return of 404
I can try to `curl`the endpoint using the `/api/latest/fleet/`version...
Both versions (v1 and latest) gives expected output when curl'ed:
Copy code
{
  "message": "Resource Not Found",
  "errors": [
    {
      "name": "base",
      "reason": "MDMEULA was not found in the datastore"
    }
  ]
}
So it is only when coming from GitHub Workflow we are getting a 400 Bad Request 🤔
l
Huh!
Worth continue troubleshooting in the Github Workflow rather than locally. Any proxy filtering for traffic coming from Github?
j
Oh wait....the error was there when running
fleetctl gitops
locally as well...I forgot. So not related to GitHub 😅
l
Ah gotcha.
j
Ok. Found something in our Kubernetes Gateway logs. It gives "400 Bad request: body_not_allowed". So the `fleetctl gitops`command adds a request body to the GET request, which the gateway does not allow. I guess GET requests shouldn't really have any data. Do you know if that is intentional, or perhaps a bug, in fleetctl? We recently changed from using Kubernetes ingresses, to using the Kubernetes Gateway API. So, maybe we need to change back as the ingress controller seems to be more lenient about having data in the body of a GET request....
l
Makes sense there's a
{}
in the GET request in GitOps.
🎯 1
j
Seems like a bug that can trip things up for others as well. Will you make an issue to have it fixed?
l
I'll create an issue to avoid sending these empty json bodies on GET requests. PS: There might be other API GET requests that do send some body (usually not rejected, but not recommended), for those we might not be able to change to not break the API (but at least we can document them for future reference).
j
Thanks for helping me get to the bottom of it anyways 🙌
l
Thanks @Jarle Kittilsen for helping with the troubleshooting! #31700 (feel free to add any details)