Hello, I'm trying to get on-demand queries to work. I have an nginx proxy that all the osquery clients talk to. In the nginx config I've got the below for the websocket connection

proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

On the other side I have fleet running on kubernetes across several clusters for redundancy. Sometimes on-demand queries will work and sometimes I'll get the below error:

Failed to load resource: the server responded with a status of 404 ()

bundle-ec4e3a98f99eeaa1006b.js:110 WebSocket connection to '<wss://fleetdm-testing.example.com/api/v1/fleet/results/198/dd3kiay2/websocket>' failed:

If I reduce the kubernetes deployments to a single deployment, everything works fine. Given that it's intermittent and works with a single cluster deployment, I suspect there is a handshake/routing issue happening at the kubernetes layer where responses are not landing where fleet expects them. Anyone got any thoughts on this?

Just to double-check... You've got all the Fleet deployments talking to the same Redis and MySQL instances, yeah?

@Stephan https://mysteryincorporated.medium.com/nginx-configuration-for-fleetdm-setups-that-want-to-catch-3m-flatties-willem-powerfish-be-proud-7f99f97fdede

@zwass yup, the same for all deployments.

@Stephan yea but I'm not doing connection upgrade or any of those things you listed, so you may need to conform your nginx config to that one to make it work.