https://github.com/osquery/osquery logo
Title
s

Stephan

08/16/2021, 11:20 PM
Hello, I'm trying to get on-demand queries to work. I have an nginx proxy that all the osquery clients talk to. In the nginx config I've got the below for the websocket connection
proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
On the other side I have fleet running on kubernetes across several clusters for redundancy. Sometimes on-demand queries will work and sometimes I'll get the below error:
Failed to load resource: the server responded with a status of 404 ()

bundle-ec4e3a98f99eeaa1006b.js:110 WebSocket connection to '<wss://fleetdm-testing.example.com/api/v1/fleet/results/198/dd3kiay2/websocket>' failed:
If I reduce the kubernetes deployments to a single deployment, everything works fine. Given that it's intermittent and works with a single cluster deployment, I suspect there is a handshake/routing issue happening at the kubernetes layer where responses are not landing where fleet expects them. Anyone got any thoughts on this?
z

zwass

08/17/2021, 1:30 AM
Just to double-check... You've got all the Fleet deployments talking to the same Redis and MySQL instances, yeah?
s

Stephan

08/17/2021, 10:22 PM
@zwass yup, the same for all deployments.
I've had a look through the above article but I'm not sure it'll address the issue. The infrastructure involved would remain the same even if I split it out (browser --> nginx proxy for SSO --> kube cluster)
m

Mystery Incorporated

08/19/2021, 2:23 AM
@Stephan yea but I'm not doing connection upgrade or any of those things you listed, so you may need to conform your nginx config to that one to make it work.