Hi! I followed @Lucas Rodriguez advice earlier and for most of it, fleet on kubernetes seems to work. The only problem I still have left is that running live queries doesn't seem to work from the interface in any shape or form. This are the logs I see from nginx as a proxy (fleet doesn't seem to show any logs at that point):

x.x.x.x -  [25/May/2023:21:39:05 +0000] "POST /api/v1/fleet/results/198/tmxrqkgm/xhr_send?t=1685050744996 HTTP/1.1" 405 0 "<https://fleet.security.pleo.io/queries/new>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 7017 0.002 [fleet-fleet-8080] [] 172.x.x.x:8080 0 0.001 405 6753df76dfb05c2be4398c1e10e4512b
x.x.x.x -  [25/May/2023:21:39:04 +0000] "POST /api/v1/fleet/results/198/gq3lvb2w/xhr_streaming?t=1685050744898 HTTP/1.1" 405 0 "<https://fleet.security.pleo.io/queries/new>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 6857 0.000 [fleet-fleet-8080] [] 172.x.x.x:8080 0 0.001 405 e615a3ca7d0c7749c443fecd81675e6c
x.x.x.x -  [25/May/2023:21:39:04 +0000] "GET /api/v1/fleet/results/198/n0ex0hts/websocket HTTP/1.1" 403 10 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 6750 0.000 [fleet-fleet-8080] [] 172.x.x.x:8080 10 0.001 403 de8df61d42f7d1f6073c1bb23ebeb3d5

And this is part of my kubernetes config:

---
apiVersion: <http://helm.toolkit.fluxcd.io/v2beta1|helm.toolkit.fluxcd.io/v2beta1>
kind: HelmRelease
metadata:
  name: fleet
spec:
  chart:
    spec:
      chart: fleet
      version: 5.0.1
      sourceRef:
        kind: HelmRepository
        name: fleet
  values:
    ingress:
      enabled: true
      className: ingress-external
      hosts:
        - host: fleet.hostname.tld
          paths:
            - path: /
              pathType: ImplementationSpecific
      annotations:
        <http://kubernetes.io/external-dns-class|kubernetes.io/external-dns-class>: ingress-external
        <http://nginx.ingress.kubernetes.io/service-upstream|nginx.ingress.kubernetes.io/service-upstream>: "true"
        <http://nginx.ingress.kubernetes.io/upstream-vhost|nginx.ingress.kubernetes.io/upstream-vhost>: internal.fleet.hostname.local
        <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "3600"
        <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "3600"
        <http://nginx.ingress.kubernetes.io/secure-backends|nginx.ingress.kubernetes.io/secure-backends>: "true"
        <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
        <http://nginx.org/websocket-services|nginx.org/websocket-services>: "fleet"
        <http://nginx.ingress.kubernetes.io/server-snippets|nginx.ingress.kubernetes.io/server-snippets>: |
          location / {
            proxy_set_header Upgrade $http_upgrade;
            proxy_http_version 1.1;
            proxy_set_header X-Forwarded-Host $http_host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header Origin "<https://fleet.hostname.tld>";
            proxy_set_header Host $host;
            proxy_set_header Connection "upgrade";
            proxy_pass_header X-XSRF-TOKEN;
            proxy_cache_bypass $http_upgrade;
            }
    fleet:
      keepalive: true
      tls:
        enabled: false
      websockets_allow_unsafe_origin: true

Are you seeing any errors in the browser's dev console? What happens if you try running a live query using

fleetctl

?

in the browser dev console I see this:

content.js:1 new website
bundle-18a618685952a79fd739.js:2 WebSocket connection to '<wss://fleet.domain.tld/api/v1/fleet/results/909/wi4yrvrv/websocket>' failed: 
e.exports @ bundle-18a618685952a79fd739.js:2
l @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._receiveInfo @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
xhr.onreadystatechange @ bundle-18a618685952a79fd739.js:2
bundle-18a618685952a79fd739.js:2     POST <https://fleet.domain.tld/api/v1/fleet/results/909/mfqtdyvh/xhr_streaming?t=1685054028554> 405 (Method Not Allowed)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
a._scheduleReceiver @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
o @ bundle-18a618685952a79fd739.js:2
s @ bundle-18a618685952a79fd739.js:2
c @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._transportClose @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
ws.onerror @ bundle-18a618685952a79fd739.js:2
error (async)
l @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._receiveInfo @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
xhr.onreadystatechange @ bundle-18a618685952a79fd739.js:2
XMLHttpRequest.send (async)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
s @ bundle-18a618685952a79fd739.js:2
p._getReceiver @ bundle-18a618685952a79fd739.js:2
p.doXhr @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
p @ bundle-18a618685952a79fd739.js:2
x @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
r @ bundle-18a618685952a79fd739.js:2
n @ bundle-18a618685952a79fd739.js:2
Promise.then (async)
a @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
m @ bundle-18a618685952a79fd739.js:2
g @ bundle-18a618685952a79fd739.js:2
x @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
ao @ bundle-18a618685952a79fd739.js:2
xl @ bundle-18a618685952a79fd739.js:2
t.unstable_runWithPriority @ bundle-18a618685952a79fd739.js:2
zi @ bundle-18a618685952a79fd739.js:2
bl @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
L @ bundle-18a618685952a79fd739.js:2
w.port1.onmessage @ bundle-18a618685952a79fd739.js:2
bundle-18a618685952a79fd739.js:2     POST <https://fleet.domain.tld/api/v1/fleet/results/909/i5naxpm1/xhr_send?t=1685054028641> 405 (Method Not Allowed)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.sendSchedule @ bundle-18a618685952a79fd739.js:2
a.send @ bundle-18a618685952a79fd739.js:2
x.send @ bundle-18a618685952a79fd739.js:2
t.onopen @ bundle-18a618685952a79fd739.js:2
t.dispatchEvent @ bundle-18a618685952a79fd739.js:2
x._open @ bundle-18a618685952a79fd739.js:2
x._transportMessage @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
n.onmessage @ bundle-18a618685952a79fd739.js:2

I think we can skip that for now since you're getting those websocket errors. It definitely looks like there's something amiss in the configuration that's causing the connection to the websocket to fail. Do you have anything set up that might be blocking the POST request:

POST <https://fleet.domain.tld/api/v1/fleet/results/909/i5naxpm1/xhr_send?t=1685054028641> 405 (Method Not Allowed)

Possibly related to

> In this case, you might need to disable the origin header (by setting this configuration to true) check or configure your reverse proxy to forward the correct Origin header.

https://fleetdm.com/docs/deploying/configuration#server-websockets-allow-unsafe-origin Had a similar issue come up for someone else not too long ago.

@Benjamin Edwards I am looking at that right now, but from the looks of it, that setting is not supported by the helmchart at this moment

its an env var for fleet server. I think you'd need to edit the spec for the fleet pod https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/

@Benjamin Edwards I did it slightly different and it actually works now!

ah there ya go. so it worked for you?

yes, comms with my agents and live queries now work!

sweet!

Thank you all for your support ❤️

yeah reverse proxies can be weird