https://github.com/osquery/osquery logo
Title
s

Sebastiaan

05/25/2023, 9:46 PM
Hi! I followed @Lucas Rodriguez advice earlier and for most of it, fleet on kubernetes seems to work. The only problem I still have left is that running live queries doesn't seem to work from the interface in any shape or form. This are the logs I see from nginx as a proxy (fleet doesn't seem to show any logs at that point):
x.x.x.x -  [25/May/2023:21:39:05 +0000] "POST /api/v1/fleet/results/198/tmxrqkgm/xhr_send?t=1685050744996 HTTP/1.1" 405 0 "<https://fleet.security.pleo.io/queries/new>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 7017 0.002 [fleet-fleet-8080] [] 172.x.x.x:8080 0 0.001 405 6753df76dfb05c2be4398c1e10e4512b
x.x.x.x -  [25/May/2023:21:39:04 +0000] "POST /api/v1/fleet/results/198/gq3lvb2w/xhr_streaming?t=1685050744898 HTTP/1.1" 405 0 "<https://fleet.security.pleo.io/queries/new>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 6857 0.000 [fleet-fleet-8080] [] 172.x.x.x:8080 0 0.001 405 e615a3ca7d0c7749c443fecd81675e6c
x.x.x.x -  [25/May/2023:21:39:04 +0000] "GET /api/v1/fleet/results/198/n0ex0hts/websocket HTTP/1.1" 403 10 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 6750 0.000 [fleet-fleet-8080] [] 172.x.x.x:8080 10 0.001 403 de8df61d42f7d1f6073c1bb23ebeb3d5
And this is part of my kubernetes config:
---
apiVersion: <http://helm.toolkit.fluxcd.io/v2beta1|helm.toolkit.fluxcd.io/v2beta1>
kind: HelmRelease
metadata:
  name: fleet
spec:
  chart:
    spec:
      chart: fleet
      version: 5.0.1
      sourceRef:
        kind: HelmRepository
        name: fleet
  values:
    ingress:
      enabled: true
      className: ingress-external
      hosts:
        - host: fleet.hostname.tld
          paths:
            - path: /
              pathType: ImplementationSpecific
      annotations:
        <http://kubernetes.io/external-dns-class|kubernetes.io/external-dns-class>: ingress-external
        <http://nginx.ingress.kubernetes.io/service-upstream|nginx.ingress.kubernetes.io/service-upstream>: "true"
        <http://nginx.ingress.kubernetes.io/upstream-vhost|nginx.ingress.kubernetes.io/upstream-vhost>: internal.fleet.hostname.local
        <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "3600"
        <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "3600"
        <http://nginx.ingress.kubernetes.io/secure-backends|nginx.ingress.kubernetes.io/secure-backends>: "true"
        <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
        <http://nginx.org/websocket-services|nginx.org/websocket-services>: "fleet"
        <http://nginx.ingress.kubernetes.io/server-snippets|nginx.ingress.kubernetes.io/server-snippets>: |
          location / {
            proxy_set_header Upgrade $http_upgrade;
            proxy_http_version 1.1;
            proxy_set_header X-Forwarded-Host $http_host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header Origin "<https://fleet.hostname.tld>";
            proxy_set_header Host $host;
            proxy_set_header Connection "upgrade";
            proxy_pass_header X-XSRF-TOKEN;
            proxy_cache_bypass $http_upgrade;
            }
    fleet:
      keepalive: true
      tls:
        enabled: false
      websockets_allow_unsafe_origin: true
k

Kathy Satterlee

05/25/2023, 10:04 PM
Are you seeing any errors in the browser's dev console? What happens if you try running a live query using
fleetctl
?
s

Sebastiaan

05/25/2023, 10:34 PM
in the browser dev console I see this:
content.js:1 new website
bundle-18a618685952a79fd739.js:2 WebSocket connection to '<wss://fleet.domain.tld/api/v1/fleet/results/909/wi4yrvrv/websocket>' failed: 
e.exports @ bundle-18a618685952a79fd739.js:2
l @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._receiveInfo @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
xhr.onreadystatechange @ bundle-18a618685952a79fd739.js:2
bundle-18a618685952a79fd739.js:2     POST <https://fleet.domain.tld/api/v1/fleet/results/909/mfqtdyvh/xhr_streaming?t=1685054028554> 405 (Method Not Allowed)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
a._scheduleReceiver @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
o @ bundle-18a618685952a79fd739.js:2
s @ bundle-18a618685952a79fd739.js:2
c @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._transportClose @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
ws.onerror @ bundle-18a618685952a79fd739.js:2
error (async)
l @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._receiveInfo @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
xhr.onreadystatechange @ bundle-18a618685952a79fd739.js:2
XMLHttpRequest.send (async)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
s @ bundle-18a618685952a79fd739.js:2
p._getReceiver @ bundle-18a618685952a79fd739.js:2
p.doXhr @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
p @ bundle-18a618685952a79fd739.js:2
x @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
r @ bundle-18a618685952a79fd739.js:2
n @ bundle-18a618685952a79fd739.js:2
Promise.then (async)
a @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
m @ bundle-18a618685952a79fd739.js:2
g @ bundle-18a618685952a79fd739.js:2
x @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
ao @ bundle-18a618685952a79fd739.js:2
xl @ bundle-18a618685952a79fd739.js:2
t.unstable_runWithPriority @ bundle-18a618685952a79fd739.js:2
zi @ bundle-18a618685952a79fd739.js:2
bl @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
L @ bundle-18a618685952a79fd739.js:2
w.port1.onmessage @ bundle-18a618685952a79fd739.js:2
bundle-18a618685952a79fd739.js:2     POST <https://fleet.domain.tld/api/v1/fleet/results/909/i5naxpm1/xhr_send?t=1685054028641> 405 (Method Not Allowed)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.sendSchedule @ bundle-18a618685952a79fd739.js:2
a.send @ bundle-18a618685952a79fd739.js:2
x.send @ bundle-18a618685952a79fd739.js:2
t.onopen @ bundle-18a618685952a79fd739.js:2
t.dispatchEvent @ bundle-18a618685952a79fd739.js:2
x._open @ bundle-18a618685952a79fd739.js:2
x._transportMessage @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
n.onmessage @ bundle-18a618685952a79fd739.js:2
@Kathy Satterlee how would I use fleetctl to run a live query, via the server, locally?
k

Kathy Satterlee

05/25/2023, 10:57 PM
I think we can skip that for now since you're getting those websocket errors. It definitely looks like there's something amiss in the configuration that's causing the connection to the websocket to fail. Do you have anything set up that might be blocking the POST request:
POST <https://fleet.domain.tld/api/v1/fleet/results/909/i5naxpm1/xhr_send?t=1685054028641> 405 (Method Not Allowed)
b

Benjamin Edwards

05/25/2023, 11:16 PM
Possibly related to
> In this case, you might need to disable the origin header (by setting this configuration to true) check or configure your reverse proxy to forward the correct Origin header.
https://fleetdm.com/docs/deploying/configuration#server-websockets-allow-unsafe-origin Had a similar issue come up for someone else not too long ago.
s

Sebastiaan

05/25/2023, 11:53 PM
@Benjamin Edwards I am looking at that right now, but from the looks of it, that setting is not supported by the helmchart at this moment
b

Benjamin Edwards

05/25/2023, 11:56 PM
its an env var for fleet server. I think you'd need to edit the spec for the fleet pod https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
s

Sebastiaan

05/25/2023, 11:57 PM
@Benjamin Edwards I did it slightly different and it actually works now!
Thank you for your input and thank you to @Kathy Satterlee as well!
the helm chart allows you to set extra env variables in the values block
as defined here:
so i did this in my config file for k8s:
values:
    environments:
      FLEET_SERVER_WEBSOCKETS_ALLOW_UNSAFE_ORIGIN: true
b

Benjamin Edwards

05/25/2023, 11:58 PM
ah there ya go. so it worked for you?
s

Sebastiaan

05/25/2023, 11:59 PM
yes, comms with my agents and live queries now work!
b

Benjamin Edwards

05/25/2023, 11:59 PM
sweet!
s

Sebastiaan

05/25/2023, 11:59 PM
Thank you all for your support ❤️
b

Benjamin Edwards

05/25/2023, 11:59 PM
yeah reverse proxies can be weird
I haven't seen nginx have this issue, but have seen it with other fancy ingress stuff