https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi! I followed <@U02J4JCKDL2> advice earlier and f...
# fleet
s
Hi! I followed @Lucas Rodriguez advice earlier and for most of it, fleet on kubernetes seems to work. The only problem I still have left is that running live queries doesn't seem to work from the interface in any shape or form. This are the logs I see from nginx as a proxy (fleet doesn't seem to show any logs at that point):
Copy code
x.x.x.x -  [25/May/2023:21:39:05 +0000] "POST /api/v1/fleet/results/198/tmxrqkgm/xhr_send?t=1685050744996 HTTP/1.1" 405 0 "<https://fleet.security.pleo.io/queries/new>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 7017 0.002 [fleet-fleet-8080] [] 172.x.x.x:8080 0 0.001 405 6753df76dfb05c2be4398c1e10e4512b
x.x.x.x -  [25/May/2023:21:39:04 +0000] "POST /api/v1/fleet/results/198/gq3lvb2w/xhr_streaming?t=1685050744898 HTTP/1.1" 405 0 "<https://fleet.security.pleo.io/queries/new>" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 6857 0.000 [fleet-fleet-8080] [] 172.x.x.x:8080 0 0.001 405 e615a3ca7d0c7749c443fecd81675e6c
x.x.x.x -  [25/May/2023:21:39:04 +0000] "GET /api/v1/fleet/results/198/n0ex0hts/websocket HTTP/1.1" 403 10 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36" 6750 0.000 [fleet-fleet-8080] [] 172.x.x.x:8080 10 0.001 403 de8df61d42f7d1f6073c1bb23ebeb3d5
And this is part of my kubernetes config:
Copy code
---
apiVersion: <http://helm.toolkit.fluxcd.io/v2beta1|helm.toolkit.fluxcd.io/v2beta1>
kind: HelmRelease
metadata:
  name: fleet
spec:
  chart:
    spec:
      chart: fleet
      version: 5.0.1
      sourceRef:
        kind: HelmRepository
        name: fleet
  values:
    ingress:
      enabled: true
      className: ingress-external
      hosts:
        - host: fleet.hostname.tld
          paths:
            - path: /
              pathType: ImplementationSpecific
      annotations:
        <http://kubernetes.io/external-dns-class|kubernetes.io/external-dns-class>: ingress-external
        <http://nginx.ingress.kubernetes.io/service-upstream|nginx.ingress.kubernetes.io/service-upstream>: "true"
        <http://nginx.ingress.kubernetes.io/upstream-vhost|nginx.ingress.kubernetes.io/upstream-vhost>: internal.fleet.hostname.local
        <http://nginx.ingress.kubernetes.io/proxy-read-timeout|nginx.ingress.kubernetes.io/proxy-read-timeout>: "3600"
        <http://nginx.ingress.kubernetes.io/proxy-send-timeout|nginx.ingress.kubernetes.io/proxy-send-timeout>: "3600"
        <http://nginx.ingress.kubernetes.io/secure-backends|nginx.ingress.kubernetes.io/secure-backends>: "true"
        <http://nginx.ingress.kubernetes.io/ssl-redirect|nginx.ingress.kubernetes.io/ssl-redirect>: "true"
        <http://nginx.org/websocket-services|nginx.org/websocket-services>: "fleet"
        <http://nginx.ingress.kubernetes.io/server-snippets|nginx.ingress.kubernetes.io/server-snippets>: |
          location / {
            proxy_set_header Upgrade $http_upgrade;
            proxy_http_version 1.1;
            proxy_set_header X-Forwarded-Host $http_host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_set_header Origin "<https://fleet.hostname.tld>";
            proxy_set_header Host $host;
            proxy_set_header Connection "upgrade";
            proxy_pass_header X-XSRF-TOKEN;
            proxy_cache_bypass $http_upgrade;
            }
    fleet:
      keepalive: true
      tls:
        enabled: false
      websockets_allow_unsafe_origin: true
k
Are you seeing any errors in the browser's dev console? What happens if you try running a live query using 
fleetctl
?
s
in the browser dev console I see this:
Copy code
content.js:1 new website
bundle-18a618685952a79fd739.js:2 WebSocket connection to '<wss://fleet.domain.tld/api/v1/fleet/results/909/wi4yrvrv/websocket>' failed: 
e.exports @ bundle-18a618685952a79fd739.js:2
l @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._receiveInfo @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
xhr.onreadystatechange @ bundle-18a618685952a79fd739.js:2
bundle-18a618685952a79fd739.js:2     POST <https://fleet.domain.tld/api/v1/fleet/results/909/mfqtdyvh/xhr_streaming?t=1685054028554> 405 (Method Not Allowed)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
a._scheduleReceiver @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
o @ bundle-18a618685952a79fd739.js:2
s @ bundle-18a618685952a79fd739.js:2
c @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._transportClose @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
ws.onerror @ bundle-18a618685952a79fd739.js:2
error (async)
l @ bundle-18a618685952a79fd739.js:2
x._connect @ bundle-18a618685952a79fd739.js:2
x._receiveInfo @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
i @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
xhr.onreadystatechange @ bundle-18a618685952a79fd739.js:2
XMLHttpRequest.send (async)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
s @ bundle-18a618685952a79fd739.js:2
p._getReceiver @ bundle-18a618685952a79fd739.js:2
p.doXhr @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
p @ bundle-18a618685952a79fd739.js:2
x @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
r @ bundle-18a618685952a79fd739.js:2
n @ bundle-18a618685952a79fd739.js:2
Promise.then (async)
a @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
m @ bundle-18a618685952a79fd739.js:2
g @ bundle-18a618685952a79fd739.js:2
x @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
ao @ bundle-18a618685952a79fd739.js:2
xl @ bundle-18a618685952a79fd739.js:2
t.unstable_runWithPriority @ bundle-18a618685952a79fd739.js:2
zi @ bundle-18a618685952a79fd739.js:2
bl @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
L @ bundle-18a618685952a79fd739.js:2
w.port1.onmessage @ bundle-18a618685952a79fd739.js:2
bundle-18a618685952a79fd739.js:2     POST <https://fleet.domain.tld/api/v1/fleet/results/909/i5naxpm1/xhr_send?t=1685054028641> 405 (Method Not Allowed)
l._start @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
setTimeout (async)
l @ bundle-18a618685952a79fd739.js:2
a @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.sendSchedule @ bundle-18a618685952a79fd739.js:2
a.send @ bundle-18a618685952a79fd739.js:2
x.send @ bundle-18a618685952a79fd739.js:2
t.onopen @ bundle-18a618685952a79fd739.js:2
t.dispatchEvent @ bundle-18a618685952a79fd739.js:2
x._open @ bundle-18a618685952a79fd739.js:2
x._transportMessage @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
(anonymous) @ bundle-18a618685952a79fd739.js:2
a.emit @ bundle-18a618685952a79fd739.js:2
n.onmessage @ bundle-18a618685952a79fd739.js:2
@Kathy Satterlee how would I use fleetctl to run a live query, via the server, locally?
k
I think we can skip that for now since you're getting those websocket errors. It definitely looks like there's something amiss in the configuration that's causing the connection to the websocket to fail. Do you have anything set up that might be blocking the POST request:
Copy code
POST <https://fleet.domain.tld/api/v1/fleet/results/909/i5naxpm1/xhr_send?t=1685054028641> 405 (Method Not Allowed)
b
Possibly related to
> In this case, you might need to disable the origin header (by setting this configuration to true) check or configure your reverse proxy to forward the correct Origin header.
https://fleetdm.com/docs/deploying/configuration#server-websockets-allow-unsafe-origin Had a similar issue come up for someone else not too long ago.
s
@Benjamin Edwards I am looking at that right now, but from the looks of it, that setting is not supported by the helmchart at this moment
b
its an env var for fleet server. I think you'd need to edit the spec for the fleet pod https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/
s
@Benjamin Edwards I did it slightly different and it actually works now!
Thank you for your input and thank you to @Kathy Satterlee as well!
the helm chart allows you to set extra env variables in the values block
as defined here:
so i did this in my config file for k8s:
Copy code
values:
    environments:
      FLEET_SERVER_WEBSOCKETS_ALLOW_UNSAFE_ORIGIN: true
b
ah there ya go. so it worked for you?
s
yes, comms with my agents and live queries now work!
b
sweet!
s
Thank you all for your support ❤️
b
yeah reverse proxies can be weird
I haven't seen nginx have this issue, but have seen it with other fancy ingress stuff
5 Views
Open in Slack
PreviousNext
Powered by