Hi all! I moved fleet from a self signed ssl cert to setting

server: tls: false

in the

config.yaml

and using nginx as the reverse proxy to expos fleet. Since then, queries aren't going through to the osquery machines and Im getting this error:

Jan 15 12:49:53 localhost fleet[33217]: ts=2025-01-15T12:49:53.411443589Z component=http err="read auth token: reading from websocket: sockjs: session not in open state" msg="failed to read auth token"

which doesn't seem to have any documentation apart from this singular forum post which seems to imply its a load balancer issue? I'm running mysql and redis in a docker container. This is my compose file:

version: '3.8'
services:
 cache:
  image: redis:6.2-alpine
  restart: always
  ports:
   - '6379:6379'
  command: redis-server --save 20 1 --loglevel warning --requirepass **********
  volumes: 
   - cache:/data

 db:
  image: mysql:8.4.3
  cap_add:
   - SYS_NICE
  restart: always
  environment:
   - MYSQL_DATABASE=fleet
   - MYSQL_ROOT_PASSWORD=**********
  ports:
   - '3306:3306'
  volumes:
   - db:/var/lib/mysql
   - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql
volumes:
 db:
  driver: local
 cache:
  driver: local

And fleet is running as a systemd servicewith this `config.yaml`:

mysql:
  address: localhost:3306
  database: fleet
  username: root
  password: ******
redis:
  address: 127.0.0.1:6379
  password: ********
server:
  #cert: /etc/ssl/certs/kolide.csr
  #key: /etc/ssl/private/kolide.key
  tls: false

My osquery flags file is as follows:

# Server
--tls_hostname=*******
--tls_server_certs=/etc/osquery/fleet.pem
# Enrollment
--host_identifier=fleet-server
--enroll_secret_path=/etc/osquery/secret.txt
--enroll_tls_endpoint=/api/osquery/enroll
# Configuration
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_refresh=10
# Live query
--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
# Logging
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10
# File carving
--disable_carver=false
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block
--carver_block_size=8000000

with

secret.txt

and

enroll

key taken from the fleet web gui in "Add hosts" > "Advanced" Any idea why this might be happening?

Hi @Smaran Jawalkar - What version of Fleet were you on and what did you upgrade or attempt to upgrade to?

I am on fleet v 4.62.0, Im not attempting any upgrades. The person who posted the same error in the forum post was upgrading, I am facing same error on fresh install of fleet v 4.62.0. I started facing this issue after shifting to using https by using nginx and certbot for ssl and passing

server: tls: false

to the fleet

config.yaml

instead of passing my own self signed certs to fleet

Ah got it. Thank you! Will have a member of the team take a look at this today

Not sure if this is relevant but this is another error being seen in the logs:

Jan 15 12:50:04 localhost fleet[33217]: level=error ts=2025-01-15T12:50:04.966243147Z component=http method=POST uri=/api/v1/osquery/distributed/write took=2.210057ms ip_addr=172.236.188.31 x_for_ip_addr=172.236.188.31 ingestion-err="campaignID=15 waiting for listener" err="error in query ingestion"

Live queries basically never complete on any hosts

try setting this value to true on the fleet server env vars https://fleetdm.com/docs/configuration/fleet-server-configuration#server-websockets-allow-unsafe-origin reverse proxy with websockets can get mixed up easily

@Smaran Jawalkar ^

Nope, I'm getting the same error even after setting

websockets_allow_unsafe_origin: true

😞

I believe you need to explicitly allow websockets in NGINX. Checking their documentation.

Changing that didn't work either. Seeing the issues with using nginx, shifted back to using the embedded web server in

fleet

and everything works fine now. Tysm for the help!