Hello, I've been troubleshooting an issue for a bit now and I'm not sure where to go. ISSUE: I have two Linux hosts (FleetDM Server itsself on Ubuntu, and a Debian server) who will not respond to queries, yet show up as valid hosts listing all software/vulns,etc... STACK • Ubuntu 24.04 LTS hosted on a major cloud provider • Apache2 for Reverse Proxy • All services running localhost (Apache, REDIS, MYSQL, FleetDM) ERROR LOGS: • syslog shows errors like '2024-08-10T123652.090409+00:00 FleetDM fleet[753]: level=error ts=2024-08-10T123652.088296762Z component=http method=POST uri=/api/v1/osquery/distributed/write took=4.885414ms ip_addr=XXX.245.XXX.41 x_for_ip_addr=XXX.245.XXX.41 ingestion-err="campaignID=10 waiting for listener" err="error in query ingestion" ' • No errors in mysql logs • No errors in redis logs • No erros in apache2 logs SCREENSHOTS: • My FleetDM Config file • FleetDM Hosts Dashboard • Queries Page • Live Query page • Ubuntu syslog errors • Apache2 Virtual Host File FIREWALL: • Configured on the cloud provider portal • Inbound Ports 22,80,443 open on all machines • No outbound ports blocked And recommendations on where to go, what to troubleshoot next, or any configuration recommendations? Thanks in advance

Thanks for the detailed information! Typically, that error occurs when a host checks in with results either before the web browser has connected to the websocket, or the client responds after the connection has closed. How long after starting the query are you seeing these errors?

Hello, it these log entries should seconds after running the query - then essentially fron the front-end it hangs forever. Meaning, I can run a querry against multiple hosts and wait 2 hours on that screen and nothing happens. Eventually some of these polling log entries stop comming however nothing ever returns. I tried Windows and Linux hosts and all the same query results. However again I can run scripts against hosts with no issues and good response time.

Do you have anything in your environment blocking Web Sockets?

Hello, no I do not. I'm using apache2 as a reverse proxy which I assume would be any web socket blocker - although its a very vanilla configuration on the server overall. Still today I can't run scheduled or live QURIES, yet software assets populate and show vulnerability data, also scheduled and live SCRIPTS run fine. Not sure where to go from here; Nothing juicy in service logs like apache, mysql, redis besides the log data I originally provided. Extra notes; • I'm using the latest "Free" FleetDM self hosted version on a major cloud provider • I've been building, deploying and managing hundreds of servers and endpoints using a plethora of tech stacks (NodeJS, Java, PHP, Python, Microservices architecture, mobile apps, Desktop apps, Security tooling, Enterprise solutions of all kinds for orgs with hundreds of users....) for over 10 years. • I'm a software developer for companies around the world, held DevOps roles for years, and currently a full time IT Manager for an Enterprise organization. Still no luck, breaking my head , and about to throw in the towel. Please send help, Cheers

If you look at network traffic in the browser's dev console, do you see any websocket traffic there?