Hi Folks I am running into the strangest issue with Fleet My

Question

Hi Folks I am running into the strangest issue with Fleet My distributed queries sometimes execute and sometimes do not I am monitoring logs both on the Fleet server and the osquery node I see in the Fleet server that a NewDistributedQueryCampaign is scheduled successfully On the osquery node I am monitoring with ` verbose` and ` tls dump` to see all data I see that the osquery node is polling the read endpoint `api v1 osquery distributed read` with the correct node key It usually just gets back an empty `queries ` response Sometimes however it gets the proper query and runs it It seems to run it about 10 of the time Also notable is that creating packs scheduling queries works 100 of the time My Fleet server is deployed behind an HAProxy LB The LB uses its own certificates signed wildcard and my Fleet server uses self signed certs The osquery node is using the public key `pem` file for the LB cert and it enrolls properly Anyone have any ideas

sundsta · Answer

Is HAProxy caching anything

sundsta · Answer

I think it s a GET request so the response may be cached with an empty queries object from when osqueryd checked in and there were no distributed queries to run

sundsta · Answer

Edit looks like it s a POST request so less likely to be cached but still something to check

Erich Stoekl · Answer

interesting i ll look into that

Erich Stoekl · Answer

Woah I just found something crazy When I m logged in to the Fleet Web UI not using the load balancer so connecting to the UI directly at the server then my distributed queries work 100 of the time The osquery nodes can be connected through the LB

Erich Stoekl · Answer

It seems that when scheduling queries through the LB endpoint the queries don t actually get scheduled

zwass · Answer

Are you referring to scheduled queries in packs or live queries

Erich Stoekl · Answer

live queries

zwass · Answer

Does your LB support websockets We try to support live queries either way but that could potentially be an issue

Erich Stoekl · Answer

I m not sure I will check Does it normally use websockets Can I explicitly disable websockets

zwass · Answer

Normally it pushes results over a websocket I don t think you can explicitly disable them

Erich Stoekl · Answer

The osquery agent or the fleet server pushes results via websocket I see some stuff in the `LiveQuery` method in `server service client live query go` but I m having trouble figuring out what the websocket is doing

Erich Stoekl · Answer

ohh it s so the Web UI can interact with the fleet server backend right

zwass · Answer

yes

Erich Stoekl · Answer

Interesting when not using the LB my queries work but I see the following error in the Chrome Console ```WebSocket connection to <wss my kolide svc my company com api v1 kolide results 207 efwfuaj3 websocket> failed WebSocket is closed before the connection is established ``` However I see in the network tab that the websocket performed the content download and it took 2 26 ms When executing the live query through the LB I see in the network tab that the websocket is initiated but it just hangs there