Mucking around with some ideas. Does Fleet remove the redis messages for live queries from redis (

results_*

) once it gets them? Based on what I am seeing in the code, it looks like it is just a normal pub/sub setup, but just wanted to check.

Yes, as it's using the pub/sub features of Redis the message is gone after the subscribers have received it.

hmmmm. When I have another subscriber, Fleet is only picking up some of the results, not all.

You're saying you have another client using a redis

SUBSCRIBE

command?

yes, I just verified it using a different client.

redis-cli

PSUBSCRIBE *

Run default query against 3 systems (select * from osquery_info) - Get 1 result from each of the 3 systems in the redis-cli, but only get 2 results in the FleetUI. (3 hosts returning 2 results)

Yep, I see it.

I will submit an issue, as it appears it must be a bug of some kind

I think it's because Redis doesn't take writes to the pubsub channel until there's at least one listener. This is fine for typical usage since osqueryd will just wait until there's a listener. In your case it's probably that some results are being read by the redis shell listener and are not made available to the UI live query listener.

sure - short version: I am playing around with the idea of being able to make a copy of the live query results and send it to another system so that the user can run further analysis beyond what is currently available in the Live Query UI. (ie stacking, filtering, etc). I actually have it working quite well already (Logstash subs to the channel and pushes the results to ES, which is then accessible to another tool). I would still like the results of the Live Query to be accessible in the WebUI as it is currently - that is why I was going down the rabbit hole of trying to figure out what is going on.

When I say internal API I'm referring to the way Fleet uses Redis to push live query results around. Fleet never expected someone to "hook in" to that point.

Once again, here is a quick idea of what I am doing: https://www.screencast.com/t/FH8G5wTI

Fleet expects a listening subscriber to indicate that the web UI is listening essentially. So the confusion is that you've got something else listening.

Thanks Zach!

Okay, so a couple of things: 1. Is this something you are ready to ship? Trying to judge whether the change needs to make it in for the release tomorrow or if it can wait a few weeks. 2. We can probably make this work for your use case. This would still be working with an internal API, so no guarantees of stability. Does that work for you?

Nope, it can wait

Okay cool can you please file an issue so we can track this? I have some ideas.

Just casting in some extra feedback - I'd love to have live query stuff logged from fleet (and possibly some other visibility) if this is something you're actively looking into @zwass. Probably several requests: • ability to log all results to wherever fleet is configured to send its logs • saving live query results, even temporarily, and providing a way to view them from the UI • ability for admins to view & possibly cancel currently-running live queries • sorting / searching of live query results (#177)

@nyanshak re: request #3, wrote up some thoughts: https://github.com/fleetdm/fleet/issues/348

@zwass Issue created: https://github.com/fleetdm/fleet/issues/349