Hi! I ran into live queries issues after 4.3.0 upd...
# fleet
a
Hi! I ran into live queries issues after 4.3.0 update. When I try to make requests in UI fleet, they hang for a long time and the result does not appear ( With fleetctl, requests are executed every other time). I have similar issues after update to 4.2.2 https://osquery.slack.com/archives/C01DXJL16D8/p1629901892389400 P.S. the results of all queries from packs are executed and accepted on the server in the file osquery_results. And then they are loaded into Elastic. I have only problems with live queries from fleet UI and fleetctl.
And fetching hosts doesn't work after update
t
hi, regarding fetching hosts, what happens if you refresh the page?
for live queries, could you tell me a bit more about your infrastructure? what redis and mysql are you running, how many nodes, etc
a
Redis version=4.0.9 mysql Ver 14.14 Distrib 5.7.31 1 node 234 hosts connectred to fleet After refresh page I see similar picture( And all of my hosts couldn't fetched after update fleet(
t
is this deployed in aws?
and could you share logs for fleet serve during these issues?
a
1. We deployed in local infrastructure 2.
Sep 15 16:36:58 z14-1897-fleet fleet[5201]: {"component":"http","level":"info","method":"POST","numHosts":1,"query_id":null,"sql":"SELECT * FROM osquery_info","took":"35.803443ms","ts":"2021-09-15T13:36:58.148541382Z","uri":"/api/v1/fleet/queries/run","user":"sychevak"}
but query does not output any results
In previous issue @zwass made a patch here, maybe it will help. Now a similar problem
t
could you run fleet serve with --logging_debug and share more logs so that we can understand what is happening overall?
a
level=debug ts=2021-09-15T14:58:15.625374827Z component=http method=POST uri=/api/v1/osquery/distributed/read took=2.697696ms ip_addr=172.31.7.3:48608 x_for_ip_addr= err="unsupported value type"
t
what's the output of
fleetctl get config
?
a
t
that error in the logs is probably a red herring in terms of the issue with live queries. Did you try running a live query in the logs above? I don't see any of the expected calls
a
I have not seen any more errors on the server side of the fleet, it seems to me that there is an error in the client side of the fleet UI. Last time there was an error in the client side of the Fleet. maybe @zwass can help us
t
Could you screenshot the Network tab in a browser while you reproduce the issue so that we can see more information about what might happening?
a
I attach the responses to the first two requests from the screenshot
maybe this issue related with my problems with database migration https://github.com/fleetdm/fleet/issues/2074 This is my issue 🙂
t
oh, so you're running 4.3.0 while not having been able to run the migrations?
a
but i cannot downmigrate(( How could I solve this problems...
I think that error in migrate only related with software on the hosts
t
did you backup the db before updating>
a
I backup server. Ok I will restore it(
g
So I just ran into this issue https://github.com/fleetdm/fleet/issues/2074 Looking at the DB the migration of the Software tables causes the resource utilisation to spike crashing the DB I was able to migrate once I upped the CPU + RAM
t
great insight! what instance were you running for the db?
g
Added comment on the original ticket now.
but 4x scale was needed.
it also took around an hour to complete.
t
we are looking into potential improvements for that migration, thank you for the data, it was very helpful
1
g
Yeah I am currently looking at the actual migration it seems that the following delete is not very nice .
Copy code
DELETE FROM software WHERE NOT EXISTS (select 1 from host_software hs where hs.software_id=software.id)
Show warnings provides the following
Copy code
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELETE FROM software WHERE NOT EXIST
z
@Gavin we just updated the migration thanks to your feedback: https://github.com/fleetdm/fleet/pull/2163
g
Awesome