Hi!
I ran into live queries issues after 4.3.0 upd...
# fleeta
Artem
09/15/2021, 8:11 AMHi!
I ran into live queries issues after 4.3.0 update.
When I try to make requests in UI fleet, they hang for a long time and the result does not appear ( With fleetctl, requests are executed every other time).
I have similar issues after update to 4.2.2
https://osquery.slack.com/archives/C01DXJL16D8/p1629901892389400
P.S. the results of all queries from packs are executed and accepted on the server in the file osquery_results. And then they are loaded into Elastic. I have only problems with live queries from fleet UI and fleetctl.
Artem
09/15/2021, 8:32 AMAnd fetching hosts doesn't work after update
t
Tomas Touceda
09/15/2021, 1:24 PMhi, regarding fetching hosts, what happens if you refresh the page?
Tomas Touceda
09/15/2021, 1:24 PMfor live queries, could you tell me a bit more about your infrastructure? what redis and mysql are you running, how many nodes, etc
a
Artem
09/15/2021, 1:30 PMRedis version=4.0.9
mysql Ver 14.14 Distrib 5.7.31
1 node
234 hosts connectred to fleet
After refresh page I see similar picture(
And all of my hosts couldn't fetched after update fleet(
t
Tomas Touceda
09/15/2021, 1:33 PMis this deployed in aws?
Tomas Touceda
09/15/2021, 1:34 PMand could you share logs for fleet serve during these issues?
a
Artem
09/15/2021, 1:39 PM1. We deployed in local infrastructure
2.
Sep 15 16:36:58 z14-1897-fleet fleet[5201]: {"component":"http","level":"info","method":"POST","numHosts":1,"query_id":null,"sql":"SELECT * FROM osquery_info","took":"35.803443ms","ts":"2021-09-15T13:36:58.148541382Z","uri":"/api/v1/fleet/queries/run","user":"sychevak"}Artem
09/15/2021, 1:44 PMIn previous issue @zwass made a patch here, maybe it will help. Now a similar problem
t
Tomas Touceda
09/15/2021, 2:13 PMcould you run fleet serve with --logging_debug and share more logs so that we can understand what is happening overall?
a
Artem
09/15/2021, 2:59 PMArtem
09/15/2021, 3:00 PMlevel=debug ts=2021-09-15T14:58:15.625374827Z component=http method=POST uri=/api/v1/osquery/distributed/read took=2.697696ms ip_addr=172.31.7.3:48608 x_for_ip_addr= err="unsupported value type"t
Tomas Touceda
09/15/2021, 3:01 PMwhat's the output of
fleetctl get configa
Artem
09/15/2021, 3:04 PMArtem
09/15/2021, 3:04 PMt
Tomas Touceda
09/15/2021, 3:39 PMthat error in the logs is probably a red herring in terms of the issue with live queries. Did you try running a live query in the logs above? I don't see any of the expected calls
a
Artem
09/16/2021, 6:22 AMI have not seen any more errors on the server side of the fleet, it seems to me that there is an error in the client side of the fleet UI. Last time there was an error in the client side of the Fleet.
maybe @zwass can help us
t
Tomas Touceda
09/16/2021, 1:11 PMCould you screenshot the Network tab in a browser while you reproduce the issue so that we can see more information about what might happening?
a
Artem
09/16/2021, 1:56 PMArtem
09/16/2021, 2:06 PMI attach the responses to the first two requests from the screenshot
Artem
09/16/2021, 2:19 PMmaybe this issue related with my problems with database migration
https://github.com/fleetdm/fleet/issues/2074
This is my issue 🙂
t
Tomas Touceda
09/16/2021, 2:20 PMoh, so you're running 4.3.0 while not having been able to run the migrations?
a
Artem
09/16/2021, 2:20 PMbut i cannot downmigrate((
How could I solve this problems...
Artem
09/16/2021, 2:22 PMI think that error in migrate only related with software on the hosts
t
Tomas Touceda
09/16/2021, 2:35 PMdid you backup the db before updating>
a
Artem
09/17/2021, 6:32 AMI backup server. Ok I will restore it(
g
Gavin
09/21/2021, 5:19 PMSo I just ran into this issue https://github.com/fleetdm/fleet/issues/2074
Looking at the DB the migration of the Software tables causes the resource utilisation to spike crashing the DB I was able to migrate once I upped the CPU + RAM
t
Tomas Touceda
09/21/2021, 5:20 PMgreat insight! what instance were you running for the db?
g
Gavin
09/21/2021, 5:22 PMAdded comment on the original ticket now.
Gavin
09/21/2021, 5:23 PMbut 4x scale was needed.
Gavin
09/21/2021, 5:31 PMit also took around an hour to complete.
t
Tomas Touceda
09/21/2021, 5:51 PMwe are looking into potential improvements for that migration, thank you for the data, it was very helpful
âž• 1
g
Gavin
09/21/2021, 6:02 PMYeah I am currently looking at the actual migration it seems that the following delete is not very nice .
Copy code
DELETE FROM software WHERE NOT EXISTS (select 1 from host_software hs where hs.software_id=software.id)Gavin
09/21/2021, 6:03 PMShow warnings provides the following
Copy code
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELETE FROM software WHERE NOT EXISTz
zwass
09/21/2021, 7:34 PM
@Gavin we just updated the migration thanks to your feedback: https://github.com/fleetdm/fleet/pull/2163
g
Gavin
09/21/2021, 7:39 PMAwesome
2 Views