https://github.com/osquery/osquery logo
Powered by
#fleet
Title
# fleet
a

Artem

09/15/2021, 8:11 AM
Hi! I ran into live queries issues after 4.3.0 update. When I try to make requests in UI fleet, they hang for a long time and the result does not appear ( With fleetctl, requests are executed every other time). I have similar issues after update to 4.2.2 https://osquery.slack.com/archives/C01DXJL16D8/p1629901892389400 P.S. the results of all queries from packs are executed and accepted on the server in the file osquery_results. And then they are loaded into Elastic. I have only problems with live queries from fleet UI and fleetctl.
And fetching hosts doesn't work after update
t

Tomas Touceda

09/15/2021, 1:24 PM
hi, regarding fetching hosts, what happens if you refresh the page?
for live queries, could you tell me a bit more about your infrastructure? what redis and mysql are you running, how many nodes, etc
a

Artem

09/15/2021, 1:30 PM
Redis version=4.0.9 mysql Ver 14.14 Distrib 5.7.31 1 node 234 hosts connectred to fleet After refresh page I see similar picture( And all of my hosts couldn't fetched after update fleet(
t

Tomas Touceda

09/15/2021, 1:33 PM
is this deployed in aws?
and could you share logs for fleet serve during these issues?
a

Artem

09/15/2021, 1:39 PM
1. We deployed in local infrastructure 2. 
Sep 15 16:36:58 z14-1897-fleet fleet[5201]: {"component":"http","level":"info","method":"POST","numHosts":1,"query_id":null,"sql":"SELECT * FROM osquery_info","took":"35.803443ms","ts":"2021-09-15T13:36:58.148541382Z","uri":"/api/v1/fleet/queries/run","user":"sychevak"}
but query does not output any results
In previous issue @zwass made a patch here, maybe it will help. Now a similar problem
t

Tomas Touceda

09/15/2021, 2:13 PM
could you run fleet serve with --logging_debug and share more logs so that we can understand what is happening overall?
a

Artem

09/15/2021, 2:59 PM
level=debug ts=2021-09-15T14:58:15.625374827Z component=http method=POST uri=/api/v1/osquery/distributed/read took=2.697696ms ip_addr=172.31.7.3:48608 x_for_ip_addr= err="unsupported value type"
t

Tomas Touceda

09/15/2021, 3:01 PM
what's the output of 
fleetctl get config
?
a

Artem

09/15/2021, 3:04 PM
t

Tomas Touceda

09/15/2021, 3:39 PM
that error in the logs is probably a red herring in terms of the issue with live queries. Did you try running a live query in the logs above? I don't see any of the expected calls
a

Artem

09/16/2021, 6:22 AM
I have not seen any more errors on the server side of the fleet, it seems to me that there is an error in the client side of the fleet UI. Last time there was an error in the client side of the Fleet. maybe @zwass can help us
t

Tomas Touceda

09/16/2021, 1:11 PM
Could you screenshot the Network tab in a browser while you reproduce the issue so that we can see more information about what might happening?
a

Artem

09/16/2021, 1:56 PM
I attach the responses to the first two requests from the screenshot
response2.txtresponse.txt
maybe this issue related with my problems with database migration https://github.com/fleetdm/fleet/issues/2074 This is my issue 🙂
t

Tomas Touceda

09/16/2021, 2:20 PM
oh, so you're running 4.3.0 while not having been able to run the migrations?
a

Artem

09/16/2021, 2:20 PM
but i cannot downmigrate(( How could I solve this problems...
I think that error in migrate only related with software on the hosts
t

Tomas Touceda

09/16/2021, 2:35 PM
did you backup the db before updating>
a

Artem

09/17/2021, 6:32 AM
I backup server. Ok I will restore it(
g

Gavin

09/21/2021, 5:19 PM
So I just ran into this issue https://github.com/fleetdm/fleet/issues/2074 Looking at the DB the migration of the Software tables causes the resource utilisation to spike crashing the DB I was able to migrate once I upped the CPU + RAM
t

Tomas Touceda

09/21/2021, 5:20 PM
great insight! what instance were you running for the db?
g

Gavin

09/21/2021, 5:22 PM
Added comment on the original ticket now.
but 4x scale was needed.
it also took around an hour to complete.
t

Tomas Touceda

09/21/2021, 5:51 PM
we are looking into potential improvements for that migration, thank you for the data, it was very helpful
âž• 1
g

Gavin

09/21/2021, 6:02 PM
Yeah I am currently looking at the actual migration it seems that the following delete is not very nice .
Copy code
DELETE FROM software WHERE NOT EXISTS (select 1 from host_software hs where hs.software_id=software.id)
Show warnings provides the following
Copy code
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELETE FROM software WHERE NOT EXIST
z

zwass

09/21/2021, 7:34 PM
@Gavin we just updated the migration thanks to your feedback: https://github.com/fleetdm/fleet/pull/2163
g

Gavin

09/21/2021, 7:39 PM
Awesome
2 Views
Powered by