Channels
android_tests
apple-silicon
arm-architecture
auditing-warroom
aws
carving
code-review
community-feeds
core
darkbytes
doorman
ebpf
eclecticiq-polylogyx-extension
extensions
file-carving
fim
fleet
fleet-dev
fleetosquery
foundation
fuzzing
general
golang
goquery
infrastructure
jobs
kolide
linen-dev
linux
macos
officehours
osctrl
plugins
process-auditing
querycon
queryhub
random
selfgroup
sql
tls
uptycs
vendor-feeds
website
windows
zeek
zentral
zercurity
Title
a
Artem
09/15/2021, 8:11 AMHi!
I ran into live queries issues after 4.3.0 update.
When I try to make requests in UI fleet, they hang for a long time and the result does not appear ( With fleetctl, requests are executed every other time).
I have similar issues after update to 4.2.2
https://osquery.slack.com/archives/C01DXJL16D8/p1629901892389400
P.S. the results of all queries from packs are executed and accepted on the server in the file osquery_results. And then they are loaded into Elastic. I have only problems with live queries from fleet UI and fleetctl.
And fetching hosts doesn't work after update
t
Tomas Touceda
09/15/2021, 1:24 PMhi, regarding fetching hosts, what happens if you refresh the page?
for live queries, could you tell me a bit more about your infrastructure? what redis and mysql are you running, how many nodes, etc
a
Artem
09/15/2021, 1:30 PMRedis version=4.0.9
mysql Ver 14.14 Distrib 5.7.31
1 node
234 hosts connectred to fleet
After refresh page I see similar picture(
And all of my hosts couldn't fetched after update fleet(
t
Tomas Touceda
09/15/2021, 1:33 PMis this deployed in aws?
and could you share logs for fleet serve during these issues?
a
Artem
09/15/2021, 1:39 PM1. We deployed in local infrastructure
2.
Sep 15 16:36:58 z14-1897-fleet fleet[5201]: {"component":"http","level":"info","method":"POST","numHosts":1,"query_id":null,"sql":"SELECT * FROM osquery_info","took":"35.803443ms","ts":"2021-09-15T13:36:58.148541382Z","uri":"/api/v1/fleet/queries/run","user":"sychevak"}In previous issue @zwass made a patch here, maybe it will help. Now a similar problem
t
Tomas Touceda
09/15/2021, 2:13 PMcould you run fleet serve with --logging_debug and share more logs so that we can understand what is happening overall?
a
Artem
09/15/2021, 2:59 PMlevel=debug ts=2021-09-15T14:58:15.625374827Z component=http method=POST uri=/api/v1/osquery/distributed/read took=2.697696ms ip_addr=172.31.7.3:48608 x_for_ip_addr= err="unsupported value type"t
Tomas Touceda
09/15/2021, 3:01 PMwhat's the output of
fleetctl get configa
Artem
09/15/2021, 3:04 PMt
Tomas Touceda
09/15/2021, 3:39 PMthat error in the logs is probably a red herring in terms of the issue with live queries. Did you try running a live query in the logs above? I don't see any of the expected calls
a
Artem
09/16/2021, 6:22 AMI have not seen any more errors on the server side of the fleet, it seems to me that there is an error in the client side of the fleet UI. Last time there was an error in the client side of the Fleet.
maybe @zwass can help us
t
Tomas Touceda
09/16/2021, 1:11 PMCould you screenshot the Network tab in a browser while you reproduce the issue so that we can see more information about what might happening?
a
Artem
09/16/2021, 1:56 PMI attach the responses to the first two requests from the screenshot
maybe this issue related with my problems with database migration
https://github.com/fleetdm/fleet/issues/2074
This is my issue 🙂
t
Tomas Touceda
09/16/2021, 2:20 PMoh, so you're running 4.3.0 while not having been able to run the migrations?
a
Artem
09/16/2021, 2:20 PMbut i cannot downmigrate((
How could I solve this problems...
I think that error in migrate only related with software on the hosts
t
Tomas Touceda
09/16/2021, 2:35 PMdid you backup the db before updating>
a
Artem
09/17/2021, 6:32 AMI backup server. Ok I will restore it(
g
Gavin
09/21/2021, 5:19 PMSo I just ran into this issue https://github.com/fleetdm/fleet/issues/2074
Looking at the DB the migration of the Software tables causes the resource utilisation to spike crashing the DB I was able to migrate once I upped the CPU + RAM
t
Tomas Touceda
09/21/2021, 5:20 PMgreat insight! what instance were you running for the db?
g
Gavin
09/21/2021, 5:22 PMAdded comment on the original ticket now.
but 4x scale was needed.
it also took around an hour to complete.
t
Tomas Touceda
09/21/2021, 5:51 PMwe are looking into potential improvements for that migration, thank you for the data, it was very helpful
âž• 1
g
Gavin
09/21/2021, 6:02 PMYeah I am currently looking at the actual migration it seems that the following delete is not very nice .
DELETE FROM software WHERE NOT EXISTS (select 1 from host_software hs where hs.software_id=software.id)Show warnings provides the following
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELETE FROM software WHERE NOT EXISTz
zwass
09/21/2021, 7:34 PM@Gavin we just updated the migration thanks to your feedback: https://github.com/fleetdm/fleet/pull/2163
g
Gavin
09/21/2021, 7:39 PMAwesome