Channels
doorman
infrastructure
random
zercurity
community-feeds
fleet-dev
code-review
queryhub
apple-silicon
carving
tls
fim
goquery
zentral
aws
querycon
golang
zeek
file-carving
fuzzing
auditing-warroom
linen-dev
fleetosquery
plugins
jobs
arm-architecture
darkbytes
process-auditing
uptycs
android_tests
selfgroup
vendor-feeds
fleet
eclecticiq-polylogyx-extension
ebpf
website
core
general
macos
kolide
osctrl
extensions
foundation
sql
officehours
linux
windows
Title
g
Gavin
09/27/2021, 4:59 PMI have not dug too deeply into this , Fleet docs suggest mysql 5.7 , I have noticed performance issues around labels , and policies on 5.7 but they’re less prominent on a 8.0 test instance I upgraded to.
Is the fleet team developing against 8 or 5.7 ?
I have yet to look yet through raw queries etc
t
Tomas Touceda
09/27/2021, 5:01 PMwe are developing against 5.7, support for 8 is not technically official yet, but it should work. Could you share a bit more about your setup, and the performance differences you are noticing?
we are working on improving performance there in different ways, the more information we have the better
g
Gavin
09/27/2021, 5:03 PMSo for 1000 hosts we’re seeing an additional 10-20% cpu utilisation per Device Policy added to the point where it no longer returns works after 5 policies are added .
On an 8 mirrored setup we’re seeing at least a response on like hardware.
This is timeouts against
policies/managet
Tomas Touceda
09/27/2021, 5:03 PMwhat version of fleet are you running?
g
Gavin
09/27/2021, 5:03 PMlatest.
t
Tomas Touceda
09/27/2021, 5:04 PM4.3.1? could you tell me the output of the following sql query:
select count(*) from policy_membership_historyg
Gavin
09/27/2021, 5:04 PMRight now I can’t but I will once the DB begins to respond.
t
Tomas Touceda
09/27/2021, 5:05 PMalso, could you tell me what type of resources do you have for the db?
g
Gavin
09/27/2021, 5:05 PMGCP 16gb RAM , 4vcpu
Fleet prior to v3 used to run with the same host count on a smaller instance of 1vcpu , 4gb ram however it was increased to handle the 3.3 migration issue.
Note I do expect additional DB load due to increased queries with new features so that is not a compliant.
t
Tomas Touceda
09/27/2021, 5:07 PMin the meantime, if I'm understanding you correctly, what fails is when you navigate to the list of policies, correct?
g
Gavin
09/27/2021, 5:07 PMYes
Timeout on that endpoint with an error page to raise a GH issue.
t
Tomas Touceda
09/27/2021, 5:07 PMyeah, new features are tricky that way
I have an idea of what might be failing, once I understand the scale of it based on your count, I can start thinking about strategies
I'm sorry you're facing this issue
g
Gavin
09/27/2021, 5:08 PMThese spikes out of context are trying to get results from compliance checks
And don’t be sorry, things happen we yolo latest to discover bugs and feedback.
🙏🏽 1
t
Tomas Touceda
09/27/2021, 5:09 PMwe might be running and storing policies too much: https://github.com/fleetdm/fleet/issues/2240
you might need to disable policies while we address this one, we have an idea of what it might be
👀 1
g
Gavin
09/27/2021, 5:11 PMYeah at this time I think I need to manually edit the db to remove the policies as it’s not a run time flag.
t
Tomas Touceda
09/27/2021, 5:13 PMdelete from policiesdelete from policy_membership_historyg
Gavin
09/27/2021, 5:13 PMOne item I am going to look at is does the number of running pods impact performance.
t
Tomas Touceda
09/27/2021, 5:27 PMwe'll publish a new version with a fix for this in the next day or two
g
Gavin
09/27/2021, 5:52 PMselect count(*) from policy_membership_historySELECT table_rows
-> FROM information_schema.tables
-> WHERE table_name='policy_membership_history';
+------------+
| table_rows |
+------------+
| 36844359 |
+------------+
1 row in set (0.04 sec)For 1000 hosts & 1 configured policy over 24 hours I don’t think this is the norm.
t
Tomas Touceda
09/27/2021, 5:57 PMyup, I'm working on a fix right now, if you run those deletes, it'll remove the policies and the history until we have the patch out