Channels
android_tests
apple-silicon
arm-architecture
auditing-warroom
aws
carving
code-review
community-feeds
core
darkbytes
doorman
ebpf
eclecticiq-polylogyx-extension
extensions
file-carving
fim
fleet
fleet-dev
fleetosquery
foundation
fuzzing
general
golang
goquery
infrastructure
jobs
kolide
linen-dev
linux
macos
officehours
osctrl
plugins
process-auditing
querycon
queryhub
random
selfgroup
sql
tls
uptycs
vendor-feeds
website
windows
zeek
zentral
zercurity
Title
r
Ryan
09/16/2021, 4:45 PMMy upgrade to 4.3.0 hasn’t gone well, it appears to be stuck on a migration, MySQL process list shows it “sending data” for the following query:
DELETE FROM software WHERE NOT EXISTS (select 1 from host_software hs where hs.software_id=software.id)t
Tomas Touceda
09/16/2021, 4:55 PMhave you checked that all the fleet instances have stopped running?
r
Ryan
09/20/2021, 4:55 PMthey were still running, but I haven’t been stopping them for previous migrations, is that something I should start doing? it does make rolling deployments more difficult.
In the end I just truncated all the software tables, there were ~ four of them. Given they are rebuilt by the system itself anyway as hosts “check in” I didn’t think they were too important, and once that was done, the migrations were able to complete and Fleet started working again.
It was a bit of a scary release though, all the others have been super smooth so I had gotten used to just running them.
t
Tomas Touceda
09/20/2021, 5:06 PMI haven’t been stopping them for previous migrations, is that something I should start doing?yes, always. Otherwise locking is very likely to happen in an active system
It was a bit of a scary release though, all the others have been super smooth so I had gotten used to just running them.I hear you. I'm sorry it was that way. Migrations were never intended to run while fleet is running, though, so the fact that you were able to so far was a lucky shot.
r
Ryan
09/20/2021, 5:14 PMhahaha fair enough 😂
I’ll make an enhancement to our automation to stop the service when migrations are needed. Is there some mechanism I can use to detect when a migration should run, or just set it up to trigger if the package version ever changes?
t
Tomas Touceda
09/20/2021, 5:16 PMthe simplest is to trigger with every update. It's a quick noop if there's nothing
r
Ryan
09/20/2021, 5:18 PMok no worries then 🙂
thanks for your help, appreciated!
👍 1