Title
#fleet
r

Ryan

09/16/2021, 4:45 PM
My upgrade to 4.3.0 hasn’t gone well, it appears to be stuck on a migration, MySQL process list shows it “sending data” for the following query:
DELETE FROM software WHERE NOT EXISTS (select 1 from host_software hs where hs.software_id=software.id)
Tomas Touceda

Tomas Touceda

09/16/2021, 4:55 PM
have you checked that all the fleet instances have stopped running?
r

Ryan

09/20/2021, 4:55 PM
they were still running, but I haven’t been stopping them for previous migrations, is that something I should start doing? it does make rolling deployments more difficult.
4:56 PM
In the end I just truncated all the software tables, there were ~ four of them. Given they are rebuilt by the system itself anyway as hosts “check in” I didn’t think they were too important, and once that was done, the migrations were able to complete and Fleet started working again.
4:57 PM
It was a bit of a scary release though, all the others have been super smooth so I had gotten used to just running them.
Tomas Touceda

Tomas Touceda

09/20/2021, 5:06 PM
I haven’t been stopping them for previous migrations, is that something I should start doing?
yes, always. Otherwise locking is very likely to happen in an active system
5:06 PM
It was a bit of a scary release though, all the others have been super smooth so I had gotten used to just running them.
I hear you. I'm sorry it was that way. Migrations were never intended to run while fleet is running, though, so the fact that you were able to so far was a lucky shot.
r

Ryan

09/20/2021, 5:14 PM
hahaha fair enough 😂
5:15 PM
I’ll make an enhancement to our automation to stop the service when migrations are needed. Is there some mechanism I can use to detect when a migration should run, or just set it up to trigger if the package version ever changes?
Tomas Touceda

Tomas Touceda

09/20/2021, 5:16 PM
the simplest is to trigger with every update. It's a quick noop if there's nothing
r

Ryan

09/20/2021, 5:18 PM
ok no worries then 🙂
5:18 PM
thanks for your help, appreciated!