https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi there, I am updating to 4.73 from 4.69 and that...
# fleet
p
Hi there, I am updating to 4.73 from 4.69 and that cause a tons of errors including 
*err*: authentication error: find host: Error 1146 (42S02): Table 'kolide.host_identity_scep_certificates' doesn't exist
Seems like there may be an issue with the schemas
k
Hi @pickmansec! Did you run into any errors while running migrations?
p
Hi Kathy, I am looking for some of thos logs but I dont see any yet. but i am going to look a bit more
we are running 
/usr/bin/fleet prepare db
k
Is that running with all Fleet instances offline?
p
hmmm thats a good question, I believe it is
k
If you could do the following and share what you see in the logs, we'll hopefully see some good data in the logs: 1. Take all Fleet instances offline 2. Bring up a single Fleet instance to run 
/usr/bin/fleet prepare db
a. Make sure this instance is connected to the production database 3. If that completes without errors, bring Fleet back up
p
Thanks for that I will give that a shot.
Also do you know why this error shows up? 
*err*: enroll failed: no matching secret found: EnrollSecret was not found in the datastore
We get it farily often but I am not sure what causes it.
Copy code
2025/09/19 17:49:29 FAIL 20250410104321_UpdateMacOSSoftwareNames.go (adding temporary index to host_software_installed_paths: Error 1061 (42000): Duplicate key name 'software_id'), quitting migration.
Migration Failed
k
*err*: enroll failed: no matching secret found: EnrollSecret was not found in the datastore
You'll see that error when you've got a client out there trying to enroll with an invalid enroll secret.
Taking a look at the migration that's throwing that error. Do you run multiple Fleet instances, or just one?
p
Going back to your previous question, we do have multiple instances and I suspect they all ran at the same time
k
That sounds likely. One started the migration, then another tried to take the same action.
Can you try it again, making sure that only one Fleet instance is brought up first?
p
Sure I can do that
Thank you again for helping me
is there a lock in place against the mysql db that would prevent this multiple update issue?
couldnt there be a flag in the mysql db saying its being updated
k
Best practice is to not start all Fleet instances with the 'prepare db' command, but to bring up a single instance to run migrations.
It's really important that there is no communication with agents happening during the migration process.
How do you have Fleet deployed?
It sounds like you may also have the --allow-missing-migrations flag enabled. If so, I'd definitely recommend turning that off as well.
p
How do you collect logs from systems while it is updating
k
Any queued up logs from osquery will flow through once Fleet is back online.
p
We brought it down to one instance and it said
Copy code
Migrations already completed. Nothing to do.
Migration Success
however when we brought it back to multiple instances it failed
We are going to try one instance again
We double checked with running this with one instance and it still fails
Probably a good idea to create a lock on the db while the update is running by inserting a short row into the db
or a file on disk
or environment variable
k
Looks like we've maybe got some manual work to do. Taking a look at some specific migrations to see what the issue may be.
Just to confirm, with only one instance running, what is the error you're seeing?
Is it this one: 
*err*: authentication error: find host: Error 1146 (42S02): Table 'kolide.host_identity_scep_certificates' doesn't exist
And do you have full access to the database? We may need to do some manual remediation.
p
im getting the above error and
Copy code
2025/09/19 17:49:29 FAIL 20250410104321_UpdateMacOSSoftwareNames.go (adding temporary index to host_software_installed_paths: Error 1061 (42000): Duplicate key name 'software_id'), quitting migration.Migration Failed`
k
If we're seeing both errors, it sounds like there may still be more than one instance coming up, We should never be getting to that authentication error if the migration is failing. Do you have a sec to hop on a call and try to look at this together?
p
Unfortunately I do not have time today
But I appreciate you offering your time
k
If you could send over logs from the start to the finish of running the migrations, I'd be more than happy to take a look.
2 Views
Open in Slack
PreviousNext
Powered by