Hi, has anyone seen issues with full text searches from Fleet causing issues with aurora RDS? We've all of a sudden started to see our clusters rebooting and AWS actually proactively notified us on two of these. Here's what they say about it:
"Your Aurora MySQL cluster foo in region us-east-1 had been restarting frequently while running SQL statements involving full text search queries on engine version 2.10.0."
They have a fix they have applied for us, but it's very vague. Also, in at least one environment this seems to have corresponded to us turning on host expiration to remove MIA hosts from Fleet. Has anyone run into something like this before?
07/29/2021, 9:57 PM
FWIW we're using global aurora rds, and haven't had that issue
we did have issues before we upgraded to the latest aurora mysql engine
07/29/2021, 10:31 PM
@Jocelyn Bothe, thanks. What version did you upgrade to? I think we were on a lower version of 2.x a few months ago and also upgraded. we had issues with fleet 3.12+ on the older version
07/30/2021, 1:28 AM
We haven't seen issues with MySQL FTS in relation to Fleet in the past. Seemed like both of your orgs ran into issues with the very old version of Aurora crashing recently, though it seemed like that was resolved with upgrading to something newer?
FWIW I don't think the usage of FTS has changed in Fleet for years.
07/30/2021, 5:14 PM
@zwass - we previously had issues when going from lower versions of Fleet to 3.11+ and upgrading Aurora from 2.07.2 to 2.10.0 (minor MySQL 5.7 update) seemed to resolve that. Most recently we had this new issue where Fleet ground to a halt because calls to the DB were timing out. AWS has confirmed we hit a bug in Aurora related to FTS causing our RDS clusters to reboot frequently. I was just curious if anyone had seen it, or if Fleet's FTS behavior is dependent on how we use Fleet, i.e. are we doing something to cause FTS searching to be heavy.
07/30/2021, 7:25 PM
Thanks for the update! Interesting to learn where Aurora seems to deviate from MySQL.
08/09/2021, 9:27 PM
@zwass - just thought I'd share the feedback from AWS on Aurora 2.10 and the FTS issue we had. Basically, they tried to make improvements to FTS by syncing the FTS cache to persistent storage. this failed due to the following bug - https://jira.mariadb.org/browse/MDEV-12676. So the 'fix' for our issues was turning off their improvement. They plan to fix the issue in a future release.