we removed pre-built because they were a pain to keep up to date, and required to be rebuilt from scratch every time we wanted to change build flags etc

This doesn’t seem right. The same ideas of ccache should be true here — hash all those input parameters, and store the artifact. Whether the artifact storage costs are worth the time savings is a different question.