Building on a machine other than the Pi sounds like a great idea. Maybe we should add a Raspberry Pi to the compile speed benchmarks thread so we can see the difference.
Would it also be feasible to build with Docker and then copy the file from the container to the Pi? I’m not very familiar with Raspberry Pi other than “it’s small and cheap”.