We’re currently running CI on three operating systems: Linux, macOS, Windows.
But Crystal supports many more platforms, though we cannot test them automatically.
We have some simple smoke tests to ensure code at least compiles for these platforms, but there’s not validation that it actually runs. That’s better than nothing, but not much.
The primary limitation for expanding CI targets is limited support in GitHub Actions. The runner only works on these three operating systems.
It’s technically possible to execute a VM on a GitHub runner with alternative operating systems in the VM.
I’ve tried to test that a couple of months ago, but it didn’t work well. Might be related to the resource demand. It was slow and unreliable. So not really a good option. Might be worth trying again to see if things have improved, but I’m not expecting much.
Of course we’re not restricted to GitHub Actions. We can run CI on a different CI service, ideally integrating with GitHub Checks.
However, it’s not even easy to find CI services with good support for BSDs and other operating systems.
Technically, we would not necessarily need a CI system at all. Just having a machine which runs a cron job to build crystal every night and run some tests would be sufficient for basic coverage. An integrated system would be much nicer though.
cirrus-ci could be a good option. It offers runner images for FreeBSD, and it should be possible to use custom VMs for other OSes. This doesn’t seem to be very well-charted territory though.
A nice feat is that cirrus supports a similar mechanism to what we’re using with runs-on for aarch64-linux: it can provision virtual machines on demand, supporting various cloud providers.
jenkins is a battle-tested CI/CD tool with a broad supports for many platforms. It comes with quite a bit complexity because we’d probably have to host it ourselves.
We have had some effort made towards that a long time ago, before GitHub Actions was a thing: Continuous Integration Infastructure · Issue #3721 · crystal-lang/crystal · GitHub (shoutout to @RX14 
). Maybe picking up from there wouldn’t be too shabby.
Anyway, I’m interested in hearing any other suggestions. Or anything useful at all.
It would be great to expand CI coverage to more platforms. At the moment we expect them to work, but don’t really have any clue about it.
