Clang builds of ASAN+UBSAN on Linux take significantly less time on the
Azure CI runners. Measured times were 82 minutes for Clang 18 and
112 minutes for GCC 13, with no cache.
To keep our coverage of Ladybird builds + testing with GCC, add a
nightly job to run through the full test cycle on GCC 13.
This was useful when building both i686 and x86_64 SerenityOS targets as
we could use a single toolchain build for both targets. But now all this
extra job does is create the opportunity for the toolchain to need to be
built twice (i.e. if the pipelines are backed up and the toolchain cache
is busted between these jobs while the x86_64 step is waiting for a VM).
As of the Clang 13 upgrade, we only need to build the toolchain once and
can use that toolchain for both x86_64 and i686. To do this, this breaks
the main Azure configuration into 3 "stages" (Lagom, Toolchain, and
Serenity), where the Serenity stage depends on the Toolchain stage.
This has the added benefit of uploading a new prebuilt toolchain cache
sooner than before, which should help alleviate pressure from PRs.
The CI system often develops a significant backlog when we have a lot
of PRs in the queue, and folks are pushing to master directly, or
other PRs are getting merged. The individual pushes to master or PR
merges to master each end up creating a dedicated master CI build.
These builds complete for machines with the normal PR validation builds.
To aid with this, Azure DevOps has a feature where they allow the CI
builds to "batch" multiple changes together, instead of running
multiple builds for each change.
Azure DevOps defines batching as:
When a pipeline is running, the system waits until the run is
completed, then starts another run with all changes that have
not yet been built.
Documentation Reference:
https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#batching-ci-runs
