about summary refs log tree commit diff
path: root/users/tazjin
diff options
context:
space:
mode:
authorsterni <sternenseemann@systemli.org>2024-03-10T17·05+0100
committerclbot <clbot@tvl.fyi>2024-04-16T16·16+0000
commit8fed8982b75d658b620447fc02bccab3bfa0cde6 (patch)
tree0c996c54660346c3ebd2476eaa14dbdce1db315e /users/tazjin
parentdce1a7480c283e5298f5f8ecdb1df95f164c7576 (diff)
feat(nix/buildkite): reflect deps between derivations in pipelines r/7940
Most of the steps in our buildkite pipeline build derivations without
doing anything else. A lot of those derivations depend on each other.
Consequently, buildkite will schedule builds of derivations whose
dependencies are still in the process of being built. The result is many
buildkite agents doing nothing but blocking on other derivations being
built. We can easily prevent this by using the dependency information we
can get from the derivation (files) of the targets we want to build and
translating them into buildkite step dependencies.

The hard part of this has already been done for a while:
//nix/dependency-analyzer finds the dependencies between a list of
“known” derivations (even if they only depend on each other through
intermediate derivations) without depending on a specific derivation
builder convention, but rather relying on `.drv` files. It still has a
few rough edges, but has been working reliably for our purposes.

Since our steps are identified by derivation hashes, we can just
directly use the available dependency data. Luckily, buildkite seems to
just takes a step as if it was completed if it is skipped, so we don't
even have to check whether dependencies have been skipped or not.

On whitby it seems that the dependency analysis costs about a minute
additionally (which is how long it takes to run
//nix/dependency-analyzer in isolation just about).

Supersedes cl/5063, cl/5060, cl/5064 and cl/5065.

Change-Id: I91d2eb2b43d60811cac0d26fa94467298f622970
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11116
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Diffstat (limited to 'users/tazjin')
0 files changed, 0 insertions, 0 deletions