feat(buildkite): avoid building extraSteps in pipeline construction r/7656

In principle we don't want to build any (later) pipeline target during
pipeline evaluation insofar they appear in extraSteps. For this reason,
we have the needsOutput mechanism which prevents the parent target of an
extraStep from being built in 🦙.

Unfortunately, this mechanism is not general purpose enough, as we use
other (i.e. non parent) targets from depot in extraSteps. As a
consequence, kind of expensive builds need to happen during pipeline
construction at the moment. The solution is to use the fact that the
command script we want to run is exposed via the readTree interface to
depot and build the script proper only when the extra step is executed.

To facilitate this, some prerequisite changes need to be made:

- We need to use a symlink different to result in case needsOutput is
  true which needs support in mkBuildCommand. We also need to avoid this
  symlink being picked up by git, as many extra steps check whether the
  tree is dirty or not. (Is there a way to have it outside the depot
  tree?)

- Since we rely on the build command printing a single store path we
  store in $command_script, we need to avoid it printing two paths
  in cases where nix-store(1) is used (nix-store(1) prints the symlink
  and readlink(1) would print the store path in a separate line).

Future work would be to remove/deprecate the needsOutput mechanism:
After this change the parent target wouldn't be built right away even if
it appeared in the script via string interpolation. Thus we could,
instead of expecting the target being available as `./result`, make our
extra steps nix-ier.

Change-Id: Idd2e88a865eadabe229ce1e05406e8cc4cb63f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10850
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI

1 files changed, 3 insertions, 0 deletions

diff --git a/.gitignore b/.gitignore
index 9066757e58..8cdaa738f3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -8,3 +8,6 @@ garbage/
 # Ignore Nix result symlinks
 result
 result-*
+# Nix result symlink used by //nix/buildkite
+/nix-buildkite-extra-step-command-script
+/nix-buildkite-extra-step-command-script-*