feat(ops/pipelines/depot): only evaluate once if possible r/2783

We currently evaluate every target twice -- once when the depot pipeline
is built and once when actually running the build step in question. Nix
evaluation is quite slow especially given heavy use of import from
derivation in depot, so avoiding the second evaluation is desireable.

Evaluating a derivation yields a `drv` file in the nix store which can
be passed to `nix-store --realise` in order to build it eliminating the
need to wait for evaluation. We can obtain the path to the `drv` file
while building the pipeline via `target.drvPath` and remember it for the
build later.

However we need to work around a flaw (or oversight) in Nix's dependency
tracking via string context: This is based on derivations, not output
path (because this is what evaluation deals with, likely). This is no
problem per se, but an issue is that Nix can't express a dependency on
a `drv` file without any of its output paths. This means for us that we
either have to build all output paths at evaluation time (which we don't
want, obviously) or to deal with the fact that the `drv` file we need
may be garbage collected at any moment after discarding the string
context -- then nix is unable to track the reference from the pipeline
to the `drv` file in the store.

So to prevent a race condition between the pipeline and the garbage
collector we fall back to the normal nix-build invocation as we did
before.

Change-Id: I9ef8bd233085dc6e30eba54f403ea03ac2d35748
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3426
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>

1 files changed, 15 insertions, 3 deletions

diff --git a/ops/pipelines/depot.nix b/ops/pipelines/depot.nix
index 6415747e6845..02a2c114b951 100644
--- a/ops/pipelines/depot.nix
+++ b/ops/pipelines/depot.nix
@@ -32,9 +32,21 @@ let
   # pipeline as failed. Buildkite has a concept of a failed pipeline
   # regardless, but this data is not accessible.
   mkStep = target: {
-    command = ''
-      nix-build -E '${mkBuildExpr target}' --show-trace || (buildkite-agent meta-data set "failure" "1"; exit 1)
-    '';
+    command = let
+      drvPath = builtins.unsafeDiscardStringContext target.drvPath;
+    in lib.concatStringsSep " " [
+      # First try to realise the drvPath of the target so we don't evaluate twice.
+      # Nix has no concept of depending on a derivation file without depending on
+      # at least one of its `outPath`s, so we need to discard the string context
+      # if we don't want to build everything during pipeline construction.
+      "nix-store --realise '${drvPath}'"
+      # However, Nix doesn't track references of store paths to derivations, so
+      # there's no guarantee that the derivation file is not garbage collected.
+      # To handle this case we fall back to an ordinary build if the derivation
+      # file is missing.
+      "|| (test ! -f '${drvPath}' && nix-build -E '${mkBuildExpr target}' --show-trace)"
+      "|| (buildkite-agent meta-data set 'failure' '1'; exit 1)"
+    ];
     label = ":nix: ${mkLabel target}";
   };