about summary refs log tree commit diff
diff options
context:
space:
mode:
authorVincent Ambo <mail@tazj.in>2021-08-26T15·02+0300
committertazjin <mail@tazj.in>2021-08-26T16·29+0000
commitd5ddfb7b96835828b4d2c38006e58f9303ef672e (patch)
tree81b7d28291b1a1a41e1a5657816d3cf655a01357
parent094dfa2ed3ac39354cfe039f3bfa2f89169b9ed3 (diff)
feat(pipelines/depot): Skip build steps if their out paths exist r/2786
Skip build steps if they have already been built, reducing pipelines
to the things that actually changed between builds. On canon all
targets are always built (we require this for anchoring).

Note that this is not perfect, garbage collection and competing
pipelines may affect each other.

Also note that we have some impure targets that change on every
commit.

Change-Id: Ic6bae3b6c8e1e7fd2116ec252f5089f471854ab6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3427
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
-rw-r--r--ops/pipelines/depot.nix12
1 files changed, 12 insertions, 0 deletions
diff --git a/ops/pipelines/depot.nix b/ops/pipelines/depot.nix
index 02a2c114b951..961d8e6ecec8 100644
--- a/ops/pipelines/depot.nix
+++ b/ops/pipelines/depot.nix
@@ -48,6 +48,18 @@ let
       "|| (buildkite-agent meta-data set 'failure' '1'; exit 1)"
     ];
     label = ":nix: ${mkLabel target}";
+
+    # Skip build steps if their out path has already been built.
+    skip = let
+      shouldSkip = with builtins;
+        # Only skip in real Buildkite builds
+        (getEnv "BUILDKITE_BUILD_ID" != "") &&
+        # Always build everything for the canon branch.
+        (getEnv "BUILDKITE_BRANCH" != "canon") &&
+        # Discard string context to avoid realising the store path during
+        # pipeline construction.
+        (pathExists (unsafeDiscardStringContext target.outPath));
+      in if shouldSkip then "Target was already built." else false;
   };
 
   # Protobuf check step which validates that changes to .proto files