about summary refs log tree commit diff
path: root/nix/buildkite/default.nix (follow)
AgeCommit message (Collapse)AuthorFilesLines
2022-02-02 r/3743 feat(nix/buildkite): Add parentOverride argument to extra stepsVincent Ambo1-9/+23
This can be used to override the parent derivation if its output is required, for example to inject versions which are only used during releases to avoid cache-busting. Change-Id: I2211496efa8f9bc98ea43b23e4f3f92c61a6da73 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5184 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-31 r/3723 style: format entire depot with nixpkgs-fmtVincent Ambo1-148/+164
This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-29 r/3701 chore(nix/buildkite): reduce chunk size to 192Vincent Ambo1-1/+1
We've seen the famous 1 minute timeout on Buildkite again, probably due to something (keys in targets?) increasing the overall payload size of our chunks. This reduces the chunk size by 25%. Lets keep an eye on it with this value ... Change-Id: I6bf0e9e4ab0d5b8de22773e6cd5da8d0959cc448 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5105 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-24 r/3668 feat(nix/buildkite): Support un-gated post-build extra stepsVincent Ambo1-2/+7
Allows users to define steps with `postBuild = true` which always run after :duck:, but do not require human approvals. This can be useful for things like unconditional release steps. Change-Id: Idbf6c48a9dedcfc6cc9b7f098423364e2fa72d2d Reviewed-on: https://cl.tvl.fyi/c/depot/+/5052 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 r/3667 refactor(nix/buildkite): Restrict step conditionals to refs onlyVincent Ambo1-11/+9
The previous `condition` abstraction which allowed the full set of Buildkite conditionals is way too leaky (it lets users to very Buildkite-specific things which we may not want to allow, and which are mostly not relevant to a pure evaluation). Supporting only the `branches` condition (native to Buildkite) should make it possible to port this to other future CI systems later. Change-Id: Ib8adcc41db4f1a3566cbeecf13a4228403105c1f Reviewed-on: https://cl.tvl.fyi/c/depot/+/5051 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com> Autosubmit: tazjin <tazjin@tvl.su>
2022-01-24 r/3666 feat(nix/buildkite): Add support for gated stepsVincent Ambo1-7/+69
Adds support for extra build steps that specify a `prompt`. These steps will be run at the end of the pipeline and will be gated by human approval. This mechanism can be used to, for example, stage releases of software released from depot that are subject to approval. Change-Id: I97bb505664a2ccf01142286f14e20a370afaa345 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5033 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com> Autosubmit: tazjin <tazjin@tvl.su>
2022-01-22 r/3657 refactor(ops/pipelines): Split build/post steps into separate chunksVincent Ambo1-16/+9
This will create `build-chunk-$n.json` files for steps that should run _before_ duck, and `post-chunk-$n.json` files for steps that should run after duck. The post steps are not yet uploaded to Buildkite, but we also don't have any right now. Change-Id: I7e1b59cf55a8bf1d97266f6e988aa496959077bf Reviewed-on: https://cl.tvl.fyi/c/depot/+/5047 Tested-by: BuildkiteCI Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com> Autosubmit: tazjin <tazjin@tvl.su>
2022-01-21 r/3652 feat(nix/buildkite): Add support for extra stepsVincent Ambo1-17/+85
This introduces a new feature to our CI system in which targets can declare extra steps in `meta.ci.extraSteps`. See the comment in //nix/buildkite/default.nix for an explanation of how these extra steps are defined. Change-Id: Icce2890c743286dd37f43024cd390dcebac8cdba Reviewed-on: https://cl.tvl.fyi/c/depot/+/5008 Tested-by: BuildkiteCI Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2022-01-17 r/3602 feat(nix/buildkite): Check target map of parent to determine skipsVincent Ambo1-22/+22
This changes the logic for build pipeline generation to inspect an (optional) parentTargetMap attribute which contains the derivation map of a target commit. Targets that existed in a parent commit with the same drv hash will be skipped, as they are not considered to have changed. This does not yet wire up any logic for retrieving the target map from storage, meaning that at this commit all targets are always built. The intention is that we will have logic to fetch the target map (initially from Buildkite artefact storage), which we then pass to the depot via externalArgs when actually generating the pipeline. Change-Id: I3373c60aaf4b56b94c6ab64e2e5eef68dea9287c Reviewed-on: https://cl.tvl.fyi/c/depot/+/4946 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-01-17 r/3601 feat(ops/pipelines): Create drvmap structure for each commitVincent Ambo1-3/+23
Always create a structure that maps all targets to derivations, and persist it as a JSON file. This relates to some of the ideas expressed in: https://docs.google.com/document/d/16A0a5oUxH1VoiSM8hyFyLW0WiUYpNo2e2D6FTW4BlH8/edit The file is always uploaded to Buildkite as an artifact. This allows for retrieving it based on the commit ID in a Buildkite GraphQL query. By default, Buildkite stores artefacts for 6 months. Storage location can be overridden (with custom retention) through some environment variables, but for now at TVL the Buildkite-managed storage is fine. See also: https://buildkite.com/docs/pipelines/artifacts In the subsequent filtering implementation, when diffing commits across a time-range that exceeds artefact retention time, we should simply default to building everything. Change-Id: I6d808461cd1c1fdd6983ba8c8ef075736d42caa7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3662 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-12-28 r/3491 refactor: Generalise pipeline generation in //nix/buildkiteVincent Ambo1-0/+165
Extracts the logic for generating our Buildkite pipeline (which has been copy&pasted and slightly modified in some places outside of depot) into a generic //nix/buildkite library. This should cause no change in functionality. Change-Id: Iad3201713945de41279b39e4f1b847f697c179f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4726 Autosubmit: tazjin <mail@tazj.in> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>