about summary refs log tree commit diff
path: root/ops/pipelines
AgeCommit message (Collapse)AuthorFilesLines
2021-08-26 r/2786 feat(pipelines/depot): Skip build steps if their out paths existVincent Ambo1-0/+12
Skip build steps if they have already been built, reducing pipelines to the things that actually changed between builds. On canon all targets are always built (we require this for anchoring). Note that this is not perfect, garbage collection and competing pipelines may affect each other. Also note that we have some impure targets that change on every commit. Change-Id: Ic6bae3b6c8e1e7fd2116ec252f5089f471854ab6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3427 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: grfn <grfn@gws.fyi>
2021-08-26 r/2783 feat(ops/pipelines/depot): only evaluate once if possiblesterni1-3/+15
We currently evaluate every target twice -- once when the depot pipeline is built and once when actually running the build step in question. Nix evaluation is quite slow especially given heavy use of import from derivation in depot, so avoiding the second evaluation is desireable. Evaluating a derivation yields a `drv` file in the nix store which can be passed to `nix-store --realise` in order to build it eliminating the need to wait for evaluation. We can obtain the path to the `drv` file while building the pipeline via `target.drvPath` and remember it for the build later. However we need to work around a flaw (or oversight) in Nix's dependency tracking via string context: This is based on derivations, not output path (because this is what evaluation deals with, likely). This is no problem per se, but an issue is that Nix can't express a dependency on a `drv` file without any of its output paths. This means for us that we either have to build all output paths at evaluation time (which we don't want, obviously) or to deal with the fact that the `drv` file we need may be garbage collected at any moment after discarding the string context -- then nix is unable to track the reference from the pipeline to the `drv` file in the store. So to prevent a race condition between the pipeline and the garbage collector we fall back to the normal nix-build invocation as we did before. Change-Id: I9ef8bd233085dc6e30eba54f403ea03ac2d35748 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3426 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-04-12 r/2492 feat(ops/pipelines): pass --show-trace to nix-buildsterni1-1/+1
--show-trace should make it easier to debug tricky evaluation errors without running nix-build -A ops.pipelines.depot locally again. Change-Id: Ice540562c3b389fc2a49ec1fc0adacb17db2a528 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2947 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-04-11 r/2480 fix(pipelines/depot): Buildkite refers to branches by full refVincent Ambo1-1/+1
This change is required to run the :anchor: step on canon builds. Change-Id: Ib3cebac67c9f5337b27a948f120b0a9ba834ef2a Reviewed-on: https://cl.tvl.fyi/c/depot/+/2932 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-11 r/2477 feat(ops/pipelines): Add gcroots for depot builds on canonVincent Ambo1-2/+22
Adds a conditional build step that only runs on the canon branch, and only if :duck: (the status reporting step) succeeds, which creates a new Nix GC root for all depot targets named `depot-canon`. In practice this might be a bit racey, as canon builds are not guaranteed to succeed in order (though it is likely). This shouldn't matter much in practice: We only want to prevent rebuilds of the whole world. This fixes b/102 Change-Id: Id3d0bf4158bffcb1ed6929888a29d31609b6ece1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2904 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2021-01-30 r/2160 fix(ops/piplines/static-pipeline): add --show-trace to nix-buildProfpatsch1-1/+1
Change-Id: Ib0473f916b1436934844e620ce981f52d11e8512 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2467 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-11-17 r/1882 feat(ops/pipelines): Check in the static pipelineVincent Ambo1-0/+15
This file represents the static pipeline which is configured in the Buildkite web UI. Updates to this file should be applied in the admin interface. These steps are responsible for launching the dynamic pipeline evaluation, or falling back to the fallback pipeline if evaluation fails. Change-Id: I6d7dd623cde65e8c69faea729f737c9bba00c2fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/2103 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-17 r/1881 feat(ops/pipelines): Add a fallback Buildkite configurationVincent Ambo1-0/+8
This adds a simple fallback Buildkite pipeline configuration which always fails the pipeline, but correctly reports back the failure status. Note that this also requires changes in the Buildkite configuration that is not in version-control. Relates to b/66. Change-Id: I6802a6f76448c3893798a06d514e6ccba0f50dd2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2102 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-31 r/1748 feat(ci): Add subtarget support for buildsVincent Ambo1-6/+13
We have naturally evolved a distinction between logical and physical targets. Physical targets are those which correspond directly to a tree location on disk and can be built with `-A path.to.files`, while logical targets are those that are exported from within an expression but do not have a corresponding file on disk. This change adds support for exporting logical targets from any tree location by adding a `meta.targets` attribute containing keys into itself, which will be consumed by the CI target gathering logic and included in the generated pipeline. Note that the labels for subtargets are syntactically different to emphasise that they do not correspond to a file location. For example, this change enables 'ops.nixos.whitbySystem' as a subtarget, which is labeled in CI as `ops/nixos:whitbySystem`. Change-Id: Ied09647a62c2ba98e3914548e3742ad422c63ecf Reviewed-on: https://cl.tvl.fyi/c/depot/+/1893 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-31 r/1747 feat(ops/pipelines): Dynamically generate CI pipeline from targetsVincent Ambo1-13/+66
Create the pipeline by outputting a file that contains nix-build invocations for each target's *derivation path*. Each invocation has a generated Nix expression passed to it with `-E` which fetches the correct target from the tree while correctly handling targets with strange characters (such as in Go-packages). This makes it possible to run target-level granular pipelines. We're getting somewhere! Change-Id: Ia6946e389dafd1d4926130bb8891446d6e17133b Reviewed-on: https://cl.tvl.fyi/c/depot/+/1855 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi> Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-26 r/1725 feat: Implement automatic CI target detection for the depotVincent Ambo1-1/+1
Automatically walk the entire depot tree and pick out things that are "buildable", then include them in the attribute `ci.targets` (which is now also the target for CI builds). A long time ago, in a land far away, we (well, I, at the time) had a prototype of this which ran into constant issues with infinite recursions while trying to walk the tree. In fact, this is why readTree originally gained the `__readTree`-attribute which marks things that were imported automatically. Based on some code edef whipped up earlier (with the breakthrough being that we also add the attribute to top-level folders, which suddenly resolves a whole bunch of problems), I've now implemented this actually working version. At the moment all builds still happen as one big bag of builds, but at some point we will granularise this. Change-Id: I86f12ce7f63dae98e7e5c6646a4e9d220de783f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1854 Tested-by: BuildkiteCI Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-17 r/1669 docs: Update README for the repository itselfVincent Ambo1-3/+5
Adds a lot more information about what is actually going on here, as well as links to relevant things such as our Monorepo document. I'm aware that I didn't make all `//...` links clickable, but I realised at some point that it might be easier to just update cheddar to support highlighting those natively :-) Change-Id: Icbb212a6c07a5cf38ab8e65d83a64bec783eb8d0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1768 Tested-by: BuildkiteCI Reviewed-by: isomer <isomer@tvl.fyi> Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-17 r/1366 feat(ci): run buf check lint in CIKane York1-0/+4
Breaking change detection will run but not enforce. Emoji of water buffalo was chosen by @pedge fiat in the bufbuild slack. Change-Id: Ie292f2bfddc0e3bc512e4a138c0b5d0fa2603bad Reviewed-on: https://cl.tvl.fyi/c/depot/+/1247 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-29 r/1117 feat(pipelines/depot): Run with --show-traceGriffin Smith1-1/+1
So if an evaluation fails we get a stacktrace Change-Id: I54cdc9e93c765ef7cf3a4d0cd79e6d067f4789d3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/733
2020-06-28 r/1106 feat(besadii): Enable automatic builds for CLsVincent Ambo1-1/+1
This expands builds to also be triggered for updates to CL refs. The message displayed on Buildkite will contain a link back to the CL (& patchset) from which the build was triggered. Change-Id: Ib36dee454aeb11d623b89c78b384359ee7ea3477 Reviewed-on: https://cl.tvl.fyi/c/depot/+/708 Reviewed-by: ericvolp12 <ericvolp12@gmail.com> Reviewed-by: isomer <isomer@tvl.fyi>
2020-06-27 r/1099 feat(ops/pipelines): Add Buildkite pipeline configurationVincent Ambo2-0/+24
This adds configuration which generates the structure expected for Buildkite pipelines, which can then be dynamically ingested by Buildkite when a pipeline is triggered. Change-Id: I61e3dc3affb19c1f2550ef827fa73b17f8d8ae47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/627 Reviewed-by: ericvolp12 <ericvolp12@gmail.com> Reviewed-by: lukegb <lukegb@tvl.fyi>