about summary refs log tree commit diff
path: root/tools/nixery
AgeCommit message (Collapse)AuthorFilesLines
2024-02-28 r/7620 refactor(nixery): expose launch script derivationVincent Ambo1-53/+51
Simplifies reusing the launch script in other use-cases than the "official" Nixery image. Relates to nixery#166 Change-Id: Iaf1dff385ce270792253551081c1b2fca6400037 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11046 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-28 r/7619 fix(nixery): use set-default for setting WEB_DIRVincent Ambo1-1/+1
Makes it possible for users to still override this using an envvar. Relates to nixery#166 Change-Id: Ief2925e03cf2e4351bc38554bf553c8ee259f1f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11045 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-22 r/7593 fix(nixery): strictly adhere to OCI image specMarkus Rudy1-2/+2
nixery.dev uses the vnd.docker.container.image.v1 format, which is recognized by the OCI [1] and originally defined by Docker [2]. The config field in this image format, which this commit is about, is even portable between the Docker and OCI formats (the Docker Golang library embeds the OCI definition [3]). The attribute names in what's called ImageConfig in [3] are specified as PascalCase, which effectively means that the names Env and Cmd used by nixery need to be capitalized. The lowercase variant is not causing a lot of issues because most container tooling is written in Golang, which allows case-insensitive matches when deserializing JSON. Languages that parse strictly either miss the configuration values, or fail due to unknown attributes. This commit capitalizes Cmd and Env to accomodate strict parsers. [1]: https://github.com/opencontainers/image-spec/blob/365fa41/media-types.md?plain=1#L70 [2]: https://github.com/moby/moby/blob/v20.10.8/image/spec/v1.2.md#image-json-description [3]: https://github.com/opencontainers/image-spec/blob/365fa41/specs-go/v1/config.go#L24 Change-Id: Ibee597a64d36c008dea83a3b7a0d8e59b8287d0d Signed-off-by: Markus Rudy <webmaster@burgerdev.de> Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11012 Autosubmit: lukegb <lukegb@tvl.fyi> Reviewed-by: lukegb <lukegb@tvl.fyi> Tested-by: BuildkiteCI
2023-10-30 r/6915 chore(tools/depotfmt): use Go version from buildGoVincent Ambo1-8/+9
This is required because Go 1.18 is actually being deleted. I've applied the formatting breakage that it introduces (such as breaking comment formatting), because I can't be bothered to try and work around broken Go stuff. Change-Id: Ica7cee0d01228845d6a766079fef36df99a3da96 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9832 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-19 r/6617 docs(nixery): occurrences typoguangwu1-1/+1
Change-Id: I3798e1c23d6b0580b99b14bb4aae1c7cfc81fb6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9366 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-04-29 r/6118 fix(nixery): allow references to packages starting with numbersVincent Ambo1-2/+15
These packages are invalid in Nix, and worked around in nixpkgs with underscores, but the underscores are invalid in the Docker registry protocol. We work around this by detecting this case and adding the underscore to yield the correct package reference. There is no case where this workaround can break something, as there can be no valid package matching the regular expression. This relates to https://github.com/tazjin/nixery/issues/158 Change-Id: I7990cdb534a8e86c2ceee2c589a2636af70a4a03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8531 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de>
2023-02-06 r/5838 fix(nixery): fix link to nixery logoVincent Ambo1-1/+1
Change-Id: Ib78659b971696feaff579bc0a31df7d8ee24e459 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8034 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-10-12 r/5116 docs(nixery): replace the Nixery mdBook with a simple web pageVincent Ambo15-569/+171
Nixery's previous landing page was an mdBook that was basically unmaintained and full of incorrect information. It also duplicated some things (like nix-1p) which actually live elsewhere. This commit removes the mdBook completely and reduces it down to a simple TVL-style landing page. The landing page has been checked in in its entirety because Nixery is frequently cloned through josh without the entirety of depot, however the page has been created by building it through depot's //web/tvl/template. See also https://github.com/tazjin/nixery/issues/156 Change-Id: I20e1d58f1e6608377207e80345c169f7d92d3847 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6930 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-10-02 r/5013 chore(nixery): use `ldflags` parameter instead of `buildFlagsArray`Vincent Ambo1-2/+5
The latter has been deprecated in nixpkgs. Relates to b/200 Change-Id: I42871ce3eb54ebf092909f033b43936b9610d982 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6836 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-09-28 r/4980 chore(3p/sources): Bump channels & overlayssterni1-1/+1
Upstream nixpkgs removed a lot of aliases this time, so we needed to do the following transformations. It's a real shame that aliases only really become discoverable easily when they are removed. * runCommandNoCC -> runCommand * gmailieer -> lieer We also need to work around the fact that home-manager hasn't catched on to this rename. * mysql -> mariadb * pkgconfig -> pkg-config This also affects our Nix fork which needs to be bumped. * prometheus_client -> prometheus-client * rxvt_unicode -> rxvt-unicode-unwrapped * nix-review -> nixpkgs-review * oauth2_proxy -> oauth2-proxy Additionally, some Go-related builders decided to drop support for passing the sha256 hash in directly, so we need to use the generic hash arguments. Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-09-19 r/4926 fix(nixery): Set correct depot ref when fetching nix-1ptalyz1-1/+4
Change-Id: Iffa49a4e8fd38d0762ed1f60bf72b9a050594a3c Reviewed-on: https://cl.tvl.fyi/c/depot/+/6697 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-09-19 r/4925 fix(nixery): Discard string context before parsing with fromJSONtalyz1-3/+3
Discard string context in prepare-image.nix before parsing input read with readFile with fromJSON. Required for compatibility with nix >2.3. Change-Id: I3830707e80fd19a700551a15f1a96d2841d0b022 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6696 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-09-19 r/4924 fix(nixery): Avoid race when the same image is fetched in paralleltalyz5-47/+60
Remove a race condition which appears when uploadHashLayer is called with the same key from multiple threads simultaneously. This can easily happen when the same image path is requested by multiple clients at the same time. When it does, a 500 status is returned and the following error message is logged: { "context": { "filePath": "github.com/google/nixery/builder/builder.go", "lineNumber": 440, "functionName": "github.com/google/nixery/builder.uploadHashLayer" }, "error": "rename /var/lib/nixery/staging/<hash> /var/lib/nixery/layers/<hash>: no such file or directory", "eventTime": "...", "layer": "<hash>", "message": "failed to move layer from staging", ... } To solve this issue, introduce a mutex keyed on the uploaded hash and move all layer caching into uploadHashLayer. This could additionally provide a small performance benefit when an already built image is requested and NIXERY_PKGS_PATH is set, since symlink layers and config layers are now also cached. Change-Id: I50788a7ec7940cb5e5760f244692e361019a9bb7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6695 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-06-16 r/4242 chore(nixery): use nix-1p from within the depotVincent Ambo2-9/+6
Since the source of nix-1p is checked in under //nix/nix-1p, we should use it from there if Nixery is being built inside of depot. Change-Id: Iddd54f7b93b398b2f909db6ee105366a9914a2ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/5882 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2022-06-16 r/4241 docs(nixery): dynamically display current nixpkgs commitVincent Ambo3-13/+25
People occasionally ask what the current nixpkgs commit is on nixery.dev (see e.g. https://github.com/tazjin/nixery/issues/153). With this change, the commit is displayed on nixery.dev if Nixery is built for the TVL deployment. Change-Id: I795220214db5a367a126c9b4bd03754e9f144940 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5881 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2022-05-26 r/4123 chore(nixery): Bump Go dependenciesVincent Ambo3-69/+106
Change-Id: Id6ff48d66368732cba0b8af6e1cbab64b0f2afbf Reviewed-on: https://cl.tvl.fyi/c/depot/+/5671 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-26 r/4121 feat(nixery): Automatically mirror subtree to GithubVincent Ambo1-0/+8
This exports the `:/tools/nixery` subtree to Github automatically after merges to `canon`. Due to the way the project was imported this continues the existing git history in the external repository. Change-Id: Ie871c14ad5d8f1019f8be86adecbe9b130ffb01a Reviewed-on: https://cl.tvl.fyi/c/depot/+/5667 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-23 r/4106 refactor(nixery): Modernise structure of binariesVincent Ambo2-28/+25
Nixery is going to gain a new binary (used for building images without a registry server); to prepare for this the server binary has moved to cmd/server and the Nix build logic has been updated to wrap this binary and set the required environment variables. Change-Id: I9b4f49f47872ae76430463e2fcb8f68114070f72 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5603 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-23 r/4105 refactor(nixery): Extract layering logic into separate packageVincent Ambo4-21/+25
This will be required for making a standalone, Nixery-style image builder function usable from Nix. Change-Id: I5e36348bd4c32d249d56f6628cd046916691319f Reviewed-on: https://cl.tvl.fyi/c/depot/+/5601 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-04-21 r/3987 fix(nixery): Avoid impure reading of .git directoryVincent Ambo1-5/+3
Change-Id: I67405f9c9bd9cc8cb34fafff80e30b2fca53a2b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5502 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su>
2022-04-20 r/3982 chore(nixery): Housekeeping for depot compatibilityVincent Ambo27-455/+185
Cleans up a whole bunch of things I wanted to get out of the door right away: * depot internal references to //third_party/nixery have been replaced with //tools/nixery * cleaned up files from Github * fixed SPDX & Copyright headers * code formatting and inclusion in //tools/depotfmt checks Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su>
2022-04-20 r/3979 refactor(nixery): Adapt Nix build instructions for readTreeVincent Ambo2-7/+11
This does not fully change the build structure of Nixery to be depot-compatible yet, but should allow most targets to be built in depot CI. This contains some hacks to work around surface incompatibilities which we'll clear away later. Change-Id: I84e7734334abbe299983956f528c0897f49fa8c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5485 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-20 r/3978 feat(tools/nixery): Absorb Nixery into depotVincent Ambo45-0/+5063
This absorbs a josh-filtered Nix subtree into depot, at //tools/nixery. This subtree was created through `josh-filter ':prefix=tools/nixery'`, which allows a filter on tools/nixery to yield the same commit hashes as the original Nixery repository (allowing for history continuity). Change-Id: Icc1a99bf1248226b91f437b0a90361d36fb0d327
2022-04-20 docs: change references to repo URLRaphael Borun Das Gupta3-5/+5
The Nixery main Git repo has moved from https://github.com/google/nixery to https://github.com/tazjin/nixery . So change it in README and on the https://nixery.dev/ website.
2021-12-27 feat: set SSL_CERT_FILE and provide a CmdJérôme Petazzoni2-7/+20
Two minor "quality of life" improvements: - automatically set SSL_CERT_FILE environment variable, so that programs relying on OpenSSL for certificate validation can actually validate certificates (the certificates are included no matter what since we add the "cacert" package to all iamges) - if the requested image includes an interactive shell (e.g. if it includes the "shell" metapackage), set the image Cmd to "bash", which allows to execute "docker run nixery.dev/shell" and get a shell) I'm happy to split this PR in two if you'd like, but since both features touch the Config structure and are rather small, I thought it would make sense to bundle them together.
2021-12-24 feat: add /tmpJérôme Petazzoni1-5/+7
Examples of programs that fail when /tmp doesn't exist: - terraform - anything using mktemp and similar helpers
2021-12-24 docs: mention arm64 metapackageEthan Davidson1-2/+4
2021-12-24 chore: fix env var name in error messageJérôme Petazzoni1-1/+1
The error message shows the wrong variable name, which might be confusing for new users.
2021-12-24 docs: update installation instructionsJérôme Petazzoni1-12/+60
These instructions were not up-to-date (they didn't mention the different storage backends, and some variables were tagged as optional while they were mandatory). With this update, they should (hopefully) be more accurate! :) I also added instructions if someone wants to run Nixery outside of the container image (I found it convenient when working on Nixery's code).
2021-10-29 chore: Bump all Go dependenciesVincent Ambo3-36/+169
Result of 'go get -u && go mod tidy'
2021-10-29 chore: Bump nixpkgs pin to nixos-unstable 2021-10-29Vincent Ambo1-2/+2
2021-10-29 chore(docs): Bump included nix-1p versionVincent Ambo1-2/+2
... basically never updated this, oops.
2021-10-29 docs: Remove note about unsupported Google projectsVincent Ambo1-2/+0
I no longer work at Google and the repo has moved, so this is no longer relevant.
2021-10-08 revert: "feat(storage): Add generic support for content-types"Jérôme Petazzoni1-10/+0
This reverts commit 7db252f36a68d875429a25e06d88fbfc804d84fd. Superseded by the implementation in #127.
2021-08-25 feat(prepare-image): Ensure /usr/bin/env is always presentVincent Ambo1-0/+12
This is required by common patterns in shell scripts. There are some caveats around this. Adding logic to filter whether coreutils is included in an image would slow down the Nix evaluation, so the link is currently created even in cases where it doesn't point to anything. Fixes #109
2021-08-06 chore(build): Allow passing in a specific commit hash when buildingVincent Ambo1-2/+3
Required for builds where the full repository isn't available (e.g. from a tarball).
2021-07-15 docs: Update build badge in READMEVincent Ambo1-1/+1
Moves the build badge to point at Github Actions, instead of the old (failing) Travis build
2021-06-26 feat(storage): Store blob content-type in extended attributesJérôme Petazzoni4-3/+20
After the discussion in #116, this stores the blob content types in extended attributes when using the filesystem backend. If the underlying filesystem doesn't support extended attributes, storing blobs won't work; also, if extended attributes get removed, blobs won't be served anymore. We can relax this behavior if needed (i.e. log errors but still accept to store or serve blobs). However, since the Docker Engine (and possibly other container engines) won't accept to pull images from a registry that doesn't use correct content types for manifest files, it could be argued that it's better to give a hard fail. (Otherwise, the container engine gives cryptic error messages like "missing signature key".) I can change that behavior (and log errors but still store/serve blobs to the filesystem) if you think it's better.
2021-06-20 feat(ci): don't mount /var/cache/nixery from tmpfs into docker containerFlorian Klink2-2/+13
With https://github.com/google/nixery/pull/127, nixery will use extended attributes to store metadata (when using local storage). Right now, our integration test mounts a tmpfs to /var/cache/nixery. However, *user* xattrs aren't supported with tmpfs [1], so setting xattrs would fail. To workaround this, use a folder in the current working directory and hope it's backed by something supporting user xattrs (which is the case for GitHub Actions). [1]: https://man7.org/linux/man-pages/man5/tmpfs.5.html#NOTES
2021-04-30 feat(build): Run `go vet` as a step in the GitHub Actions workflowVincent Ambo2-1/+3
2021-04-30 refactor(build): Pin dependencies using Go modulesVincent Ambo5-164/+553
Drops the go2nix configuration in favour of pkgs.buildGoModule. Note that the go.sum file is bloated by issues with cyclic dependencies in some Google projects, but this large number of dependencies is not actually built.
2021-04-30 chore(build): Use current git commit hash as build versionVincent Ambo1-6/+4
2021-04-30 chore: Update default NixOS channel to nixos-20.09Vincent Ambo6-8/+8
2021-04-30 chore(ci): Remove unnecessary commands from new CI setupVincent Ambo1-7/+2
* remove a step that was not supposed to be committed ("Do we have Docker?") * remove setup of old temporary storage directory (now done in integration script test instead) * skip creation of out-link for initial Nixery build (to avoid cache-busting on the second build)
2021-04-29 docs: document unset GOOGLE_APPLICATION_CREDENTIALSFlorian Klink2-0/+8
In case the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is not set, a redirect to storage.googleapis.com is issued, which means the underlying bucket objects need to be publicly accessible. This wasn't really obvious until now, so further clarify it.
2021-04-29 feat(ci): add integration tests to GitHub Actions, remove .travis.yamlFlorian Klink3-78/+53
This copies the integration tests from `.travis.yaml` into a script, documents the assumptions it makes, and wires it into GitHub Actions. Contrary to the travis version, we don't use Nixery's GCS backend, as handing out access to the bucket used, especially for PRs, needs to be done carefully. Adding back GCS to the integration test can be done at a later point, either by using a mock server, or by only exposing the credentials for master builds (and have the test script decide on whether GOOGLE_APPLICATION_CREDENTIALS is set or not). The previous travis version had some complicated post-mortem log gathering - instead of doing this, we can just `docker run` nixery, but fork it into the background with the shell - causing it to still be able to log its output as it's running. An additional `--rm` is appended, so the container gets cleaned up on termination - this allows subsequent runs on non-CI infrastructure (like developer laptops), without having to manually clean up containers. Fixes #119.
2021-04-29 feat(ci): remove unneeded permissions: read-allFlorian Klink1-1/+0
We don't intend to label, authenticate or whatever with the GITHUB_TOKEN, so there's not really a reason to give any broader permissions than the defaults.
2021-04-28 feat(ci): Configure initial GitHub Actions setupVincent Ambo1-0/+29
Travis is being deprecated, and this might be the best option for now.
2021-04-27 feat(storage): Add generic support for content-typesJerome Petazzoni1-0/+10
When serving a manifest, it is important to set the content-type correctly (otherwise pulling an image is likely to give a cryptic error message, "Error response from daemon: missing signature key"). This makes sure that we set the content-type properly for both manifests and layers.
2021-04-14 chore(nix): update channel URLJerome Petazzoni1-1/+1
It looks like NixPkgs channels have moved. Fixing this URL allows using nixos-20.09, for instance.
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-02T04·34+0000