about summary refs log tree commit diff
path: root/tools/nixery
AgeCommit message (Collapse)AuthorFilesLines
2022-04-20 r/3979 refactor(nixery): Adapt Nix build instructions for readTreeVincent Ambo2-7/+11
This does not fully change the build structure of Nixery to be depot-compatible yet, but should allow most targets to be built in depot CI. This contains some hacks to work around surface incompatibilities which we'll clear away later. Change-Id: I84e7734334abbe299983956f528c0897f49fa8c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5485 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-20 r/3978 feat(tools/nixery): Absorb Nixery into depotVincent Ambo45-0/+5063
This absorbs a josh-filtered Nix subtree into depot, at //tools/nixery. This subtree was created through `josh-filter ':prefix=tools/nixery'`, which allows a filter on tools/nixery to yield the same commit hashes as the original Nixery repository (allowing for history continuity). Change-Id: Icc1a99bf1248226b91f437b0a90361d36fb0d327
2022-04-20 docs: change references to repo URLRaphael Borun Das Gupta3-5/+5
The Nixery main Git repo has moved from https://github.com/google/nixery to https://github.com/tazjin/nixery . So change it in README and on the https://nixery.dev/ website.
2021-12-27 feat: set SSL_CERT_FILE and provide a CmdJérôme Petazzoni2-7/+20
Two minor "quality of life" improvements: - automatically set SSL_CERT_FILE environment variable, so that programs relying on OpenSSL for certificate validation can actually validate certificates (the certificates are included no matter what since we add the "cacert" package to all iamges) - if the requested image includes an interactive shell (e.g. if it includes the "shell" metapackage), set the image Cmd to "bash", which allows to execute "docker run nixery.dev/shell" and get a shell) I'm happy to split this PR in two if you'd like, but since both features touch the Config structure and are rather small, I thought it would make sense to bundle them together.
2021-12-24 feat: add /tmpJérôme Petazzoni1-5/+7
Examples of programs that fail when /tmp doesn't exist: - terraform - anything using mktemp and similar helpers
2021-12-24 docs: mention arm64 metapackageEthan Davidson1-2/+4
2021-12-24 chore: fix env var name in error messageJérôme Petazzoni1-1/+1
The error message shows the wrong variable name, which might be confusing for new users.
2021-12-24 docs: update installation instructionsJérôme Petazzoni1-12/+60
These instructions were not up-to-date (they didn't mention the different storage backends, and some variables were tagged as optional while they were mandatory). With this update, they should (hopefully) be more accurate! :) I also added instructions if someone wants to run Nixery outside of the container image (I found it convenient when working on Nixery's code).
2021-10-29 chore: Bump all Go dependenciesVincent Ambo3-36/+169
Result of 'go get -u && go mod tidy'
2021-10-29 chore: Bump nixpkgs pin to nixos-unstable 2021-10-29Vincent Ambo1-2/+2
2021-10-29 chore(docs): Bump included nix-1p versionVincent Ambo1-2/+2
... basically never updated this, oops.
2021-10-29 docs: Remove note about unsupported Google projectsVincent Ambo1-2/+0
I no longer work at Google and the repo has moved, so this is no longer relevant.
2021-10-08 revert: "feat(storage): Add generic support for content-types"Jérôme Petazzoni1-10/+0
This reverts commit 7db252f36a68d875429a25e06d88fbfc804d84fd. Superseded by the implementation in #127.
2021-08-25 feat(prepare-image): Ensure /usr/bin/env is always presentVincent Ambo1-0/+12
This is required by common patterns in shell scripts. There are some caveats around this. Adding logic to filter whether coreutils is included in an image would slow down the Nix evaluation, so the link is currently created even in cases where it doesn't point to anything. Fixes #109
2021-08-06 chore(build): Allow passing in a specific commit hash when buildingVincent Ambo1-2/+3
Required for builds where the full repository isn't available (e.g. from a tarball).
2021-07-15 docs: Update build badge in READMEVincent Ambo1-1/+1
Moves the build badge to point at Github Actions, instead of the old (failing) Travis build
2021-06-26 feat(storage): Store blob content-type in extended attributesJérôme Petazzoni4-3/+20
After the discussion in #116, this stores the blob content types in extended attributes when using the filesystem backend. If the underlying filesystem doesn't support extended attributes, storing blobs won't work; also, if extended attributes get removed, blobs won't be served anymore. We can relax this behavior if needed (i.e. log errors but still accept to store or serve blobs). However, since the Docker Engine (and possibly other container engines) won't accept to pull images from a registry that doesn't use correct content types for manifest files, it could be argued that it's better to give a hard fail. (Otherwise, the container engine gives cryptic error messages like "missing signature key".) I can change that behavior (and log errors but still store/serve blobs to the filesystem) if you think it's better.
2021-06-20 feat(ci): don't mount /var/cache/nixery from tmpfs into docker containerFlorian Klink2-2/+13
With https://github.com/google/nixery/pull/127, nixery will use extended attributes to store metadata (when using local storage). Right now, our integration test mounts a tmpfs to /var/cache/nixery. However, *user* xattrs aren't supported with tmpfs [1], so setting xattrs would fail. To workaround this, use a folder in the current working directory and hope it's backed by something supporting user xattrs (which is the case for GitHub Actions). [1]: https://man7.org/linux/man-pages/man5/tmpfs.5.html#NOTES
2021-04-30 feat(build): Run `go vet` as a step in the GitHub Actions workflowVincent Ambo2-1/+3
2021-04-30 refactor(build): Pin dependencies using Go modulesVincent Ambo5-164/+553
Drops the go2nix configuration in favour of pkgs.buildGoModule. Note that the go.sum file is bloated by issues with cyclic dependencies in some Google projects, but this large number of dependencies is not actually built.
2021-04-30 chore(build): Use current git commit hash as build versionVincent Ambo1-6/+4
2021-04-30 chore: Update default NixOS channel to nixos-20.09Vincent Ambo6-8/+8
2021-04-30 chore(ci): Remove unnecessary commands from new CI setupVincent Ambo1-7/+2
* remove a step that was not supposed to be committed ("Do we have Docker?") * remove setup of old temporary storage directory (now done in integration script test instead) * skip creation of out-link for initial Nixery build (to avoid cache-busting on the second build)
2021-04-29 docs: document unset GOOGLE_APPLICATION_CREDENTIALSFlorian Klink2-0/+8
In case the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is not set, a redirect to storage.googleapis.com is issued, which means the underlying bucket objects need to be publicly accessible. This wasn't really obvious until now, so further clarify it.
2021-04-29 feat(ci): add integration tests to GitHub Actions, remove .travis.yamlFlorian Klink3-78/+53
This copies the integration tests from `.travis.yaml` into a script, documents the assumptions it makes, and wires it into GitHub Actions. Contrary to the travis version, we don't use Nixery's GCS backend, as handing out access to the bucket used, especially for PRs, needs to be done carefully. Adding back GCS to the integration test can be done at a later point, either by using a mock server, or by only exposing the credentials for master builds (and have the test script decide on whether GOOGLE_APPLICATION_CREDENTIALS is set or not). The previous travis version had some complicated post-mortem log gathering - instead of doing this, we can just `docker run` nixery, but fork it into the background with the shell - causing it to still be able to log its output as it's running. An additional `--rm` is appended, so the container gets cleaned up on termination - this allows subsequent runs on non-CI infrastructure (like developer laptops), without having to manually clean up containers. Fixes #119.
2021-04-29 feat(ci): remove unneeded permissions: read-allFlorian Klink1-1/+0
We don't intend to label, authenticate or whatever with the GITHUB_TOKEN, so there's not really a reason to give any broader permissions than the defaults.
2021-04-28 feat(ci): Configure initial GitHub Actions setupVincent Ambo1-0/+29
Travis is being deprecated, and this might be the best option for now.
2021-04-27 feat(storage): Add generic support for content-typesJerome Petazzoni1-0/+10
When serving a manifest, it is important to set the content-type correctly (otherwise pulling an image is likely to give a cryptic error message, "Error response from daemon: missing signature key"). This makes sure that we set the content-type properly for both manifests and layers.
2021-04-14 chore(nix): update channel URLJerome Petazzoni1-1/+1
It looks like NixPkgs channels have moved. Fixing this URL allows using nixos-20.09, for instance.
2020-12-05 docs: Update README with a link to the NixCon talkVincent Ambo1-4/+10
2020-12-04 docs(config): Fix comment typoDave Nicponski1-1/+1
2020-10-29 feat(storage): Add support for content-types (GCS only)Vincent Ambo7-13/+34
Extends storage.Persist to accept a Content-Type argument, which in the GCS backend is persisted with the object to ensure that the object is served back with this content-type. This is not yet implemented for the filesystem backend, where the parameter is simply ignored. This should help in the case of clients which expect the returned objects to have content-types set when, for example, fetching layers by digest.
2020-10-27 docs: Add a note about a Nix-native builder to the roadmapVincent Ambo1-0/+6
... if I don't mention this somewhere I'll probably never do it!
2020-10-27 feat(main): Implement caching of manifests in CASVincent Ambo1-0/+35
To ensure that registry clients which attempt to pull manifests by their content hash can interact with Nixery, this change implements persisting image manifests in the CAS in the same way as image layers. In combination with the previous refactorings this means that Nixery's serving flow is now compatible with containerd. I have verified this locally, but CI currently only runs against Docker and not containerd, which is something I plan to address in a subsequent PR. This fixes #102
2020-10-27 feat(main): Implement serving of manifests by digestVincent Ambo1-11/+12
Modifies the layer serving endpoint to be a generic blob-serving endpoint that can handle both manifest and layer object "types". Note that this commit does not yet populate the CAS with any manifests.
2020-10-27 refactor(storage): Rename ServeLayer -> ServeVincent Ambo4-12/+12
This is going to be used for general content-addressed objects, and is not layer specific anymore.
2020-10-27 refactor(main): Split HTTP handlers into separate functionsVincent Ambo2-57/+62
There is a new handler coming up to fix #102 and I want to avoid falling into the classic Go trap of creating thousand-line functions.
2020-10-27 fix(build): Work around arbitrary new maxLayers restrictionVincent Ambo1-1/+1
2020-10-27 fix(build): Completely remove Cachix from build setupVincent Ambo1-2/+2
Installing Cachix started failing on ARM64.
2020-07-25 fix(build): Don't use Cachix as the binary cache during buildsVincent Ambo1-1/+0
Permission changes in the Travis CI Nix builders have caused this to start failing, as the build user now has insufficient permissions to use caches. There may be a way to change the permissions instead, but in the meantime we will just cause things to rebuild.
2020-07-25 chore(build): Update pinned Go dependenciesVincent Ambo1-41/+50
2020-07-25 fix(popcount): Accommodate upstream changes on nixos.orgVincent Ambo1-2/+9
Channel serving has moved to a new subdomain, and the redirect semantics have changed. Instead of serving temporary redirects, permanent redirects are now issued. I've reported this upstream as a bug, but this workaround will fix it in the meantime.
2020-07-25 chore(build): Change pin for default nixpkgs used to build NixeryVincent Ambo3-2/+6
This moves the pin from just being in the Travis configuration to also being set in a nixpkgs-pin.nix file, which makes it trivial to build at the right commit when performing local builds.
2020-05-01 chore(nix): update channel 19.03 -> 20.03Raphael Borun Das Gupta4-5/+5
Use a NixOS / NixPkgs release that's actually being supported and regularly updated.
2020-02-26 fix(popcount): Fix nix-build -A nixery-popcountFlorian Klink1-8/+6
Previously, this was failing as follows: ``` these derivations will be built: /nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv building '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv'... building warning: GOPATH set to GOROOT (/nix/store/4859cp1v7zqcqh43jkqsayl4wrz3g6hp-go-1.13.4/share/go) has no effect failed to initialize build cache at /homeless-shelter/.cache/go-build: mkdir /homeless-shelter: permission denied builder for '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed with exit code 1 error: build of '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed ```
2020-01-19 fix(builder): Fix minor logging switcharooVincent Ambo1-1/+1
2019-11-27 refactor: Reshuffle file structure for better code layoutVincent Ambo21-114/+83
This gets rid of the package called "server" and instead moves everything into the project root, such that Go actually builds us a binary called `nixery`. This is the first step towards factoring out CLI-based functionality for Nixery.
2019-11-27 fix(builder): Ensure "solo-metapackages" do not break buildsVincent Ambo1-1/+7
The previous logic failed because single meta-packages such as "nixery.dev/shell" would not end up removing the meta-package itself from the list of packages passed to Nix, causing a build failure. This was a regression introduced in 827468a.
2019-11-27 test(builder): Add test coverage for name->image conversionVincent Ambo1-0/+123
Adds tests to cover that packages & metapackages are parsed into image names correctly.
2019-11-09 chore(build): Use significantly fewer layers for Nixery itselfVincent Ambo2-3/+5
Nixery itself is built with the buildLayeredImage system, which takes some time to create large numbers of layers. This adjusts the default number of image layers from 96 to 20. Additionally Nixery's image is often loaded with `docker load -i`, which ignores layer cache hits anyways. Additionaly the CI build is configured to use only 1, which speeds up CI runs.
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-28T05·46+0000