about summary refs log tree commit diff
path: root/tools
AgeCommit message (Collapse)AuthorFilesLines
2021-06-20 feat(ci): don't mount /var/cache/nixery from tmpfs into docker containerFlorian Klink2-2/+13
With https://github.com/google/nixery/pull/127, nixery will use extended attributes to store metadata (when using local storage). Right now, our integration test mounts a tmpfs to /var/cache/nixery. However, *user* xattrs aren't supported with tmpfs [1], so setting xattrs would fail. To workaround this, use a folder in the current working directory and hope it's backed by something supporting user xattrs (which is the case for GitHub Actions). [1]: https://man7.org/linux/man-pages/man5/tmpfs.5.html#NOTES
2021-04-30 feat(build): Run `go vet` as a step in the GitHub Actions workflowVincent Ambo2-1/+3
2021-04-30 refactor(build): Pin dependencies using Go modulesVincent Ambo5-164/+553
Drops the go2nix configuration in favour of pkgs.buildGoModule. Note that the go.sum file is bloated by issues with cyclic dependencies in some Google projects, but this large number of dependencies is not actually built.
2021-04-30 chore(build): Use current git commit hash as build versionVincent Ambo1-6/+4
2021-04-30 chore: Update default NixOS channel to nixos-20.09Vincent Ambo6-8/+8
2021-04-30 chore(ci): Remove unnecessary commands from new CI setupVincent Ambo1-7/+2
* remove a step that was not supposed to be committed ("Do we have Docker?") * remove setup of old temporary storage directory (now done in integration script test instead) * skip creation of out-link for initial Nixery build (to avoid cache-busting on the second build)
2021-04-29 docs: document unset GOOGLE_APPLICATION_CREDENTIALSFlorian Klink2-0/+8
In case the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is not set, a redirect to storage.googleapis.com is issued, which means the underlying bucket objects need to be publicly accessible. This wasn't really obvious until now, so further clarify it.
2021-04-29 feat(ci): add integration tests to GitHub Actions, remove .travis.yamlFlorian Klink3-78/+53
This copies the integration tests from `.travis.yaml` into a script, documents the assumptions it makes, and wires it into GitHub Actions. Contrary to the travis version, we don't use Nixery's GCS backend, as handing out access to the bucket used, especially for PRs, needs to be done carefully. Adding back GCS to the integration test can be done at a later point, either by using a mock server, or by only exposing the credentials for master builds (and have the test script decide on whether GOOGLE_APPLICATION_CREDENTIALS is set or not). The previous travis version had some complicated post-mortem log gathering - instead of doing this, we can just `docker run` nixery, but fork it into the background with the shell - causing it to still be able to log its output as it's running. An additional `--rm` is appended, so the container gets cleaned up on termination - this allows subsequent runs on non-CI infrastructure (like developer laptops), without having to manually clean up containers. Fixes #119.
2021-04-29 feat(ci): remove unneeded permissions: read-allFlorian Klink1-1/+0
We don't intend to label, authenticate or whatever with the GITHUB_TOKEN, so there's not really a reason to give any broader permissions than the defaults.
2021-04-28 feat(ci): Configure initial GitHub Actions setupVincent Ambo1-0/+29
Travis is being deprecated, and this might be the best option for now.
2021-04-27 feat(storage): Add generic support for content-typesJerome Petazzoni1-0/+10
When serving a manifest, it is important to set the content-type correctly (otherwise pulling an image is likely to give a cryptic error message, "Error response from daemon: missing signature key"). This makes sure that we set the content-type properly for both manifests and layers.
2021-04-14 chore(nix): update channel URLJerome Petazzoni1-1/+1
It looks like NixPkgs channels have moved. Fixing this URL allows using nixos-20.09, for instance.
2020-12-05 docs: Update README with a link to the NixCon talkVincent Ambo1-4/+10
2020-12-04 docs(config): Fix comment typoDave Nicponski1-1/+1
2020-10-29 feat(storage): Add support for content-types (GCS only)Vincent Ambo7-13/+34
Extends storage.Persist to accept a Content-Type argument, which in the GCS backend is persisted with the object to ensure that the object is served back with this content-type. This is not yet implemented for the filesystem backend, where the parameter is simply ignored. This should help in the case of clients which expect the returned objects to have content-types set when, for example, fetching layers by digest.
2020-10-27 docs: Add a note about a Nix-native builder to the roadmapVincent Ambo1-0/+6
... if I don't mention this somewhere I'll probably never do it!
2020-10-27 feat(main): Implement caching of manifests in CASVincent Ambo1-0/+35
To ensure that registry clients which attempt to pull manifests by their content hash can interact with Nixery, this change implements persisting image manifests in the CAS in the same way as image layers. In combination with the previous refactorings this means that Nixery's serving flow is now compatible with containerd. I have verified this locally, but CI currently only runs against Docker and not containerd, which is something I plan to address in a subsequent PR. This fixes #102
2020-10-27 feat(main): Implement serving of manifests by digestVincent Ambo1-11/+12
Modifies the layer serving endpoint to be a generic blob-serving endpoint that can handle both manifest and layer object "types". Note that this commit does not yet populate the CAS with any manifests.
2020-10-27 refactor(storage): Rename ServeLayer -> ServeVincent Ambo4-12/+12
This is going to be used for general content-addressed objects, and is not layer specific anymore.
2020-10-27 refactor(main): Split HTTP handlers into separate functionsVincent Ambo2-57/+62
There is a new handler coming up to fix #102 and I want to avoid falling into the classic Go trap of creating thousand-line functions.
2020-10-27 fix(build): Work around arbitrary new maxLayers restrictionVincent Ambo1-1/+1
2020-10-27 fix(build): Completely remove Cachix from build setupVincent Ambo1-2/+2
Installing Cachix started failing on ARM64.
2020-07-25 fix(build): Don't use Cachix as the binary cache during buildsVincent Ambo1-1/+0
Permission changes in the Travis CI Nix builders have caused this to start failing, as the build user now has insufficient permissions to use caches. There may be a way to change the permissions instead, but in the meantime we will just cause things to rebuild.
2020-07-25 chore(build): Update pinned Go dependenciesVincent Ambo1-41/+50
2020-07-25 fix(popcount): Accommodate upstream changes on nixos.orgVincent Ambo1-2/+9
Channel serving has moved to a new subdomain, and the redirect semantics have changed. Instead of serving temporary redirects, permanent redirects are now issued. I've reported this upstream as a bug, but this workaround will fix it in the meantime.
2020-07-25 chore(build): Change pin for default nixpkgs used to build NixeryVincent Ambo3-2/+6
This moves the pin from just being in the Travis configuration to also being set in a nixpkgs-pin.nix file, which makes it trivial to build at the right commit when performing local builds.
2020-05-01 chore(nix): update channel 19.03 -> 20.03Raphael Borun Das Gupta4-5/+5
Use a NixOS / NixPkgs release that's actually being supported and regularly updated.
2020-02-26 fix(popcount): Fix nix-build -A nixery-popcountFlorian Klink1-8/+6
Previously, this was failing as follows: ``` these derivations will be built: /nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv building '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv'... building warning: GOPATH set to GOROOT (/nix/store/4859cp1v7zqcqh43jkqsayl4wrz3g6hp-go-1.13.4/share/go) has no effect failed to initialize build cache at /homeless-shelter/.cache/go-build: mkdir /homeless-shelter: permission denied builder for '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed with exit code 1 error: build of '/nix/store/7rbrf06phkiyz31dwpq88x920zjhnw0c-nixery-popcount.drv' failed ```
2020-01-19 fix(builder): Fix minor logging switcharooVincent Ambo1-1/+1
2019-11-27 refactor: Reshuffle file structure for better code layoutVincent Ambo21-114/+83
This gets rid of the package called "server" and instead moves everything into the project root, such that Go actually builds us a binary called `nixery`. This is the first step towards factoring out CLI-based functionality for Nixery.
2019-11-27 fix(builder): Ensure "solo-metapackages" do not break buildsVincent Ambo1-1/+7
The previous logic failed because single meta-packages such as "nixery.dev/shell" would not end up removing the meta-package itself from the list of packages passed to Nix, causing a build failure. This was a regression introduced in 827468a.
2019-11-27 test(builder): Add test coverage for name->image conversionVincent Ambo1-0/+123
Adds tests to cover that packages & metapackages are parsed into image names correctly.
2019-11-09 chore(build): Use significantly fewer layers for Nixery itselfVincent Ambo2-3/+5
Nixery itself is built with the buildLayeredImage system, which takes some time to create large numbers of layers. This adjusts the default number of image layers from 96 to 20. Additionally Nixery's image is often loaded with `docker load -i`, which ignores layer cache hits anyways. Additionaly the CI build is configured to use only 1, which speeds up CI runs.
2019-11-09 feat(build): Integration test on both CPU architecturesVincent Ambo1-1/+22
2019-11-09 feat(build): Include arm64 in build matrixVincent Ambo1-0/+4
2019-11-09 fix(build-image): Allow "cross-builds" of images for different archVincent Ambo1-4/+9
Imports the package set twice in the builder expression: Once configured for the target system, once configured for the native system. This makes it possible to fetch the actual image contents for the required architecture, but use local tools to assemble the symlink layer and metadata.
2019-11-09 feat(builder): Support 'arm64' meta-packageVincent Ambo1-12/+29
Specifying this meta-package toggles support for ARM64 images, for example: # Pull a default x86_64 image docker pull nixery.dev/hello # Pull an ARM64 image docker pull nixery.dev/arm64/hello
2019-11-09 refactor(builder): Parameterise CPU architecture to use for imagesVincent Ambo3-6/+33
Adds the CPU architecture to the image configuration. This will make it possible to let users toggle architecture via meta-packages. Relates to #13
2019-11-03 chore(build): Add nixery-popcount to top-level package setVincent Ambo2-0/+28
2019-11-03 feat(popcount): Cache seen narinfos on diskVincent Ambo1-0/+14
2019-11-03 chore: Add missing copyright headers to source filesVincent Ambo8-0/+110
2019-11-03 feat(popcount): Add new narinfo-based popcount implementationVincent Ambo4-67/+256
Adds an implementation of popcount that, instead of realising derivations locally, just queries the cache's narinfo files. The downside of this is that calculating popularity for arbitrary Nix package sets is not possible with this implementation. The upside is that calculating the popularity for an entire Nix channel can now be done in ~10 seconds[0]. This fixes #65. [0]: Assuming a /fast/ internet connection.
2019-10-29 chore(server): Remove outdated TODOVincent Ambo1-5/+1
Real-life experience has shown that the weighting of the metric produced here is appropriate.
2019-10-29 refactor(server): Handle non-error errors safelyVincent Ambo1-5/+6
This case should not be possible unless something manually constructs a logrus entry with a non-error value in the log.ErrorKey field, but it's better to be safe than sorry.
2019-10-28 chore(build): Configure build to use new GCS configuration optionsVincent Ambo1-3/+2
2019-10-28 fix(server): Use correct scope for GCS tokensVincent Ambo1-1/+1
2019-10-28 docs(under-the-hood): Update builder & storage backend informationVincent Ambo1-28/+51
Both of these no longer matched the reality of what was actually going on in Nixery.
2019-10-28 docs: Add storage configuration options to READMEVincent Ambo1-1/+25
2019-10-28 fix(server): Thread request context to all relevant placesVincent Ambo6-23/+20
Previously background contexts where created where necessary (e.g. in GCS interactions). Should I begin to use request timeouts or other context-dependent things in the future, it's useful to have the actual HTTP request context around. This threads the request context through the application to all places that need it.
2019-10-28 chore(server): Move cache miss log statement to debug levelVincent Ambo2-2/+2
This is very annoying otherwise.
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-29T02·53+0000