about summary refs log tree commit diff
path: root/tools/nixery/default.nix
AgeCommit message (Collapse)AuthorFilesLines
2024-02-28 r/7620 refactor(nixery): expose launch script derivationVincent Ambo1-53/+51
Simplifies reusing the launch script in other use-cases than the "official" Nixery image. Relates to nixery#166 Change-Id: Iaf1dff385ce270792253551081c1b2fca6400037 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11046 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-28 r/7619 fix(nixery): use set-default for setting WEB_DIRVincent Ambo1-1/+1
Makes it possible for users to still override this using an envvar. Relates to nixery#166 Change-Id: Ief2925e03cf2e4351bc38554bf553c8ee259f1f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11045 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-10-12 r/5116 docs(nixery): replace the Nixery mdBook with a simple web pageVincent Ambo1-30/+5
Nixery's previous landing page was an mdBook that was basically unmaintained and full of incorrect information. It also duplicated some things (like nix-1p) which actually live elsewhere. This commit removes the mdBook completely and reduces it down to a simple TVL-style landing page. The landing page has been checked in in its entirety because Nixery is frequently cloned through josh without the entirety of depot, however the page has been created by building it through depot's //web/tvl/template. See also https://github.com/tazjin/nixery/issues/156 Change-Id: I20e1d58f1e6608377207e80345c169f7d92d3847 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6930 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-10-02 r/5013 chore(nixery): use `ldflags` parameter instead of `buildFlagsArray`Vincent Ambo1-2/+5
The latter has been deprecated in nixpkgs. Relates to b/200 Change-Id: I42871ce3eb54ebf092909f033b43936b9610d982 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6836 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-09-28 r/4980 chore(3p/sources): Bump channels & overlayssterni1-1/+1
Upstream nixpkgs removed a lot of aliases this time, so we needed to do the following transformations. It's a real shame that aliases only really become discoverable easily when they are removed. * runCommandNoCC -> runCommand * gmailieer -> lieer We also need to work around the fact that home-manager hasn't catched on to this rename. * mysql -> mariadb * pkgconfig -> pkg-config This also affects our Nix fork which needs to be bumped. * prometheus_client -> prometheus-client * rxvt_unicode -> rxvt-unicode-unwrapped * nix-review -> nixpkgs-review * oauth2_proxy -> oauth2-proxy Additionally, some Go-related builders decided to drop support for passing the sha256 hash in directly, so we need to use the generic hash arguments. Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-09-19 r/4926 fix(nixery): Set correct depot ref when fetching nix-1ptalyz1-1/+4
Change-Id: Iffa49a4e8fd38d0762ed1f60bf72b9a050594a3c Reviewed-on: https://cl.tvl.fyi/c/depot/+/6697 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-09-19 r/4924 fix(nixery): Avoid race when the same image is fetched in paralleltalyz1-1/+1
Remove a race condition which appears when uploadHashLayer is called with the same key from multiple threads simultaneously. This can easily happen when the same image path is requested by multiple clients at the same time. When it does, a 500 status is returned and the following error message is logged: { "context": { "filePath": "github.com/google/nixery/builder/builder.go", "lineNumber": 440, "functionName": "github.com/google/nixery/builder.uploadHashLayer" }, "error": "rename /var/lib/nixery/staging/<hash> /var/lib/nixery/layers/<hash>: no such file or directory", "eventTime": "...", "layer": "<hash>", "message": "failed to move layer from staging", ... } To solve this issue, introduce a mutex keyed on the uploaded hash and move all layer caching into uploadHashLayer. This could additionally provide a small performance benefit when an already built image is requested and NIXERY_PKGS_PATH is set, since symlink layers and config layers are now also cached. Change-Id: I50788a7ec7940cb5e5760f244692e361019a9bb7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6695 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-06-16 r/4242 chore(nixery): use nix-1p from within the depotVincent Ambo1-0/+5
Since the source of nix-1p is checked in under //nix/nix-1p, we should use it from there if Nixery is being built inside of depot. Change-Id: Iddd54f7b93b398b2f909db6ee105366a9914a2ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/5882 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2022-06-16 r/4241 docs(nixery): dynamically display current nixpkgs commitVincent Ambo1-2/+18
People occasionally ask what the current nixpkgs commit is on nixery.dev (see e.g. https://github.com/tazjin/nixery/issues/153). With this change, the commit is displayed on nixery.dev if Nixery is built for the TVL deployment. Change-Id: I795220214db5a367a126c9b4bd03754e9f144940 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5881 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2022-05-26 r/4123 chore(nixery): Bump Go dependenciesVincent Ambo1-1/+1
Change-Id: Id6ff48d66368732cba0b8af6e1cbab64b0f2afbf Reviewed-on: https://cl.tvl.fyi/c/depot/+/5671 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-26 r/4121 feat(nixery): Automatically mirror subtree to GithubVincent Ambo1-0/+8
This exports the `:/tools/nixery` subtree to Github automatically after merges to `canon`. Due to the way the project was imported this continues the existing git history in the external repository. Change-Id: Ie871c14ad5d8f1019f8be86adecbe9b130ffb01a Reviewed-on: https://cl.tvl.fyi/c/depot/+/5667 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-05-23 r/4106 refactor(nixery): Modernise structure of binariesVincent Ambo1-28/+25
Nixery is going to gain a new binary (used for building images without a registry server); to prepare for this the server binary has moved to cmd/server and the Nix build logic has been updated to wrap this binary and set the required environment variables. Change-Id: I9b4f49f47872ae76430463e2fcb8f68114070f72 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5603 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-04-21 r/3987 fix(nixery): Avoid impure reading of .git directoryVincent Ambo1-5/+3
Change-Id: I67405f9c9bd9cc8cb34fafff80e30b2fca53a2b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5502 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su>
2022-04-20 r/3982 chore(nixery): Housekeeping for depot compatibilityVincent Ambo1-67/+60
Cleans up a whole bunch of things I wanted to get out of the door right away: * depot internal references to //third_party/nixery have been replaced with //tools/nixery * cleaned up files from Github * fixed SPDX & Copyright headers * code formatting and inclusion in //tools/depotfmt checks Change-Id: Iea79f0fdf3aa04f71741d4f4032f88605ae415bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/5486 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su>
2022-04-20 r/3979 refactor(nixery): Adapt Nix build instructions for readTreeVincent Ambo1-3/+11
This does not fully change the build structure of Nixery to be depot-compatible yet, but should allow most targets to be built in depot CI. This contains some hacks to work around surface incompatibilities which we'll clear away later. Change-Id: I84e7734334abbe299983956f528c0897f49fa8c2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5485 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-20 r/3978 feat(tools/nixery): Absorb Nixery into depotVincent Ambo1-0/+125
This absorbs a josh-filtered Nix subtree into depot, at //tools/nixery. This subtree was created through `josh-filter ':prefix=tools/nixery'`, which allows a filter on tools/nixery to yield the same commit hashes as the original Nixery repository (allowing for history continuity). Change-Id: Icc1a99bf1248226b91f437b0a90361d36fb0d327
2021-10-29 chore: Bump all Go dependenciesVincent Ambo1-1/+1
Result of 'go get -u && go mod tidy'
2021-08-06 chore(build): Allow passing in a specific commit hash when buildingVincent Ambo1-2/+3
Required for builds where the full repository isn't available (e.g. from a tarball).
2021-06-26 feat(storage): Store blob content-type in extended attributesJérôme Petazzoni1-1/+1
After the discussion in #116, this stores the blob content types in extended attributes when using the filesystem backend. If the underlying filesystem doesn't support extended attributes, storing blobs won't work; also, if extended attributes get removed, blobs won't be served anymore. We can relax this behavior if needed (i.e. log errors but still accept to store or serve blobs). However, since the Docker Engine (and possibly other container engines) won't accept to pull images from a registry that doesn't use correct content types for manifest files, it could be argued that it's better to give a hard fail. (Otherwise, the container engine gives cryptic error messages like "missing signature key".) I can change that behavior (and log errors but still store/serve blobs to the filesystem) if you think it's better.
2021-04-30 refactor(build): Pin dependencies using Go modulesVincent Ambo1-25/+7
Drops the go2nix configuration in favour of pkgs.buildGoModule. Note that the go.sum file is bloated by issues with cyclic dependencies in some Google projects, but this large number of dependencies is not actually built.
2021-04-30 chore(build): Use current git commit hash as build versionVincent Ambo1-6/+4
2020-07-25 chore(build): Change pin for default nixpkgs used to build NixeryVincent Ambo1-1/+1
This moves the pin from just being in the Travis configuration to also being set in a nixpkgs-pin.nix file, which makes it trivial to build at the right commit when performing local builds.
2019-11-27 refactor: Reshuffle file structure for better code layoutVincent Ambo1-7/+37
This gets rid of the package called "server" and instead moves everything into the project root, such that Go actually builds us a binary called `nixery`. This is the first step towards factoring out CLI-based functionality for Nixery.
2019-11-09 chore(build): Use significantly fewer layers for Nixery itselfVincent Ambo1-2/+4
Nixery itself is built with the buildLayeredImage system, which takes some time to create large numbers of layers. This adjusts the default number of image layers from 96 to 20. Additionally Nixery's image is often loaded with `docker load -i`, which ignores layer cache hits anyways. Additionaly the CI build is configured to use only 1, which speeds up CI runs.
2019-11-03 chore(build): Add nixery-popcount to top-level package setVincent Ambo1-0/+2
2019-10-28 refactor(build): Do not expose nixery-server attributeVincent Ambo1-5/+3
In most cases this is not useful for users without the wrapper script, so users should always build nixery-bin anyways.
2019-10-11 fix(build): Only take the first matching hash for source hashingVincent Ambo1-1/+1
Some Nix download mechanisms will add a second hash in the store path, which had been added to the source hash output (breaking argument interpolation).
2019-10-06 feat(server): Use hash of Nixery source as versionVincent Ambo1-1/+10
Uses a hash of Nixery's sources as the version displayed when Nixery launches or logs an error. This makes it possible to distinguish between errors logged from different versions. The source hashes should be reproducible between different checkouts of the same source tree.
2019-10-06 fix(server): Amend package path for Go tooling compatibilityVincent Ambo1-1/+1
With these changes it is possible to keep Nixery in $GOPATH and build the server in there, while still having things work correctly via Nix.
2019-10-03 chore(build-image): Remove nixery-build-layersVincent Ambo1-6/+3
This functionality has been rolled into the server component and is no longer required.
2019-10-03 fix(nix): Minor fixes to derivations for new build processVincent Ambo1-1/+2
2019-10-03 feat(nix): Add derivation to create layer tars from a store path setVincent Ambo1-2/+5
This introduces a new Nix derivation that, given an attribute set of layer hashes mapped to store paths, will create a layer tarball for each of the store paths. This is going to be used by the builder to create layers that are not present in the cache. Relates to #50.
2019-10-03 chore(build-image): Simplify wrapper build & remove layer groupingVincent Ambo1-8/+3
Simplifies the wrapper script used to invoke Nix builds from Nixery to just contain the essentials, since the layer grouping logic is moving into the server itself.
2019-09-22 refactor(build): Add group-layers to top-level Nix derivationsVincent Ambo1-5/+7
This makes CI build the group-layers tool (and cache it to Cachix!)
2019-09-21 fix(build): Ensure launch script compatibility with other runtimesVincent Ambo1-1/+2
Fixes two launch script compatibility issues with other container runtimes (such as gvisor): * don't fail if /tmp already exists * don't fail if the environment becomes unset
2019-09-21 fix(build): Ensure nixery-build-image is on Nixery's PATHVincent Ambo1-0/+1
This is useful when running Nixery locally.
2019-09-10 refactor(server): Move package source management logic to serverVincent Ambo1-1/+2
Introduces three new types representing each of the possible package sources and moves the logic for specifying the package source to the server. Concrete changes: * Determining whether a specified git reference is a commit vs. a branch/tag is now done in the server, and is done more precisely by using a regular expression. * Package sources now have a new `CacheKey` function which can be used to retrieve a key under which a build manifest can be cached *if* the package source is not a moving target (i.e. a full git commit hash of either nixpkgs or a private repository). This function is not yet used. * Users *must* now specify a package source, Nixery no longer defaults to anything and will fail to launch if no source is configured.
2019-08-21 chore(build): Add iana-etc to Nixery's own imageVincent Ambo1-0/+1
This package is used by a variety of programs that users may want to embed into Nixery in addition, for example cachix, but those packages don't refer to it explicitly.
2019-08-19 feat(build): Add 'extraPackages' parameterVincent Ambo1-2/+4
This makes it possible to inject additional programs (e.g. Cachix) into a Nixery container.
2019-08-17 fix(build): Ensure root user is known inside of containerVincent Ambo1-0/+4
This is required by git in cases where Nixery is configured with a custom git repository. I've also added a shell back into the image to make debugging a running Nixery easier. It turns out some of the dependencies already pull in bash anyways, so this is just surfacing it to $PATH.
2019-08-14 refactor(build-image): Extract package set loading into helperVincent Ambo1-1/+3
Some upcoming changes might require the Nix build to be split into multiple separate nix-build invocations of different expressions, thus splitting this out is useful. It also fixes an issue where `build-image/default.nix` might be called in an environment where no Nix channels are configured.
2019-08-14 style: Apply nixfmt to trivial Nix filesVincent Ambo1-5/+4
ALl the ones except for build-image.nix are considered trivial. On the latter, nixfmt makes some useful changes but by-and-large it is not ready for that code yet.
2019-08-14 feat(build-image): Introduce a terrifying hack to build group-layersVincent Ambo1-1/+1
The issue is described in detail in a comment in `build-image/default.nix`, please read it.
2019-08-14 refactor(server): Use wrapper script to avoid path dependencyVincent Ambo1-1/+3
Instead of requiring the server component to be made aware of the location of the Nix builder via environment variables, this commit introduces a wrapper script for the builder that can simply exist on the builders $PATH. This is one step towards a slightly nicer out-of-the-box experience when using `nix-build -A nixery-bin`.
2019-08-14 chore(docs): Move mdBook derivation to docs/default.nixVincent Ambo1-22/+1
2019-08-14 refactor: Move registry server to a subfolderVincent Ambo1-20/+2
2019-08-08 chore(nix): Increase maximum number of layers to 96Vincent Ambo1-1/+1
This uses a significantly larger percentage of the total available layers (125) than before, which means that cache hits for layers become more likely between images.
2019-08-05 feat(build): Build Nixery book and embed it into Nixery imageVincent Ambo1-8/+10
Executes the previously added mdBook on the previously added book source to yield a directory that can be served by Nixery on its index page. This is one of those 'I <3 Nix' things due to how easy it is to do.
2019-08-05 feat(build): Add mdBook 0.3.1 to build environmentVincent Ambo1-0/+18
Upstream nixpkgs currently only has an older versin of mdBook. Until that changes, we keep a different version in here.
2019-08-04 feat(build): Support additional pre-launch commands in imageVincent Ambo1-1/+4
This makes it possible for users to hook basically arbitrary things into the Nixery container image.
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-18T21·10+0000