From 7c41a7a8723c8bd6606c0c2f3fed3a546f1efb24 Mon Sep 17 00:00:00 2001 From: Vincent Ambo Date: Sun, 4 Aug 2019 22:38:51 +0100 Subject: docs: Replace static page with mdBook site Uses mdBook[1] to generate a documentation overview page instead of the previous HTML site. This makes it possible to add more elaborate documentation without having to deal with finicky markup. [1]: https://github.com/rust-lang-nursery/mdBook --- tools/nixery/docs/src/SUMMARY.md | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 tools/nixery/docs/src/SUMMARY.md (limited to 'tools/nixery/docs/src/SUMMARY.md') diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md new file mode 100644 index 0000000000..5d680b82e8 --- /dev/null +++ b/tools/nixery/docs/src/SUMMARY.md @@ -0,0 +1,4 @@ +# Summary + +- [Nixery](./nixery.md) +- [Nix, the language](./nix-1p.md) -- cgit 1.4.1 From a3f6278913d756c74d4f5c636c88b91a1b3748f6 Mon Sep 17 00:00:00 2001 From: Vincent Ambo Date: Sun, 4 Aug 2019 23:42:57 +0100 Subject: docs: Add an "under-the-hood" page explaining the build process This page describes the various steps that Nixery goes through when "procuring" an image. The intention is to give users some more visibility into what is going on and to make it clear that this is not just an image storage service. --- tools/nixery/docs/src/SUMMARY.md | 1 + tools/nixery/docs/src/nixery.md | 8 +-- tools/nixery/docs/src/under-the-hood.md | 105 ++++++++++++++++++++++++++++++++ 3 files changed, 110 insertions(+), 4 deletions(-) create mode 100644 tools/nixery/docs/src/under-the-hood.md (limited to 'tools/nixery/docs/src/SUMMARY.md') diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md index 5d680b82e8..f5ba3e9b08 100644 --- a/tools/nixery/docs/src/SUMMARY.md +++ b/tools/nixery/docs/src/SUMMARY.md @@ -1,4 +1,5 @@ # Summary - [Nixery](./nixery.md) +- [Under the hood](./under-the-hood.md) - [Nix, the language](./nix-1p.md) diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md index d3d1911d28..83e1aac52b 100644 --- a/tools/nixery/docs/src/nixery.md +++ b/tools/nixery/docs/src/nixery.md @@ -52,10 +52,6 @@ The instance at `nixery.dev` tracks a recent NixOS channel, currently NixOS Private registries might be configured to track a different channel (such as `nixos-unstable`) or even track a git repository with custom packages. -### Is this an official Google project? - -**No.** Nixery is not officially supported by Google. - ### Should I depend on `nixery.dev` in production? While we appreciate the enthusiasm, if you would like to use Nixery in your @@ -63,6 +59,10 @@ production project we recommend setting up a private instance. The public Nixery at `nixery.dev` is run on a best-effort basis and we make no guarantees about availability. +### Is this an official Google project? + +**No.** Nixery is not officially supported by Google. + ### Who made this? Nixery was written mostly by [tazjin][]. diff --git a/tools/nixery/docs/src/under-the-hood.md b/tools/nixery/docs/src/under-the-hood.md new file mode 100644 index 0000000000..3791707b1c --- /dev/null +++ b/tools/nixery/docs/src/under-the-hood.md @@ -0,0 +1,105 @@ +# Under the hood + +This page serves as a quick explanation of what happens under-the-hood when an +image is requested from Nixery. + + + +- [1. The image manifest is requested](#1-the-image-manifest-is-requested) +- [2. Nix builds the image](#2-nix-builds-the-image) +- [3. Layers are uploaded to Nixery's storage](#3-layers-are-uploaded-to-nixerys-storage) +- [4. The image manifest is sent back](#4-the-image-manifest-is-sent-back) +- [5. Image layers are requested](#5-image-layers-are-requested) + + + +-------- + +## 1. The image manifest is requested + +When container registry clients such as Docker pull an image, the first thing +they do is ask for the image manifest. This is a JSON document describing which +layers are contained in an image, as well as some additional auxiliary +information. + +This request is of the form `GET /v2/$imageName/manifests/$imageTag`. + +Nixery receives this request and begins by splitting the image name into its +path components and substituting meta-packages (such as `shell`) for their +contents. + +For example, requesting `shell/htop/git` results in Nixery expanding the image +name to `["bashInteractive", "coreutils", "htop", "git"]`. + +If Nixery is configured with a private Nix repository, it also looks at the +image tag and substitutes `latest` with `master`. + +It then invokes Nix with three parameters: + +1. image contents (as above) +2. image tag +3. configured package set source + +## 2. Nix builds the image + +Using the parameters above, Nix imports the package set and begins by mapping +the image names to attributes in the package set. + +A special case during this process is packages with uppercase characters in +their name, for example anything under `haskellPackages`. The registry protocol +does not allow uppercase characters, so the Nix code will translate something +like `haskellpackages` (lowercased) to the correct attribute name. + +After identifying all contents, Nix determines the contents of each layer while +optimising for the best possible cache efficiency. + +Finally it builds each layer, assembles the image manifest as JSON structure, +and yields this manifest back to the web server. + +*Note:* While this step is running (which can take some time in the case of +large first-time image builds), the registry client is left hanging waiting for +an HTTP response. Unfortunately the registry protocol does not allow for any +feedback back to the user at this point, so from the user's perspective things +just ... hang, for a moment. + +## 3. Layers are uploaded to Nixery's storage + +Nixery inspects the returned manifest and uploads each layer to the configured +[Google Cloud Storage][gcs] bucket. To avoid unnecessary uploading, it will +first check whether layers are already present in the bucket and - just to be +safe - compare their MD5-hashes against what was built. + +## 4. The image manifest is sent back + +If everything went well at this point, Nixery responds to the registry client +with the image manifest. + +The client now inspects the manifest and basically sees a list of SHA256-hashes, +each corresponding to one layer of the image. Most clients will now consult +their local layer storage and determine which layers they are missing. + +Each of the missing layers is then requested from Nixery. + +## 5. Image layers are requested + +For each image layer that it needs to retrieve, the registry client assembles a +request that looks like this: + +`GET /v2/${imageName}/blob/sha256:${layerHash}` + +Nixery receives these requests and *rewrites* them to Google Cloud Storage URLs, +responding with an `HTTP 303 See Other` status code and the actual download URL +of the layer. + +Nixery supports using private buckets which are not generally world-readable, in +which case [signed URLs][] are constructed using a private key. These allow the +registry client to download each layer without needing to care about how the +underlying authentication works. + +--------- + +That's it. After these five steps the registry client has retrieved all it needs +to run the image produced by Nixery. + +[gcs]: https://cloud.google.com/storage/ +[signed URLs]: https://cloud.google.com/storage/docs/access-control/signed-urls -- cgit 1.4.1 From 6293d69fd94f709d029c5c121956900cad3d24c1 Mon Sep 17 00:00:00 2001 From: Vincent Ambo Date: Mon, 5 Aug 2019 00:27:39 +0100 Subject: docs: Add a section on running your own Nixery --- tools/nixery/docs/src/SUMMARY.md | 6 +- tools/nixery/docs/src/run-your-own.md | 141 ++++++++++++++++++++++++++++++++++ 2 files changed, 145 insertions(+), 2 deletions(-) create mode 100644 tools/nixery/docs/src/run-your-own.md (limited to 'tools/nixery/docs/src/SUMMARY.md') diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md index f5ba3e9b08..677f328972 100644 --- a/tools/nixery/docs/src/SUMMARY.md +++ b/tools/nixery/docs/src/SUMMARY.md @@ -1,5 +1,7 @@ # Summary - [Nixery](./nixery.md) -- [Under the hood](./under-the-hood.md) -- [Nix, the language](./nix-1p.md) + - [Under the hood](./under-the-hood.md) + - [Run your own Nixery](./run-your-own.md) +- [Nix](./nix.md) + - [Nix, the language](./nix-1p.md) diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md new file mode 100644 index 0000000000..0539ab02fc --- /dev/null +++ b/tools/nixery/docs/src/run-your-own.md @@ -0,0 +1,141 @@ +## Run your own Nixery + + + +- [0. Prerequisites](#0-prerequisites) +- [1. Choose a package set](#1-choose-a-package-set) +- [2. Build Nixery itself](#2-build-nixery-itself) +- [3. Prepare configuration](#3-prepare-configuration) +- [4. Deploy Nixery](#4-deploy-nixery) +- [5. Productionise](#5-productionise) + + + + +--------- + +⚠ This page is still under construction! ⚠ + +-------- + +Running your own Nixery is not difficult, but requires some setup. Follow the +steps below to get up & running. + +*Note:* Nixery can be run inside of a [GKE][] cluster, providing a local service +from which images can be requested. Documentation for how to set this up is +forthcoming, please see [nixery#4][]. + +## 0. Prerequisites + +To run Nixery, you must have: + +* [Nix][] (to build Nixery itself) +* Somewhere to run it (your own server, Google AppEngine, a Kubernetes cluster, + whatever!) +* A [Google Cloud Storage][gcs] bucket in which to store & serve layers + +## 1. Choose a package set + +When running your own Nixery you need to decide which package set you want to +serve. By default, Nixery builds packages from a recent NixOS channel which +ensures that most packages are cached upstream and no expensive builds need to +be performed for trivial things. + +However if you are running a private Nixery, chances are high that you intend to +use it with your own packages. There are three options available: + +1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-19.03` or + `nixos-unstable`. +2. Specify your own git-repository with a custom package set[^2]. This makes it + possible to pull different tags, branches or commits by modifying the image + tag. +3. Specify a local file path containing a Nix package set. Where this comes from + or what it contains is up to you. + +## 2. Build Nixery itself + +Building Nixery creates a container image. This section assumes that the +container runtime used is Docker, please modify instructions correspondingly if +you are using something else. + +With a working Nix installation, building Nixery is done by invoking `nix-build +-A nixery-image` from a checkout of the [Nixery repository][repo]. + +This will create a `result`-symlink which points to a tarball containing the +image. In Docker, this tarball can be loaded by using `docker load -i result`. + +## 3. Prepare configuration + +Nixery is configured via environment variables. + +You must set *all* of these: + +* `BUCKET`: [Google Cloud Storage][gcs] bucket to store & serve image layers +* `PORT`: HTTP port on which Nixery should listen + +You may set *one* of these, if unset Nixery defaults to `nixos-19.03`: + +* `NIXERY_CHANNEL`: The name of a Nix/NixOS channel to use for building +* `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses + locally configured SSH/git credentials) +* `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use + for building + +You may set *both* of these: + +* `GCS_SIGNING_KEY`: A Google service account key (in PEM format) that can be + used to [sign Cloud Storage URLs][signed-urls] +* `GCS_SIGNING_ACCOUNT`: Google service account ID that the signing key belongs + to + +To authenticate to the configured GCS bucket, Nixery uses Google's [Application +Default Credentials][ADC]. Depending on your environment this may require +additional configuration. + +## 4. Deploy Nixery + +With the above environment variables configured, you can run the image that was +built in step 2. + +How this works depends on the environment you are using and is, for now, outside +of the scope of this tutorial. + +Once Nixery is running you can immediately start requesting images from it. + +## 5. Productionise + +(⚠ Here be dragons! ⚠) + +Nixery is still an early project and has not yet been deployed in any production +environments and some caveats apply. + +Notably, Nixery currently does not support any authentication methods, so anyone +with network access to the registry can retrieve images. + +Running a Nixery inside of a fenced-off environment (such as internal to a +Kubernetes cluster) should be fine, but you should consider to do all of the +following: + +* Issue a TLS certificate for the hostname you are assigning to Nixery. In fact, + Docker will refuse to pull images from registries that do not use TLS (with + the exception of `.local` domains). +* Configure signed GCS URLs to avoid having to make your bucket world-readable. +* Configure request timeouts for Nixery if you have your own web server in front + of it. This will be natively supported by Nixery in the future. + +------- + +[^1]: Nixery will not work with Nix channels older than `nixos-19.03`. + +[^2]: This documentation will be updated with instructions on how to best set up + a custom Nix repository. Nixery expects custom package sets to be a superset + of `nixpkgs`, as it uses `lib` and other features from `nixpkgs` + extensively. + +[GKE]: https://cloud.google.com/kubernetes-engine/ +[nixery#4]: https://github.com/google/nixery/issues/4 +[Nix]: https://nixos.org/nix +[gcs]: https://cloud.google.com/storage/ +[repo]: https://github.com/google/nixery +[signed-urls]: under-the-hood.html#5-image-layers-are-requested +[ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically -- cgit 1.4.1 From 2c8ef634f67f91c8efc1cf6a58a271f4c44544dd Mon Sep 17 00:00:00 2001 From: Vincent Ambo Date: Mon, 30 Sep 2019 11:51:36 +0100 Subject: docs(caching): Add information about Nixery's caching strategies --- tools/nixery/docs/src/SUMMARY.md | 1 + tools/nixery/docs/src/caching.md | 70 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+) create mode 100644 tools/nixery/docs/src/caching.md (limited to 'tools/nixery/docs/src/SUMMARY.md') diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md index 677f328972..f1d68a3ac4 100644 --- a/tools/nixery/docs/src/SUMMARY.md +++ b/tools/nixery/docs/src/SUMMARY.md @@ -2,6 +2,7 @@ - [Nixery](./nixery.md) - [Under the hood](./under-the-hood.md) + - [Caching](./caching.md) - [Run your own Nixery](./run-your-own.md) - [Nix](./nix.md) - [Nix, the language](./nix-1p.md) diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md new file mode 100644 index 0000000000..175fe04d70 --- /dev/null +++ b/tools/nixery/docs/src/caching.md @@ -0,0 +1,70 @@ +# Caching in Nixery + +This page gives a quick overview over the caching done by Nixery. All cache data +is written to Nixery's storage bucket and is based on deterministic identifiers +or content-addressing, meaning that cache entries under the same key *never +change*. + +## Manifests + +Manifests of builds are cached at `$BUCKET/manifests/$KEY`. The effect of this +cache is that multiple instances of Nixery do not need to rebuild the same +manifest from scratch. + +Since the manifest cache is populated only *after* layers are uploaded, Nixery +can immediately return the manifest to its clients without needing to check +whether layers have been uploaded already. + +`$KEY` is generated by creating a SHA1 hash of the requested content of a +manifest plus the package source specification. + +Manifests are *only* cached if the package source specification is *not* a +moving target. + +Manifest caching *only* applies in the following cases: + +* package source specification is a specific git commit +* package source specification is a specific NixOS/nixpkgs commit + +Manifest caching *never* applies in the following cases: + +* package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`) +* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-19.03`) +* package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`) + +It is thus always preferable to request images from a fully-pinned package +source. + +Manifests can be removed from the manifest cache without negative consequences. + +## Layer tarballs + +Layer tarballs are the files that Nixery clients retrieve from the storage +bucket to download an image. + +They are stored content-addressably at `$BUCKET/layers/$SHA256HASH` and layer +requests sent to Nixery will redirect directly to this storage location. + +The effect of this cache is that Nixery does not need to upload identical layers +repeatedly. When Nixery notices that a layer already exists in GCS, it will use +the object metadata to compare its MD5-hash with the locally computed one and +skip uploading. + +Removing layers from the cache is *potentially problematic* if there are cached +manifests or layer builds referencing those layers. + +To clean up layers, a user must ensure that no other cached resources still +reference these layers. + +## Layer builds + +Layer builds are cached at `$BUCKET/builds/$HASH`, where `$HASH` is a SHA1 of +the Nix store paths included in the layer. + +The content of the cached entries is a JSON-object that contains the MD5 and +SHA256 hashes of the built layer. + +The effect of this cache is that different instances of Nixery will not build, +hash and upload layers that have identical contents across different instances. + +Layer builds can be removed from the cache without negative consequences. -- cgit 1.4.1