8 files changed, 513 insertions, 0 deletions
diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md
new file mode 100644
index 000000000000..f1d68a3ac451
--- /dev/null
+++ b/tools/nixery/docs/src/SUMMARY.md
@@ -0,0 +1,8 @@
+# Summary
+
+- [Nixery](./nixery.md)
+  - [Under the hood](./under-the-hood.md)
+  - [Caching](./caching.md)
+  - [Run your own Nixery](./run-your-own.md)
+- [Nix](./nix.md)
+  - [Nix, the language](./nix-1p.md)
diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md
new file mode 100644
index 000000000000..05ea68ef6083
--- /dev/null
+++ b/tools/nixery/docs/src/caching.md
@@ -0,0 +1,69 @@
+# Caching in Nixery
+
+This page gives a quick overview over the caching done by Nixery. All cache data
+is written to Nixery's storage bucket and is based on deterministic identifiers
+or content-addressing, meaning that cache entries under the same key *never
+change*.
+
+## Manifests
+
+Manifests of builds are cached at `$BUCKET/manifests/$KEY`. The effect of this
+cache is that multiple instances of Nixery do not need to rebuild the same
+manifest from scratch.
+
+Since the manifest cache is populated only *after* layers are uploaded, Nixery
+can immediately return the manifest to its clients without needing to check
+whether layers have been uploaded already.
+
+`$KEY` is generated by creating a SHA1 hash of the requested content of a
+manifest plus the package source specification.
+
+Manifests are *only* cached if the package source specification is *not* a
+moving target.
+
+Manifest caching *only* applies in the following cases:
+
+* package source specification is a specific git commit
+* package source specification is a specific NixOS/nixpkgs commit
+
+Manifest caching *never* applies in the following cases:
+
+* package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`)
+* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-20.09`)
+* package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`)
+
+It is thus always preferable to request images from a fully-pinned package
+source.
+
+Manifests can be removed from the manifest cache without negative consequences.
+
+## Layer tarballs
+
+Layer tarballs are the files that Nixery clients retrieve from the storage
+bucket to download an image.
+
+They are stored content-addressably at `$BUCKET/layers/$SHA256HASH` and layer
+requests sent to Nixery will redirect directly to this storage location.
+
+The effect of this cache is that Nixery does not need to upload identical layers
+repeatedly. When Nixery notices that a layer already exists in GCS it will skip
+uploading this layer.
+
+Removing layers from the cache is *potentially problematic* if there are cached
+manifests or layer builds referencing those layers.
+
+To clean up layers, a user must ensure that no other cached resources still
+reference these layers.
+
+## Layer builds
+
+Layer builds are cached at `$BUCKET/builds/$HASH`, where `$HASH` is a SHA1 of
+the Nix store paths included in the layer.
+
+The content of the cached entries is a JSON-object that contains the SHA256
+hashes and sizes of the built layer.
+
+The effect of this cache is that different instances of Nixery will not build,
+hash and upload layers that have identical contents across different instances.
+
+Layer builds can be removed from the cache without negative consequences.
diff --git a/tools/nixery/docs/src/nix-1p.md b/tools/nixery/docs/src/nix-1p.md
new file mode 100644
index 000000000000..a21234150fc7
--- /dev/null
+++ b/tools/nixery/docs/src/nix-1p.md
@@ -0,0 +1,2 @@
+This page is a placeholder. During the build process, it is replaced by the
+actual `nix-1p` guide from https://github.com/tazjin/nix-1p
diff --git a/tools/nixery/docs/src/nix.md b/tools/nixery/docs/src/nix.md
new file mode 100644
index 000000000000..2bfd75a6925c
--- /dev/null
+++ b/tools/nixery/docs/src/nix.md
@@ -0,0 +1,31 @@
+# Nix
+
+These sections are designed to give some background information on what Nix is.
+If you've never heard of Nix before looking at Nixery, this might just be the
+page for you!
+
+[Nix][] is a functional package-manager that comes with a number of advantages
+over traditional package managers, such as side-by-side installs of different
+package versions, atomic updates, easy customisability, simple binary caching
+and much more. Feel free to explore the [Nix website][Nix] for an overview of
+Nix itself.
+
+Nix uses a custom programming language also called Nix, which is explained here
+[on its own page][nix-1p].
+
+In addition to the package manager and language, the Nix project also maintains
+[NixOS][] - a Linux distribution built entirely on Nix. On NixOS, users can
+declaratively describe the *entire* configuration of their system and perform
+updates/rollbacks to other system configurations with ease.
+
+Most Nix packages are tracked in the [Nix package set][nixpkgs], usually simply
+referred to as `nixpkgs`. It contains tens of thousands of packages already!
+
+Nixery (which you are looking at!) provides an easy & simple way to get started
+with Nix, in fact you don't even need to know that you're using Nix to make use
+of Nixery.
+
+[Nix]: https://nixos.org/nix/
+[nix-1p]: nix-1p.html
+[NixOS]: https://nixos.org/
+[nixpkgs]: https://github.com/nixos/nixpkgs
diff --git a/tools/nixery/docs/src/nixery-logo.png b/tools/nixery/docs/src/nixery-logo.png
new file mode 100644
index 000000000000..fcf77df3d6a9
--- /dev/null
+++ b/tools/nixery/docs/src/nixery-logo.png
Binary files differdiff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
new file mode 100644
index 000000000000..d9ba179010f6
--- /dev/null
+++ b/tools/nixery/docs/src/nixery.md
@@ -0,0 +1,80 @@
+![Nixery](./nixery-logo.png)
+
+------------
+
+Welcome to this instance of [Nixery][]. It provides ad-hoc container images that
+contain packages from the [Nix][] package manager. Images with arbitrary
+packages can be requested via the image name.
+
+Nix not only provides the packages to include in the images, but also builds the
+images themselves by using a special [layering strategy][] that optimises for
+cache efficiency.
+
+For general information on why using Nix makes sense for container images, check
+out [this blog post][layers].
+
+## Demo
+
+<script src="https://asciinema.org/a/262583.js" id="asciicast-262583" async data-autoplay="true" data-loop="true"></script>
+
+## Quick start
+
+Simply pull an image from this registry, separating each package you want
+included by a slash:
+
+    docker pull nixery.dev/shell/git/htop
+
+This gives you an image with `git`, `htop` and an interactively configured
+shell. You could run it like this:
+
+    docker run -ti nixery.dev/shell/git/htop bash
+
+Each path segment corresponds either to a key in the Nix package set, or a
+meta-package that automatically expands to several other packages.
+
+Meta-packages **must** be the first path component if they are used. Currently
+there are only two meta-packages:
+- `shell`, which provides a `bash`-shell with interactive configuration and 
+  standard tools like `coreutils`.
+- `arm64`, which provides ARM64 binaries.
+
+**Tip:** When pulling from a private Nixery instance, replace `nixery.dev` in
+the above examples with your registry address.
+
+## FAQ
+
+If you have a question that is not answered here, feel free to file an issue on
+Github so that we can get it included in this section. The volume of questions
+is quite low, thus by definition your question is already frequently asked.
+
+### Where is the source code for this?
+
+Over [on Github][Nixery]. It is licensed under the Apache 2.0 license. Consult
+the documentation entries in the sidebar for information on how to set up your
+own instance of Nixery.
+
+### Which revision of `nixpkgs` is used for the builds?
+
+The instance at `nixery.dev` tracks a recent NixOS channel, currently NixOS
+20.09. The channel is updated several times a day.
+
+Private registries might be configured to track a different channel (such as
+`nixos-unstable`) or even track a git repository with custom packages.
+
+### Should I depend on `nixery.dev` in production?
+
+While we appreciate the enthusiasm, if you would like to use Nixery in your
+production project we recommend setting up a private instance. The public Nixery
+at `nixery.dev` is run on a best-effort basis and we make no guarantees about
+availability.
+
+### Who made this?
+
+Nixery was written by [tazjin][], but many people have contributed to Nix over
+time, maybe you could become one of them?
+
+[Nixery]: https://github.com/tazjin/nixery
+[Nix]: https://nixos.org/nix
+[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
+[layers]: https://grahamc.com/blog/nix-and-layered-docker-images
+[tazjin]: https://tazj.in
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
new file mode 100644
index 000000000000..7ed8bdd0bc0a
--- /dev/null
+++ b/tools/nixery/docs/src/run-your-own.md
@@ -0,0 +1,194 @@
+## Run your own Nixery
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+
+- [0. Prerequisites](#0-prerequisites)
+- [1. Choose a package set](#1-choose-a-package-set)
+- [2. Build Nixery itself](#2-build-nixery-itself)
+- [3. Prepare configuration](#3-prepare-configuration)
+- [4. Deploy Nixery](#4-deploy-nixery)
+- [5. Productionise](#5-productionise)
+
+<!-- markdown-toc end -->
+
+
+---------
+
+⚠ This page is still under construction! ⚠
+
+--------
+
+Running your own Nixery is not difficult, but requires some setup. Follow the
+steps below to get up & running.
+
+*Note:* Nixery can be run inside of a [GKE][] cluster, providing a local service
+from which images can be requested. Documentation for how to set this up is
+forthcoming, please see [nixery#4][].
+
+## 0. Prerequisites
+
+To run Nixery, you must have:
+
+* [Nix][] (to build Nixery itself)
+* Somewhere to run it (your own server, Google AppEngine, a Kubernetes cluster,
+  whatever!)
+* *Either* a [Google Cloud Storage][gcs] bucket in which to store & serve layers,
+  *or* a comfortable amount of disk space
+
+Note that while the main Nixery process is a server written in Go,
+it invokes a script that itself relies on Nix to be available.
+You can compile the main Nixery daemon without Nix, but it won't
+work without Nix.
+
+(If you are completely new to Nix and don't know how to get
+started, check the [Nix installation documentation][nixinstall].)
+
+## 1. Choose a package set
+
+When running your own Nixery you need to decide which package set you want to
+serve. By default, Nixery builds packages from a recent NixOS channel which
+ensures that most packages are cached upstream and no expensive builds need to
+be performed for trivial things.
+
+However if you are running a private Nixery, chances are high that you intend to
+use it with your own packages. There are three options available:
+
+1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-20.09` or
+   `nixos-unstable`.
+2. Specify your own git-repository with a custom package set[^2]. This makes it
+   possible to pull different tags, branches or commits by modifying the image
+   tag.
+3. Specify a local file path containing a Nix package set. Where this comes from
+   or what it contains is up to you.
+
+## 2. Build Nixery itself
+
+### 2.1. With a container image
+
+The easiest way to run Nixery is to build a container image. This
+section assumes that the container runtime used is Docker, please
+modify instructions accordingly if you are using something else.
+
+With a working Nix installation, you can clone and build the Nixery
+image like this:
+
+```
+git clone https://code.tvl.fyi/depot.git:/tools/nixery.git
+nix-build -A nixery-image
+```
+
+This will create a `result`-symlink which points to a tarball containing the
+image. In Docker, this tarball can be loaded by using `docker load -i result`.
+
+### 2.2. Without a container image
+
+*This method might be more convenient if you intend to work on
+the code of the Nixery server itself, because you won't have to
+rebuild (and reload) an image each time to test your changes.*
+
+You will need to run the two following commands at the root of the repo:
+
+* `go build` to build the `nixery` binary;
+* `nix-env --install --file prepare-image/default.nix` to build
+  the required helpers.
+
+## 3. Prepare configuration
+
+Nixery is configured via environment variables.
+
+You must set *all* of these:
+
+* `NIXERY_STORAGE_BACKEND` (must be set to `gcs` or `filesystem`)
+* `PORT`: HTTP port on which Nixery should listen
+* `WEB_DIR`: directory containing static files (see below)
+
+You must set *one* of these:
+
+* `NIXERY_CHANNEL`: The name of a [Nix/NixOS channel][nixchannel] to use for building,
+  for instance `nixos-21.05`
+* `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses
+  locally configured SSH/git credentials)
+* `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use
+  for building
+
+If `NIXERY_STORAGE_BACKEND` is set to `filesystem`, then `STORAGE_PATH`
+must be set to the directory that will hold the registry blobs.
+That directory must be located on a filesystem that supports extended
+attributes (which means that on most systems, `/tmp` won't work).
+
+If `NIXERY_STORAGE_BACKEND` is set to `gcs`, then `GCS_BUCKET`
+must be set to the [Google Cloud Storage][gcs] bucket that will be
+used to store & serve image layers.
+
+You may set *all* of these:
+
+* `NIX_TIMEOUT`: Number of seconds that any Nix builder is allowed to run
+  (defaults to 60)
+
+To authenticate to the configured GCS bucket, Nixery uses Google's [Application
+Default Credentials][ADC]. Depending on your environment this may require
+additional configuration.
+
+If the `GOOGLE_APPLICATION_CREDENTIALS` environment is configured, the service
+account's private key will be used to create [signed URLs for
+layers][signed-urls].
+
+## 4. Start Nixery
+
+Run the image that was built in step 2.1 with all the environment variables
+mentioned above. Alternatively, set all the environment variables and run
+the Nixery server that was built in step 2.2.
+
+Once Nixery is running you can immediately start requesting images from it.
+
+## 5. Productionise
+
+(⚠ Here be dragons! ⚠)
+
+Nixery is still an early project and has not yet been deployed in any production
+environments and some caveats apply.
+
+Notably, Nixery currently does not support any authentication methods, so anyone
+with network access to the registry can retrieve images.
+
+Running a Nixery inside of a fenced-off environment (such as internal to a
+Kubernetes cluster) should be fine, but you should consider to do all of the
+following:
+
+* Issue a TLS certificate for the hostname you are assigning to Nixery. In fact,
+  Docker will refuse to pull images from registries that do not use TLS (with
+  the exception of `.local` domains).
+* Configure signed GCS URLs to avoid having to make your bucket world-readable.
+* Configure request timeouts for Nixery if you have your own web server in front
+  of it. This will be natively supported by Nixery in the future.
+
+## 6. `WEB_DIR`
+
+All the URLs accessed by Docker registry clients start with `/v2/`.
+This means that it is possible to serve a static website from Nixery
+itself (as long as you don't want to serve anything starting with `/v2`).
+This is how, for instance, https://nixery.dev shows the website for Nixery,
+while it is also possible to e.g. `docker pull nixery.dev/shell`.
+
+When running Nixery, you must set the `WEB_DIR` environment variable.
+When Nixery receives requests that don't look like registry requests,
+it tries to serve them using files in the directory indicated by `WEB_DIR`.
+If the directory doesn't exist, Nixery will run fine but serve 404.
+
+-------
+
+[^1]: Nixery will not work with Nix channels older than `nixos-19.03`.
+
+[^2]: This documentation will be updated with instructions on how to best set up
+    a custom Nix repository. Nixery expects custom package sets to be a superset
+    of `nixpkgs`, as it uses `lib` and other features from `nixpkgs`
+    extensively.
+
+[GKE]: https://cloud.google.com/kubernetes-engine/
+[nixery#4]: https://github.com/tazjin/nixery/issues/4
+[Nix]: https://nixos.org/nix
+[gcs]: https://cloud.google.com/storage/
+[signed-urls]: under-the-hood.html#5-image-layers-are-requested
+[ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
+[nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html
+[nixchannel]: https://nixos.wiki/wiki/Nix_channels
diff --git a/tools/nixery/docs/src/under-the-hood.md b/tools/nixery/docs/src/under-the-hood.md
new file mode 100644
index 000000000000..4b798300100b
--- /dev/null
+++ b/tools/nixery/docs/src/under-the-hood.md
@@ -0,0 +1,129 @@
+# Under the hood
+
+This page serves as a quick explanation of what happens under-the-hood when an
+image is requested from Nixery.
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+
+- [1. The image manifest is requested](#1-the-image-manifest-is-requested)
+- [2. Nix fetches and prepares image content](#2-nix-fetches-and-prepares-image-content)
+- [3. Layers are grouped, created, hashed, and persisted](#3-layers-are-grouped-created-hashed-and-persisted)
+- [4. The manifest is assembled and returned to the client](#4-the-manifest-is-assembled-and-returned-to-the-client)
+- [5. Image layers are requested](#5-image-layers-are-requested)
+
+<!-- markdown-toc end -->
+
+--------
+
+## 1. The image manifest is requested
+
+When container registry clients such as Docker pull an image, the first thing
+they do is ask for the image manifest. This is a JSON document describing which
+layers are contained in an image, as well as some additional auxiliary
+information.
+
+This request is of the form `GET /v2/$imageName/manifests/$imageTag`.
+
+Nixery receives this request and begins by splitting the image name into its
+path components and substituting meta-packages (such as `shell`) for their
+contents.
+
+For example, requesting `shell/htop/git` results in Nixery expanding the image
+name to `["bashInteractive", "coreutils", "htop", "git"]`.
+
+If Nixery is configured with a private Nix repository, it also looks at the
+image tag and substitutes `latest` with `master`.
+
+It then invokes Nix with three parameters:
+
+1. image contents (as above)
+2. image tag
+3. configured package set source
+
+## 2. Nix fetches and prepares image content
+
+Using the parameters above, Nix imports the package set and begins by mapping
+the image names to attributes in the package set.
+
+A special case during this process is packages with uppercase characters in
+their name, for example anything under `haskellPackages`. The registry protocol
+does not allow uppercase characters, so the Nix code will translate something
+like `haskellpackages` (lowercased) to the correct attribute name.
+
+After identifying all contents, Nix uses the `symlinkJoin` function to
+create a special layer with the "symlink farm" required to let the
+image function like a normal disk image.
+
+Nix then returns information about the image contents as well as the
+location of the special layer to Nixery.
+
+## 3. Layers are grouped, created, hashed, and persisted
+
+With the information received from Nix, Nixery determines the contents
+of each layer while optimising for the best possible cache efficiency
+(see the [layering design doc][] for details).
+
+With the grouped layers, Nixery then begins to create compressed
+tarballs with all required contents for each layer. As these tarballs
+are being created, they are simultaneously being hashed (as the image
+manifest must contain the content-hashes of all layers) and persisted
+to storage.
+
+Storage can be either a remote [Google Cloud Storage][gcs] bucket, or
+a local filesystem path.
+
+During this step, Nixery checks its build cache (see [Caching][]) to
+determine whether a layer needs to be built or is already cached from
+a previous build.
+
+*Note:* While this step is running (which can take some time in the case of
+large first-time image builds), the registry client is left hanging waiting for
+an HTTP response. Unfortunately the registry protocol does not allow for any
+feedback back to the user at this point, so from the user's perspective things
+just ... hang, for a moment.
+
+## 4. The manifest is assembled and returned to the client
+
+Once armed with the hashes of all required layers, Nixery assembles
+the OCI Container Image manifest which describes the structure of the
+built image and names all of its layers by their content hash.
+
+This manifest is returned to the client.
+
+## 5. Image layers are requested
+
+The client now inspects the manifest and determines which of the
+layers it is currently missing based on their content hashes. Note
+that different container runtimes will handle this differently, and in
+the case of certain engine and storage driver combinations (e.g.
+Docker with OverlayFS) layers might be downloaded again even if they
+are already present.
+
+For each of the missing layers, the client now issues a request to
+Nixery that looks like this:
+
+`GET /v2/${imageName}/blob/sha256:${layerHash}`
+
+Nixery receives these requests and handles them based on the
+configured storage backend.
+
+If the storage backend is GCS, it *redirects* them to Google Cloud
+Storage URLs, responding with an `HTTP 303 See Other` status code and
+the actual download URL of the layer.
+
+Nixery supports using private buckets which are not generally world-readable, in
+which case [signed URLs][] are constructed using a private key. These allow the
+registry client to download each layer without needing to care about how the
+underlying authentication works.
+
+If the storage backend is the local filesystem, Nixery will attempt to
+serve the layer back to the client from disk.
+
+---------
+
+That's it. After these five steps the registry client has retrieved all it needs
+to run the image produced by Nixery.
+
+[gcs]: https://cloud.google.com/storage/
+[signed URLs]: https://cloud.google.com/storage/docs/access-control/signed-urls
+[layering design doc]: https://storage.googleapis.com/nixdoc/nixery-layers.html