about summary refs log tree commit diff
path: root/tools/nixery/docs/src
diff options
context:
space:
mode:
Diffstat (limited to 'tools/nixery/docs/src')
-rw-r--r--tools/nixery/docs/src/SUMMARY.md8
-rw-r--r--tools/nixery/docs/src/caching.md69
-rw-r--r--tools/nixery/docs/src/nix-1p.md2
-rw-r--r--tools/nixery/docs/src/nix.md31
-rw-r--r--tools/nixery/docs/src/nixery-logo.pngbin194098 -> 0 bytes
-rw-r--r--tools/nixery/docs/src/nixery.md72
-rw-r--r--tools/nixery/docs/src/run-your-own.md194
-rw-r--r--tools/nixery/docs/src/under-the-hood.md129
8 files changed, 0 insertions, 505 deletions
diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md
deleted file mode 100644
index f1d68a3ac451..000000000000
--- a/tools/nixery/docs/src/SUMMARY.md
+++ /dev/null
@@ -1,8 +0,0 @@
-# Summary
-
-- [Nixery](./nixery.md)
-  - [Under the hood](./under-the-hood.md)
-  - [Caching](./caching.md)
-  - [Run your own Nixery](./run-your-own.md)
-- [Nix](./nix.md)
-  - [Nix, the language](./nix-1p.md)
diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md
deleted file mode 100644
index 05ea68ef6083..000000000000
--- a/tools/nixery/docs/src/caching.md
+++ /dev/null
@@ -1,69 +0,0 @@
-# Caching in Nixery
-
-This page gives a quick overview over the caching done by Nixery. All cache data
-is written to Nixery's storage bucket and is based on deterministic identifiers
-or content-addressing, meaning that cache entries under the same key *never
-change*.
-
-## Manifests
-
-Manifests of builds are cached at `$BUCKET/manifests/$KEY`. The effect of this
-cache is that multiple instances of Nixery do not need to rebuild the same
-manifest from scratch.
-
-Since the manifest cache is populated only *after* layers are uploaded, Nixery
-can immediately return the manifest to its clients without needing to check
-whether layers have been uploaded already.
-
-`$KEY` is generated by creating a SHA1 hash of the requested content of a
-manifest plus the package source specification.
-
-Manifests are *only* cached if the package source specification is *not* a
-moving target.
-
-Manifest caching *only* applies in the following cases:
-
-* package source specification is a specific git commit
-* package source specification is a specific NixOS/nixpkgs commit
-
-Manifest caching *never* applies in the following cases:
-
-* package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`)
-* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-20.09`)
-* package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`)
-
-It is thus always preferable to request images from a fully-pinned package
-source.
-
-Manifests can be removed from the manifest cache without negative consequences.
-
-## Layer tarballs
-
-Layer tarballs are the files that Nixery clients retrieve from the storage
-bucket to download an image.
-
-They are stored content-addressably at `$BUCKET/layers/$SHA256HASH` and layer
-requests sent to Nixery will redirect directly to this storage location.
-
-The effect of this cache is that Nixery does not need to upload identical layers
-repeatedly. When Nixery notices that a layer already exists in GCS it will skip
-uploading this layer.
-
-Removing layers from the cache is *potentially problematic* if there are cached
-manifests or layer builds referencing those layers.
-
-To clean up layers, a user must ensure that no other cached resources still
-reference these layers.
-
-## Layer builds
-
-Layer builds are cached at `$BUCKET/builds/$HASH`, where `$HASH` is a SHA1 of
-the Nix store paths included in the layer.
-
-The content of the cached entries is a JSON-object that contains the SHA256
-hashes and sizes of the built layer.
-
-The effect of this cache is that different instances of Nixery will not build,
-hash and upload layers that have identical contents across different instances.
-
-Layer builds can be removed from the cache without negative consequences.
diff --git a/tools/nixery/docs/src/nix-1p.md b/tools/nixery/docs/src/nix-1p.md
deleted file mode 100644
index a21234150fc7..000000000000
--- a/tools/nixery/docs/src/nix-1p.md
+++ /dev/null
@@ -1,2 +0,0 @@
-This page is a placeholder. During the build process, it is replaced by the
-actual `nix-1p` guide from https://github.com/tazjin/nix-1p
diff --git a/tools/nixery/docs/src/nix.md b/tools/nixery/docs/src/nix.md
deleted file mode 100644
index 2bfd75a6925c..000000000000
--- a/tools/nixery/docs/src/nix.md
+++ /dev/null
@@ -1,31 +0,0 @@
-# Nix
-
-These sections are designed to give some background information on what Nix is.
-If you've never heard of Nix before looking at Nixery, this might just be the
-page for you!
-
-[Nix][] is a functional package-manager that comes with a number of advantages
-over traditional package managers, such as side-by-side installs of different
-package versions, atomic updates, easy customisability, simple binary caching
-and much more. Feel free to explore the [Nix website][Nix] for an overview of
-Nix itself.
-
-Nix uses a custom programming language also called Nix, which is explained here
-[on its own page][nix-1p].
-
-In addition to the package manager and language, the Nix project also maintains
-[NixOS][] - a Linux distribution built entirely on Nix. On NixOS, users can
-declaratively describe the *entire* configuration of their system and perform
-updates/rollbacks to other system configurations with ease.
-
-Most Nix packages are tracked in the [Nix package set][nixpkgs], usually simply
-referred to as `nixpkgs`. It contains tens of thousands of packages already!
-
-Nixery (which you are looking at!) provides an easy & simple way to get started
-with Nix, in fact you don't even need to know that you're using Nix to make use
-of Nixery.
-
-[Nix]: https://nixos.org/nix/
-[nix-1p]: nix-1p.html
-[NixOS]: https://nixos.org/
-[nixpkgs]: https://github.com/nixos/nixpkgs
diff --git a/tools/nixery/docs/src/nixery-logo.png b/tools/nixery/docs/src/nixery-logo.png
deleted file mode 100644
index fcf77df3d6a9..000000000000
--- a/tools/nixery/docs/src/nixery-logo.png
+++ /dev/null
Binary files differdiff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
deleted file mode 100644
index 0d55cfb545d9..000000000000
--- a/tools/nixery/docs/src/nixery.md
+++ /dev/null
@@ -1,72 +0,0 @@
-![Nixery](./nixery-logo.png)
-
-------------
-
-Welcome to this instance of [Nixery][]. It provides ad-hoc container images that
-contain packages from the [Nix][] package manager. Images with arbitrary
-packages can be requested via the image name.
-
-Nix not only provides the packages to include in the images, but also builds the
-images themselves by using a special [layering strategy][] that optimises for
-cache efficiency.
-
-For general information on why using Nix makes sense for container images, check
-out [this blog post][layers].
-
-## Demo
-
-<script src="https://asciinema.org/a/262583.js" id="asciicast-262583" async data-autoplay="true" data-loop="true"></script>
-
-## Quick start
-
-Simply pull an image from this registry, separating each package you want
-included by a slash:
-
-    docker pull nixery.dev/shell/git/htop
-
-This gives you an image with `git`, `htop` and an interactively configured
-shell. You could run it like this:
-
-    docker run -ti nixery.dev/shell/git/htop bash
-
-Each path segment corresponds either to a key in the Nix package set, or a
-meta-package that automatically expands to several other packages.
-
-Meta-packages **must** be the first path component if they are used. Currently
-there are only two meta-packages:
-- `shell`, which provides a `bash`-shell with interactive configuration and
-  standard tools like `coreutils`.
-- `arm64`, which provides ARM64 binaries.
-
-**Tip:** When pulling from a private Nixery instance, replace `nixery.dev` in
-the above examples with your registry address.
-
-## FAQ
-
-If you have a question that is not answered here, feel free to file an issue on
-Github so that we can get it included in this section. The volume of questions
-is quite low, thus by definition your question is already frequently asked.
-
-### Where is the source code for this?
-
-Over [on Github][Nixery]. It is licensed under the Apache 2.0 license. Consult
-the documentation entries in the sidebar for information on how to set up your
-own instance of Nixery.
-
-### Should I depend on `nixery.dev` in production?
-
-While we appreciate the enthusiasm, if you would like to use Nixery in your
-production project we recommend setting up a private instance. The public Nixery
-at `nixery.dev` is run on a best-effort basis and we make no guarantees about
-availability.
-
-### Who made this?
-
-Nixery was written by [tazjin][], but many people have contributed to Nix over
-time, maybe you could become one of them?
-
-[Nixery]: https://github.com/tazjin/nixery
-[Nix]: https://nixos.org/nix
-[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
-[layers]: https://grahamc.com/blog/nix-and-layered-docker-images
-[tazjin]: https://tazj.in
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
deleted file mode 100644
index 7ed8bdd0bc0a..000000000000
--- a/tools/nixery/docs/src/run-your-own.md
+++ /dev/null
@@ -1,194 +0,0 @@
-## Run your own Nixery
-
-<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
-
-- [0. Prerequisites](#0-prerequisites)
-- [1. Choose a package set](#1-choose-a-package-set)
-- [2. Build Nixery itself](#2-build-nixery-itself)
-- [3. Prepare configuration](#3-prepare-configuration)
-- [4. Deploy Nixery](#4-deploy-nixery)
-- [5. Productionise](#5-productionise)
-
-<!-- markdown-toc end -->
-
-
----------
-
-⚠ This page is still under construction! ⚠
-
---------
-
-Running your own Nixery is not difficult, but requires some setup. Follow the
-steps below to get up & running.
-
-*Note:* Nixery can be run inside of a [GKE][] cluster, providing a local service
-from which images can be requested. Documentation for how to set this up is
-forthcoming, please see [nixery#4][].
-
-## 0. Prerequisites
-
-To run Nixery, you must have:
-
-* [Nix][] (to build Nixery itself)
-* Somewhere to run it (your own server, Google AppEngine, a Kubernetes cluster,
-  whatever!)
-* *Either* a [Google Cloud Storage][gcs] bucket in which to store & serve layers,
-  *or* a comfortable amount of disk space
-
-Note that while the main Nixery process is a server written in Go,
-it invokes a script that itself relies on Nix to be available.
-You can compile the main Nixery daemon without Nix, but it won't
-work without Nix.
-
-(If you are completely new to Nix and don't know how to get
-started, check the [Nix installation documentation][nixinstall].)
-
-## 1. Choose a package set
-
-When running your own Nixery you need to decide which package set you want to
-serve. By default, Nixery builds packages from a recent NixOS channel which
-ensures that most packages are cached upstream and no expensive builds need to
-be performed for trivial things.
-
-However if you are running a private Nixery, chances are high that you intend to
-use it with your own packages. There are three options available:
-
-1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-20.09` or
-   `nixos-unstable`.
-2. Specify your own git-repository with a custom package set[^2]. This makes it
-   possible to pull different tags, branches or commits by modifying the image
-   tag.
-3. Specify a local file path containing a Nix package set. Where this comes from
-   or what it contains is up to you.
-
-## 2. Build Nixery itself
-
-### 2.1. With a container image
-
-The easiest way to run Nixery is to build a container image. This
-section assumes that the container runtime used is Docker, please
-modify instructions accordingly if you are using something else.
-
-With a working Nix installation, you can clone and build the Nixery
-image like this:
-
-```
-git clone https://code.tvl.fyi/depot.git:/tools/nixery.git
-nix-build -A nixery-image
-```
-
-This will create a `result`-symlink which points to a tarball containing the
-image. In Docker, this tarball can be loaded by using `docker load -i result`.
-
-### 2.2. Without a container image
-
-*This method might be more convenient if you intend to work on
-the code of the Nixery server itself, because you won't have to
-rebuild (and reload) an image each time to test your changes.*
-
-You will need to run the two following commands at the root of the repo:
-
-* `go build` to build the `nixery` binary;
-* `nix-env --install --file prepare-image/default.nix` to build
-  the required helpers.
-
-## 3. Prepare configuration
-
-Nixery is configured via environment variables.
-
-You must set *all* of these:
-
-* `NIXERY_STORAGE_BACKEND` (must be set to `gcs` or `filesystem`)
-* `PORT`: HTTP port on which Nixery should listen
-* `WEB_DIR`: directory containing static files (see below)
-
-You must set *one* of these:
-
-* `NIXERY_CHANNEL`: The name of a [Nix/NixOS channel][nixchannel] to use for building,
-  for instance `nixos-21.05`
-* `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses
-  locally configured SSH/git credentials)
-* `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use
-  for building
-
-If `NIXERY_STORAGE_BACKEND` is set to `filesystem`, then `STORAGE_PATH`
-must be set to the directory that will hold the registry blobs.
-That directory must be located on a filesystem that supports extended
-attributes (which means that on most systems, `/tmp` won't work).
-
-If `NIXERY_STORAGE_BACKEND` is set to `gcs`, then `GCS_BUCKET`
-must be set to the [Google Cloud Storage][gcs] bucket that will be
-used to store & serve image layers.
-
-You may set *all* of these:
-
-* `NIX_TIMEOUT`: Number of seconds that any Nix builder is allowed to run
-  (defaults to 60)
-
-To authenticate to the configured GCS bucket, Nixery uses Google's [Application
-Default Credentials][ADC]. Depending on your environment this may require
-additional configuration.
-
-If the `GOOGLE_APPLICATION_CREDENTIALS` environment is configured, the service
-account's private key will be used to create [signed URLs for
-layers][signed-urls].
-
-## 4. Start Nixery
-
-Run the image that was built in step 2.1 with all the environment variables
-mentioned above. Alternatively, set all the environment variables and run
-the Nixery server that was built in step 2.2.
-
-Once Nixery is running you can immediately start requesting images from it.
-
-## 5. Productionise
-
-(⚠ Here be dragons! ⚠)
-
-Nixery is still an early project and has not yet been deployed in any production
-environments and some caveats apply.
-
-Notably, Nixery currently does not support any authentication methods, so anyone
-with network access to the registry can retrieve images.
-
-Running a Nixery inside of a fenced-off environment (such as internal to a
-Kubernetes cluster) should be fine, but you should consider to do all of the
-following:
-
-* Issue a TLS certificate for the hostname you are assigning to Nixery. In fact,
-  Docker will refuse to pull images from registries that do not use TLS (with
-  the exception of `.local` domains).
-* Configure signed GCS URLs to avoid having to make your bucket world-readable.
-* Configure request timeouts for Nixery if you have your own web server in front
-  of it. This will be natively supported by Nixery in the future.
-
-## 6. `WEB_DIR`
-
-All the URLs accessed by Docker registry clients start with `/v2/`.
-This means that it is possible to serve a static website from Nixery
-itself (as long as you don't want to serve anything starting with `/v2`).
-This is how, for instance, https://nixery.dev shows the website for Nixery,
-while it is also possible to e.g. `docker pull nixery.dev/shell`.
-
-When running Nixery, you must set the `WEB_DIR` environment variable.
-When Nixery receives requests that don't look like registry requests,
-it tries to serve them using files in the directory indicated by `WEB_DIR`.
-If the directory doesn't exist, Nixery will run fine but serve 404.
-
--------
-
-[^1]: Nixery will not work with Nix channels older than `nixos-19.03`.
-
-[^2]: This documentation will be updated with instructions on how to best set up
-    a custom Nix repository. Nixery expects custom package sets to be a superset
-    of `nixpkgs`, as it uses `lib` and other features from `nixpkgs`
-    extensively.
-
-[GKE]: https://cloud.google.com/kubernetes-engine/
-[nixery#4]: https://github.com/tazjin/nixery/issues/4
-[Nix]: https://nixos.org/nix
-[gcs]: https://cloud.google.com/storage/
-[signed-urls]: under-the-hood.html#5-image-layers-are-requested
-[ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
-[nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html
-[nixchannel]: https://nixos.wiki/wiki/Nix_channels
diff --git a/tools/nixery/docs/src/under-the-hood.md b/tools/nixery/docs/src/under-the-hood.md
deleted file mode 100644
index 4b798300100b..000000000000
--- a/tools/nixery/docs/src/under-the-hood.md
+++ /dev/null
@@ -1,129 +0,0 @@
-# Under the hood
-
-This page serves as a quick explanation of what happens under-the-hood when an
-image is requested from Nixery.
-
-<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
-
-- [1. The image manifest is requested](#1-the-image-manifest-is-requested)
-- [2. Nix fetches and prepares image content](#2-nix-fetches-and-prepares-image-content)
-- [3. Layers are grouped, created, hashed, and persisted](#3-layers-are-grouped-created-hashed-and-persisted)
-- [4. The manifest is assembled and returned to the client](#4-the-manifest-is-assembled-and-returned-to-the-client)
-- [5. Image layers are requested](#5-image-layers-are-requested)
-
-<!-- markdown-toc end -->
-
---------
-
-## 1. The image manifest is requested
-
-When container registry clients such as Docker pull an image, the first thing
-they do is ask for the image manifest. This is a JSON document describing which
-layers are contained in an image, as well as some additional auxiliary
-information.
-
-This request is of the form `GET /v2/$imageName/manifests/$imageTag`.
-
-Nixery receives this request and begins by splitting the image name into its
-path components and substituting meta-packages (such as `shell`) for their
-contents.
-
-For example, requesting `shell/htop/git` results in Nixery expanding the image
-name to `["bashInteractive", "coreutils", "htop", "git"]`.
-
-If Nixery is configured with a private Nix repository, it also looks at the
-image tag and substitutes `latest` with `master`.
-
-It then invokes Nix with three parameters:
-
-1. image contents (as above)
-2. image tag
-3. configured package set source
-
-## 2. Nix fetches and prepares image content
-
-Using the parameters above, Nix imports the package set and begins by mapping
-the image names to attributes in the package set.
-
-A special case during this process is packages with uppercase characters in
-their name, for example anything under `haskellPackages`. The registry protocol
-does not allow uppercase characters, so the Nix code will translate something
-like `haskellpackages` (lowercased) to the correct attribute name.
-
-After identifying all contents, Nix uses the `symlinkJoin` function to
-create a special layer with the "symlink farm" required to let the
-image function like a normal disk image.
-
-Nix then returns information about the image contents as well as the
-location of the special layer to Nixery.
-
-## 3. Layers are grouped, created, hashed, and persisted
-
-With the information received from Nix, Nixery determines the contents
-of each layer while optimising for the best possible cache efficiency
-(see the [layering design doc][] for details).
-
-With the grouped layers, Nixery then begins to create compressed
-tarballs with all required contents for each layer. As these tarballs
-are being created, they are simultaneously being hashed (as the image
-manifest must contain the content-hashes of all layers) and persisted
-to storage.
-
-Storage can be either a remote [Google Cloud Storage][gcs] bucket, or
-a local filesystem path.
-
-During this step, Nixery checks its build cache (see [Caching][]) to
-determine whether a layer needs to be built or is already cached from
-a previous build.
-
-*Note:* While this step is running (which can take some time in the case of
-large first-time image builds), the registry client is left hanging waiting for
-an HTTP response. Unfortunately the registry protocol does not allow for any
-feedback back to the user at this point, so from the user's perspective things
-just ... hang, for a moment.
-
-## 4. The manifest is assembled and returned to the client
-
-Once armed with the hashes of all required layers, Nixery assembles
-the OCI Container Image manifest which describes the structure of the
-built image and names all of its layers by their content hash.
-
-This manifest is returned to the client.
-
-## 5. Image layers are requested
-
-The client now inspects the manifest and determines which of the
-layers it is currently missing based on their content hashes. Note
-that different container runtimes will handle this differently, and in
-the case of certain engine and storage driver combinations (e.g.
-Docker with OverlayFS) layers might be downloaded again even if they
-are already present.
-
-For each of the missing layers, the client now issues a request to
-Nixery that looks like this:
-
-`GET /v2/${imageName}/blob/sha256:${layerHash}`
-
-Nixery receives these requests and handles them based on the
-configured storage backend.
-
-If the storage backend is GCS, it *redirects* them to Google Cloud
-Storage URLs, responding with an `HTTP 303 See Other` status code and
-the actual download URL of the layer.
-
-Nixery supports using private buckets which are not generally world-readable, in
-which case [signed URLs][] are constructed using a private key. These allow the
-registry client to download each layer without needing to care about how the
-underlying authentication works.
-
-If the storage backend is the local filesystem, Nixery will attempt to
-serve the layer back to the client from disk.
-
----------
-
-That's it. After these five steps the registry client has retrieved all it needs
-to run the image produced by Nixery.
-
-[gcs]: https://cloud.google.com/storage/
-[signed URLs]: https://cloud.google.com/storage/docs/access-control/signed-urls
-[layering design doc]: https://storage.googleapis.com/nixdoc/nixery-layers.html