diff options
Diffstat (limited to 'tools/nixery/docs/src/run-your-own.md')
| -rw-r--r-- | tools/nixery/docs/src/run-your-own.md | 194 |
1 files changed, 194 insertions, 0 deletions
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md new file mode 100644 index 000000000000..7ed8bdd0bc0a --- /dev/null +++ b/tools/nixery/docs/src/run-your-own.md @@ -0,0 +1,194 @@ +## Run your own Nixery + +<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc --> + +- [0. Prerequisites](#0-prerequisites) +- [1. Choose a package set](#1-choose-a-package-set) +- [2. Build Nixery itself](#2-build-nixery-itself) +- [3. Prepare configuration](#3-prepare-configuration) +- [4. Deploy Nixery](#4-deploy-nixery) +- [5. Productionise](#5-productionise) + +<!-- markdown-toc end --> + + +--------- + +⚠ This page is still under construction! ⚠ + +-------- + +Running your own Nixery is not difficult, but requires some setup. Follow the +steps below to get up & running. + +*Note:* Nixery can be run inside of a [GKE][] cluster, providing a local service +from which images can be requested. Documentation for how to set this up is +forthcoming, please see [nixery#4][]. + +## 0. Prerequisites + +To run Nixery, you must have: + +* [Nix][] (to build Nixery itself) +* Somewhere to run it (your own server, Google AppEngine, a Kubernetes cluster, + whatever!) +* *Either* a [Google Cloud Storage][gcs] bucket in which to store & serve layers, + *or* a comfortable amount of disk space + +Note that while the main Nixery process is a server written in Go, +it invokes a script that itself relies on Nix to be available. +You can compile the main Nixery daemon without Nix, but it won't +work without Nix. + +(If you are completely new to Nix and don't know how to get +started, check the [Nix installation documentation][nixinstall].) + +## 1. Choose a package set + +When running your own Nixery you need to decide which package set you want to +serve. By default, Nixery builds packages from a recent NixOS channel which +ensures that most packages are cached upstream and no expensive builds need to +be performed for trivial things. + +However if you are running a private Nixery, chances are high that you intend to +use it with your own packages. There are three options available: + +1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-20.09` or + `nixos-unstable`. +2. Specify your own git-repository with a custom package set[^2]. This makes it + possible to pull different tags, branches or commits by modifying the image + tag. +3. Specify a local file path containing a Nix package set. Where this comes from + or what it contains is up to you. + +## 2. Build Nixery itself + +### 2.1. With a container image + +The easiest way to run Nixery is to build a container image. This +section assumes that the container runtime used is Docker, please +modify instructions accordingly if you are using something else. + +With a working Nix installation, you can clone and build the Nixery +image like this: + +``` +git clone https://code.tvl.fyi/depot.git:/tools/nixery.git +nix-build -A nixery-image +``` + +This will create a `result`-symlink which points to a tarball containing the +image. In Docker, this tarball can be loaded by using `docker load -i result`. + +### 2.2. Without a container image + +*This method might be more convenient if you intend to work on +the code of the Nixery server itself, because you won't have to +rebuild (and reload) an image each time to test your changes.* + +You will need to run the two following commands at the root of the repo: + +* `go build` to build the `nixery` binary; +* `nix-env --install --file prepare-image/default.nix` to build + the required helpers. + +## 3. Prepare configuration + +Nixery is configured via environment variables. + +You must set *all* of these: + +* `NIXERY_STORAGE_BACKEND` (must be set to `gcs` or `filesystem`) +* `PORT`: HTTP port on which Nixery should listen +* `WEB_DIR`: directory containing static files (see below) + +You must set *one* of these: + +* `NIXERY_CHANNEL`: The name of a [Nix/NixOS channel][nixchannel] to use for building, + for instance `nixos-21.05` +* `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses + locally configured SSH/git credentials) +* `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use + for building + +If `NIXERY_STORAGE_BACKEND` is set to `filesystem`, then `STORAGE_PATH` +must be set to the directory that will hold the registry blobs. +That directory must be located on a filesystem that supports extended +attributes (which means that on most systems, `/tmp` won't work). + +If `NIXERY_STORAGE_BACKEND` is set to `gcs`, then `GCS_BUCKET` +must be set to the [Google Cloud Storage][gcs] bucket that will be +used to store & serve image layers. + +You may set *all* of these: + +* `NIX_TIMEOUT`: Number of seconds that any Nix builder is allowed to run + (defaults to 60) + +To authenticate to the configured GCS bucket, Nixery uses Google's [Application +Default Credentials][ADC]. Depending on your environment this may require +additional configuration. + +If the `GOOGLE_APPLICATION_CREDENTIALS` environment is configured, the service +account's private key will be used to create [signed URLs for +layers][signed-urls]. + +## 4. Start Nixery + +Run the image that was built in step 2.1 with all the environment variables +mentioned above. Alternatively, set all the environment variables and run +the Nixery server that was built in step 2.2. + +Once Nixery is running you can immediately start requesting images from it. + +## 5. Productionise + +(⚠ Here be dragons! ⚠) + +Nixery is still an early project and has not yet been deployed in any production +environments and some caveats apply. + +Notably, Nixery currently does not support any authentication methods, so anyone +with network access to the registry can retrieve images. + +Running a Nixery inside of a fenced-off environment (such as internal to a +Kubernetes cluster) should be fine, but you should consider to do all of the +following: + +* Issue a TLS certificate for the hostname you are assigning to Nixery. In fact, + Docker will refuse to pull images from registries that do not use TLS (with + the exception of `.local` domains). +* Configure signed GCS URLs to avoid having to make your bucket world-readable. +* Configure request timeouts for Nixery if you have your own web server in front + of it. This will be natively supported by Nixery in the future. + +## 6. `WEB_DIR` + +All the URLs accessed by Docker registry clients start with `/v2/`. +This means that it is possible to serve a static website from Nixery +itself (as long as you don't want to serve anything starting with `/v2`). +This is how, for instance, https://nixery.dev shows the website for Nixery, +while it is also possible to e.g. `docker pull nixery.dev/shell`. + +When running Nixery, you must set the `WEB_DIR` environment variable. +When Nixery receives requests that don't look like registry requests, +it tries to serve them using files in the directory indicated by `WEB_DIR`. +If the directory doesn't exist, Nixery will run fine but serve 404. + +------- + +[^1]: Nixery will not work with Nix channels older than `nixos-19.03`. + +[^2]: This documentation will be updated with instructions on how to best set up + a custom Nix repository. Nixery expects custom package sets to be a superset + of `nixpkgs`, as it uses `lib` and other features from `nixpkgs` + extensively. + +[GKE]: https://cloud.google.com/kubernetes-engine/ +[nixery#4]: https://github.com/tazjin/nixery/issues/4 +[Nix]: https://nixos.org/nix +[gcs]: https://cloud.google.com/storage/ +[signed-urls]: under-the-hood.html#5-image-layers-are-requested +[ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically +[nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html +[nixchannel]: https://nixos.wiki/wiki/Nix_channels |