From 7c41a7a8723c8bd6606c0c2f3fed3a546f1efb24 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Sun, 4 Aug 2019 22:38:51 +0100
Subject: docs: Replace static page with mdBook site

Uses mdBook[1] to generate a documentation overview page instead of
the previous HTML site.

This makes it possible to add more elaborate documentation without
having to deal with finicky markup.

[1]: https://github.com/rust-lang-nursery/mdBook
---
 tools/nixery/docs/.gitignore          |   1 +
 tools/nixery/docs/book.toml           |   8 ++++
 tools/nixery/docs/src/SUMMARY.md      |   4 ++
 tools/nixery/docs/src/nix-1p.md       |   2 +
 tools/nixery/docs/src/nixery-logo.png | Bin 0 -> 194098 bytes
 tools/nixery/docs/src/nixery.md       |  77 ++++++++++++++++++++++++++++++++++
 tools/nixery/docs/theme/favicon.png   | Bin 0 -> 16053 bytes
 tools/nixery/docs/theme/nixery.css    |   3 ++
 8 files changed, 95 insertions(+)
 create mode 100644 tools/nixery/docs/.gitignore
 create mode 100644 tools/nixery/docs/book.toml
 create mode 100644 tools/nixery/docs/src/SUMMARY.md
 create mode 100644 tools/nixery/docs/src/nix-1p.md
 create mode 100644 tools/nixery/docs/src/nixery-logo.png
 create mode 100644 tools/nixery/docs/src/nixery.md
 create mode 100644 tools/nixery/docs/theme/favicon.png
 create mode 100644 tools/nixery/docs/theme/nixery.css

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/.gitignore b/tools/nixery/docs/.gitignore
new file mode 100644
index 000000000000..7585238efedf
--- /dev/null
+++ b/tools/nixery/docs/.gitignore
@@ -0,0 +1 @@
+book
diff --git a/tools/nixery/docs/book.toml b/tools/nixery/docs/book.toml
new file mode 100644
index 000000000000..bf6ccbb27f35
--- /dev/null
+++ b/tools/nixery/docs/book.toml
@@ -0,0 +1,8 @@
+[book]
+authors = ["Vincent Ambo <tazjin@google.com>"]
+language = "en"
+multilingual = false
+src = "src"
+
+[output.html]
+additional-css = ["theme/nixery.css"]
diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md
new file mode 100644
index 000000000000..5d680b82e8d9
--- /dev/null
+++ b/tools/nixery/docs/src/SUMMARY.md
@@ -0,0 +1,4 @@
+# Summary
+
+- [Nixery](./nixery.md)
+- [Nix, the language](./nix-1p.md)
diff --git a/tools/nixery/docs/src/nix-1p.md b/tools/nixery/docs/src/nix-1p.md
new file mode 100644
index 000000000000..a21234150fc7
--- /dev/null
+++ b/tools/nixery/docs/src/nix-1p.md
@@ -0,0 +1,2 @@
+This page is a placeholder. During the build process, it is replaced by the
+actual `nix-1p` guide from https://github.com/tazjin/nix-1p
diff --git a/tools/nixery/docs/src/nixery-logo.png b/tools/nixery/docs/src/nixery-logo.png
new file mode 100644
index 000000000000..fcf77df3d6a9
Binary files /dev/null and b/tools/nixery/docs/src/nixery-logo.png differ
diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
new file mode 100644
index 000000000000..d3d1911d2880
--- /dev/null
+++ b/tools/nixery/docs/src/nixery.md
@@ -0,0 +1,77 @@
+![Nixery](./nixery-logo.png)
+
+------------
+
+Welcome to this instance of [Nixery][]. It provides ad-hoc container images that
+contain packages from the [Nix][] package manager. Images with arbitrary
+packages can be requested via the image name.
+
+Nix not only provides the packages to include in the images, but also builds the
+images themselves by using an interesting layering strategy described in [this
+blog post][layers].
+
+## Quick start
+
+Simply pull an image from this registry, separating each package you want
+included by a slash:
+
+    docker pull nixery.dev/shell/git/htop
+
+This gives you an image with `git`, `htop` and an interactively configured
+shell. You could run it like this:
+
+    docker run -ti nixery.dev/shell/git/htop bash
+
+Each path segment corresponds either to a key in the Nix package set, or a
+meta-package that automatically expands to several other packages.
+
+Meta-packages **must** be the first path component if they are used. Currently
+the only meta-package is `shell`, which provides a `bash`-shell with interactive
+configuration and standard tools like `coreutils`.
+
+**Tip:** When pulling from a private Nixery instance, replace `nixery.dev` in
+the above examples with your registry address.
+
+## FAQ
+
+If you have a question that is not answered here, feel free to file an issue on
+Github so that we can get it included in this section. The volume of questions
+is quite low, thus by definition your question is already frequently asked.
+
+### Where is the source code for this?
+
+Over [on Github][Nixery]. It is licensed under the Apache 2.0 license. Consult
+the documentation entries in the sidebar for information on how to set up your
+own instance of Nixery.
+
+### Which revision of `nixpkgs` is used for the builds?
+
+The instance at `nixery.dev` tracks a recent NixOS channel, currently NixOS
+19.03. The channel is updated several times a day.
+
+Private registries might be configured to track a different channel (such as
+`nixos-unstable`) or even track a git repository with custom packages.
+
+### Is this an official Google project?
+
+**No.** Nixery is not officially supported by Google.
+
+### Should I depend on `nixery.dev` in production?
+
+While we appreciate the enthusiasm, if you would like to use Nixery in your
+production project we recommend setting up a private instance. The public Nixery
+at `nixery.dev` is run on a best-effort basis and we make no guarantees about
+availability.
+
+### Who made this?
+
+Nixery was written mostly by [tazjin][].
+
+[grahamc][] authored the image layering strategy. Many people have contributed
+to Nix over time, maybe you could become one of them?
+
+[Nixery]: https://github.com/google/nixery
+[Nix]: https://nixos.org/nix
+[layers]: https://grahamc.com/blog/nix-and-layered-docker-images
+[tazjin]: https://github.com/tazjin
+[grahamc]: https://github.com/grahamc
diff --git a/tools/nixery/docs/theme/favicon.png b/tools/nixery/docs/theme/favicon.png
new file mode 100644
index 000000000000..f510bde197ac
Binary files /dev/null and b/tools/nixery/docs/theme/favicon.png differ
diff --git a/tools/nixery/docs/theme/nixery.css b/tools/nixery/docs/theme/nixery.css
new file mode 100644
index 000000000000..c240e693d550
--- /dev/null
+++ b/tools/nixery/docs/theme/nixery.css
@@ -0,0 +1,3 @@
+h2, h3 {
+  margin-top: 1em;
+}
-- 
cgit 1.4.1


From 2bef0ba2403f20826fbd615b0abb91cb2aff0350 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Sun, 4 Aug 2019 22:45:23 +0100
Subject: feat(build): Build Nixery book and embed it into Nixery image

Executes the previously added mdBook on the previously added book
source to yield a directory that can be served by Nixery on its index
page.

This is one of those 'I <3 Nix' things due to how easy it is to do.
---
 tools/nixery/default.nix      | 18 ++++++++++--------
 tools/nixery/docs/default.nix | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 8 deletions(-)
 create mode 100644 tools/nixery/docs/default.nix

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/default.nix b/tools/nixery/default.nix
index 28b94af5bd59..4f0b14c90394 100644
--- a/tools/nixery/default.nix
+++ b/tools/nixery/default.nix
@@ -48,13 +48,6 @@ rec {
     cat ${./build-registry-image.nix} > $out
   '';
 
-  # Static files to serve on the Nixery index. This is used primarily
-  # for the demo instance running at nixery.dev and provides some
-  # background information for what Nixery is.
-  nixery-static = runCommand "nixery-static" {} ''
-    mkdir $out
-    cp ${./static}/* $out
-  '';
   # nixpkgs currently has an old version of mdBook. A new version is
   # built here, but eventually the update will be upstreamed
   # (nixpkgs#65890)
@@ -73,6 +66,10 @@ rec {
     cargoSha256 = "0qwhc42a86jpvjcaysmfcw8kmwa150lmz01flmlg74g6qnimff5m";
   };
 
+  # Use mdBook to build a static asset page which Nixery can then
+  # serve. This is primarily used for the public instance at
+  # nixery.dev.
+  nixery-book = callPackage ./docs { inherit mdbook; };
 
   # Wrapper script running the Nixery server with the above two data
   # dependencies configured.
@@ -81,7 +78,7 @@ rec {
   # are installing Nixery directly.
   nixery-bin = writeShellScriptBin "nixery" ''
     export NIX_BUILDER="${nixery-builder}"
-    export WEB_DIR="${nixery-static}"
+    export WEB_DIR="${nixery-book}"
     exec ${nixery-server}/bin/nixery
   '';
 
@@ -107,6 +104,11 @@ rec {
       mkdir -p /etc/nix
       echo 'sandbox = false' >> /etc/nix/nix.conf
 
+      # In some cases users building their own image might want to
+      # customise something on the inside (e.g. set up an environment
+      # for keys or whatever).
+      #
+      # This can be achieved by setting a 'preLaunch' script.
       ${preLaunch}
 
       exec ${nixery-bin}/bin/nixery
diff --git a/tools/nixery/docs/default.nix b/tools/nixery/docs/default.nix
new file mode 100644
index 000000000000..ba652cef9c26
--- /dev/null
+++ b/tools/nixery/docs/default.nix
@@ -0,0 +1,36 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Builds the documentation page using the Rust project's 'mdBook'
+# tool.
+#
+# Some of the documentation is pulled in and included from other
+# sources.
+
+{ fetchFromGitHub, mdbook, runCommand }:
+
+let
+  nix-1p = fetchFromGitHub {
+    owner  = "tazjin";
+    repo   = "nix-1p";
+    rev    = "aab846cd3d79fcd092b1bfea1346c587b2a56095";
+    sha256 = "12dl0xrwgg2d4wyv9zxgdn0hzqnanczjg23vqn3356rywxlzzwak";
+  };
+in runCommand "nixery-book" {} ''
+  mkdir -p $out
+  cp -r ${./.}/* .
+  chmod -R a+w src
+  cp ${nix-1p}/README.md src/nix-1p.md
+  ${mdbook}/bin/mdbook build -d $out
+''
-- 
cgit 1.4.1


From a3f6278913d756c74d4f5c636c88b91a1b3748f6 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Sun, 4 Aug 2019 23:42:57 +0100
Subject: docs: Add an "under-the-hood" page explaining the build process

This page describes the various steps that Nixery goes through when
"procuring" an image.

The intention is to give users some more visibility into what is going
on and to make it clear that this is not just an image storage
service.
---
 tools/nixery/docs/src/SUMMARY.md        |   1 +
 tools/nixery/docs/src/nixery.md         |   8 +--
 tools/nixery/docs/src/under-the-hood.md | 105 ++++++++++++++++++++++++++++++++
 3 files changed, 110 insertions(+), 4 deletions(-)
 create mode 100644 tools/nixery/docs/src/under-the-hood.md

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md
index 5d680b82e8d9..f5ba3e9b084a 100644
--- a/tools/nixery/docs/src/SUMMARY.md
+++ b/tools/nixery/docs/src/SUMMARY.md
@@ -1,4 +1,5 @@
 # Summary
 
 - [Nixery](./nixery.md)
+- [Under the hood](./under-the-hood.md)
 - [Nix, the language](./nix-1p.md)
diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
index d3d1911d2880..83e1aac52bdf 100644
--- a/tools/nixery/docs/src/nixery.md
+++ b/tools/nixery/docs/src/nixery.md
@@ -52,10 +52,6 @@ The instance at `nixery.dev` tracks a recent NixOS channel, currently NixOS
 Private registries might be configured to track a different channel (such as
 `nixos-unstable`) or even track a git repository with custom packages.
 
-### Is this an official Google project?
-
-**No.** Nixery is not officially supported by Google.
-
 ### Should I depend on `nixery.dev` in production?
 
 While we appreciate the enthusiasm, if you would like to use Nixery in your
@@ -63,6 +59,10 @@ production project we recommend setting up a private instance. The public Nixery
 at `nixery.dev` is run on a best-effort basis and we make no guarantees about
 availability.
 
+### Is this an official Google project?
+
+**No.** Nixery is not officially supported by Google.
+
 ### Who made this?
 
 Nixery was written mostly by [tazjin][].
diff --git a/tools/nixery/docs/src/under-the-hood.md b/tools/nixery/docs/src/under-the-hood.md
new file mode 100644
index 000000000000..3791707b1cd2
--- /dev/null
+++ b/tools/nixery/docs/src/under-the-hood.md
@@ -0,0 +1,105 @@
+# Under the hood
+
+This page serves as a quick explanation of what happens under-the-hood when an
+image is requested from Nixery.
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+
+- [1. The image manifest is requested](#1-the-image-manifest-is-requested)
+- [2. Nix builds the image](#2-nix-builds-the-image)
+- [3. Layers are uploaded to Nixery's storage](#3-layers-are-uploaded-to-nixerys-storage)
+- [4. The image manifest is sent back](#4-the-image-manifest-is-sent-back)
+- [5. Image layers are requested](#5-image-layers-are-requested)
+
+<!-- markdown-toc end -->
+
+--------
+
+## 1. The image manifest is requested
+
+When container registry clients such as Docker pull an image, the first thing
+they do is ask for the image manifest. This is a JSON document describing which
+layers are contained in an image, as well as some additional auxiliary
+information.
+
+This request is of the form `GET /v2/$imageName/manifests/$imageTag`.
+
+Nixery receives this request and begins by splitting the image name into its
+path components and substituting meta-packages (such as `shell`) for their
+contents.
+
+For example, requesting `shell/htop/git` results in Nixery expanding the image
+name to `["bashInteractive", "coreutils", "htop", "git"]`.
+
+If Nixery is configured with a private Nix repository, it also looks at the
+image tag and substitutes `latest` with `master`.
+
+It then invokes Nix with three parameters:
+
+1. image contents (as above)
+2. image tag
+3. configured package set source
+
+## 2. Nix builds the image
+
+Using the parameters above, Nix imports the package set and begins by mapping
+the image names to attributes in the package set.
+
+A special case during this process is packages with uppercase characters in
+their name, for example anything under `haskellPackages`. The registry protocol
+does not allow uppercase characters, so the Nix code will translate something
+like `haskellpackages` (lowercased) to the correct attribute name.
+
+After identifying all contents, Nix determines the contents of each layer while
+optimising for the best possible cache efficiency.
+
+Finally it builds each layer, assembles the image manifest as JSON structure,
+and yields this manifest back to the web server.
+
+*Note:* While this step is running (which can take some time in the case of
+large first-time image builds), the registry client is left hanging waiting for
+an HTTP response. Unfortunately the registry protocol does not allow for any
+feedback back to the user at this point, so from the user's perspective things
+just ... hang, for a moment.
+
+## 3. Layers are uploaded to Nixery's storage
+
+Nixery inspects the returned manifest and uploads each layer to the configured
+[Google Cloud Storage][gcs] bucket. To avoid unnecessary uploading, it will
+first check whether layers are already present in the bucket and - just to be
+safe - compare their MD5-hashes against what was built.
+
+## 4. The image manifest is sent back
+
+If everything went well at this point, Nixery responds to the registry client
+with the image manifest.
+
+The client now inspects the manifest and basically sees a list of SHA256-hashes,
+each corresponding to one layer of the image. Most clients will now consult
+their local layer storage and determine which layers they are missing.
+
+Each of the missing layers is then requested from Nixery.
+
+## 5. Image layers are requested
+
+For each image layer that it needs to retrieve, the registry client assembles a
+request that looks like this:
+
+`GET /v2/${imageName}/blob/sha256:${layerHash}`
+
+Nixery receives these requests and *rewrites* them to Google Cloud Storage URLs,
+responding with an `HTTP 303 See Other` status code and the actual download URL
+of the layer.
+
+Nixery supports using private buckets which are not generally world-readable, in
+which case [signed URLs][] are constructed using a private key. These allow the
+registry client to download each layer without needing to care about how the
+underlying authentication works.
+
+---------
+
+That's it. After these five steps the registry client has retrieved all it needs
+to run the image produced by Nixery.
+
+[gcs]: https://cloud.google.com/storage/
+[signed URLs]: https://cloud.google.com/storage/docs/access-control/signed-urls
-- 
cgit 1.4.1


From 6293d69fd94f709d029c5c121956900cad3d24c1 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Mon, 5 Aug 2019 00:27:39 +0100
Subject: docs: Add a section on running your own Nixery

---
 tools/nixery/docs/src/SUMMARY.md      |   6 +-
 tools/nixery/docs/src/run-your-own.md | 141 ++++++++++++++++++++++++++++++++++
 2 files changed, 145 insertions(+), 2 deletions(-)
 create mode 100644 tools/nixery/docs/src/run-your-own.md

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md
index f5ba3e9b084a..677f328972bb 100644
--- a/tools/nixery/docs/src/SUMMARY.md
+++ b/tools/nixery/docs/src/SUMMARY.md
@@ -1,5 +1,7 @@
 # Summary
 
 - [Nixery](./nixery.md)
-- [Under the hood](./under-the-hood.md)
-- [Nix, the language](./nix-1p.md)
+  - [Under the hood](./under-the-hood.md)
+  - [Run your own Nixery](./run-your-own.md)
+- [Nix](./nix.md)
+  - [Nix, the language](./nix-1p.md)
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
new file mode 100644
index 000000000000..0539ab02fcff
--- /dev/null
+++ b/tools/nixery/docs/src/run-your-own.md
@@ -0,0 +1,141 @@
+## Run your own Nixery
+
+<!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
+
+- [0. Prerequisites](#0-prerequisites)
+- [1. Choose a package set](#1-choose-a-package-set)
+- [2. Build Nixery itself](#2-build-nixery-itself)
+- [3. Prepare configuration](#3-prepare-configuration)
+- [4. Deploy Nixery](#4-deploy-nixery)
+- [5. Productionise](#5-productionise)
+
+<!-- markdown-toc end -->
+
+
+---------
+
+⚠ This page is still under construction! ⚠
+
+--------
+
+Running your own Nixery is not difficult, but requires some setup. Follow the
+steps below to get up & running.
+
+*Note:* Nixery can be run inside of a [GKE][] cluster, providing a local service
+from which images can be requested. Documentation for how to set this up is
+forthcoming, please see [nixery#4][].
+
+## 0. Prerequisites
+
+To run Nixery, you must have:
+
+* [Nix][] (to build Nixery itself)
+* Somewhere to run it (your own server, Google AppEngine, a Kubernetes cluster,
+  whatever!)
+* A [Google Cloud Storage][gcs] bucket in which to store & serve layers
+
+## 1. Choose a package set
+
+When running your own Nixery you need to decide which package set you want to
+serve. By default, Nixery builds packages from a recent NixOS channel which
+ensures that most packages are cached upstream and no expensive builds need to
+be performed for trivial things.
+
+However if you are running a private Nixery, chances are high that you intend to
+use it with your own packages. There are three options available:
+
+1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-19.03` or
+   `nixos-unstable`.
+2. Specify your own git-repository with a custom package set[^2]. This makes it
+   possible to pull different tags, branches or commits by modifying the image
+   tag.
+3. Specify a local file path containing a Nix package set. Where this comes from
+   or what it contains is up to you.
+
+## 2. Build Nixery itself
+
+Building Nixery creates a container image. This section assumes that the
+container runtime used is Docker, please modify instructions correspondingly if
+you are using something else.
+
+With a working Nix installation, building Nixery is done by invoking `nix-build
+-A nixery-image` from a checkout of the [Nixery repository][repo].
+
+This will create a `result`-symlink which points to a tarball containing the
+image. In Docker, this tarball can be loaded by using `docker load -i result`.
+
+## 3. Prepare configuration
+
+Nixery is configured via environment variables.
+
+You must set *all* of these:
+
+* `BUCKET`: [Google Cloud Storage][gcs] bucket to store & serve image layers
+* `PORT`: HTTP port on which Nixery should listen
+
+You may set *one* of these, if unset Nixery defaults to `nixos-19.03`:
+
+* `NIXERY_CHANNEL`: The name of a Nix/NixOS channel to use for building
+* `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses
+  locally configured SSH/git credentials)
+* `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use
+  for building
+
+You may set *both* of these:
+
+* `GCS_SIGNING_KEY`: A Google service account key (in PEM format) that can be
+  used to [sign Cloud Storage URLs][signed-urls]
+* `GCS_SIGNING_ACCOUNT`: Google service account ID that the signing key belongs
+  to
+
+To authenticate to the configured GCS bucket, Nixery uses Google's [Application
+Default Credentials][ADC]. Depending on your environment this may require
+additional configuration.
+
+## 4. Deploy Nixery
+
+With the above environment variables configured, you can run the image that was
+built in step 2.
+
+How this works depends on the environment you are using and is, for now, outside
+of the scope of this tutorial.
+
+Once Nixery is running you can immediately start requesting images from it.
+
+## 5. Productionise
+
+(⚠ Here be dragons! ⚠)
+
+Nixery is still an early project and has not yet been deployed in any production
+environments and some caveats apply.
+
+Notably, Nixery currently does not support any authentication methods, so anyone
+with network access to the registry can retrieve images.
+
+Running a Nixery inside of a fenced-off environment (such as internal to a
+Kubernetes cluster) should be fine, but you should consider to do all of the
+following:
+
+* Issue a TLS certificate for the hostname you are assigning to Nixery. In fact,
+  Docker will refuse to pull images from registries that do not use TLS (with
+  the exception of `.local` domains).
+* Configure signed GCS URLs to avoid having to make your bucket world-readable.
+* Configure request timeouts for Nixery if you have your own web server in front
+  of it. This will be natively supported by Nixery in the future.
+
+-------
+
+[^1]: Nixery will not work with Nix channels older than `nixos-19.03`.
+
+[^2]: This documentation will be updated with instructions on how to best set up
+    a custom Nix repository. Nixery expects custom package sets to be a superset
+    of `nixpkgs`, as it uses `lib` and other features from `nixpkgs`
+    extensively.
+
+[GKE]: https://cloud.google.com/kubernetes-engine/
+[nixery#4]: https://github.com/google/nixery/issues/4
+[Nix]: https://nixos.org/nix
+[gcs]: https://cloud.google.com/storage/
+[repo]: https://github.com/google/nixery
+[signed-urls]: under-the-hood.html#5-image-layers-are-requested
+[ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
-- 
cgit 1.4.1


From d87662b7b562d68561775adeefd29fffb065465c Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Mon, 5 Aug 2019 00:27:47 +0100
Subject: docs: Add a section on Nix itself

---
 tools/nixery/docs/src/nix.md | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 tools/nixery/docs/src/nix.md

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/nix.md b/tools/nixery/docs/src/nix.md
new file mode 100644
index 000000000000..2bfd75a6925c
--- /dev/null
+++ b/tools/nixery/docs/src/nix.md
@@ -0,0 +1,31 @@
+# Nix
+
+These sections are designed to give some background information on what Nix is.
+If you've never heard of Nix before looking at Nixery, this might just be the
+page for you!
+
+[Nix][] is a functional package-manager that comes with a number of advantages
+over traditional package managers, such as side-by-side installs of different
+package versions, atomic updates, easy customisability, simple binary caching
+and much more. Feel free to explore the [Nix website][Nix] for an overview of
+Nix itself.
+
+Nix uses a custom programming language also called Nix, which is explained here
+[on its own page][nix-1p].
+
+In addition to the package manager and language, the Nix project also maintains
+[NixOS][] - a Linux distribution built entirely on Nix. On NixOS, users can
+declaratively describe the *entire* configuration of their system and perform
+updates/rollbacks to other system configurations with ease.
+
+Most Nix packages are tracked in the [Nix package set][nixpkgs], usually simply
+referred to as `nixpkgs`. It contains tens of thousands of packages already!
+
+Nixery (which you are looking at!) provides an easy & simple way to get started
+with Nix, in fact you don't even need to know that you're using Nix to make use
+of Nixery.
+
+[Nix]: https://nixos.org/nix/
+[nix-1p]: nix-1p.html
+[NixOS]: https://nixos.org/
+[nixpkgs]: https://github.com/nixos/nixpkgs
-- 
cgit 1.4.1


From 3e385dc379de4a1e8dd619a7a47714925e99490a Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Thu, 8 Aug 2019 21:02:08 +0100
Subject: docs: Update embedded nix-1p

The new commit has an operator table, which is nice to have!
---
 tools/nixery/docs/default.nix | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/default.nix b/tools/nixery/docs/default.nix
index ba652cef9c26..deebdffd7fd9 100644
--- a/tools/nixery/docs/default.nix
+++ b/tools/nixery/docs/default.nix
@@ -24,8 +24,8 @@ let
   nix-1p = fetchFromGitHub {
     owner  = "tazjin";
     repo   = "nix-1p";
-    rev    = "aab846cd3d79fcd092b1bfea1346c587b2a56095";
-    sha256 = "12dl0xrwgg2d4wyv9zxgdn0hzqnanczjg23vqn3356rywxlzzwak";
+    rev    = "3cd0f7d7b4f487d04a3f1e3ca8f2eb1ab958c49b";
+    sha256 = "02lpda03q580gyspkbmlgnb2cbm66rrcgqsv99aihpbwyjym81af";
   };
 in runCommand "nixery-book" {} ''
   mkdir -p $out
-- 
cgit 1.4.1


From 819b4602788195cacde48cf8bb36ab242d240512 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Mon, 12 Aug 2019 17:10:34 +0100
Subject: chore(docs): Move mdBook derivation to docs/default.nix

---
 tools/nixery/default.nix      | 23 +----------------------
 tools/nixery/docs/default.nix | 20 +++++++++++++++++++-
 2 files changed, 20 insertions(+), 23 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/default.nix b/tools/nixery/default.nix
index dee5713c64af..fe5afdb8ed8b 100644
--- a/tools/nixery/default.nix
+++ b/tools/nixery/default.nix
@@ -25,32 +25,11 @@ rec {
   # data dependencies.
   nixery-server = callPackage ./server {};
 
-  # Nix expression (unimported!) which is used by Nixery to build
-  # container images.
-  nixery-builder = ./build-registry-image.nix;
-
-  # nixpkgs currently has an old version of mdBook. A new version is
-  # built here, but eventually the update will be upstreamed
-  # (nixpkgs#65890)
-  mdbook = rustPlatform.buildRustPackage rec {
-    name = "mdbook-${version}";
-    version = "0.3.1";
-    doCheck = false;
-
-    src = fetchFromGitHub {
-      owner = "rust-lang-nursery";
-      repo = "mdBook";
-      rev = "v${version}";
-      sha256 = "0py69267jbs6b7zw191hcs011cm1v58jz8mglqx3ajkffdfl3ghw";
-    };
-
-    cargoSha256 = "0qwhc42a86jpvjcaysmfcw8kmwa150lmz01flmlg74g6qnimff5m";
-  };
 
   # Use mdBook to build a static asset page which Nixery can then
   # serve. This is primarily used for the public instance at
   # nixery.dev.
-  nixery-book = callPackage ./docs { inherit mdbook; };
+  nixery-book = callPackage ./docs {};
 
   # Wrapper script running the Nixery server with the above two data
   # dependencies configured.
diff --git a/tools/nixery/docs/default.nix b/tools/nixery/docs/default.nix
index deebdffd7fd9..6a31be4fd4e0 100644
--- a/tools/nixery/docs/default.nix
+++ b/tools/nixery/docs/default.nix
@@ -18,9 +18,27 @@
 # Some of the documentation is pulled in and included from other
 # sources.
 
-{ fetchFromGitHub, mdbook, runCommand }:
+{ fetchFromGitHub, mdbook, runCommand, rustPlatform }:
 
 let
+  # nixpkgs currently has an old version of mdBook. A new version is
+  # built here, but eventually the update will be upstreamed
+  # (nixpkgs#65890)
+  mdbook = rustPlatform.buildRustPackage rec {
+    name = "mdbook-${version}";
+    version = "0.3.1";
+    doCheck = false;
+
+    src = fetchFromGitHub {
+      owner = "rust-lang-nursery";
+      repo = "mdBook";
+      rev = "v${version}";
+      sha256 = "0py69267jbs6b7zw191hcs011cm1v58jz8mglqx3ajkffdfl3ghw";
+    };
+
+    cargoSha256 = "0qwhc42a86jpvjcaysmfcw8kmwa150lmz01flmlg74g6qnimff5m";
+  };
+
   nix-1p = fetchFromGitHub {
     owner  = "tazjin";
     repo   = "nix-1p";
-- 
cgit 1.4.1


From 3939722063f3d08a547fa98e17aac609f7f765ac Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Tue, 13 Aug 2019 00:35:42 +0100
Subject: style: Apply nixfmt to trivial Nix files

ALl the ones except for build-image.nix are considered trivial. On the
latter, nixfmt makes some useful changes but by-and-large it is not
ready for that code yet.
---
 tools/nixery/build-image/go-deps.nix | 20 +++++++++-----------
 tools/nixery/default.nix             |  9 ++++-----
 tools/nixery/docs/default.nix        |  8 ++++----
 tools/nixery/server/default.nix      |  4 ++--
 tools/nixery/shell.nix               |  7 ++-----
 5 files changed, 21 insertions(+), 27 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/build-image/go-deps.nix b/tools/nixery/build-image/go-deps.nix
index 235c3c4c6dbe..0f22a7088f52 100644
--- a/tools/nixery/build-image/go-deps.nix
+++ b/tools/nixery/build-image/go-deps.nix
@@ -1,12 +1,10 @@
 # This file was generated by https://github.com/kamilchm/go2nix v1.3.0
-[
-  {
-    goPackagePath = "gonum.org/v1/gonum";
-    fetch = {
-      type = "git";
-      url = "https://github.com/gonum/gonum";
-      rev = "ced62fe5104b907b6c16cb7e575c17b2e62ceddd";
-      sha256 = "1b7q6haabnp53igpmvr6a2414yralhbrldixx4kbxxg1apy8jdjg";
-    };
-  }
-]
+[{
+  goPackagePath = "gonum.org/v1/gonum";
+  fetch = {
+    type = "git";
+    url = "https://github.com/gonum/gonum";
+    rev = "ced62fe5104b907b6c16cb7e575c17b2e62ceddd";
+    sha256 = "1b7q6haabnp53igpmvr6a2414yralhbrldixx4kbxxg1apy8jdjg";
+  };
+}]
diff --git a/tools/nixery/default.nix b/tools/nixery/default.nix
index 926ab0d19f6b..686c230553f0 100644
--- a/tools/nixery/default.nix
+++ b/tools/nixery/default.nix
@@ -11,8 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-{ pkgs ? import <nixpkgs> {}
-, preLaunch ? "" }:
+{ pkgs ? import <nixpkgs> { }, preLaunch ? "" }:
 
 with pkgs;
 
@@ -23,7 +22,7 @@ rec {
   # Users will usually not want to use this directly, instead see the
   # 'nixery' derivation below, which automatically includes runtime
   # data dependencies.
-  nixery-server = callPackage ./server {};
+  nixery-server = callPackage ./server { };
 
   # Implementation of the image building & layering logic
   nixery-build-image = (import ./build-image { inherit pkgs; }).wrapper;
@@ -31,7 +30,7 @@ rec {
   # Use mdBook to build a static asset page which Nixery can then
   # serve. This is primarily used for the public instance at
   # nixery.dev.
-  nixery-book = callPackage ./docs {};
+  nixery-book = callPackage ./docs { };
 
   # Wrapper script running the Nixery server with the above two data
   # dependencies configured.
@@ -76,7 +75,7 @@ rec {
     '';
   in dockerTools.buildLayeredImage {
     name = "nixery";
-    config.Cmd = ["${nixery-launch-script}/bin/nixery"];
+    config.Cmd = [ "${nixery-launch-script}/bin/nixery" ];
     maxLayers = 96;
     contents = [
       cacert
diff --git a/tools/nixery/docs/default.nix b/tools/nixery/docs/default.nix
index 6a31be4fd4e0..aae2fdde42a5 100644
--- a/tools/nixery/docs/default.nix
+++ b/tools/nixery/docs/default.nix
@@ -40,12 +40,12 @@ let
   };
 
   nix-1p = fetchFromGitHub {
-    owner  = "tazjin";
-    repo   = "nix-1p";
-    rev    = "3cd0f7d7b4f487d04a3f1e3ca8f2eb1ab958c49b";
+    owner = "tazjin";
+    repo = "nix-1p";
+    rev = "3cd0f7d7b4f487d04a3f1e3ca8f2eb1ab958c49b";
     sha256 = "02lpda03q580gyspkbmlgnb2cbm66rrcgqsv99aihpbwyjym81af";
   };
-in runCommand "nixery-book" {} ''
+in runCommand "nixery-book" { } ''
   mkdir -p $out
   cp -r ${./.}/* .
   chmod -R a+w src
diff --git a/tools/nixery/server/default.nix b/tools/nixery/server/default.nix
index 0d0c056a56f4..05ad64261fa5 100644
--- a/tools/nixery/server/default.nix
+++ b/tools/nixery/server/default.nix
@@ -15,9 +15,9 @@
 { buildGoPackage, lib }:
 
 buildGoPackage {
-  name   = "nixery-server";
+  name = "nixery-server";
   goDeps = ./go-deps.nix;
-  src    = ./.;
+  src = ./.;
 
   goPackagePath = "github.com/google/nixery";
 
diff --git a/tools/nixery/shell.nix b/tools/nixery/shell.nix
index 49d0e5581971..93cd1f4cec62 100644
--- a/tools/nixery/shell.nix
+++ b/tools/nixery/shell.nix
@@ -14,14 +14,11 @@
 
 # Configures a shell environment that builds required local packages to
 # run Nixery.
-{pkgs ? import <nixpkgs> {} }:
+{ pkgs ? import <nixpkgs> { } }:
 
 let nixery = import ./default.nix { inherit pkgs; };
 in pkgs.stdenv.mkDerivation {
   name = "nixery-dev-shell";
 
-  buildInputs = with pkgs;[
-    jq
-    nixery.nixery-build-image
-  ];
+  buildInputs = with pkgs; [ jq nixery.nixery-build-image ];
 }
-- 
cgit 1.4.1


From 0ec369d76c2b151fa82839230e6eb4d58015b1dc Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Thu, 15 Aug 2019 11:50:23 +0100
Subject: docs(book): Update information on new layering strategy

---
 tools/nixery/docs/src/nixery.md         | 15 ++++++++-------
 tools/nixery/docs/src/under-the-hood.md |  4 +++-
 2 files changed, 11 insertions(+), 8 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
index 83e1aac52bdf..3eaeb6be4e97 100644
--- a/tools/nixery/docs/src/nixery.md
+++ b/tools/nixery/docs/src/nixery.md
@@ -7,8 +7,11 @@ contain packages from the [Nix][] package manager. Images with arbitrary
 packages can be requested via the image name.
 
 Nix not only provides the packages to include in the images, but also builds the
-images themselves by using an interesting layering strategy described in [this
-blog post][layers].
+images themselves by using a special [layering strategy][] that optimises for
+cache efficiency.
+
+For general information on why using Nix makes sense for container images, check
+out [this blog post][layers].
 
 ## Quick start
 
@@ -65,13 +68,11 @@ availability.
 
 ### Who made this?
 
-Nixery was written mostly by [tazjin][].
-
-[grahamc][] authored the image layering strategy. Many people have contributed
-to Nix over time, maybe you could become one of them?
+Nixery was written by [tazjin][], but many people have contributed to Nix over
+time, maybe you could become one of them?
 
 [Nixery]: https://github.com/google/nixery
 [Nix]: https://nixos.org/nix
+[layering-strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
 [layers]: https://grahamc.com/blog/nix-and-layered-docker-images
 [tazjin]: https://github.com/tazjin
-[grahamc]: https://github.com/grahamc
diff --git a/tools/nixery/docs/src/under-the-hood.md b/tools/nixery/docs/src/under-the-hood.md
index 3791707b1cd2..6b5e5e9bbf21 100644
--- a/tools/nixery/docs/src/under-the-hood.md
+++ b/tools/nixery/docs/src/under-the-hood.md
@@ -51,7 +51,8 @@ does not allow uppercase characters, so the Nix code will translate something
 like `haskellpackages` (lowercased) to the correct attribute name.
 
 After identifying all contents, Nix determines the contents of each layer while
-optimising for the best possible cache efficiency.
+optimising for the best possible cache efficiency (see the [layering design
+doc][] for details).
 
 Finally it builds each layer, assembles the image manifest as JSON structure,
 and yields this manifest back to the web server.
@@ -103,3 +104,4 @@ to run the image produced by Nixery.
 
 [gcs]: https://cloud.google.com/storage/
 [signed URLs]: https://cloud.google.com/storage/docs/access-control/signed-urls
+[layering design doc]: https://storage.googleapis.com/nixdoc/nixery-layers.html
-- 
cgit 1.4.1


From 3f232e017075f5b45c7d11cdf27225a51f47c085 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Thu, 15 Aug 2019 12:00:36 +0100
Subject: docs: Add asciinema demo to README & book

---
 tools/nixery/README.md          | 25 ++++++++-----------------
 tools/nixery/docs/src/nixery.md |  4 ++++
 2 files changed, 12 insertions(+), 17 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/README.md b/tools/nixery/README.md
index 9c323a57fa12..8225e430b36e 100644
--- a/tools/nixery/README.md
+++ b/tools/nixery/README.md
@@ -25,23 +25,14 @@ images. The design for this is outlined in [a public gist][gist].
 
 This is not an officially supported Google project.
 
-## Usage example
-
-Using the publicly available Nixery instance at `nixery.dev`, one could
-retrieve a container image containing `curl` and an interactive shell like this:
-
-```shell
-tazjin@tazbox:~$ sudo docker run -ti nixery.dev/shell/curl bash
-Unable to find image 'nixery.dev/shell/curl:latest' locally
-latest: Pulling from shell/curl
-7734b79e1ba1: Already exists
-b0d2008d18cd: Pull complete
-< ... some layers omitted ...>
-Digest: sha256:178270bfe84f74548b6a43347d73524e5c2636875b673675db1547ec427cf302
-Status: Downloaded newer image for nixery.dev/shell/curl:latest
-bash-4.4# curl --version
-curl 7.64.0 (x86_64-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.0.2q zlib/1.2.11 libssh2/1.8.0 nghttp2/1.35.1
-```
+## Demo
+
+Click the image to see an example in which an image containing an interactive
+shell and GNU `hello` is downloaded.
+
+[![asciicast](https://asciinema.org/a/262583.png)](https://asciinema.org/a/262583?autoplay=1)
+
+To try it yourself, head to [nixery.dev][public]!
 
 The special meta-package `shell` provides an image base with many core
 components (such as `bash` and `coreutils`) that users commonly expect in
diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
index 3eaeb6be4e97..edea53bab62e 100644
--- a/tools/nixery/docs/src/nixery.md
+++ b/tools/nixery/docs/src/nixery.md
@@ -13,6 +13,10 @@ cache efficiency.
 For general information on why using Nix makes sense for container images, check
 out [this blog post][layers].
 
+## Demo
+
+<script src="https://asciinema.org/a/262583.js" id="asciicast-262583" async data-autoplay="true" data-loop="true"></script>
+
 ## Quick start
 
 Simply pull an image from this registry, separating each package you want
-- 
cgit 1.4.1


From ffae282eac86f4df48a5c39f61d82dd88ccca1ec Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Sun, 18 Aug 2019 02:44:34 +0100
Subject: fix(docs): Correct link to layering strategy

---
 tools/nixery/docs/src/nixery.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
index edea53bab62e..6cc16431c9e8 100644
--- a/tools/nixery/docs/src/nixery.md
+++ b/tools/nixery/docs/src/nixery.md
@@ -77,6 +77,6 @@ time, maybe you could become one of them?
 
 [Nixery]: https://github.com/google/nixery
 [Nix]: https://nixos.org/nix
-[layering-strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
+[layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
 [layers]: https://grahamc.com/blog/nix-and-layered-docker-images
 [tazjin]: https://github.com/tazjin
-- 
cgit 1.4.1


From bb5427a47a29161f854bd9b57e388849ea26e818 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Wed, 21 Aug 2019 10:21:45 +0100
Subject: chore(docs): Update embedded nix-1p version

The new version of the document has syntactic fixes that render pipes
in code blocks in tables correctly across dialects.

Fixes #44
---
 tools/nixery/docs/default.nix | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/default.nix b/tools/nixery/docs/default.nix
index aae2fdde42a5..11eda3ff7052 100644
--- a/tools/nixery/docs/default.nix
+++ b/tools/nixery/docs/default.nix
@@ -42,8 +42,8 @@ let
   nix-1p = fetchFromGitHub {
     owner = "tazjin";
     repo = "nix-1p";
-    rev = "3cd0f7d7b4f487d04a3f1e3ca8f2eb1ab958c49b";
-    sha256 = "02lpda03q580gyspkbmlgnb2cbm66rrcgqsv99aihpbwyjym81af";
+    rev = "e0a051a016b9118bea90ec293d6cd346b9707e77";
+    sha256 = "0d1lfkxg03lki8dc3229g1cgqiq3nfrqgrknw99p6w0zk1pjd4dj";
   };
 in runCommand "nixery-book" { } ''
   mkdir -p $out
-- 
cgit 1.4.1


From 496a4ab84742279fb7bbd1c8ac9d4ebd1a44b148 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Mon, 2 Sep 2019 23:35:50 +0100
Subject: docs: Add information about NIX_TIMEOUT variable

---
 tools/nixery/README.md                | 2 ++
 tools/nixery/docs/src/run-your-own.md | 4 +++-
 2 files changed, 5 insertions(+), 1 deletion(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/README.md b/tools/nixery/README.md
index cdae23bc4b72..3bc6df845a7f 100644
--- a/tools/nixery/README.md
+++ b/tools/nixery/README.md
@@ -81,6 +81,8 @@ variables:
   locally configured SSH/git credentials)
 * `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use
   for building
+* `NIX_TIMEOUT`: Number of seconds that any Nix builder is allowed to run
+  (defaults to 60
 * `GCS_SIGNING_KEY`: A Google service account key (in PEM format) that can be
   used to sign Cloud Storage URLs
 * `GCS_SIGNING_ACCOUNT`: Google service account ID that the signing key belongs
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
index 0539ab02fcff..7a294f56055e 100644
--- a/tools/nixery/docs/src/run-your-own.md
+++ b/tools/nixery/docs/src/run-your-own.md
@@ -81,8 +81,10 @@ You may set *one* of these, if unset Nixery defaults to `nixos-19.03`:
 * `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use
   for building
 
-You may set *both* of these:
+You may set *all* of these:
 
+* `NIX_TIMEOUT`: Number of seconds that any Nix builder is allowed to run
+  (defaults to 60)
 * `GCS_SIGNING_KEY`: A Google service account key (in PEM format) that can be
   used to [sign Cloud Storage URLs][signed-urls]
 * `GCS_SIGNING_ACCOUNT`: Google service account ID that the signing key belongs
-- 
cgit 1.4.1


From e4d03fdb17041ead530aa3e115f84988148a3b21 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Sat, 21 Sep 2019 12:04:04 +0100
Subject: chore(docs): Remove mdbook override

The change has been upstreamed in Nixpkgs.
---
 tools/nixery/docs/default.nix | 18 ------------------
 1 file changed, 18 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/default.nix b/tools/nixery/docs/default.nix
index 11eda3ff7052..fdf0e6ff9ea5 100644
--- a/tools/nixery/docs/default.nix
+++ b/tools/nixery/docs/default.nix
@@ -21,24 +21,6 @@
 { fetchFromGitHub, mdbook, runCommand, rustPlatform }:
 
 let
-  # nixpkgs currently has an old version of mdBook. A new version is
-  # built here, but eventually the update will be upstreamed
-  # (nixpkgs#65890)
-  mdbook = rustPlatform.buildRustPackage rec {
-    name = "mdbook-${version}";
-    version = "0.3.1";
-    doCheck = false;
-
-    src = fetchFromGitHub {
-      owner = "rust-lang-nursery";
-      repo = "mdBook";
-      rev = "v${version}";
-      sha256 = "0py69267jbs6b7zw191hcs011cm1v58jz8mglqx3ajkffdfl3ghw";
-    };
-
-    cargoSha256 = "0qwhc42a86jpvjcaysmfcw8kmwa150lmz01flmlg74g6qnimff5m";
-  };
-
   nix-1p = fetchFromGitHub {
     owner = "tazjin";
     repo = "nix-1p";
-- 
cgit 1.4.1


From 2c8ef634f67f91c8efc1cf6a58a271f4c44544dd Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Mon, 30 Sep 2019 11:51:36 +0100
Subject: docs(caching): Add information about Nixery's caching strategies

---
 tools/nixery/docs/src/SUMMARY.md |  1 +
 tools/nixery/docs/src/caching.md | 70 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 71 insertions(+)
 create mode 100644 tools/nixery/docs/src/caching.md

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/SUMMARY.md b/tools/nixery/docs/src/SUMMARY.md
index 677f328972bb..f1d68a3ac451 100644
--- a/tools/nixery/docs/src/SUMMARY.md
+++ b/tools/nixery/docs/src/SUMMARY.md
@@ -2,6 +2,7 @@
 
 - [Nixery](./nixery.md)
   - [Under the hood](./under-the-hood.md)
+  - [Caching](./caching.md)
   - [Run your own Nixery](./run-your-own.md)
 - [Nix](./nix.md)
   - [Nix, the language](./nix-1p.md)
diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md
new file mode 100644
index 000000000000..175fe04d7084
--- /dev/null
+++ b/tools/nixery/docs/src/caching.md
@@ -0,0 +1,70 @@
+# Caching in Nixery
+
+This page gives a quick overview over the caching done by Nixery. All cache data
+is written to Nixery's storage bucket and is based on deterministic identifiers
+or content-addressing, meaning that cache entries under the same key *never
+change*.
+
+## Manifests
+
+Manifests of builds are cached at `$BUCKET/manifests/$KEY`. The effect of this
+cache is that multiple instances of Nixery do not need to rebuild the same
+manifest from scratch.
+
+Since the manifest cache is populated only *after* layers are uploaded, Nixery
+can immediately return the manifest to its clients without needing to check
+whether layers have been uploaded already.
+
+`$KEY` is generated by creating a SHA1 hash of the requested content of a
+manifest plus the package source specification.
+
+Manifests are *only* cached if the package source specification is *not* a
+moving target.
+
+Manifest caching *only* applies in the following cases:
+
+* package source specification is a specific git commit
+* package source specification is a specific NixOS/nixpkgs commit
+
+Manifest caching *never* applies in the following cases:
+
+* package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`)
+* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-19.03`)
+* package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`)
+
+It is thus always preferable to request images from a fully-pinned package
+source.
+
+Manifests can be removed from the manifest cache without negative consequences.
+
+## Layer tarballs
+
+Layer tarballs are the files that Nixery clients retrieve from the storage
+bucket to download an image.
+
+They are stored content-addressably at `$BUCKET/layers/$SHA256HASH` and layer
+requests sent to Nixery will redirect directly to this storage location.
+
+The effect of this cache is that Nixery does not need to upload identical layers
+repeatedly. When Nixery notices that a layer already exists in GCS, it will use
+the object metadata to compare its MD5-hash with the locally computed one and
+skip uploading.
+
+Removing layers from the cache is *potentially problematic* if there are cached
+manifests or layer builds referencing those layers.
+
+To clean up layers, a user must ensure that no other cached resources still
+reference these layers.
+
+## Layer builds
+
+Layer builds are cached at `$BUCKET/builds/$HASH`, where `$HASH` is a SHA1 of
+the Nix store paths included in the layer.
+
+The content of the cached entries is a JSON-object that contains the MD5 and
+SHA256 hashes of the built layer.
+
+The effect of this cache is that different instances of Nixery will not build,
+hash and upload layers that have identical contents across different instances.
+
+Layer builds can be removed from the cache without negative consequences.
-- 
cgit 1.4.1


From 53906024ff0612b6946cff4122dc28e85a414b6b Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Thu, 3 Oct 2019 12:11:46 +0100
Subject: refactor: Remove remaining MD5-hash mentions and computations

---
 tools/nixery/build-image/build-image.nix |  5 ++---
 tools/nixery/docs/src/caching.md         |  9 ++++-----
 tools/nixery/docs/src/under-the-hood.md  |  3 +--
 tools/nixery/server/builder/builder.go   | 24 +++++++++---------------
 tools/nixery/server/manifest/manifest.go |  7 ++-----
 5 files changed, 18 insertions(+), 30 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/build-image/build-image.nix b/tools/nixery/build-image/build-image.nix
index 33500dbb9e80..70049885ab1c 100644
--- a/tools/nixery/build-image/build-image.nix
+++ b/tools/nixery/build-image/build-image.nix
@@ -137,11 +137,10 @@ let
     buildInputs = with pkgs; [ coreutils jq openssl ];
   }''
     layerSha256=$(sha256sum ${symlinkLayer} | cut -d ' ' -f1)
-    layerMd5=$(openssl dgst -md5 -binary ${symlinkLayer} | openssl enc -base64)
     layerSize=$(stat --printf '%s' ${symlinkLayer})
 
-    jq -n -c --arg sha256 $layerSha256 --arg md5 $layerMd5 --arg size $layerSize --arg path ${symlinkLayer} \
-      '{ size: ($size | tonumber), sha256: $sha256, md5: $md5, path: $path }' >> $out
+    jq -n -c --arg sha256 $layerSha256 --arg size $layerSize --arg path ${symlinkLayer} \
+      '{ size: ($size | tonumber), sha256: $sha256, path: $path }' >> $out
   ''));
 
   # Final output structure returned to Nixery if the build succeeded
diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md
index 175fe04d7084..b07d9e22f046 100644
--- a/tools/nixery/docs/src/caching.md
+++ b/tools/nixery/docs/src/caching.md
@@ -46,9 +46,8 @@ They are stored content-addressably at `$BUCKET/layers/$SHA256HASH` and layer
 requests sent to Nixery will redirect directly to this storage location.
 
 The effect of this cache is that Nixery does not need to upload identical layers
-repeatedly. When Nixery notices that a layer already exists in GCS, it will use
-the object metadata to compare its MD5-hash with the locally computed one and
-skip uploading.
+repeatedly. When Nixery notices that a layer already exists in GCS it will skip
+uploading this layer.
 
 Removing layers from the cache is *potentially problematic* if there are cached
 manifests or layer builds referencing those layers.
@@ -61,8 +60,8 @@ reference these layers.
 Layer builds are cached at `$BUCKET/builds/$HASH`, where `$HASH` is a SHA1 of
 the Nix store paths included in the layer.
 
-The content of the cached entries is a JSON-object that contains the MD5 and
-SHA256 hashes of the built layer.
+The content of the cached entries is a JSON-object that contains the SHA256
+hashes and sizes of the built layer.
 
 The effect of this cache is that different instances of Nixery will not build,
 hash and upload layers that have identical contents across different instances.
diff --git a/tools/nixery/docs/src/under-the-hood.md b/tools/nixery/docs/src/under-the-hood.md
index 6b5e5e9bbf21..b58a21d0d4ec 100644
--- a/tools/nixery/docs/src/under-the-hood.md
+++ b/tools/nixery/docs/src/under-the-hood.md
@@ -67,8 +67,7 @@ just ... hang, for a moment.
 
 Nixery inspects the returned manifest and uploads each layer to the configured
 [Google Cloud Storage][gcs] bucket. To avoid unnecessary uploading, it will
-first check whether layers are already present in the bucket and - just to be
-safe - compare their MD5-hashes against what was built.
+check whether layers are already present in the bucket.
 
 ## 4. The image manifest is sent back
 
diff --git a/tools/nixery/server/builder/builder.go b/tools/nixery/server/builder/builder.go
index f3342f9918f8..64cfed14399b 100644
--- a/tools/nixery/server/builder/builder.go
+++ b/tools/nixery/server/builder/builder.go
@@ -21,7 +21,6 @@ import (
 	"bufio"
 	"bytes"
 	"context"
-	"crypto/md5"
 	"crypto/sha256"
 	"encoding/json"
 	"fmt"
@@ -108,7 +107,6 @@ type ImageResult struct {
 	SymlinkLayer struct {
 		Size   int    `json:"size"`
 		SHA256 string `json:"sha256"`
-		MD5    string `json:"md5"`
 		Path   string `json:"path"`
 	} `json:"symlinkLayer"`
 }
@@ -328,8 +326,7 @@ func uploadHashLayer(ctx context.Context, s *State, key string, data io.Reader)
 	// algorithms and uploads to the bucket
 	sw := staging.NewWriter(ctx)
 	shasum := sha256.New()
-	md5sum := md5.New()
-	multi := io.MultiWriter(sw, shasum, md5sum)
+	multi := io.MultiWriter(sw, shasum)
 
 	size, err := io.Copy(multi, data)
 	if err != nil {
@@ -342,27 +339,24 @@ func uploadHashLayer(ctx context.Context, s *State, key string, data io.Reader)
 		return nil, err
 	}
 
-	build := Build{
-		SHA256: fmt.Sprintf("%x", shasum.Sum([]byte{})),
-		MD5:    fmt.Sprintf("%x", md5sum.Sum([]byte{})),
-	}
+	sha256sum := fmt.Sprintf("%x", shasum.Sum([]byte{}))
 
 	// Hashes are now known and the object is in the bucket, what
 	// remains is to move it to the correct location and cache it.
-	err = renameObject(ctx, s, "staging/"+key, "layers/"+build.SHA256)
+	err = renameObject(ctx, s, "staging/"+key, "layers/"+sha256sum)
 	if err != nil {
 		log.Printf("failed to move layer '%s' from staging: %s\n", key, err)
 		return nil, err
 	}
 
-	cacheBuild(ctx, s, key, build)
-
-	log.Printf("Uploaded layer sha256:%s (%v bytes written)", build.SHA256, size)
+	log.Printf("Uploaded layer sha256:%s (%v bytes written)", sha256sum, size)
 
-	return &manifest.Entry{
-		Digest: "sha256:" + build.SHA256,
+	entry := manifest.Entry{
+		Digest: "sha256:" + sha256sum,
 		Size:   size,
-	}, nil
+	}
+
+	return &entry, nil
 }
 
 func BuildImage(ctx context.Context, s *State, image *Image) (*BuildResult, error) {
diff --git a/tools/nixery/server/manifest/manifest.go b/tools/nixery/server/manifest/manifest.go
index 61d280a7fbab..f777e3f585df 100644
--- a/tools/nixery/server/manifest/manifest.go
+++ b/tools/nixery/server/manifest/manifest.go
@@ -3,7 +3,6 @@
 package manifest
 
 import (
-	"crypto/md5"
 	"crypto/sha256"
 	"encoding/json"
 	"fmt"
@@ -52,12 +51,11 @@ type imageConfig struct {
 }
 
 // ConfigLayer represents the configuration layer to be included in
-// the manifest, containing its JSON-serialised content and the SHA256
-// & MD5 hashes of its input.
+// the manifest, containing its JSON-serialised content and SHA256
+// hash.
 type ConfigLayer struct {
 	Config []byte
 	SHA256 string
-	MD5    string
 }
 
 // imageConfig creates an image configuration with the values set to
@@ -78,7 +76,6 @@ func configLayer(hashes []string) ConfigLayer {
 	return ConfigLayer{
 		Config: j,
 		SHA256: fmt.Sprintf("%x", sha256.Sum256(j)),
-		MD5:    fmt.Sprintf("%x", md5.Sum(j)),
 	}
 }
 
-- 
cgit 1.4.1


From 7b7d21205fb5288f1772d6ea4baff080565ebd9e Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Sun, 27 Oct 2019 13:42:24 +0100
Subject: docs: Update GCS signing key documentation

This key is now taken straight from the configured service account
key.
---
 tools/nixery/README.md                | 18 ++++++++++--------
 tools/nixery/docs/src/run-your-own.md |  8 ++++----
 2 files changed, 14 insertions(+), 12 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/README.md b/tools/nixery/README.md
index 3026451c74e0..1574d5950a22 100644
--- a/tools/nixery/README.md
+++ b/tools/nixery/README.md
@@ -79,15 +79,17 @@ variables:
 * `NIXERY_CHANNEL`: The name of a Nix/NixOS channel to use for building
 * `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses
   locally configured SSH/git credentials)
-* `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use
-  for building
+* `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to
+  use for building
 * `NIX_TIMEOUT`: Number of seconds that any Nix builder is allowed to run
-  (defaults to 60
-* `NIX_POPULARITY_URL`: URL to a file containing popularity data for the package set (see `popcount/`)
-* `GCS_SIGNING_KEY`: A Google service account key (in PEM format) that can be
-  used to sign Cloud Storage URLs
-* `GCS_SIGNING_ACCOUNT`: Google service account ID that the signing key belongs
-  to
+  (defaults to 60)
+* `NIX_POPULARITY_URL`: URL to a file containing popularity data for
+  the package set (see `popcount/`)
+
+If the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is set to a service
+account key, Nixery will also use this key to create [signed URLs][] for layers
+in the storage bucket. This makes it possible to serve layers from a bucket
+without having to make them publicly available.
 
 ## Roadmap
 
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
index 7a294f56055e..ffddec32db5f 100644
--- a/tools/nixery/docs/src/run-your-own.md
+++ b/tools/nixery/docs/src/run-your-own.md
@@ -85,15 +85,15 @@ You may set *all* of these:
 
 * `NIX_TIMEOUT`: Number of seconds that any Nix builder is allowed to run
   (defaults to 60)
-* `GCS_SIGNING_KEY`: A Google service account key (in PEM format) that can be
-  used to [sign Cloud Storage URLs][signed-urls]
-* `GCS_SIGNING_ACCOUNT`: Google service account ID that the signing key belongs
-  to
 
 To authenticate to the configured GCS bucket, Nixery uses Google's [Application
 Default Credentials][ADC]. Depending on your environment this may require
 additional configuration.
 
+If the `GOOGLE_APPLICATION_CREDENTIALS` environment is configured, the service
+account's private key will be used to create [signed URLs for
+layers][signed-urls].
+
 ## 4. Deploy Nixery
 
 With the above environment variables configured, you can run the image that was
-- 
cgit 1.4.1


From 3611baf040f19e234b81309822ac63723690a51d Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Mon, 28 Oct 2019 18:49:01 +0100
Subject: docs(under-the-hood): Update builder & storage backend information

Both of these no longer matched the reality of what was actually going
on in Nixery.
---
 tools/nixery/docs/src/under-the-hood.md | 79 +++++++++++++++++++++------------
 1 file changed, 51 insertions(+), 28 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/under-the-hood.md b/tools/nixery/docs/src/under-the-hood.md
index b58a21d0d4ec..4b798300100b 100644
--- a/tools/nixery/docs/src/under-the-hood.md
+++ b/tools/nixery/docs/src/under-the-hood.md
@@ -6,9 +6,9 @@ image is requested from Nixery.
 <!-- markdown-toc start - Don't edit this section. Run M-x markdown-toc-refresh-toc -->
 
 - [1. The image manifest is requested](#1-the-image-manifest-is-requested)
-- [2. Nix builds the image](#2-nix-builds-the-image)
-- [3. Layers are uploaded to Nixery's storage](#3-layers-are-uploaded-to-nixerys-storage)
-- [4. The image manifest is sent back](#4-the-image-manifest-is-sent-back)
+- [2. Nix fetches and prepares image content](#2-nix-fetches-and-prepares-image-content)
+- [3. Layers are grouped, created, hashed, and persisted](#3-layers-are-grouped-created-hashed-and-persisted)
+- [4. The manifest is assembled and returned to the client](#4-the-manifest-is-assembled-and-returned-to-the-client)
 - [5. Image layers are requested](#5-image-layers-are-requested)
 
 <!-- markdown-toc end -->
@@ -40,7 +40,7 @@ It then invokes Nix with three parameters:
 2. image tag
 3. configured package set source
 
-## 2. Nix builds the image
+## 2. Nix fetches and prepares image content
 
 Using the parameters above, Nix imports the package set and begins by mapping
 the image names to attributes in the package set.
@@ -50,12 +50,31 @@ their name, for example anything under `haskellPackages`. The registry protocol
 does not allow uppercase characters, so the Nix code will translate something
 like `haskellpackages` (lowercased) to the correct attribute name.
 
-After identifying all contents, Nix determines the contents of each layer while
-optimising for the best possible cache efficiency (see the [layering design
-doc][] for details).
+After identifying all contents, Nix uses the `symlinkJoin` function to
+create a special layer with the "symlink farm" required to let the
+image function like a normal disk image.
 
-Finally it builds each layer, assembles the image manifest as JSON structure,
-and yields this manifest back to the web server.
+Nix then returns information about the image contents as well as the
+location of the special layer to Nixery.
+
+## 3. Layers are grouped, created, hashed, and persisted
+
+With the information received from Nix, Nixery determines the contents
+of each layer while optimising for the best possible cache efficiency
+(see the [layering design doc][] for details).
+
+With the grouped layers, Nixery then begins to create compressed
+tarballs with all required contents for each layer. As these tarballs
+are being created, they are simultaneously being hashed (as the image
+manifest must contain the content-hashes of all layers) and persisted
+to storage.
+
+Storage can be either a remote [Google Cloud Storage][gcs] bucket, or
+a local filesystem path.
+
+During this step, Nixery checks its build cache (see [Caching][]) to
+determine whether a layer needs to be built or is already cached from
+a previous build.
 
 *Note:* While this step is running (which can take some time in the case of
 large first-time image builds), the registry client is left hanging waiting for
@@ -63,39 +82,43 @@ an HTTP response. Unfortunately the registry protocol does not allow for any
 feedback back to the user at this point, so from the user's perspective things
 just ... hang, for a moment.
 
-## 3. Layers are uploaded to Nixery's storage
-
-Nixery inspects the returned manifest and uploads each layer to the configured
-[Google Cloud Storage][gcs] bucket. To avoid unnecessary uploading, it will
-check whether layers are already present in the bucket.
+## 4. The manifest is assembled and returned to the client
 
-## 4. The image manifest is sent back
+Once armed with the hashes of all required layers, Nixery assembles
+the OCI Container Image manifest which describes the structure of the
+built image and names all of its layers by their content hash.
 
-If everything went well at this point, Nixery responds to the registry client
-with the image manifest.
-
-The client now inspects the manifest and basically sees a list of SHA256-hashes,
-each corresponding to one layer of the image. Most clients will now consult
-their local layer storage and determine which layers they are missing.
-
-Each of the missing layers is then requested from Nixery.
+This manifest is returned to the client.
 
 ## 5. Image layers are requested
 
-For each image layer that it needs to retrieve, the registry client assembles a
-request that looks like this:
+The client now inspects the manifest and determines which of the
+layers it is currently missing based on their content hashes. Note
+that different container runtimes will handle this differently, and in
+the case of certain engine and storage driver combinations (e.g.
+Docker with OverlayFS) layers might be downloaded again even if they
+are already present.
+
+For each of the missing layers, the client now issues a request to
+Nixery that looks like this:
 
 `GET /v2/${imageName}/blob/sha256:${layerHash}`
 
-Nixery receives these requests and *rewrites* them to Google Cloud Storage URLs,
-responding with an `HTTP 303 See Other` status code and the actual download URL
-of the layer.
+Nixery receives these requests and handles them based on the
+configured storage backend.
+
+If the storage backend is GCS, it *redirects* them to Google Cloud
+Storage URLs, responding with an `HTTP 303 See Other` status code and
+the actual download URL of the layer.
 
 Nixery supports using private buckets which are not generally world-readable, in
 which case [signed URLs][] are constructed using a private key. These allow the
 registry client to download each layer without needing to care about how the
 underlying authentication works.
 
+If the storage backend is the local filesystem, Nixery will attempt to
+serve the layer back to the client from disk.
+
 ---------
 
 That's it. After these five steps the registry client has retrieved all it needs
-- 
cgit 1.4.1


From bdda24a77287e09cb855eee7019148ee8bbd1cd9 Mon Sep 17 00:00:00 2001
From: Raphael Borun Das Gupta <git@raphael.dasgupta.ch>
Date: Fri, 1 May 2020 01:38:25 +0200
Subject: chore(nix): update channel 19.03 -> 20.03

Use a NixOS / NixPkgs release that's actually being supported
and regularly updated.
---
 tools/nixery/docs/src/caching.md             | 2 +-
 tools/nixery/docs/src/nixery.md              | 2 +-
 tools/nixery/docs/src/run-your-own.md        | 4 ++--
 tools/nixery/prepare-image/prepare-image.nix | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md
index b07d9e22f046..05ea6d89b655 100644
--- a/tools/nixery/docs/src/caching.md
+++ b/tools/nixery/docs/src/caching.md
@@ -29,7 +29,7 @@ Manifest caching *only* applies in the following cases:
 Manifest caching *never* applies in the following cases:
 
 * package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`)
-* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-19.03`)
+* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-20.03`)
 * package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`)
 
 It is thus always preferable to request images from a fully-pinned package
diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
index 6cc16431c9e8..185c84630889 100644
--- a/tools/nixery/docs/src/nixery.md
+++ b/tools/nixery/docs/src/nixery.md
@@ -54,7 +54,7 @@ own instance of Nixery.
 ### Which revision of `nixpkgs` is used for the builds?
 
 The instance at `nixery.dev` tracks a recent NixOS channel, currently NixOS
-19.03. The channel is updated several times a day.
+20.03. The channel is updated several times a day.
 
 Private registries might be configured to track a different channel (such as
 `nixos-unstable`) or even track a git repository with custom packages.
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
index ffddec32db5f..9c20e3f2cde7 100644
--- a/tools/nixery/docs/src/run-your-own.md
+++ b/tools/nixery/docs/src/run-your-own.md
@@ -44,7 +44,7 @@ be performed for trivial things.
 However if you are running a private Nixery, chances are high that you intend to
 use it with your own packages. There are three options available:
 
-1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-19.03` or
+1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-20.03` or
    `nixos-unstable`.
 2. Specify your own git-repository with a custom package set[^2]. This makes it
    possible to pull different tags, branches or commits by modifying the image
@@ -73,7 +73,7 @@ You must set *all* of these:
 * `BUCKET`: [Google Cloud Storage][gcs] bucket to store & serve image layers
 * `PORT`: HTTP port on which Nixery should listen
 
-You may set *one* of these, if unset Nixery defaults to `nixos-19.03`:
+You may set *one* of these, if unset Nixery defaults to `nixos-20.03`:
 
 * `NIXERY_CHANNEL`: The name of a Nix/NixOS channel to use for building
 * `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses
diff --git a/tools/nixery/prepare-image/prepare-image.nix b/tools/nixery/prepare-image/prepare-image.nix
index 4393f2b859a6..7b73b92bfd9b 100644
--- a/tools/nixery/prepare-image/prepare-image.nix
+++ b/tools/nixery/prepare-image/prepare-image.nix
@@ -24,7 +24,7 @@
 {
   # Description of the package set to be used (will be loaded by load-pkgs.nix)
   srcType ? "nixpkgs",
-  srcArgs ? "nixos-19.03",
+  srcArgs ? "nixos-20.03",
   system ? "x86_64-linux",
   importArgs ? { },
   # Path to load-pkgs.nix
-- 
cgit 1.4.1


From 7520f2cb96159ae3cbee80b4822900f9a92f7f53 Mon Sep 17 00:00:00 2001
From: Vincent Ambo <tazjin@google.com>
Date: Fri, 23 Apr 2021 11:00:06 +0200
Subject: chore: Update default NixOS channel to nixos-20.09

---
 tools/nixery/.github/workflows/build-and-test.yaml | 2 +-
 tools/nixery/docs/src/caching.md                   | 2 +-
 tools/nixery/docs/src/nixery.md                    | 2 +-
 tools/nixery/docs/src/run-your-own.md              | 4 ++--
 tools/nixery/nixpkgs-pin.nix                       | 4 ++--
 tools/nixery/prepare-image/prepare-image.nix       | 2 +-
 6 files changed, 8 insertions(+), 8 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/.github/workflows/build-and-test.yaml b/tools/nixery/.github/workflows/build-and-test.yaml
index 86a2c43637ff..4563239a852f 100644
--- a/tools/nixery/.github/workflows/build-and-test.yaml
+++ b/tools/nixery/.github/workflows/build-and-test.yaml
@@ -6,7 +6,7 @@ on:
       - master
   pull_request: {}
 env:
-  NIX_PATH: "nixpkgs=https://github.com/NixOS/nixpkgs-channels/archive/0a40a3999eb4d577418515da842a2622a64880c5.tar.gz"
+  NIX_PATH: "nixpkgs=https://github.com/NixOS/nixpkgs/archive/4263ba5e133cc3fc699c1152ab5ee46ef668e675.tar.gz"
 jobs:
   build-and-test:
     runs-on: ubuntu-latest
diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md
index 05ea6d89b655..05ea68ef6083 100644
--- a/tools/nixery/docs/src/caching.md
+++ b/tools/nixery/docs/src/caching.md
@@ -29,7 +29,7 @@ Manifest caching *only* applies in the following cases:
 Manifest caching *never* applies in the following cases:
 
 * package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`)
-* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-20.03`)
+* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-20.09`)
 * package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`)
 
 It is thus always preferable to request images from a fully-pinned package
diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
index 185c84630889..5f6dcb7e361c 100644
--- a/tools/nixery/docs/src/nixery.md
+++ b/tools/nixery/docs/src/nixery.md
@@ -54,7 +54,7 @@ own instance of Nixery.
 ### Which revision of `nixpkgs` is used for the builds?
 
 The instance at `nixery.dev` tracks a recent NixOS channel, currently NixOS
-20.03. The channel is updated several times a day.
+20.09. The channel is updated several times a day.
 
 Private registries might be configured to track a different channel (such as
 `nixos-unstable`) or even track a git repository with custom packages.
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
index 9c20e3f2cde7..4ffb8c4d4b6e 100644
--- a/tools/nixery/docs/src/run-your-own.md
+++ b/tools/nixery/docs/src/run-your-own.md
@@ -44,7 +44,7 @@ be performed for trivial things.
 However if you are running a private Nixery, chances are high that you intend to
 use it with your own packages. There are three options available:
 
-1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-20.03` or
+1. Specify an upstream Nix/NixOS channel[^1], such as `nixos-20.09` or
    `nixos-unstable`.
 2. Specify your own git-repository with a custom package set[^2]. This makes it
    possible to pull different tags, branches or commits by modifying the image
@@ -73,7 +73,7 @@ You must set *all* of these:
 * `BUCKET`: [Google Cloud Storage][gcs] bucket to store & serve image layers
 * `PORT`: HTTP port on which Nixery should listen
 
-You may set *one* of these, if unset Nixery defaults to `nixos-20.03`:
+You may set *one* of these, if unset Nixery defaults to `nixos-20.09`:
 
 * `NIXERY_CHANNEL`: The name of a Nix/NixOS channel to use for building
 * `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses
diff --git a/tools/nixery/nixpkgs-pin.nix b/tools/nixery/nixpkgs-pin.nix
index ea1b37bfe7a0..ffedc92ed384 100644
--- a/tools/nixery/nixpkgs-pin.nix
+++ b/tools/nixery/nixpkgs-pin.nix
@@ -1,4 +1,4 @@
 import (builtins.fetchTarball {
-  url = "https://github.com/NixOS/nixpkgs-channels/archive/0a40a3999eb4d577418515da842a2622a64880c5.tar.gz";
-  sha256 = "1j8gy2d61lmrp5gzi1a2jmb2v2pbk4b9666y8pf1pjg3jiqkzf7m";
+  url = "https://github.com/NixOS/nixpkgs/archive/4263ba5e133cc3fc699c1152ab5ee46ef668e675.tar.gz";
+  sha256 = "1nzqrdw0lhbldbs9r651zmgqpwhjhh9sssykhcl2155kgsfsrk7i";
 }) {}
diff --git a/tools/nixery/prepare-image/prepare-image.nix b/tools/nixery/prepare-image/prepare-image.nix
index 7b73b92bfd9b..316bfbbf2712 100644
--- a/tools/nixery/prepare-image/prepare-image.nix
+++ b/tools/nixery/prepare-image/prepare-image.nix
@@ -24,7 +24,7 @@
 {
   # Description of the package set to be used (will be loaded by load-pkgs.nix)
   srcType ? "nixpkgs",
-  srcArgs ? "nixos-20.03",
+  srcArgs ? "nixos-20.09",
   system ? "x86_64-linux",
   importArgs ? { },
   # Path to load-pkgs.nix
-- 
cgit 1.4.1


From f4daffbb50f947a49ec17e51d846f1ebb15fd41b Mon Sep 17 00:00:00 2001
From: Vincent Ambo <mail@tazj.in>
Date: Fri, 29 Oct 2021 17:29:33 +0200
Subject: chore(docs): Bump included nix-1p version

... basically never updated this, oops.
---
 tools/nixery/docs/default.nix | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/default.nix b/tools/nixery/docs/default.nix
index fdf0e6ff9ea5..d27cbe5b3e9e 100644
--- a/tools/nixery/docs/default.nix
+++ b/tools/nixery/docs/default.nix
@@ -24,8 +24,8 @@ let
   nix-1p = fetchFromGitHub {
     owner = "tazjin";
     repo = "nix-1p";
-    rev = "e0a051a016b9118bea90ec293d6cd346b9707e77";
-    sha256 = "0d1lfkxg03lki8dc3229g1cgqiq3nfrqgrknw99p6w0zk1pjd4dj";
+    rev = "9f0baf5e270128d9101ba4446cf6844889e399a2";
+    sha256 = "1pf9i90gn98vz67h296w5lnwhssk62dc6pij983dff42dbci7lhj";
   };
 in runCommand "nixery-book" { } ''
   mkdir -p $out
-- 
cgit 1.4.1


From 1dd342161550adfc58f3a2ea5e6c5843dc2c5ca9 Mon Sep 17 00:00:00 2001
From: Jérôme Petazzoni <jerome.petazzoni@gmail.com>
Date: Thu, 23 Dec 2021 12:10:39 +0100
Subject: docs: update installation instructions

These instructions were not up-to-date (they didn't mention
the different storage backends, and some variables were
tagged as optional while they were mandatory). With this
update, they should (hopefully) be more accurate! :)

I also added instructions if someone wants to run Nixery
outside of the container image (I found it convenient when
working on Nixery's code).
---
 tools/nixery/docs/src/run-your-own.md | 72 +++++++++++++++++++++++++++++------
 1 file changed, 60 insertions(+), 12 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
index 4ffb8c4d4b6e..eb7d494ee3e2 100644
--- a/tools/nixery/docs/src/run-your-own.md
+++ b/tools/nixery/docs/src/run-your-own.md
@@ -32,7 +32,16 @@ To run Nixery, you must have:
 * [Nix][] (to build Nixery itself)
 * Somewhere to run it (your own server, Google AppEngine, a Kubernetes cluster,
   whatever!)
-* A [Google Cloud Storage][gcs] bucket in which to store & serve layers
+* *Either* a [Google Cloud Storage][gcs] bucket in which to store & serve layers,
+  *or* a comfortable amount of disk space
+
+Note that while the main Nixery process is a server written in Go,
+it invokes a script that itself relies on Nix to be available.
+You can compile the main Nixery daemon without Nix, but it won't
+work without Nix.
+
+(If you are completely new to Nix and don't know how to get
+started, check the [Nix installation documentation][nixinstall].)
 
 ## 1. Choose a package set
 
@@ -54,8 +63,11 @@ use it with your own packages. There are three options available:
 
 ## 2. Build Nixery itself
 
-Building Nixery creates a container image. This section assumes that the
-container runtime used is Docker, please modify instructions correspondingly if
+### 2.1. With a container image
+
+The easiest way to run Nixery is to build a container image.
+This section assumes that the container runtime used is Docker,
+please modify instructions accordingly if
 you are using something else.
 
 With a working Nix installation, building Nixery is done by invoking `nix-build
@@ -64,23 +76,46 @@ With a working Nix installation, building Nixery is done by invoking `nix-build
 This will create a `result`-symlink which points to a tarball containing the
 image. In Docker, this tarball can be loaded by using `docker load -i result`.
 
+### 2.2. Without a container image
+
+*This method might be more convenient if you intend to work on
+the code of the Nixery server itself, because you won't have to
+rebuild (and reload) an image each time to test your changes.*
+
+You will need to run the two following commands at the root of the repo:
+
+* `go build` to build the `nixery` binary;
+* `nix-env --install --file prepare-image/default.nix` to build
+  the required helpers.
+
 ## 3. Prepare configuration
 
 Nixery is configured via environment variables.
 
 You must set *all* of these:
 
-* `BUCKET`: [Google Cloud Storage][gcs] bucket to store & serve image layers
+* `NIXERY_STORAGE_BACKEND` (must be set to `gcs` or `filesystem`)
 * `PORT`: HTTP port on which Nixery should listen
+* `WEB_DIR`: directory containing static files (see below)
 
-You may set *one* of these, if unset Nixery defaults to `nixos-20.09`:
+You must set *one* of these:
 
-* `NIXERY_CHANNEL`: The name of a Nix/NixOS channel to use for building
+* `NIXERY_CHANNEL`: The name of a [Nix/NixOS channel][nixchannel] to use for building,
+  for instance `nixos-21.05`
 * `NIXERY_PKGS_REPO`: URL of a git repository containing a package set (uses
   locally configured SSH/git credentials)
 * `NIXERY_PKGS_PATH`: A local filesystem path containing a Nix package set to use
   for building
 
+If `NIXERY_STORAGE_BACKEND` is set to `filesystem`, then `STORAGE_PATH`
+must be set to the directory that will hold the registry blobs.
+That directory must be located on a filesystem that supports extended
+attributes (which means that on most systems, `/tmp` won't work).
+
+If `NIXERY_STORAGE_BACKEND` is set to `gcs`, then `GCS_BUCKET`
+must be set to the [Google Cloud Storage][gcs] bucket that will be
+used to store & serve image layers.
+
 You may set *all* of these:
 
 * `NIX_TIMEOUT`: Number of seconds that any Nix builder is allowed to run
@@ -94,13 +129,11 @@ If the `GOOGLE_APPLICATION_CREDENTIALS` environment is configured, the service
 account's private key will be used to create [signed URLs for
 layers][signed-urls].
 
-## 4. Deploy Nixery
-
-With the above environment variables configured, you can run the image that was
-built in step 2.
+## 4. Start Nixery
 
-How this works depends on the environment you are using and is, for now, outside
-of the scope of this tutorial.
+Run the image that was built in step 2.1 with all the environment variables
+mentioned above. Alternatively, set all the environment variables and run
+the Nixery server that was built in step 2.2.
 
 Once Nixery is running you can immediately start requesting images from it.
 
@@ -125,6 +158,19 @@ following:
 * Configure request timeouts for Nixery if you have your own web server in front
   of it. This will be natively supported by Nixery in the future.
 
+## 6. `WEB_DIR`
+
+All the URLs accessed by Docker registry clients start with `/v2/`.
+This means that it is possible to serve a static website from Nixery
+itself (as long as you don't want to serve anything starting with `/v2`).
+This is how, for instance, https://nixery.dev shows the website for Nixery,
+while it is also possible to e.g. `docker pull nixery.dev/shell`.
+
+When running Nixery, you must set the `WEB_DIR` environment variable.
+When Nixery receives requests that don't look like registry requests,
+it tries to serve them using files in the directory indicated by `WEB_DIR`.
+If the directory doesn't exist, Nixery will run fine but serve 404.
+
 -------
 
 [^1]: Nixery will not work with Nix channels older than `nixos-19.03`.
@@ -141,3 +187,5 @@ following:
 [repo]: https://github.com/google/nixery
 [signed-urls]: under-the-hood.html#5-image-layers-are-requested
 [ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
+[nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html
+[nixchannel]: https://nixos.wiki/wiki/Nix_channels
-- 
cgit 1.4.1


From 15f79e1364410bc78d1d734f4af7617f2ab8e432 Mon Sep 17 00:00:00 2001
From: Ethan Davidson <ethanmdavidson@gmail.com>
Date: Thu, 9 Dec 2021 13:42:35 -0500
Subject: docs: mention arm64 metapackage

---
 tools/nixery/docs/src/nixery.md | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
index 5f6dcb7e361c..3f68311dabb4 100644
--- a/tools/nixery/docs/src/nixery.md
+++ b/tools/nixery/docs/src/nixery.md
@@ -33,8 +33,10 @@ Each path segment corresponds either to a key in the Nix package set, or a
 meta-package that automatically expands to several other packages.
 
 Meta-packages **must** be the first path component if they are used. Currently
-the only meta-package is `shell`, which provides a `bash`-shell with interactive
-configuration and standard tools like `coreutils`.
+there are only two meta-packages:
+- `shell`, which provides a `bash`-shell with interactive configuration and 
+  standard tools like `coreutils`.
+- `arm64`, which provides ARM64 binaries.
 
 **Tip:** When pulling from a private Nixery instance, replace `nixery.dev` in
 the above examples with your registry address.
-- 
cgit 1.4.1


From 3d26ea9e636e9cd137d9430dd36f672e83239e7b Mon Sep 17 00:00:00 2001
From: Raphael Borun Das Gupta <git@raphael.dasgupta.ch>
Date: Tue, 19 Apr 2022 21:32:46 +0200
Subject: docs: change references to repo URL

The Nixery main Git repo has moved
from https://github.com/google/nixery
to https://github.com/tazjin/nixery .

So change it in README and on the https://nixery.dev/ website.
---
 tools/nixery/README.md                | 4 ++--
 tools/nixery/docs/src/nixery.md       | 2 +-
 tools/nixery/docs/src/run-your-own.md | 4 ++--
 3 files changed, 5 insertions(+), 5 deletions(-)

(limited to 'tools/nixery/docs')

diff --git a/tools/nixery/README.md b/tools/nixery/README.md
index 90799fac3feb..cba8ce6b14f6 100644
--- a/tools/nixery/README.md
+++ b/tools/nixery/README.md
@@ -4,7 +4,7 @@
 
 -----------------
 
-[![Build Status](https://github.com/google/nixery/actions/workflows/build-and-test.yaml/badge.svg)](https://github.com/google/nixery/actions/workflows/build-and-test.yaml)
+[![Build Status](https://github.com/tazjin/nixery/actions/workflows/build-and-test.yaml/badge.svg)](https://github.com/tazjin/nixery/actions/workflows/build-and-test.yaml)
 
 **Nixery** is a Docker-compatible container registry that is capable of
 transparently building and serving container images using [Nix][].
@@ -130,7 +130,7 @@ outlined in [a public gist][gist].
 It should be trivial to deploy Nixery inside of a Kubernetes cluster with
 correct caching behaviour, addressing and so on.
 
-See [issue #4](https://github.com/google/nixery/issues/4).
+See [issue #4](https://github.com/tazjin/nixery/issues/4).
 
 ### Nix-native builder
 
diff --git a/tools/nixery/docs/src/nixery.md b/tools/nixery/docs/src/nixery.md
index 3f68311dabb4..7b78ddf5aaf8 100644
--- a/tools/nixery/docs/src/nixery.md
+++ b/tools/nixery/docs/src/nixery.md
@@ -77,7 +77,7 @@ availability.
 Nixery was written by [tazjin][], but many people have contributed to Nix over
 time, maybe you could become one of them?
 
-[Nixery]: https://github.com/google/nixery
+[Nixery]: https://github.com/tazjin/nixery
 [Nix]: https://nixos.org/nix
 [layering strategy]: https://storage.googleapis.com/nixdoc/nixery-layers.html
 [layers]: https://grahamc.com/blog/nix-and-layered-docker-images
diff --git a/tools/nixery/docs/src/run-your-own.md b/tools/nixery/docs/src/run-your-own.md
index eb7d494ee3e2..cf4dc2ce6166 100644
--- a/tools/nixery/docs/src/run-your-own.md
+++ b/tools/nixery/docs/src/run-your-own.md
@@ -181,10 +181,10 @@ If the directory doesn't exist, Nixery will run fine but serve 404.
     extensively.
 
 [GKE]: https://cloud.google.com/kubernetes-engine/
-[nixery#4]: https://github.com/google/nixery/issues/4
+[nixery#4]: https://github.com/tazjin/nixery/issues/4
 [Nix]: https://nixos.org/nix
 [gcs]: https://cloud.google.com/storage/
-[repo]: https://github.com/google/nixery
+[repo]: https://github.com/tazjin/nixery
 [signed-urls]: under-the-hood.html#5-image-layers-are-requested
 [ADC]: https://cloud.google.com/docs/authentication/production#finding_credentials_automatically
 [nixinstall]: https://nixos.org/manual/nix/stable/installation/installing-binary.html
-- 
cgit 1.4.1