feat(tools/nixery): Absorb Nixery into depot r/3978

This absorbs a josh-filtered Nix subtree into depot, at
//tools/nixery.

This subtree was created through `josh-filter ':prefix=tools/nixery'`,
which allows a filter on tools/nixery to yield the same commit hashes
as the original Nixery repository (allowing for history continuity).

Change-Id: Icc1a99bf1248226b91f437b0a90361d36fb0d327

1 files changed, 69 insertions, 0 deletions

diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md
new file mode 100644
index 0000000000..05ea68ef60
--- /dev/null
+++ b/tools/nixery/docs/src/caching.md
@@ -0,0 +1,69 @@
+# Caching in Nixery
+
+This page gives a quick overview over the caching done by Nixery. All cache data
+is written to Nixery's storage bucket and is based on deterministic identifiers
+or content-addressing, meaning that cache entries under the same key *never
+change*.
+
+## Manifests
+
+Manifests of builds are cached at `$BUCKET/manifests/$KEY`. The effect of this
+cache is that multiple instances of Nixery do not need to rebuild the same
+manifest from scratch.
+
+Since the manifest cache is populated only *after* layers are uploaded, Nixery
+can immediately return the manifest to its clients without needing to check
+whether layers have been uploaded already.
+
+`$KEY` is generated by creating a SHA1 hash of the requested content of a
+manifest plus the package source specification.
+
+Manifests are *only* cached if the package source specification is *not* a
+moving target.
+
+Manifest caching *only* applies in the following cases:
+
+* package source specification is a specific git commit
+* package source specification is a specific NixOS/nixpkgs commit
+
+Manifest caching *never* applies in the following cases:
+
+* package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`)
+* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-20.09`)
+* package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`)
+
+It is thus always preferable to request images from a fully-pinned package
+source.
+
+Manifests can be removed from the manifest cache without negative consequences.
+
+## Layer tarballs
+
+Layer tarballs are the files that Nixery clients retrieve from the storage
+bucket to download an image.
+
+They are stored content-addressably at `$BUCKET/layers/$SHA256HASH` and layer
+requests sent to Nixery will redirect directly to this storage location.
+
+The effect of this cache is that Nixery does not need to upload identical layers
+repeatedly. When Nixery notices that a layer already exists in GCS it will skip
+uploading this layer.
+
+Removing layers from the cache is *potentially problematic* if there are cached
+manifests or layer builds referencing those layers.
+
+To clean up layers, a user must ensure that no other cached resources still
+reference these layers.
+
+## Layer builds
+
+Layer builds are cached at `$BUCKET/builds/$HASH`, where `$HASH` is a SHA1 of
+the Nix store paths included in the layer.
+
+The content of the cached entries is a JSON-object that contains the SHA256
+hashes and sizes of the built layer.
+
+The effect of this cache is that different instances of Nixery will not build,
+hash and upload layers that have identical contents across different instances.
+
+Layer builds can be removed from the cache without negative consequences.