diff options
Diffstat (limited to 'tools/nixery/docs/src/caching.md')
-rw-r--r-- | tools/nixery/docs/src/caching.md | 70 |
1 files changed, 70 insertions, 0 deletions
diff --git a/tools/nixery/docs/src/caching.md b/tools/nixery/docs/src/caching.md new file mode 100644 index 000000000000..175fe04d7084 --- /dev/null +++ b/tools/nixery/docs/src/caching.md @@ -0,0 +1,70 @@ +# Caching in Nixery + +This page gives a quick overview over the caching done by Nixery. All cache data +is written to Nixery's storage bucket and is based on deterministic identifiers +or content-addressing, meaning that cache entries under the same key *never +change*. + +## Manifests + +Manifests of builds are cached at `$BUCKET/manifests/$KEY`. The effect of this +cache is that multiple instances of Nixery do not need to rebuild the same +manifest from scratch. + +Since the manifest cache is populated only *after* layers are uploaded, Nixery +can immediately return the manifest to its clients without needing to check +whether layers have been uploaded already. + +`$KEY` is generated by creating a SHA1 hash of the requested content of a +manifest plus the package source specification. + +Manifests are *only* cached if the package source specification is *not* a +moving target. + +Manifest caching *only* applies in the following cases: + +* package source specification is a specific git commit +* package source specification is a specific NixOS/nixpkgs commit + +Manifest caching *never* applies in the following cases: + +* package source specification is a local file path (i.e. `NIXERY_PKGS_PATH`) +* package source specification is a NixOS channel (e.g. `NIXERY_CHANNEL=nixos-19.03`) +* package source specification is a git branch or tag (e.g. `staging`, `master` or `latest`) + +It is thus always preferable to request images from a fully-pinned package +source. + +Manifests can be removed from the manifest cache without negative consequences. + +## Layer tarballs + +Layer tarballs are the files that Nixery clients retrieve from the storage +bucket to download an image. + +They are stored content-addressably at `$BUCKET/layers/$SHA256HASH` and layer +requests sent to Nixery will redirect directly to this storage location. + +The effect of this cache is that Nixery does not need to upload identical layers +repeatedly. When Nixery notices that a layer already exists in GCS, it will use +the object metadata to compare its MD5-hash with the locally computed one and +skip uploading. + +Removing layers from the cache is *potentially problematic* if there are cached +manifests or layer builds referencing those layers. + +To clean up layers, a user must ensure that no other cached resources still +reference these layers. + +## Layer builds + +Layer builds are cached at `$BUCKET/builds/$HASH`, where `$HASH` is a SHA1 of +the Nix store paths included in the layer. + +The content of the cached entries is a JSON-object that contains the MD5 and +SHA256 hashes of the built layer. + +The effect of this cache is that different instances of Nixery will not build, +hash and upload layers that have identical contents across different instances. + +Layer builds can be removed from the cache without negative consequences. |