diff options
Diffstat (limited to 'tvix/docs')
| -rw-r--r-- | tvix/docs/.gitignore | 2 | ||||
| -rw-r--r-- | tvix/docs/Makefile | 12 | ||||
| -rw-r--r-- | tvix/docs/component-flow.puml | 74 | ||||
| -rw-r--r-- | tvix/docs/components.md | 114 | ||||
| -rw-r--r-- | tvix/docs/default.nix | 39 |
5 files changed, 241 insertions, 0 deletions
diff --git a/tvix/docs/.gitignore b/tvix/docs/.gitignore new file mode 100644 index 000000000000..77699ee8a3f7 --- /dev/null +++ b/tvix/docs/.gitignore @@ -0,0 +1,2 @@ +*.svg +*.html diff --git a/tvix/docs/Makefile b/tvix/docs/Makefile new file mode 100644 index 000000000000..ba9e2bdef6d3 --- /dev/null +++ b/tvix/docs/Makefile @@ -0,0 +1,12 @@ +all: build + +puml: + plantuml *.puml -tsvg + +html: + pandoc *.md -f markdown --self-contained -t html -s -o tvix.html --csl=${CSL} + +build: puml html + +clean: + rm -f *.tex *.pdf *.png *.svg diff --git a/tvix/docs/component-flow.puml b/tvix/docs/component-flow.puml new file mode 100644 index 000000000000..3bcddbe7464e --- /dev/null +++ b/tvix/docs/component-flow.puml @@ -0,0 +1,74 @@ +@startuml + +title Tvix build flow + +actor User +participant CLI +participant "Coordinator" as Coord +participant "Evaluator" as Eval +database Store +participant "Builder" as Build + +note over CLI,Eval + Typically runs locally on the invoking machine +end note +/ note over Store, Build + Can be either local or remote +end note + +User-->CLI: User initiates build of `hello` (analogous to `nix-build -f '<nixpkgs>' -A hello`) + +CLI-->Coord: CLI invokes coordinator + +Coord-->Eval: Sends message to start evaluation of `<nixpkgs>` (path lookup) with attribute `hello` +note right: The paths to the evaluator are local file system paths + +Coord<--Eval: Yields derivations to be built +note right + Immediately starts streaming derivations as they are instantiated across + the dependency graph so they can be built while the evaluation is still running. + + There are two types of build requests: One for regular "fire and forget" builds + and another for IFD (import from derivation). + + These are distinct because IFD needs to be fed back into the evaluator for + further processing while a regular build does not. +end note + +loop while has more derivations + + Coord-->Store: Check if desired paths are in store + alt Store has path + Coord<--Store: Success response + else Store does not have path + Coord-->Build: Request derivation to be built + note left + The build request optionally includes a desired store. + If a builder is aware of how to push to the store it will do so + directly when the build is finished. + + If the store is not known by the builder results will be streamed + back to the coordinator for store addition. + end note + + alt Build failure + Coord<--Build: Fail response + note left: It's up to the coordinator whether to exit on build failure + else Build success + alt Known store + Build-->Store: Push outputs to store + Build<--Coord: Send success & pushed response + else Unknown store + Build<--Coord: Send success & not pushed response + Coord<--Build: Stream build outputs + Coord-->Store: Push outputs to store + end + end + + end +end + +CLI<--Coord: Respond success/fail +User<--CLI: Exit success/fail + +@enduml diff --git a/tvix/docs/components.md b/tvix/docs/components.md new file mode 100644 index 000000000000..19e7baa3ec8a --- /dev/null +++ b/tvix/docs/components.md @@ -0,0 +1,114 @@ +--- +title: "Tvix - Architecture & data flow" +numbersections: true +author: +- adisbladis +- flokli +- tazjin +email: +- adis@blad.is +- mail@tazj.in +lang: en-GB +classoption: +- twocolumn +header-includes: +- \usepackage{caption, graphicx, tikz, aeguill, pdflscape} +--- + +# Background + +We intend for Tvix tooling to be more decoupled than the existing, +monolithic Nix implementation. In practice, we expect to gain several +benefits from this, such as: + +- Ability to use different builders +- Ability to use different store implementations +- No monopolisation of the implementation, allowing users to replace + components that they are unhappy with (up to and including the + language evaluator) +- Less hidden intra-dependencies between tools due to explicit RPC/IPC + boundaries + +Communication between different components of the system will use +gRPC. The rest of this document outlines the components. + +# Components + +## Coordinator + +*Purpose:* The coordinator (in the simplest case, the Tvix CLI tool) +oversees the flow of a build process and delegates tasks to the right +subcomponents. For example, if a user runs the equivalent of +`nix-build` in a folder containing a `default.nix` file, the +coordinator will invoke the evaluator, pass the resulting derivations +to the builder and coordinate any necessary store interactions (for +substitution and other purposes). + +While many users are likely to use the CLI tool as their primary +method of interacting with Tvix, it is not unlikely that alternative +coordinators (e.g. for a distributed, "Nix-native" CI system) would be +implemented. To facilitate this, we are considering implementing the +coordinator on top of a state-machine model that would make it +possible to reuse the FSM logic without tying it to any particular +kind of application. + +## Evaluator + +*Purpose:* Eval takes care of evaluating Nix code. In a typical build +flow it would be responsible for producing derivations. It can also be +used as a standalone tool, for example, in use-cases where Nix is used +to generate configuration without any build or store involvement. + +*Requirements:* For now, it will run on the machine invoking the build +command itself. We give it filesystem access to handle things like +imports or `builtins.readFile`. + +In the future, we might abstract away raw filesystem access by +allowing the evaluator to request files from the coordinator (which +will query the store for it). This might get messy, and the benefits +are questionable. We might be okay with running the evaluator with +filesystem access for now and can extend the interface if the need +arises. + +## Builder + +*Purpose:* A builder receives derivations from the coordinator and +builds them. + +By making builder a standardised interface it's possible to make the +sandboxing mechanism used by the build process pluggable. + +Nix is currently using a hard-coded +[libseccomp](https://github.com/seccomp/libseccomp) based sandboxing +mechanism and another one based on +[sandboxd](https://www.unix.com/man-page/mojave/8/sandboxd/) on macOS. +These are only separated by [compiler preprocessor +macros](https://gcc.gnu.org/onlinedocs/cpp/Ifdef.html) within the same +source files despite having very little in common with each other. + +This makes experimentation with alternative backends difficult and +porting Nix to other platforms harder than it has to be. We want to +write a new Linux builder which uses +[OCI](https://github.com/opencontainers/runtime-spec), the current +dominant Linux containerisation technology, by default. + +With a well-defined builder abstraction, it's also easy to imagine +other backends such as a Kubernetes-based one in the future. + +## Store + +*Purpose:* Store takes care of storing build results. It provides a +unified interface to get file paths and upload new ones. + +Most likely, we will end up with multiple implementations of store, a +few possible ones that come to mind are: + +- Local +- SSH +- GCP +- S3 +- Ceph + +# Figures + + diff --git a/tvix/docs/default.nix b/tvix/docs/default.nix new file mode 100644 index 000000000000..606027424a11 --- /dev/null +++ b/tvix/docs/default.nix @@ -0,0 +1,39 @@ +{ pkgs, lib, ... }: + +let + + tl = pkgs.texlive.combine { + inherit (pkgs.texlive) scheme-medium wrapfig ulem capt-of + titlesec preprint enumitem paralist ctex environ svg + beamer trimspaces zhnumber changepage framed pdfpages + fvextra minted upquote ifplatform xstring; + }; + + csl = pkgs.fetchurl { + name = "numeric.csl"; + url = "https://gist.githubusercontent.com/bwiernik/8c6f39cf51ceb3a03107/raw/1d75c2d62113ffbba6ed03a47ad99bde86934f2b/APA%2520Numeric"; + sha256 = "1yfhhnhbzvhrv93baz98frmgsx5y442nzhb0l956l4j35fb0cc3h"; + }; + +in pkgs.stdenv.mkDerivation { + pname = "tvix-doc"; + version = "0.1"; + + src = lib.cleanSource ./.; + + CSL = csl; + + nativeBuildInputs = [ + pkgs.pandoc + pkgs.plantuml + tl + ]; + + installPhase = '' + runHook preInstall + mkdir -p $out + cp -v *.html $out/ + runHook postSubmit + ''; + +} |