6 files changed, 326 insertions, 0 deletions
diff --git a/tvix/docs/.gitignore b/tvix/docs/.gitignore
new file mode 100644
index 000000000000..77699ee8a3f7
--- /dev/null
+++ b/tvix/docs/.gitignore
@@ -0,0 +1,2 @@
+*.svg
+*.html
diff --git a/tvix/docs/Makefile b/tvix/docs/Makefile
new file mode 100644
index 000000000000..ba9e2bdef6d3
--- /dev/null
+++ b/tvix/docs/Makefile
@@ -0,0 +1,12 @@
+all: build
+
+puml:
+	plantuml *.puml -tsvg
+
+html:
+	pandoc *.md -f markdown --self-contained -t html -s -o tvix.html --csl=${CSL}
+
+build: puml html
+
+clean:
+	rm -f *.tex *.pdf *.png *.svg
diff --git a/tvix/docs/component-flow.puml b/tvix/docs/component-flow.puml
new file mode 100644
index 000000000000..3bcddbe7464e
--- /dev/null
+++ b/tvix/docs/component-flow.puml
@@ -0,0 +1,74 @@
+@startuml
+
+title Tvix build flow
+
+actor User
+participant CLI
+participant "Coordinator" as Coord
+participant "Evaluator" as Eval
+database Store
+participant "Builder" as Build
+
+note over CLI,Eval
+    Typically runs locally on the invoking machine
+end note
+/ note over Store, Build
+    Can be either local or remote
+end note
+
+User-->CLI: User initiates build of `hello` (analogous to `nix-build -f '<nixpkgs>' -A hello`)
+
+CLI-->Coord: CLI invokes coordinator
+
+Coord-->Eval: Sends message to start evaluation of `<nixpkgs>` (path lookup) with attribute `hello`
+note right: The paths to the evaluator are local file system paths
+
+Coord<--Eval: Yields derivations to be built
+note right
+    Immediately starts streaming derivations as they are instantiated across
+    the dependency graph so they can be built while the evaluation is still running.
+
+    There are two types of build requests: One for regular "fire and forget" builds
+    and another for IFD (import from derivation).
+
+    These are distinct because IFD needs to be fed back into the evaluator for
+    further processing while a regular build does not.
+end note
+
+loop while has more derivations
+
+    Coord-->Store: Check if desired paths are in store
+    alt Store has path
+        Coord<--Store: Success response
+    else Store does not have path
+        Coord-->Build: Request derivation to be built
+        note left
+            The build request optionally includes a desired store.
+            If a builder is aware of how to push to the store it will do so
+            directly when the build is finished.
+
+            If the store is not known by the builder results will be streamed
+            back to the coordinator for store addition.
+        end note
+
+        alt Build failure
+            Coord<--Build: Fail response
+            note left: It's up to the coordinator whether to exit on build failure
+        else Build success
+            alt Known store
+                Build-->Store: Push outputs to store
+                Build<--Coord: Send success & pushed response
+            else Unknown store
+                Build<--Coord: Send success & not pushed response
+                Coord<--Build: Stream build outputs
+                Coord-->Store: Push outputs to store
+            end
+        end
+
+    end
+end
+
+CLI<--Coord: Respond success/fail
+User<--CLI: Exit success/fail
+
+@enduml
diff --git a/tvix/docs/components.md b/tvix/docs/components.md
new file mode 100644
index 000000000000..19e7baa3ec8a
--- /dev/null
+++ b/tvix/docs/components.md
@@ -0,0 +1,114 @@
+---
+title: "Tvix - Architecture & data flow"
+numbersections: true
+author:
+- adisbladis
+- flokli
+- tazjin
+email:
+- adis@blad.is
+- mail@tazj.in
+lang: en-GB
+classoption:
+- twocolumn
+header-includes:
+- \usepackage{caption, graphicx, tikz, aeguill, pdflscape}
+---
+
+# Background
+
+We intend for Tvix tooling to be more decoupled than the existing,
+monolithic Nix implementation. In practice, we expect to gain several
+benefits from this, such as:
+
+- Ability to use different builders
+- Ability to use different store implementations
+- No monopolisation of the implementation, allowing users to replace
+  components that they are unhappy with (up to and including the
+  language evaluator)
+- Less hidden intra-dependencies between tools due to explicit RPC/IPC
+  boundaries
+
+Communication between different components of the system will use
+gRPC. The rest of this document outlines the components.
+
+# Components
+
+## Coordinator
+
+*Purpose:* The coordinator (in the simplest case, the Tvix CLI tool)
+oversees the flow of a build process and delegates tasks to the right
+subcomponents. For example, if a user runs the equivalent of
+`nix-build` in a folder containing a `default.nix` file, the
+coordinator will invoke the evaluator, pass the resulting derivations
+to the builder and coordinate any necessary store interactions (for
+substitution and other purposes).
+
+While many users are likely to use the CLI tool as their primary
+method of interacting with Tvix, it is not unlikely that alternative
+coordinators (e.g. for a distributed, "Nix-native" CI system) would be
+implemented. To facilitate this, we are considering implementing the
+coordinator on top of a state-machine model that would make it
+possible to reuse the FSM logic without tying it to any particular
+kind of application.
+
+## Evaluator
+
+*Purpose:* Eval takes care of evaluating Nix code. In a typical build
+flow it would be responsible for producing derivations. It can also be
+used as a standalone tool, for example, in use-cases where Nix is used
+to generate configuration without any build or store involvement.
+
+*Requirements:* For now, it will run on the machine invoking the build
+command itself. We give it filesystem access to handle things like
+imports or `builtins.readFile`.
+
+In the future, we might abstract away raw filesystem access by
+allowing the evaluator to request files from the coordinator (which
+will query the store for it). This might get messy, and the benefits
+are questionable. We might be okay with running the evaluator with
+filesystem access for now and can extend the interface if the need
+arises.
+
+## Builder
+
+*Purpose:* A builder receives derivations from the coordinator and
+builds them.
+
+By making builder a standardised interface it's possible to make the
+sandboxing mechanism used by the build process pluggable.
+
+Nix is currently using a hard-coded
+[libseccomp](https://github.com/seccomp/libseccomp) based sandboxing
+mechanism and another one based on
+[sandboxd](https://www.unix.com/man-page/mojave/8/sandboxd/) on macOS.
+These are only separated by [compiler preprocessor
+macros](https://gcc.gnu.org/onlinedocs/cpp/Ifdef.html) within the same
+source files despite having very little in common with each other.
+
+This makes experimentation with alternative backends difficult and
+porting Nix to other platforms harder than it has to be. We want to
+write a new Linux builder which uses
+[OCI](https://github.com/opencontainers/runtime-spec), the current
+dominant Linux containerisation technology, by default.
+
+With a well-defined builder abstraction, it's also easy to imagine
+other backends such as a Kubernetes-based one in the future.
+
+## Store
+
+*Purpose:* Store takes care of storing build results. It provides a
+unified interface to get file paths and upload new ones.
+
+Most likely, we will end up with multiple implementations of store, a
+few possible ones that come to mind are:
+
+- Local
+- SSH
+- GCP
+- S3
+- Ceph
+
+# Figures
+
+![component flow](./component-flow.svg)
diff --git a/tvix/docs/default.nix b/tvix/docs/default.nix
new file mode 100644
index 000000000000..4b09f8d2dae2
--- /dev/null
+++ b/tvix/docs/default.nix
@@ -0,0 +1,46 @@
+{ pkgs, lib, ... }:
+
+let
+
+  tl = pkgs.texlive.combine {
+    inherit (pkgs.texlive) scheme-medium wrapfig ulem capt-of
+    titlesec preprint enumitem paralist ctex environ svg
+    beamer trimspaces zhnumber changepage framed pdfpages
+    fvextra minted upquote ifplatform xstring;
+  };
+
+  csl = pkgs.fetchurl {
+    name = "numeric.csl";
+    url = "https://gist.githubusercontent.com/bwiernik/8c6f39cf51ceb3a03107/raw/1d75c2d62113ffbba6ed03a47ad99bde86934f2b/APA%2520Numeric";
+    sha256 = "1yfhhnhbzvhrv93baz98frmgsx5y442nzhb0l956l4j35fb0cc3h";
+  };
+
+in pkgs.stdenv.mkDerivation {
+  pname = "tvix-doc";
+  version = "0.1";
+
+  outputs = [ "out" "svg" ];
+
+  src = lib.cleanSource ./.;
+
+  CSL = csl;
+
+  nativeBuildInputs = [
+    pkgs.pandoc
+    pkgs.plantuml
+    tl
+  ];
+
+  installPhase = ''
+    runHook preInstall
+
+    mkdir -p $out
+    cp -v *.html $out/
+
+    mkdir -p $svg
+    cp -v *.svg $svg/
+
+    runHook postSubmit
+  '';
+
+}
diff --git a/tvix/docs/language-spec.md b/tvix/docs/language-spec.md
new file mode 100644
index 000000000000..a71437493307
--- /dev/null
+++ b/tvix/docs/language-spec.md
@@ -0,0 +1,78 @@
+---
+title: "Specification of the Nix language"
+numbersections: true
+author:
+- tazjin
+email:
+- tazjin@tvl.su
+lang: en-GB
+---
+
+The Nix Language
+================
+
+WARNING: This document is a work in progress. Please keep an eye on
+[`topic:nix-spec`](https://cl.tvl.fyi/q/topic:nix-spec) for ongoing
+CLs.
+
+Nix is a general-purpose, functional programming language which this
+document aims to describe.
+
+## Background
+
+Nix was designed and implemented as part of the [Nix package
+manager](https://nixos.org/nix). It is primarily used for generating
+so-called [*derivations*](#derivations), which are data structures
+describing how to build a package.
+
+The language has been described in the
+[thesis](https://edolstra.github.io/pubs/phd-thesis.pdf) introducing
+the package manager, but only on a high-level. At the time of writing,
+Nix is informally specified (via its only complete implementation in
+the package manager) and there is no complete overview over its -
+sometimes surprising - semantics.
+
+The primary project written in Nix is
+[nixpkgs](https://github.com/NixOS/nixpkgs/). Uncertainties in the
+process of writing this specification are resolved by investigating
+patterns in nixpkgs, which we consider canonical. The code in nixpkgs
+uses a reasonable subset of the features exposed by the current
+implementation, some of which are *accidental*, and is thus more
+useful for specifying how the language should work.
+
+## Introduction to Nix
+
+Nix is a general-purpose, partially lazy, functional programming
+language which provides higher-order functions, type reflection,
+primitive data types such as integers, strings and floats, and
+compound data structures such as lists and attribute sets.
+
+Nix has syntactic sugar for common operations, such as those for
+attribute sets, and also provides a wide range of built-in functions
+which have organically accumulated over time.
+
+Nix has a variety of legacy features that are not in practical use,
+but are documented in sections of this specification for the sake of
+completeness.
+
+This document describes the syntax and abstract semantics of the Nix
+language, but leaves out implementation details about how Nix can be
+interpreted/compiled/analysed etc.
+
+### Program structure
+
+This section describes the semantic structure of Nix, and how it
+relates to the rest of the specification.
+
+Each Nix program is a single [*expression*](#expressions) denoting a
+[*value*](#values) (commonly a [*function*](#functions)). Each value
+has a [*type*](#types), however this type is not statically known.
+
+Nix code is modularised through the use of the
+[*import*](#builtins-import) built-in function. No separate module
+system exists.
+
+In addition to chapters describing the building blocks mentioned
+above, this specificiation also describes the [*syntax*](#syntax), the
+available [built-in functions](#builtins), [*error handling*](#errors)
+and known [*deficiencies*](#deficiencies) in the language.