---
author:
- Florian Klink
date: 2023-09-09
title: "Tvix: Status update"
theme: moon
revealOptions:
transition: 'fade'
---
# Tvix: Status update
2023-09-09
Florian Klink
---
## Whoami
- flokli
- nixpkgs contributor since 2018, maintaining systemd, nsncd and some
more stuff
- Freelance Nix/DevOps consultant
- I spend too much time on computers :-)
---
## What is Tvix?
- A new implementation of Nix
- modular
- written in Rust
- developed in the [TVL](https://tvl.fyi) monorepo
- subtree exported to [github:tvlfyi/tvix](https://github.com/tvlfyi/tvix)
---
## Structure
- strong separation between **Evaluator**, **Store** and **Builder**
- Defined interfaces between different components (Protobuf/gRPC)
- Allows adding to/combining with your own components
- A lot of helper code for some of the Nix internals in the `nix-compat` crate
Note: Derivation types, serializers. NAR writers, nixbase32 enc/dec, Nix Hash function, stringparsing etc.
----
---
## Evaluator: Design
-
Nix code is parsed via [rnix](https://github.com/nix-community/rnix-parser)
-
AST traversal, generate bytecode (with some transformations)
-
Bytecode is executed by an abstract machine with a Nix-specific instruction set
----
## Evaluator: Design
-
Builtins are separate from the "evaluator core"
-
inject your own builtins
-
this includes `builtins.derivation`!
-
IO is nicely abstracted away
-
We can run a Nixlang subset without IO in wasm (see [tvixbolt](https://bolt.tvix.dev/)),
or parse Nix into a config struct with `tvix-serde`
----
Tvixbolt Demo
----
### Evaluator: Current Work
-
Current goal: **evaluate nixpkgs the same way as Nix does**
-
Checked by comparing the calculated output paths, which checks correctness of all \"parent\" output paths too.
-
Required implementing a lot of Nix internals in `nix-compat`, and `tvix-store` (A-Term, hash modulo, NAR writer/hasher)
Note: Getting output hashing correct, and exposing this in a re-usable fashion took quite some iterations to get right.
----
### Evaluator: Current Work (cont.)
-
🎉 Already correct for (and continously checked by CI on every commit):
-
`stdenv`, `hello`
-
`pkgsCross.aarch64-multiplatform.stdenv`, `pkgsCross.aarch64-multiplatform.hello`
-
Some work left for more complicated expressions
-
infinite recursion [when inheriting from a `builtins.tryEval` multiple times](https://b.tvl.fyi/281)
-
small details around file imports
-
Not too much performance finetuning until we're correct first.
----
### Evaluator: Demo
[](https://asciinema.org/a/MH4tuVPLsKewJSGJUYEyIKUpj)
---
## Store: Design
-
Uses a very different underlying data model:
-
Nix stores on a per- `StorePath` granularity
-
tvix-store uses a Merkle DAG of directories, similar to git trees, but with [BLAKE3](https://github.com/BLAKE3-team/BLAKE3) digests as pointers.
-
Compat layer in front to still render/calculate NAR on demand where needed
-
Substitution, caching, ... possible to describe via composition/layering
----
----
### Store: Advantages
-
Less downloading of data that didn't change
-
Less duplicate data on disk/storage
-
Inherently content-addressed, so P2P substitution possible
-
Allows doing verified blob streaming ([BAO](https://github.com/oconnor663/bao), [bao-tree](https://github.com/n0-computer/bao-tree))
-
Protocol has some \"smarter\" methods to avoid roundtrips, but could all be statically pre-rendered
-
Very little data that needs to be fetched from a trusted party (or be signed)
Note: Our way of addressing blobs by their raw blake3 digest is natively compatible with iroh, the IPFS Re-implementation in Rust
----
### Store: Status
-
Whole Merkle-based store implementation (and Nix NAR compat layer) is there
-
exercised by the output path CI tests, and a test suite.
-
three backends (Sled, in-memory, gRPC client)
-
more backends and more test suites planned.
-
FUSE filesystem to expose the store (to Tvix Builders, \"appliances\")
Note: backends: RocksDB, sqlite, s3. fuse: lazy fetching of build input files | think about a minimal initrd to bring up network and mount the store, then run any closure.
----
### Store: Demo
[](https://asciinema.org/a/YFB9yycHdx0OUH9N0WdAkIYua)
----
### Store: Status (cont.)
-
More work on store composition needed (necessary for substition and caching)
-
More work on more granular blob substititon needed.
-
More work on bridges with Nix needed
-
Get Nix to talk to a tvix-store
-
Expose existing binary caches to tvix-store
---
### Builder: Design
-
Build requests/Build protocol is less Nix-specific
-
allows reusing builders for other usages (non-sandboxed builds, other build systems, playing with other addressing mechanisms)
-
Distinction between a **specific build attempt** and the **general build recipe**
-
allows keeping metadata about failed builds
-
stats (memory/cpu consumption)
-
comparing different produced binary outputs (r11y)
----
### Builder: Design
-
Invididual builds can be run in your desired container/virt engine/scheduler, as long as it speaks the same Build API
-
Build API composition/proxying, similar to Store composition
-
allows "unattended building" (evaluate nixpkgs locally and send all build requests to a remote builder)
-
allows tailing logs from currently/already running builds
----
### Builder: Status
-
Dummy Builder implementation in `go-nix` (using OCI)
-
Some scribble notes on the Build Protocol
-
Glue code to trigger builds from inside `builtins.derivation` needs to be written
-
Builder implementation (using `systemd-nspwan` or some container engine needs to be written.
-
Web interface to visualize store contents and build graphs/builds/logs
---
## Contributing
-
Join the IRC channel (`#tvl` on `hackint`), bridged to Matrix and XMPP
-
Check our issue tracker
-
Try to use it and tell us how you broke it!
-
Add various Nix bits to `nix-compat`
Note: or if you like what you seeing and want to sponsor parts, that's also cool :-)
---
# Thanks to...
-
all TVL contributors (some drive-by, some long-term contributors)
-
countless Nix community members for their input on the architecture and rubberducking
-
NLNET and others to sponsor parts of this
----
# Questions?
