diff options
Diffstat (limited to 'web/tvl/blog/2024-02-tvix-update.md')
| -rw-r--r-- | web/tvl/blog/2024-02-tvix-update.md | 333 |
1 files changed, 333 insertions, 0 deletions
diff --git a/web/tvl/blog/2024-02-tvix-update.md b/web/tvl/blog/2024-02-tvix-update.md new file mode 100644 index 0000000000..ce9bbf547f --- /dev/null +++ b/web/tvl/blog/2024-02-tvix-update.md @@ -0,0 +1,333 @@ +We've now been working on our rewrite of Nix, [Tvix][], for a little more than +two years. + +Our last written update was in September 2023, and although we did publish a +couple of things in the meantime (flokli's talk on Tvix at [NixCon +2023][nixcon2023], our interview at the [Nix Developer +Dialogues][nix-dev-dialogues-tvix], or tazjin's [talk on +tvix-eval][tvix-eval-ru] (in Russian)), we never found the time to write +something down. + +In the meantime a lot of stuff has happened though, so it's time to change that +:-) + +Note: This blog post is intended for a technical audience that is already +intimately familiar with Nix, and knows what things like derivations or store +paths are. If you're new to Nix, this will not make a lot of sense to you! + +## Evaluation regression testing + +Most of the evaluator work has been driven by evaluating `nixpkgs`, and ensuring +that we produce the same derivations, and that their build results end up in the +same store paths. + +Builds are not hooked up all the way to the evaluator yet, but for Nix code +without IFD (such as `nixpkgs`!) we can verify this property without building. +An evaluated Nix derivation's `outPath` (and `drvPath`) can be compared with +what C++ Nix produces for the same code, to determine whether we evaluated the +package (and all of its dependencies!) correctly [^1]. + +We added integration tests in CI that ensure that the paths we calculate match +C++ Nix, and are successfully evaluating fairly complicated expressions in them. +For example, we test against the Firefox derivation, which exercises some of the +more hairy bits in `nixpkgs` (like WASM cross-compilation infrastructure). Yay! + +Although we're avoiding fine-grained optimization until we're sure Tvix +evaluates all of `nixpkgs` correctly, we still want to have an idea about +evaluation performance and how our work affects it over time. + +For this we extended our benchmark suite and integrated it with +[Windtunnel][windtunnel], which now regularly runs benchmarks and provides a +view into how the timings change from commit to commit. + +In the future, we plan to run this as a part of code review, before changes are +applied to our canonical branch, to provide this as an additional signal to +authors and reviewers without having to run the benchmarks manually. + +## ATerms, output path calculation, and `builtins.derivation` + +We've implemented all of these features, which comprise the components needed to +construct derivations in the Nix language, and to allow us to perform the path +comparisons we mentioned before. + +As an interesting side note, in C++ Nix `builtins.derivation` is not actually a +builtin! It is a piece of [bundled Nix code][nixcpp-builtins-derivation], that +massages some parameters and then calls the *actual* builtin: +`derivationStrict`. We've decided to keep this setup, and implemented support in +Tvix to have builtins defined in `.nix` source code. + +These builtins return attribute sets with the previously mentioned `outPath` and +`drvPath` fields. Implementing them correctly meant that we needed to implement +output path calculation *exactly* the same way as Nix does (bit-by-bit). + +Very little of how this output path calculation works is documented anywhere in +C++ Nix. It uses a subset of [ATerm][aterm] internally, produces "fingerprints" +containing hashes of these ATerms, which are then hashed again. The intermediate +hashes are not printed out anywhere (except if you [patch +Nix][nixcpp-patch-hashes] to do so). + +We already did parts of this correctly while starting this work on +[go-nix][go-nix-outpath] some while ago, but found some more edge cases and +ultimately came up with a nicer interface for Tvix. + +All the Derivation internal data model, ATerm serialization and output path +calculation have been sliced out into a more general-purpose +[nix-compat][nix-compat-derivation] crate, alongside with more documentation +unit tests and a Derivation ATerm parser, so hopefully this will now be more +accessible for everyone now. + +Note our builtin does *not* yet persist the Derivation anywhere "on +disk" (though we have a debug CL that does write it to a temporary directory, +in case we want to track down differences). + +## `tvix-[ca]store` +Tvix now has a store implementation! + +### The Nix model +Inside Nix, store path contents are normally hashed and communicated in NAR +format, which is very coarse and often wasteful - a single bit of change in one +file in a large store path causes a new NAR file to be uploaded to the binary +cache, which then needs to be downloaded. + +Additionally, identifying everything by the SHA256 digest of its NAR +representation makes Nix store paths very incompatible with other +content-addressed systems, as it's a very Nix-specific format. + +### The more granular Tvix model +After experimenting with some concepts and ideas in Golang, mostly around how to +improve binary cache performance[^3], both on-disk as well as over the network, +we settled on a more granular, content-addressed and general-purpose format. + +Internally, it behaves very similar to how git handles tree objects, except +blobs are identified by their raw BLAKE3 digests rather than some custom +encoding, and similarly, tree/directory objects use the BLAKE3 digest of its +canonical protobuf serialization as identifiers. + +This provides some immediate benefits: + - We only need to keep the same data once, even if it's used across different + store paths. + - Transfers can be more granular and only need to fetch the data that's + needed. Due to everything being content-addressed, it can be fetched from + anything supporting BLAKE3 digests, immediately making it compatible with + other P2P systems (IPFS blake3 blobs, …), or general-purpose + content-addressed caches ([bazel-remote]). + +There's a lot more details about the data model, certain decisions etc. in +[the docs][castore-docs]. + +### Compatibility +We however still want to stay compatible with Nix, as in calculating +"NAR-addressed" store paths the same, support substituting from regular Nix +binary caches, as well as storing all the other additional metadata about store +paths. + +We accomplished this by splitting the two different concerns into two separate +`tvix-store` and `tvix-castore` crates, with the former one holding all +Nix-specific metadata and functionality, and the latter being a general-purpose +content-addressed blob and filesystem tree storage system, which is usable in a +lot of contexts outside of Tvix too. For example, if you want to use +tvix-castore to write your own git alternative, or provide granular and +authenticated access into large scientific datasets, you could! + +### Backends +In addition to a gRPC API and client bindings, there's support for local +filesystem-based backends, as well as for sled, an embedded K/V database. + +We're also currently working on a backend supporting most common object +storages, as well as on more granular seeking and content-defined chunking for +blobs. + +### FUSE/virtiofs +A tvix-store can be mounted via FUSE, or exposed through virtiofs[^4]. +While doing the obvious thing - allowing mounting and browsing the contents +of the store, this will allow lazy substitution of builds on remote builders, be +in containerized or virtualized workloads. + +We have an [example][tvix-boot-readme] in the repository seeding gnu hello into +a throwaway store, then booting a MicroVM and executing it. + +### nar-bridge, bridging binary caches +`nar-bridge` and the `NixHTTPPathInfoService` bridge `tvix-[ca]store` with +existing Nix binary caches and Nix. + +The former exposes a `tvix-[ca]store` over the common Nix HTTP Binary Cache +interface (both read and write). + +The latter allows Tvix to substitute from regular Nix HTTP Binary caches, +unpacking NARs and ingesting them on-the-fly into the castore model. +The necessary parsers for NARInfo, signatures etc are also available in the +[nix-compat crate][nix-compat-narinfo]. + +## EvalIO / builtins interacting with the store more closely +tvix-eval itself is designed to be quite pure when it comes to IO - it doesn't +do any IO directly on its own, but for the very little IO functionality it +does as part of "basic interaction with paths" (like importing other +`.nix` files), it goes through an `EvalIO` interface, which is provided to the +Evaluator struct on instantiation. + +This allows us to be a bit more flexible with how IO looks like in practice, +which becomes interesting for specific store implementations that might not +expose a POSIX filesystem directly, or targets where we don't have a filesystem +at all (like WASM). + +Using the `EvalIO` trait also lets `tvix-eval` avoid becoming too strongly +coupled to a specific store implementation, hashing scheme etc[^2]. As we can +extend the set of builtins available to the evaluator with "foreign builtins", +these can live in other crates. + +Following this pattern, we started implementing some of the "basic" builtins +that deal with path access in `tvix-eval`, like: + + - `builtins.pathExists` + - `builtins.readFile` + +We also recently started working on more complicated builtins like +`builtins.filterSource` and `builtins.path`, which are also used in `nixpkgs`. + +Both import a path into the store, and allow passing a Nix expression that's +used as a filter function for each path. `builtins.path` can also ensuring the +imported contents match a certain hash. + +This required the builtin to interact with the store and evaluator in a very +tight fashion, as the filter function (written in Nix) needs to be repeatedly +executed for each path, and its return value is able to cause the store to skip +over certain paths (which it previously couldn't). + +Getting the abstractions right there required some back-and-forth, but the +remaining changes should land quite soon. + +## Catchables / tryEval + +Nix has a limited exception system for dealing with user-generated errors: +`builtins.tryEval` can be used to detect if an expression fails (if +`builtins.throw` or `assert` are used to generate it). This feature requires +extra support in any Nix implementation, as errors may not necessarily cause the +Nix program to abort. + +The C++ Nix implementation reuses the C++ language-provided Exception system for +`builtins.tryEval` which Tvix can't (even if Rust had an equivalent system): + +In C++ Nix the runtime representation of the program in execution corresponds +to the Nix expression tree of the relevant source files. This means that an +exception raised in C++ code will automatically bubble up correctly since the +C++ and Nix call stacks are equivalent to each other. + +Tvix compiles the Nix expressions to a byte code program which may be mutated by +extra optimization rules (for example, we hope to eliminate as many thunks as +possible in the future). This means that such a correspondence between the state +of the runtime and the original Nix code is not guaranteed. + +Previously, `builtins.tryEval` (which is implemented in Rust and can access VM +internals) just allowed the VM to recover from certain kinds of errors. This +proved to be insufficient as it [blew up as soon as a `builtins.tryEval`-ed +thunk is forced again][tryeval-infrec] – extra bookkeeping was needed. As a +solution, we now store recoverable errors as a separate runtime value type. + +As you can imagine, storing evaluation failures as "normal" values quickly leads +to all sorts of bugs because most VM/builtins code is written with only ordinary +values like attribute sets, strings etc. in mind. + +While ironing those out, we made sure to supplement those fixes with as many +test cases for `builtins.tryEval` as possible. This will hopefully prevent any +regressions if or rather when we touch this system again. We already have some +ideas for replacing the `Catchable` value type with a cleaner representation, +but first we want to pin down all the unspoken behaviour. + +## String contexts + +For a long time, we had the [working theory][refscan-string-contexts] that we +could get away with not implementing string contexts, and instead do reference +scanning on a set of "known paths" (and not implement +`builtins.unsafeDiscardStringContext`). + +Unfortunately, we discovered that while this is *conceptually* true, due to a +[bug in Nix][string-contexts-nix-bug] that's worked around in the +`stdenv.mkDerivation` implementation, we can't currently do this and calculate +the same hashes. + +Because hash compatibility is important for us at this point, we bit the bullet +and added support for string contexts into our `NixString` implementation, +implemented the context-related builtins, and added more unit tests that verify +string context behaviour of various builtins. + +## Strings as byte strings + +C++ Nix uses C-style zero-terminated strings internally - however, until +recently, Tvix has used standard Rust strings for string values. Since those are +required to be valid UTF-8, we haven't been able to properly represent all the +string values that Nix supports. + +We recently converted our internal representation to byte strings, which allows +us to treat a `Vec<u8>` as a "string-like" value. + +## JSON/TOML/XML + +We added support for the `toJSON`, `toXML`, `fromJSON` and `fromTOML` builtins. + +`toXML` is particularly exciting, as it's the only format that allows expressing +(partially applied) functions. It's also used in some of Nix' own test suite, so +we can now include these in our unit test suite (and pass, yay!). + +## Builder protocol, drv->builder + +We've been working on the builder protocol, and Tvix's internal build +representation. + +Nix uses derivations (encoded in ATerm) as nodes in its build graph, but it +refers to other store paths used in that build by these store paths *only*. As +mentioned before, store paths only address the inputs - and not the content. + +This poses a big problem in Nix as soon as builds are scheduled on remote +builders: There is no guarantee that files at the same store path on the remote +builder actually have the same contents as on the machine orchestrating the +build. If a package is not binary reproducible, this can lead to so-called +[frankenbuilds][frankenbuild]. + +This also introduces a dependency on the state that's present on the remote +builder machine: Whatever is in its store and matches the paths will be used, +even if it was maliciously placed there. + +To eliminate this hermiticity problem and increase the integrity of builds, +we've decided to use content-addressing in the builder protocol. + +We're currently hacking on this at [Thaigersprint](https://thaigersprint.org/) +and might have some more news to share soon! + +-------------- + +That's it for now, try out Tvix and hit us up on IRC or on our mailing list if +you run into any snags, or have any questions. + +เจอกันนะ :) + +[^1]: We know that we calculated all dependencies correctly because of how their + hashes are included in the hashes of their dependents, and so on. More on + path calculation and input-addressed paths in the next section! +[^2]: That's the same reason why `builtins.derivation[Strict]` also lives in + `tvix-glue`, not in `tvix-eval`. +[^3]: See [nix-casync](https://discourse.nixos.org/t/nix-casync-a-more-efficient-way-to-store-and-substitute-nix-store-paths/16539) + for one example - investing content-defined chunking (while still keeping + the NAR format) +[^4]: Strictly speaking, not limited to tvix-store - literally anything + providing a listing into tvix-castore nodes. + +[Tvix]: https://tvix.dev +[aterm]: http://program-transformation.org/Tools/ATermFormat.html +[bazel-remote]: https://github.com/buchgr/bazel-remote/pull/715 +[castore-docs]: https://cs.tvl.fyi/depot/-/blob/tvix/castore/docs +[frankenbuild]: https://blog.layus.be/posts/2021-06-25-frankenbuilds.html +[go-nix-outpath]: https://github.com/nix-community/go-nix/blob/93cb24a868562714f1691840e94d54ef57bc0a5a/pkg/derivation/hashes.go#L52 +[nix-compat-derivation]: https://docs.tvix.dev/rust/nix_compat/derivation/struct.Derivation.html +[nix-compat-narinfo]: https://docs.tvix.dev/rust/nix_compat/narinfo/index.html +[nix-dev-dialogues-tvix]: https://www.youtube.com/watch?v=ZYG3T4l8RU8 +[nixcon2023]: https://www.youtube.com/watch?v=j67prAPYScY +[tvix-eval-ru]: https://tazj.in/blog/tvix-eval-talk-2023 +[nixcpp-builtins-derivation]: https://github.com/NixOS/nix/blob/49cf090cb2f51d6935756a6cf94d568cab063f81/src/libexpr/primops/derivation.nix#L4 +[nixcpp-patch-hashes]: https://github.com/adisbladis/nix/tree/hash-tracing +[refscan-string-contexts]: https://inbox.tvl.su/depot/20230316120039.j4fkp3puzrtbjcpi@tp/T/#t +[store-docs]: https://cs.tvl.fyi/depot/-/blob/tvix/store/docs/api.md +[string-contexts-nix-bug]: https://github.com/NixOS/nix/issues/4629 +[tryeval-infrec]: https://b.tvl.fyi/issues/281 +[tvix-boot-readme]: https://cs.tvl.fyi/depot/-/blob/tvix/boot/README.md +[why-string-contexts-now]: https://cl.tvl.fyi/c/depot/+/10446/7/tvix/eval/docs/build-references.md +[windtunnel]: https://staging.windtunnel.ci/tvl/tvix |