diff options
Diffstat (limited to 'web/tvl/blog')
| -rw-r--r-- | web/tvl/blog/2024-02-tvix-update.md | 333 | ||||
| -rw-r--r-- | web/tvl/blog/default.nix | 19 | ||||
| -rw-r--r-- | web/tvl/blog/tvix-status-202209.md | 165 |
3 files changed, 516 insertions, 1 deletions
diff --git a/web/tvl/blog/2024-02-tvix-update.md b/web/tvl/blog/2024-02-tvix-update.md new file mode 100644 index 0000000000..ce9bbf547f --- /dev/null +++ b/web/tvl/blog/2024-02-tvix-update.md @@ -0,0 +1,333 @@ +We've now been working on our rewrite of Nix, [Tvix][], for a little more than +two years. + +Our last written update was in September 2023, and although we did publish a +couple of things in the meantime (flokli's talk on Tvix at [NixCon +2023][nixcon2023], our interview at the [Nix Developer +Dialogues][nix-dev-dialogues-tvix], or tazjin's [talk on +tvix-eval][tvix-eval-ru] (in Russian)), we never found the time to write +something down. + +In the meantime a lot of stuff has happened though, so it's time to change that +:-) + +Note: This blog post is intended for a technical audience that is already +intimately familiar with Nix, and knows what things like derivations or store +paths are. If you're new to Nix, this will not make a lot of sense to you! + +## Evaluation regression testing + +Most of the evaluator work has been driven by evaluating `nixpkgs`, and ensuring +that we produce the same derivations, and that their build results end up in the +same store paths. + +Builds are not hooked up all the way to the evaluator yet, but for Nix code +without IFD (such as `nixpkgs`!) we can verify this property without building. +An evaluated Nix derivation's `outPath` (and `drvPath`) can be compared with +what C++ Nix produces for the same code, to determine whether we evaluated the +package (and all of its dependencies!) correctly [^1]. + +We added integration tests in CI that ensure that the paths we calculate match +C++ Nix, and are successfully evaluating fairly complicated expressions in them. +For example, we test against the Firefox derivation, which exercises some of the +more hairy bits in `nixpkgs` (like WASM cross-compilation infrastructure). Yay! + +Although we're avoiding fine-grained optimization until we're sure Tvix +evaluates all of `nixpkgs` correctly, we still want to have an idea about +evaluation performance and how our work affects it over time. + +For this we extended our benchmark suite and integrated it with +[Windtunnel][windtunnel], which now regularly runs benchmarks and provides a +view into how the timings change from commit to commit. + +In the future, we plan to run this as a part of code review, before changes are +applied to our canonical branch, to provide this as an additional signal to +authors and reviewers without having to run the benchmarks manually. + +## ATerms, output path calculation, and `builtins.derivation` + +We've implemented all of these features, which comprise the components needed to +construct derivations in the Nix language, and to allow us to perform the path +comparisons we mentioned before. + +As an interesting side note, in C++ Nix `builtins.derivation` is not actually a +builtin! It is a piece of [bundled Nix code][nixcpp-builtins-derivation], that +massages some parameters and then calls the *actual* builtin: +`derivationStrict`. We've decided to keep this setup, and implemented support in +Tvix to have builtins defined in `.nix` source code. + +These builtins return attribute sets with the previously mentioned `outPath` and +`drvPath` fields. Implementing them correctly meant that we needed to implement +output path calculation *exactly* the same way as Nix does (bit-by-bit). + +Very little of how this output path calculation works is documented anywhere in +C++ Nix. It uses a subset of [ATerm][aterm] internally, produces "fingerprints" +containing hashes of these ATerms, which are then hashed again. The intermediate +hashes are not printed out anywhere (except if you [patch +Nix][nixcpp-patch-hashes] to do so). + +We already did parts of this correctly while starting this work on +[go-nix][go-nix-outpath] some while ago, but found some more edge cases and +ultimately came up with a nicer interface for Tvix. + +All the Derivation internal data model, ATerm serialization and output path +calculation have been sliced out into a more general-purpose +[nix-compat][nix-compat-derivation] crate, alongside with more documentation +unit tests and a Derivation ATerm parser, so hopefully this will now be more +accessible for everyone now. + +Note our builtin does *not* yet persist the Derivation anywhere "on +disk" (though we have a debug CL that does write it to a temporary directory, +in case we want to track down differences). + +## `tvix-[ca]store` +Tvix now has a store implementation! + +### The Nix model +Inside Nix, store path contents are normally hashed and communicated in NAR +format, which is very coarse and often wasteful - a single bit of change in one +file in a large store path causes a new NAR file to be uploaded to the binary +cache, which then needs to be downloaded. + +Additionally, identifying everything by the SHA256 digest of its NAR +representation makes Nix store paths very incompatible with other +content-addressed systems, as it's a very Nix-specific format. + +### The more granular Tvix model +After experimenting with some concepts and ideas in Golang, mostly around how to +improve binary cache performance[^3], both on-disk as well as over the network, +we settled on a more granular, content-addressed and general-purpose format. + +Internally, it behaves very similar to how git handles tree objects, except +blobs are identified by their raw BLAKE3 digests rather than some custom +encoding, and similarly, tree/directory objects use the BLAKE3 digest of its +canonical protobuf serialization as identifiers. + +This provides some immediate benefits: + - We only need to keep the same data once, even if it's used across different + store paths. + - Transfers can be more granular and only need to fetch the data that's + needed. Due to everything being content-addressed, it can be fetched from + anything supporting BLAKE3 digests, immediately making it compatible with + other P2P systems (IPFS blake3 blobs, …), or general-purpose + content-addressed caches ([bazel-remote]). + +There's a lot more details about the data model, certain decisions etc. in +[the docs][castore-docs]. + +### Compatibility +We however still want to stay compatible with Nix, as in calculating +"NAR-addressed" store paths the same, support substituting from regular Nix +binary caches, as well as storing all the other additional metadata about store +paths. + +We accomplished this by splitting the two different concerns into two separate +`tvix-store` and `tvix-castore` crates, with the former one holding all +Nix-specific metadata and functionality, and the latter being a general-purpose +content-addressed blob and filesystem tree storage system, which is usable in a +lot of contexts outside of Tvix too. For example, if you want to use +tvix-castore to write your own git alternative, or provide granular and +authenticated access into large scientific datasets, you could! + +### Backends +In addition to a gRPC API and client bindings, there's support for local +filesystem-based backends, as well as for sled, an embedded K/V database. + +We're also currently working on a backend supporting most common object +storages, as well as on more granular seeking and content-defined chunking for +blobs. + +### FUSE/virtiofs +A tvix-store can be mounted via FUSE, or exposed through virtiofs[^4]. +While doing the obvious thing - allowing mounting and browsing the contents +of the store, this will allow lazy substitution of builds on remote builders, be +in containerized or virtualized workloads. + +We have an [example][tvix-boot-readme] in the repository seeding gnu hello into +a throwaway store, then booting a MicroVM and executing it. + +### nar-bridge, bridging binary caches +`nar-bridge` and the `NixHTTPPathInfoService` bridge `tvix-[ca]store` with +existing Nix binary caches and Nix. + +The former exposes a `tvix-[ca]store` over the common Nix HTTP Binary Cache +interface (both read and write). + +The latter allows Tvix to substitute from regular Nix HTTP Binary caches, +unpacking NARs and ingesting them on-the-fly into the castore model. +The necessary parsers for NARInfo, signatures etc are also available in the +[nix-compat crate][nix-compat-narinfo]. + +## EvalIO / builtins interacting with the store more closely +tvix-eval itself is designed to be quite pure when it comes to IO - it doesn't +do any IO directly on its own, but for the very little IO functionality it +does as part of "basic interaction with paths" (like importing other +`.nix` files), it goes through an `EvalIO` interface, which is provided to the +Evaluator struct on instantiation. + +This allows us to be a bit more flexible with how IO looks like in practice, +which becomes interesting for specific store implementations that might not +expose a POSIX filesystem directly, or targets where we don't have a filesystem +at all (like WASM). + +Using the `EvalIO` trait also lets `tvix-eval` avoid becoming too strongly +coupled to a specific store implementation, hashing scheme etc[^2]. As we can +extend the set of builtins available to the evaluator with "foreign builtins", +these can live in other crates. + +Following this pattern, we started implementing some of the "basic" builtins +that deal with path access in `tvix-eval`, like: + + - `builtins.pathExists` + - `builtins.readFile` + +We also recently started working on more complicated builtins like +`builtins.filterSource` and `builtins.path`, which are also used in `nixpkgs`. + +Both import a path into the store, and allow passing a Nix expression that's +used as a filter function for each path. `builtins.path` can also ensuring the +imported contents match a certain hash. + +This required the builtin to interact with the store and evaluator in a very +tight fashion, as the filter function (written in Nix) needs to be repeatedly +executed for each path, and its return value is able to cause the store to skip +over certain paths (which it previously couldn't). + +Getting the abstractions right there required some back-and-forth, but the +remaining changes should land quite soon. + +## Catchables / tryEval + +Nix has a limited exception system for dealing with user-generated errors: +`builtins.tryEval` can be used to detect if an expression fails (if +`builtins.throw` or `assert` are used to generate it). This feature requires +extra support in any Nix implementation, as errors may not necessarily cause the +Nix program to abort. + +The C++ Nix implementation reuses the C++ language-provided Exception system for +`builtins.tryEval` which Tvix can't (even if Rust had an equivalent system): + +In C++ Nix the runtime representation of the program in execution corresponds +to the Nix expression tree of the relevant source files. This means that an +exception raised in C++ code will automatically bubble up correctly since the +C++ and Nix call stacks are equivalent to each other. + +Tvix compiles the Nix expressions to a byte code program which may be mutated by +extra optimization rules (for example, we hope to eliminate as many thunks as +possible in the future). This means that such a correspondence between the state +of the runtime and the original Nix code is not guaranteed. + +Previously, `builtins.tryEval` (which is implemented in Rust and can access VM +internals) just allowed the VM to recover from certain kinds of errors. This +proved to be insufficient as it [blew up as soon as a `builtins.tryEval`-ed +thunk is forced again][tryeval-infrec] – extra bookkeeping was needed. As a +solution, we now store recoverable errors as a separate runtime value type. + +As you can imagine, storing evaluation failures as "normal" values quickly leads +to all sorts of bugs because most VM/builtins code is written with only ordinary +values like attribute sets, strings etc. in mind. + +While ironing those out, we made sure to supplement those fixes with as many +test cases for `builtins.tryEval` as possible. This will hopefully prevent any +regressions if or rather when we touch this system again. We already have some +ideas for replacing the `Catchable` value type with a cleaner representation, +but first we want to pin down all the unspoken behaviour. + +## String contexts + +For a long time, we had the [working theory][refscan-string-contexts] that we +could get away with not implementing string contexts, and instead do reference +scanning on a set of "known paths" (and not implement +`builtins.unsafeDiscardStringContext`). + +Unfortunately, we discovered that while this is *conceptually* true, due to a +[bug in Nix][string-contexts-nix-bug] that's worked around in the +`stdenv.mkDerivation` implementation, we can't currently do this and calculate +the same hashes. + +Because hash compatibility is important for us at this point, we bit the bullet +and added support for string contexts into our `NixString` implementation, +implemented the context-related builtins, and added more unit tests that verify +string context behaviour of various builtins. + +## Strings as byte strings + +C++ Nix uses C-style zero-terminated strings internally - however, until +recently, Tvix has used standard Rust strings for string values. Since those are +required to be valid UTF-8, we haven't been able to properly represent all the +string values that Nix supports. + +We recently converted our internal representation to byte strings, which allows +us to treat a `Vec<u8>` as a "string-like" value. + +## JSON/TOML/XML + +We added support for the `toJSON`, `toXML`, `fromJSON` and `fromTOML` builtins. + +`toXML` is particularly exciting, as it's the only format that allows expressing +(partially applied) functions. It's also used in some of Nix' own test suite, so +we can now include these in our unit test suite (and pass, yay!). + +## Builder protocol, drv->builder + +We've been working on the builder protocol, and Tvix's internal build +representation. + +Nix uses derivations (encoded in ATerm) as nodes in its build graph, but it +refers to other store paths used in that build by these store paths *only*. As +mentioned before, store paths only address the inputs - and not the content. + +This poses a big problem in Nix as soon as builds are scheduled on remote +builders: There is no guarantee that files at the same store path on the remote +builder actually have the same contents as on the machine orchestrating the +build. If a package is not binary reproducible, this can lead to so-called +[frankenbuilds][frankenbuild]. + +This also introduces a dependency on the state that's present on the remote +builder machine: Whatever is in its store and matches the paths will be used, +even if it was maliciously placed there. + +To eliminate this hermiticity problem and increase the integrity of builds, +we've decided to use content-addressing in the builder protocol. + +We're currently hacking on this at [Thaigersprint](https://thaigersprint.org/) +and might have some more news to share soon! + +-------------- + +That's it for now, try out Tvix and hit us up on IRC or on our mailing list if +you run into any snags, or have any questions. + +เจอกันนะ :) + +[^1]: We know that we calculated all dependencies correctly because of how their + hashes are included in the hashes of their dependents, and so on. More on + path calculation and input-addressed paths in the next section! +[^2]: That's the same reason why `builtins.derivation[Strict]` also lives in + `tvix-glue`, not in `tvix-eval`. +[^3]: See [nix-casync](https://discourse.nixos.org/t/nix-casync-a-more-efficient-way-to-store-and-substitute-nix-store-paths/16539) + for one example - investing content-defined chunking (while still keeping + the NAR format) +[^4]: Strictly speaking, not limited to tvix-store - literally anything + providing a listing into tvix-castore nodes. + +[Tvix]: https://tvix.dev +[aterm]: http://program-transformation.org/Tools/ATermFormat.html +[bazel-remote]: https://github.com/buchgr/bazel-remote/pull/715 +[castore-docs]: https://cs.tvl.fyi/depot/-/blob/tvix/castore/docs +[frankenbuild]: https://blog.layus.be/posts/2021-06-25-frankenbuilds.html +[go-nix-outpath]: https://github.com/nix-community/go-nix/blob/93cb24a868562714f1691840e94d54ef57bc0a5a/pkg/derivation/hashes.go#L52 +[nix-compat-derivation]: https://docs.tvix.dev/rust/nix_compat/derivation/struct.Derivation.html +[nix-compat-narinfo]: https://docs.tvix.dev/rust/nix_compat/narinfo/index.html +[nix-dev-dialogues-tvix]: https://www.youtube.com/watch?v=ZYG3T4l8RU8 +[nixcon2023]: https://www.youtube.com/watch?v=j67prAPYScY +[tvix-eval-ru]: https://tazj.in/blog/tvix-eval-talk-2023 +[nixcpp-builtins-derivation]: https://github.com/NixOS/nix/blob/49cf090cb2f51d6935756a6cf94d568cab063f81/src/libexpr/primops/derivation.nix#L4 +[nixcpp-patch-hashes]: https://github.com/adisbladis/nix/tree/hash-tracing +[refscan-string-contexts]: https://inbox.tvl.su/depot/20230316120039.j4fkp3puzrtbjcpi@tp/T/#t +[store-docs]: https://cs.tvl.fyi/depot/-/blob/tvix/store/docs/api.md +[string-contexts-nix-bug]: https://github.com/NixOS/nix/issues/4629 +[tryeval-infrec]: https://b.tvl.fyi/issues/281 +[tvix-boot-readme]: https://cs.tvl.fyi/depot/-/blob/tvix/boot/README.md +[why-string-contexts-now]: https://cl.tvl.fyi/c/depot/+/10446/7/tvix/eval/docs/build-references.md +[windtunnel]: https://staging.windtunnel.ci/tvl/tvix diff --git a/web/tvl/blog/default.nix b/web/tvl/blog/default.nix index 963bb635e3..2a1dfe85cc 100644 --- a/web/tvl/blog/default.nix +++ b/web/tvl/blog/default.nix @@ -7,12 +7,29 @@ baseUrl = "https://tvl.fyi/blog"; }; - posts = [ + posts = builtins.sort (a: b: a.date > b.date) [ { key = "rewriting-nix"; title = "Tvix: We are rewriting Nix"; date = 1638381387; content = ./rewriting-nix.md; + author = "tazjin"; + } + + { + key = "tvix-status-september-22"; + title = "Tvix Status - September '22"; + date = 1662995534; + content = ./tvix-status-202209.md; + author = "tazjin"; + } + + { + key = "tvix-update-february-24"; + title = "Tvix Status - February '24"; + date = 1707472132; + content = ./2024-02-tvix-update.md; + author = "flokli"; } ]; } diff --git a/web/tvl/blog/tvix-status-202209.md b/web/tvl/blog/tvix-status-202209.md new file mode 100644 index 0000000000..dae1dae194 --- /dev/null +++ b/web/tvl/blog/tvix-status-202209.md @@ -0,0 +1,165 @@ +We've now been working on our rewrite of Nix, [Tvix][], for over a +year. + +As you can imagine, this past year has been turbulent, to say the +least, given the regions where many of us live. As a result we haven't +had as much time to work on fun things (like open-source software +projects!) as we'd like. + +We've all been fortunate enough to continue making progress, but we +just haven't had the bandwidth to communicate with you and keep you up +to speed on what's going on. That's what this blog post is for. + +## Nix language evaluator + +The most significant progress in the past six months has been on our +Nix language evaluator. To answer the most important question: yes, +you can play with it right now – in [Tvixbolt][]! + +We got the evaluator into its current state by first listing all the +problems we were likely to encounter, then solving them independently, +and finally assembling all those small-scale solutions into a coherent +whole. As a result, we briefly had an impractically large private +source tree, which we have since [integrated][] into our monorepo. + +This process was much slower than we would have liked, due to code +review bandwidth... which is to say, we're all volunteers. People have +lives, bottlenecks happen. + +Most of this code was either written or reviewed by [grfn][], +[sterni][] and [tazjin][] (that's me!). + +### How much of eval is working? + +*Most of it*! You can enter most (but not *all*, sorry! Not yet, +anyway.) Nix language expressions in [Tvixbolt][] and observe how they +are evaluated. + +There's a lot of interesting stuff going on under the hood, such as: + +* The Tvix compiler can emit warnings and errors without failing + early, and retains as much source information as possible. This will + enable you to use Tvix as the basis for developer tooling, such as + language servers. + +* The Tvix compiler performs in-depth scope analysis, so it can both + generate efficient bytecode for accessing identifiers, and alert you + about problems in your code before runtime. + +* The runtime supports tail-call optimisation in many (but – again – + not yet all) cases, so you can evaluate recursive expressions in + constant stack space. + +* The runtime can give you different backing representations for the + same Nix type. For example, an attribute set is represented + differently depending on whether you've constructed an empty one, a + `name/value` pair, or a larger set. This lets us optimise frequent, + well-known use-cases without impacting the general case much. + +We've run some initial benchmarks against C++ Nix (using the features +that are ready), and in most cases Tvix evaluation is an order of +magnitude faster. To be fair, though, these benchmarks are in no way +indicative of real-life performance for things like `nixpkgs`. More +information is coming... eventually. + +### How does it all work? + +Tvix's evaluator uses a custom abstract machine with a Nix-specific +instruction set, and a compiler that traverses a parsed Nix AST to +emit this bytecode and perform a set of optimisations and other +analysis. The most important benefit of this is that we can plan and +lay out the execution of a program in a way that is better suited to +an efficient runtime than directly traversing the AST. + +TIP: You can see the generated bytecode in [Tvixbolt][]! + +This is all written in about 4000 lines of Rust (naturally), some of +which – especially around scope-handling – are deceptively simple. + +As part of our CI suite, we run the evaluator against some tests we +wrote ourselves, as well as against the upstream Nix test suite (which +we don't *quite* pass yet. We're working on it!). + +### What's next for tvix-eval? + +Despite all our progress, there are still some unfinished feature +areas, and some of them are pretty important: + +1. The majority of Nix's builtins – including fundamental ones like + `import` and `derivation` – aren't implemented yet. + +2. Neither are recursive attribute sets (`rec`). This isn't because of + a problem with the recursion itself, but because of the handling of + nested keys (such as `a.b`). We have a lackluster solution already, + but are designing a more efficient one. + +In both cases, we've mostly figured out what to do; now it's just a +matter of finding the time to do it. Our progress is steady, and can +be tracked [in the source][src] (viewer without Javascript +[here][src-noscript]). + +Apart from that, the next steps are: + +* Comprehensive benchmarking. We're standing up an infrastructure for + continuous benchmarking to measure the impact of changes. It'll also + let us identify and optimise hotspots + +* Implementing known optimisations. There are some areas of the code + that have the potential for significant speed gains, but we're + holding off implementing those until the evaluator is feature + complete and passes the Nix test suite. + +* Finishing our language specification. Based on what we've learned, + we're writing a specification of the Nix language that captures its + various behaviours in all their tricky subtlety and subtle trickery. + +Once we can evaluate `nixpkgs`, we're likely to shift our focus +towards the other areas of Tvix. + +## The Other Areas of Tvix + +Speaking of these other areas (most importantly, the builder and store +implementation), we've made some nice progress there also. + +While we've yet to start assembling the actual pieces, [flokli][] and +[adisbladis][] have been hard at work on [go-nix][], which aims to +implement many of the low-level primitives required for the Nix store +and builder (hashing and encoding schemes, archive formats, reference +scanning ...). + +We're looking forward to telling you more in the next Tvix status +update! + +## Outro ... + +We'd be delighted to onboard new contributors to Tvix! Please take a +look at the main [TVL page](https://tvl.fyi) to find out how to get in +touch with us if you'd like to join! + +Thanks also, of course, to [NLNet](https://nlnet.nl/) for sponsoring +some of this work! + +And finally, we would like to thank and pay our respects to jD91mZM2 – +the original author of +[rnix-parser](https://github.com/nix-community/rnix-parser) – who has +sadly passed away. Please, tell people how important they are to you. + +We use `rnix-parser` in our compiler, and its well-designed internals +(also thanks to its new maintainers!) have saved us a lot of time. + +That's it for this update. Go play with [Tvixbolt][], have fun +figuring out weird ways to break it – and if you do, let us know. + +We'll see you around! + +[Tvix]: https://tvl.fyi/blog/rewriting-nix +[Tvixbolt]: https://bolt.tvix.dev +[integrated]: https://cl.tvl.fyi/q/status:merged+%2522tvix/eval%2522+mergedbefore:2022-09-09 +[src]: https://cs.tvl.fyi/depot/-/tree/tvix/eval +[src-noscript]: https://code.tvl.fyi/tree/tvix/eval +[tazjin]: https://tazj.in +[grfn]: https://gws.fyi/ +[sterni]: https://github.com/sternenseemann +[go-nix]: https://github.com/nix-community/go-nix +[flokli]: https://flokli.de/ +[adisbladis]: https://github.com/adisbladis |