--- author: - Florian Klink date: 2024-10-25 title: "Tvix: Status update" theme: solarized revealOptions: transition: 'fade' --- # Tvix: Status update ![Tvix Logo](tvix-logo.png) 2023-10-25 Florian Klink (flokli) --- ## Whoami - flokli - nixpkgs contributor since 2018, maintaining systemd, nsncd and more - Freelance Nix/DevOps consultant --- ## What is Tvix? --- ![tvix solves this](./tvix_solves_this.jpg) --- ## What is Tvix? - A Rust re-implementation of the components of the Nix package manager - Uses different underlying approaches while retaining Nix compatibility "on the surface" - modular architecture, allowing to recombine aspects to solve your usecase - No "end-user CLI" for now, focus on getting the foundational architecture right and 100% correctness with Nix --- ## Topics - Component overview - Recent developments - Next steps Note: Component overview, to understand a bit better how it's different / updates since the last talk / outlook on roadmap --- ## Structure - **tvix-castore**, the granular data storage/syncing engine. - **tvix-store**, the "Nix store implementation" on top of tvix-castore - **nix-compat**, a library providing access to data formats, protocols and concepts - **tvix-eval**, a bytecode interpreter evaluator and core Nix *language* concepts and builtins - **tvix-build**, a generic builder interface - **tvix-glue**, combines tvix-eval with tvix-[ca]store and tvix-build Note: one big cargo workspace / go into detail later! / nix-compat: concepts like output path calculation, doesn't depend on tvix crates tvix-eval: language concepts being Scopes, Thunks, Nix values, "core builtins" ---- ![tvix-castore](./tvix-castore.png) Note: Nix does do content addressing on a store path level, we're on a per-file/chunk in file. That model allows granular syncing and reuse of parts, which will speedup substitution/copying. Because everything is ca, it'll also allow decentralization and local p2p substitution. Think about everyone in the same room serving chunks. verified streaming. ---- ![tvix-store](./tvix-store.png) Note: track all metadata about store paths (think about the sqlite db), and link to castore instead of storing NAR. NarCalculation: this computation, is super nice to cache, as this info never needs to expire, and is reconstructable in a pure CA manner. ---- ![tvix-store](./nix-compat.png) Note: kept as a somewhat separate library, Tvix "first consumer". But use from your code, it doesn't depend on Tvix bits. Regularly factoring out Nix concepts into this library ---- ![tvix-eval](./tvix-eval.png) Note: only includes basic builtins, like string manipulation, math, …. Other crates can bring their own builtins ---- ![tvix-build](./tvix-build.png) Note: Not aware of Nix, store paths etc. just flexible enough to express everything in there. Use it for your own build system! ---- ![tvix-glue](./tvix-glue.png) Note: connects the evaluator to store and builders ---- ![nar-bridge](./nar-bridge.png) Note: Example on how to use this: Nix HTTP binary cache lens into tvix-[ca]store, allows Nix to download from and push into. It renders NARs on-the-fly --- ## Updates Rough overview. Check blog posts and `git log` for details! ---- ## Updates (#1) - Fixes on error catchability and context behaviour - More compact Nix Value types (memory-wise) - `tvix-cli` REPL global scope manipulation (assign variables and use them in next command) - `firefox.outPath` and `pkgsCross.aarch64-multiplatform.firefox.outPath` correct and added to CI 🎉 Notes: catchability/context to align behaviour with nix / … / assign variables and use them in the next REPL command ---- ## Updates (#2) - OpenTelemetry integration, trace propagation throughout the entire stack - more backends in tvix-[ca]store (`object_store`/`local fs`/`redb`/`bigtable`/…) - nar-bridge RIIR, was deployed as a fetch-through cache for cache.nixos.org at Bornhack - store composition/combinators - wiring up of builds (without reference propagation yet, but reference scanning) Notes: o11y already proven super helpful for debugging where time is spent / first version of composition / builds waiting on reference propagation ---- ## Other Updates - A lot of the tooling for `cache.nixos.org` usage / closure analysis making use of and contributing to `nix-compat` (`@edef`) 🎉 - "Replit using `tvix-[ca]store` and reporting 10x storage reduction" (`@cbrewster`) 🎉 - "Devenv is switching to Tvix" (Talk on Saturday 12:55, `@domenkozar`) 🎉 --- ## Next steps (in no specific order): - Test suite classification system, to decouple test cases from test runner and share with other Nix impls - (Continuous) Docs deployment, website restructuring - Interactions with the evaluator (LSP, DAP) - Persistent deployment of nar-bridge, as fetch-through cache for `cache.nixos.org` - Blob / Chunking protocol improvements (use local chunks where possible, allow readahead) Note: Allows filtering, reusing test cases in Nixcpp and Lix. Mention some behavorial changes found. Mention fetchTree Make it easier for new contributors to get started LSP / Debug adapter protocol / tvix-eval jobs / … / is gonna improve performance for IO into store paths ---- ## Next steps (in no specific order) (cont.): - "Build/Fetch realization goal engine" (tradeoff network bandwidth and CPU time) - More per-store metrics and instance names - More backends (p2p discovery, ipfs, …) Note: needed for builds / to get better insights into cache hit ratios etc. --- ## Contributing - Join the IRC channel (`#tvix-dev` on `hackint`), bridged to Matrix and XMPP - Check our issue tracker (b.tvl.fyi), as well as `tvix/docs/src/TODO.md` (but ask!) - Try to use it and tell us how you broke it! - Sponsoring Note: make sure to ask in the channel to ensure noone is already working on this --- # Thanks to... - all Tvix contributors - Nix community members for their input on the architecture - Sponsors Note: some drive-by, some sticking around longer / NLNET / Clan ---- # Questions?
Florian Klink / flokli.de
Tvix / tvix.dev