about summary refs log tree commit diff
path: root/tvix/crate-hashes.json (follow)
AgeCommit message (Collapse)AuthorFilesLines
2023-09-20 r/6621 refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rsConnor Brewster1-0/+1
This switches the FUSE implementation from fuser to fuse-backend-rs. fuse-backend-rs is designed to work with both FUSE and virtiofs. Virtiofs support will make it possible to plug the tvix-store into a microvm and have `/nix/store` access without having to setup FUSE inside the guest. Additionally fuse-backend-rs has nice support for running multiple FUSE threads and has some async support. The goal of this commit is to mechanically switch over to fuse-backend-rs with minimal changes. I did have to add some locks here and there because fuse-backend-rs uses `&self` on all methods whereas fuser uses `&mut self`. `&self` is required for concurrent access to the FUSE server, so this makes sense. We can consider switching to concurrent maps and use some other techniques to reduce lock contention and critical section size. Issue: https://b.tvl.fyi/issues/305 Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-07-31 r/6453 refactor(tvix/cli/refscan): use wu-manber crate with &[u8] supportFlorian Klink1-1/+1
PR'ed at https://github.com/tvlfyi/wu-manber/pull/1, and now merged. Change-Id: I8c71e359196396a1d42a3ea2ab7ac15b137b2db0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8992 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-02-04 r/5836 fix(tvix/cli): use tvlfyi/wu-manber fork for refscannerVincent Ambo1-1/+1
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1) but it's not clear whether upstream will accept patches, so for now lets point this directly at our fork. Change-Id: Iccdcedae3e9a8b783241431787c952561d032694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-02 r/5825 refactor(tvix/cli): use Wu-Manber string scanning for drv referencesVincent Ambo1-1/+2
Switch out the string-scanning algorithm used in the reference scanner. The construction of aho-corasick automata made up the vast majority of runtime when evaluating nixpkgs previously. While the actual scanning with a constructed automaton is relatively fast, we almost never scan for the same set of strings twice and the cost is not worth it. An algorithm that better matches our needs is the Wu-Manber multiple string match algorithm, which works efficiently on *long* and *random* strings of the *same length*, which describes store paths (up to their hash component). This switches the refscanner crate to a Rust implementation[0][1] of this algorithm. This has several implications: 1. This crate does not provide a way to scan streams. I'm not sure if this is an inherent problem with the algorithm (probably not, but it would need buffering). Either way, related functions and tests (which were actually unused) have been removed. 2. All strings need to be of the same length. For this reason, we truncate the known paths after their hash part (they are still unique, of course). 3. Passing an empty set of matches, or a match that is shorter than the length of a store path, causes the crate to panic. We safeguard against this by completely skipping the refscanning if there are no known paths (i.e. when evaluating the first derivation of an eval), and by bailing out of scanning a string that is shorter than a store path. On the upside, this reduces overall runtime to less 1/5 of what it was before when evaluating `pkgs.stdenv.drvPath`. [0]: Frankly, it's a random, research-grade MIT-licensed crate that I found on Github: https://github.com/jneem/wu-manber [1]: We probably want to rewrite or at least fork the above crate, and add things like a three-byte wide scanner. Evaluating large portions of nixpkgs can easily lead to more than 65k derivations being scanned for. Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-21 r/5720 chore(tvix/store): add tonic-mockFlorian Klink1-1/+2
Upstream seems to be dead, so we're using https://github.com/tyrchen/ tonic-mock/pull/3 here. According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment- 1241164173, we might not need this crate at all, but for now, it gets the job done and is less code to write in the tests. Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-21 r/5442 fix(tvix/eval): use test-generator fork that supports workspacesVincent Ambo1-0/+3
This should make no difference in Nix builds, but allows running tests locally again with `cargo test` for //tvix/eval. Change-Id: I97d61840143d5c14db61d5862781bf635f9a28e7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7590 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>