about summary refs log tree commit diff
path: root/tvix/glue/src/tvix_store_io.rs
AgeCommit message (Collapse)AuthorFilesLines
2024-02-02 r/7467 refactor(tvix/eval): Box Value::StringAspen Smith1-2/+2
NixString is *quite* large - like 80 bytes - because of the extra capacity value for BString and because of the context. We want to keep Value small since we're passing it around a lot, so let's box the NixString inside Value::String to save on some memory, and make cloning ostensibly a little cheaper Change-Id: I343c8b4e7f61dc3dcbbaba4382efb3b3e5bbabb2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10729 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2024-01-31 r/7460 fix(tvix): Represent strings as byte arraysAspen Smith1-2/+3
C++ nix uses C-style zero-terminated char pointers to represent strings internally - however, up to this point, tvix has used Rust `String` and `str` for string values. Since those are required to be valid utf-8, we haven't been able to properly represent all the string values that Nix supports. To fix that, this change converts the internal representation of the NixString struct from `Box<str>` to `BString`, from the `bstr` crate - this is a wrapper around a `Vec<u8>` with extra functions for treating that byte vector as a "morally string-like" value, which is basically exactly what we need. Since this changes a pretty fundamental assumption about a pretty core type, there are a *lot* of changes in a lot of places to make this work, but I've tried to keep the general philosophy and intent of most of the code in most places intact. Most notably, there's nothing that's been done to make the derivation stuff in //tvix/glue work with non-utf8 strings everywhere, instead opting to just convert to String/str when passing things into that - there *might* be something to be done there, but I don't know what the rules should be and I don't want to figure them out in this change. To deal with OS-native paths in a way that also works in WASM for tvixbolt, this also adds a dependency on the "os_str_bytes" crate. Fixes: b/189 Fixes: b/337 Change-Id: I5e6eb29c62f47dd91af954f5e12bfc3d186f5526 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10200 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-01-20 r/7433 feat(tvix/store): enable `name` customization in the storeRyan Lahfa1-0/+1
Sometimes, Nix lets someone customize the `name` in the store for a path, this is the case for `builtins.path` which takes a `name` argument, we leave it to the caller to choose the name, which can be the basename by default of the path. Change-Id: Icdbf71d1d8f2dca5716b99d20aac885aab905b80 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10653 Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 r/7432 refactor(tvix/store): `import_path` → `import_path_as_nar_ca`Ryan Lahfa1-1/+1
Add multiple additional helpers such as: - `path_to_name`: derive the basename of a given path - `derive_nar_ca_path_info`: derive the `PathInfo` for a content addressed NAR which isolates further the tree walking feature and the ingestion feature. Additionally, we don't `expect` anymore and propagate properly ingestion errors up. Change-Id: I60edb5b633911c58ade7e19f5002e6f75f90e262 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10574 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-18 r/7411 refactor(tvix/glue): add BuildService to TvixStoreIOFlorian Klink1-0/+8
TvixStoreIO triggers builds whenever IO into a not-yet-built store path is requested, if it knows how to build that path. Change-Id: If30e9db6be2f2a30cbc9d0576f357f3ecfa0d35a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10645 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-18 r/7410 feat(tvix/glue): use TvixStoreIO as derivation builtin stateRyan Lahfa1-32/+34
We propagate a `TvixStoreIO` as the `state` of our derivation-specific builtins in the glue crate. The evaluators `io_handle` itself is using a Rc<dyn EvalIO>. An earlier version of TvixStoreIO was also introducing generics over the different internal services themselves, but we opted for instead hardcoding this to Arc<dyn …> for the sake of less macro voodoo. Change-Id: I535c476f06b840858fa3070c4a237ece47f7a15b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10636 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-12 r/7372 refactor(tvix/glue/tvix_store_io): async store_path_to_nodeFlorian Klink1-17/+23
Provide a store_path_to_node_sync function which uses the runtime handle to block on the async function internally, but make store_path_to_node itself async, so it can call async functions internally. We'll use that later when triggering builds and waiting on their results. Change-Id: Idae9da7aa5b0878e0d3a2eba34ea2623e1ba84b2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10607 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 r/7361 refactor(tvix/castore/directorysvc): AsRef traverse_toFlorian Klink1-2/+1
Change-Id: I641bd4ab3de591a013f03137f1e16295946315f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10579 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-09 r/7359 refactor(tvix): use AsRef<dyn …> instead of Deref<Target= …>Florian Klink1-16/+14
Removes some more needs for Arcs. Change-Id: I9a9f4b81641c271de260e9ffa98313a32944d760 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10578 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-05 r/7352 refactor(tvix/glue): simplify TvixStoreIO furtherFlorian Klink1-45/+43
We don't need to spawn in all these places, we can just block_on directly, this is all IO bound. This also means, we don't need to clone any of the service handles (except preserving clone-ability of the BlobService). Change-Id: I7d90f4d6a263a98491caa071ada538a5197a5472 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10540 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-05 r/7351 refactor(tvix/glue): simplify store_path_to_[root]_nodeFlorian Klink1-41/+36
This was wrongly named, it returns a specific node at a subpath. Also, this code can be simplified a lot - we don't need to spawn additional tasks, and can get rid of some clones too. This is also where we need a certain build - so add some TODO to block / fetch here. Change-Id: Id26d7bd80f7a2095121e642b3f7716de78d6b6a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10539 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-01 r/7302 refactor(tvix/glue/tvix_store_io): use utils::import_pathFlorian Klink1-98/+25
This is doing the exact same thing. Change-Id: Iadc5e13dd192efc91cc3d36b2bdf4b8b99a312b2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10512 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7299 feat(nix-compat/store_path): build_ca_path may failFlorian Klink1-1/+6
Change-Id: Ia74ee870f38b7966501458bace541092256c3213 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10509 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-31 r/7289 refactor(tvix/eval): remove code and location from structFlorian Klink1-2/+2
Instead, it's passed in the evaluate/compile_only functions, which feels more naturally. It lets us set up the Evaluation struct long before we actually feed it with data to evaluate. Now that Evaluation::new() would be accepting an empty list of arguments, we can simply implement Default, making things a bit more idiomatic. Change-Id: I4369658634909a0c504fdffa18242a130daa0239 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10475 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-12-14 r/7219 test(tvix/glue): add nonexisting_path_without_import testFlorian Klink1-0/+18
This moves the failing example from cl/10285 into its separate test case. There were multiple complications: tvix-[ca]store was panicking in some places, rather than returning an error. This is now fixed. It needs to live in tvix-glue, so we actually have a "proper" EvalIO interface doing something. > toString ({ line = 42; col = 42; file = /deep/thought; }.file) Should not cause an error, because it shouldn't trigger an import, but leave the path as-is, and not care about it not being present. Change-Id: I76f70b3cb1f73a0fb05870375710fd9f67d5603c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10342 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-13 r/7213 feat(tvix/glue): add single-file import testsFlorian Klink1-0/+23
This ensures importing these paths also behave the same way as Nix. Change-Id: Icaa507bbe3d9867a301fc7a300c5d2b3f9feb911 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10355 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-12-13 r/7212 feat(tvix/glue): add some import testsFlorian Klink1-0/+90
This creates a directory with a .keep file inside, and uses "${path/to/there}" to coerce it to a string (and import it into the store), ensuring it calculates the same store paths as Nix does. Change-Id: Ie14ae075104ce278bc4f2cce93aab5762a2734d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10343 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 r/7210 fix(tvix/glue/tvix_store_io): don't unwrap ingest_path errorsFlorian Klink1-1/+1
Fixes b/344. Change-Id: I1446726e3be3a8fc20801d466a964c4d6b8cbc02 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10331 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-12 r/7170 refactor(tvix): use io::Result for EvalIOFlorian Klink1-6/+6
This is just a alias for Result<_, io::Error>, but shorter. Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-09 r/7128 fix(tvix/glue): do not panic if PathInfoService returns ErrAdam Joseph1-1/+1
Nixpkgs tries to `import` the value returned by `builtins.unsafeGetAttrPos`, which in our case is the file `/deep/thought`. Since that doesn't exist, tvix-glue panics, but there's no interpreter backtrace to follow. Let's return an Err instead of panicking. ------------------------------------------------------------------------------ Before: thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27: called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'main' panicked at src/tvix_store_io.rs:276:58: called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(41580), ...) Command exited with non-zero status 101 ------------------------------------------------------------------------------ After: thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27: called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace note: while evaluating this Nix code --> [code]:1:1 | 1 | (import /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src {}).pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:6:78 | 6 | runCommand "rocm-llvm-binutils-${llvm.version}" { preferLocalBuild = true; } '' | ______________________________________________________________________________^ 7 | | mkdir -p $out/bin 8 | | 9 | | for prog in ${lld}/bin/*; do ... | 27 | | ln -s ${lld}/bin/lld $out/bin/ld 28 | | '' | |__^ note: while evaluating this as native code (coerce_to_string) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:27:9 | 27 | ln -s ${lld}/bin/lld $out/bin/ld | ^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:23 | 249 | outPath = assert condition; drv.${outputName}.outPath; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (force) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:30 | 249 | outPath = assert condition; drv.${outputName}.outPath; | ^^^^^^^^^ note: while evaluating this as native code (throw) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:262:8 | 262 | in handler msg; | ^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14 | 254 | else '' | ______________^ 255 | | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. 256 | | 257 | | '' + (builtins.getAttr reason remediation) attrs; | |________________________________________________________^ note: while evaluating this as native code (force) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14 | 254 | else '' | ______________^ 255 | | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. 256 | | 257 | | '' + (builtins.getAttr reason remediation) attrs; | |__________^ error[E029]: I/O error: /deep/thought:42: task panicked --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:255:41 | 255 | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. | ^^^^^^^^^^^^^^^ Command exited with non-zero status 1 Benchmark: {"pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath":{"kbytes":"26613180","system":"22.35","user":"140.62"}} Change-Id: I587b57e9e49d1f3ecdc0fc9cf996d179a3548f34 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10223 Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-16 r/7024 docs(tvix/glue): fix doc-comment referenceFlorian Klink1-1/+1
This has been renamed to descend_to in cl/9373. Change-Id: Ia6201fb81c7d4fa953d311451cfff95373549a50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10045 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-04 r/6936 refactor(tvix): move tvix glue code into glue crateFlorian Klink1-0/+359
There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-13T07·40+0000