about summary refs log tree commit diff
path: root/tvix/glue
AgeCommit message (Collapse)AuthorFilesLines
2024-01-03 r/7312 feat(tvix/build/protos): add additional_files fieldFlorian Klink1-0/+4
Nix builds also have some other inputs, outside of inputs_dir, used in derivations setting `passAsFile` and `__structuredAttrs`. This extends our `BuildRequest` structure to accomodate these usecases. Change-Id: I8de6516eb467fa48a4961d88933ebd548f131049 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10519 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2024-01-03 r/7311 feat(tvix/glue): emit a warning in case of bad SRI hashesFlorian Klink3-3/+42
And include a test to ensure we show the warning. Change-Id: Ib6a436dbba2592b398b54e44f15a48d1aa345099 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10470 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7302 refactor(tvix/glue/tvix_store_io): use utils::import_pathFlorian Klink1-98/+25
This is doing the exact same thing. Change-Id: Iadc5e13dd192efc91cc3d36b2bdf4b8b99a312b2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10512 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7299 feat(nix-compat/store_path): build_ca_path may failFlorian Klink1-1/+6
Change-Id: Ia74ee870f38b7966501458bace541092256c3213 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10509 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-31 r/7289 refactor(tvix/eval): remove code and location from structFlorian Klink3-6/+6
Instead, it's passed in the evaluate/compile_only functions, which feels more naturally. It lets us set up the Evaluation struct long before we actually feed it with data to evaluate. Now that Evaluation::new() would be accepting an empty list of arguments, we can simply implement Default, making things a bit more idiomatic. Change-Id: I4369658634909a0c504fdffa18242a130daa0239 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10475 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-12-29 r/7283 feat(tvix/build): don't force outputs to be in store_dirFlorian Klink1-23/+16
There's no need to require builds to also put their outputs in store_dir. So rename store_dir to inputs_dir and make outputs consume paths, not basenames. Also document the possibility for builds to write to inputs_dir. Let's see if we want to make this a constraint later. Change-Id: Ib4df20afcdde2d771c269c422f04c7e95587cd0f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10447 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-12-26 r/7265 feat(tvix/build/protos): add some missing fieldsFlorian Klink2-12/+96
- directory in which the castore input nodes are mounted - working directory for the build command - scratch paths - network access y/n - whether a (static) /bin/sh should be provided Populate these fields appropriately, and extend the tests in tvix-glue with a FOD example. Change-Id: I4f9de1483d6696d74694a09784910c407acb0be0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10412 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-21 r/7244 docs(tvix/glue): fix docstringFlorian Klink1-1/+1
This is not an unclosed <html> tag. Change-Id: I2bd2426fc600de2d96dbab47743f1c7bd5fed35e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10398 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
2023-12-14 r/7219 test(tvix/glue): add nonexisting_path_without_import testFlorian Klink1-0/+18
This moves the failing example from cl/10285 into its separate test case. There were multiple complications: tvix-[ca]store was panicking in some places, rather than returning an error. This is now fixed. It needs to live in tvix-glue, so we actually have a "proper" EvalIO interface doing something. > toString ({ line = 42; col = 42; file = /deep/thought; }.file) Should not cause an error, because it shouldn't trigger an import, but leave the path as-is, and not care about it not being present. Change-Id: I76f70b3cb1f73a0fb05870375710fd9f67d5603c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10342 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-14 r/7218 fix(tvix/eval): remove incorrect imports when coercingsterni1-5/+14
The default behavior of string coercion in C++ Nix is to weakly coerce and import to store if necessary. There is a flag to make it strongly coerce (coerceMore) and a flag that controls whether path values have the corresponding file/directory imported into the store before returning the (store) path as a string (copyToStore). We need to implement our equivalent to the copyToStore (import_paths) flag for the benefit of weak coercions that don't import into the store (dirOf, baseNameOf, readFile, ...) and strong coercions that don't import into the store (toString). This makes coerce_to_string as well as CoercionKind weirder and more versatile, but prevents us from reimplementing parts of the coercion logic constantly as can be seen in the case of baseNameOf. Note that it is not possible to test this properly in //tvix/eval tests due to the lack of an appropriate EvalIO implementation being available. Tests should be added to //tvix/glue down the line. Change-Id: I8fb8ab99c7fe08e311d2ba1c36960746bf22f566 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10361 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-13 r/7213 feat(tvix/glue): add single-file import testsFlorian Klink1-0/+23
This ensures importing these paths also behave the same way as Nix. Change-Id: Icaa507bbe3d9867a301fc7a300c5d2b3f9feb911 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10355 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-12-13 r/7212 feat(tvix/glue): add some import testsFlorian Klink2-0/+91
This creates a directory with a .keep file inside, and uses "${path/to/there}" to coerce it to a string (and import it into the store), ensuring it calculates the same store paths as Nix does. Change-Id: Ie14ae075104ce278bc4f2cce93aab5762a2734d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10343 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 r/7210 fix(tvix/glue/tvix_store_io): don't unwrap ingest_path errorsFlorian Klink1-1/+1
Fixes b/344. Change-Id: I1446726e3be3a8fc20801d466a964c4d6b8cbc02 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10331 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-12 r/7197 feat(tvix/glue): make unimplemented-structuredAttrs catchableAdam Joseph1-1/+3
This commit adjusts the error produced by STRUCTURED_ATTRS so that it is catchable. This way we are able to enumerate the release packageset, and the enumeration process will simply treat the few derivations using structured attributes as being broken, rather than killing the whole eval session. Change-Id: I2e17638b8e3227f88543c3718aaf505deaec22ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/10306 Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-12-12 r/7170 refactor(tvix): use io::Result for EvalIOFlorian Klink2-10/+10
This is just a alias for Result<_, io::Error>, but shorter. Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 r/7161 feat(tvix/glue): add nixpkgs eval benchmarkFlorian Klink2-0/+73
This adds a criterion.rs-based testbench into tvix-glue. It can be invoked by running `cargo bench` from inside the `tvix-glue` crate. `target/criterion/report/index.html` contains nice graphs. It's able to diff against the previous run, so you can invoke `cargo bench` before and after a certain change to reason about the impact in evaluation performance. Currently, we need to create a bunch of Evaluator resources inside the benchmark loop itself, which is a bit annoying, as it leaks into the things we benchmark. This should become better with b/262. Fixes b/322. Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 r/7160 chore(tvix/glue): allow unused_variablesFlorian Klink1-0/+1
cl/9364 did introduce a warning here, which is visible when building in release mode - or invoking `cargo bench` in tvix-glue. Change-Id: Ia82082a58543f0fdd32866fdfcd37d0a5fdfda9c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10261 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-11 r/7154 feat(tvix/build): add derivation_to_build_requestFlorian Klink4-0/+255
This function converts from a nix_compat::derivation::Derivation to a BuildRequest. In addition to the Derivation itself, it needs two lookup functions to map input paths to their castore nodes. Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-10 r/7142 fix(tvix/cli): panic on root cause of the fetchurl(non-boot) bugAdam Joseph1-1/+27
Currently we produce wrong drvPaths for a large number of packages that use fetchurl (but not fetchurlBoot, which is what stdenv uses). A simple reproducer is `pkgs.perl538`. I debugged this down to the root cause, which is the fact that tvix doesn't realize that the mapping from FOD-paths to outputHash is *NOT* a 1:1 mapping. It is a many-to-one mapping. You can have lots of different FODs with the same outputHash or even the same outPath. For example, perl538.src and perldevel.src use the same source tarball but a different `version`. Anyways, I have found the root cause but have run out of time for a while, so I've added a panic!() to in the spot where we have a logic bug in order to call it out. Change-Id: I9766b39cfe2fe7eafec84945b2ad6cc28f9c4b7d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9364 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Adam Joseph <adam@westernsemico.com>
2023-12-09 r/7128 fix(tvix/glue): do not panic if PathInfoService returns ErrAdam Joseph1-1/+1
Nixpkgs tries to `import` the value returned by `builtins.unsafeGetAttrPos`, which in our case is the file `/deep/thought`. Since that doesn't exist, tvix-glue panics, but there's no interpreter backtrace to follow. Let's return an Err instead of panicking. ------------------------------------------------------------------------------ Before: thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27: called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace thread 'main' panicked at src/tvix_store_io.rs:276:58: called `Result::unwrap()` on an `Err` value: JoinError::Panic(Id(41580), ...) Command exited with non-zero status 101 ------------------------------------------------------------------------------ After: thread 'tokio-runtime-worker' panicked at /source/src/import.rs:164:27: called `Result::unwrap()` on an `Err` value: Error { depth: 0, inner: Io { path: Some("/deep/thought:42"), err: Os { code: 2, kind: NotFound, message: "No such file or directory" } } } note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace note: while evaluating this Nix code --> [code]:1:1 | 1 | (import /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src {}).pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:6:78 | 6 | runCommand "rocm-llvm-binutils-${llvm.version}" { preferLocalBuild = true; } '' | ______________________________________________________________________________^ 7 | | mkdir -p $out/bin 8 | | 9 | | for prog in ${lld}/bin/*; do ... | 27 | | ln -s ${lld}/bin/lld $out/bin/ld 28 | | '' | |__^ note: while evaluating this as native code (coerce_to_string) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/development/rocm-modules/5/llvm/stage-2/bintools-unwrapped.nix:27:9 | 27 | ln -s ${lld}/bin/lld $out/bin/ld | ^^^^^^ note: while evaluating this as native code (getAttr) --> <src-builtins/derivation.nix>:26:19 | 26 | outPath = builtins.getAttr outputName strict; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (derivationStrict) --> <src-builtins/derivation.nix>:14:12 | 14 | strict = derivationStrict drvAttrs; | ^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:23 | 249 | outPath = assert condition; drv.${outputName}.outPath; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: while evaluating this as native code (force) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/lib/customisation.nix:249:30 | 249 | outPath = assert condition; drv.${outputName}.outPath; | ^^^^^^^^^ note: while evaluating this as native code (throw) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:262:8 | 262 | in handler msg; | ^^^^^^^^^^^ note: while evaluating this Nix code --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14 | 254 | else '' | ______________^ 255 | | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. 256 | | 257 | | '' + (builtins.getAttr reason remediation) attrs; | |________________________________________________________^ note: while evaluating this as native code (force) --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:254:14 | 254 | else '' | ______________^ 255 | | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. 256 | | 257 | | '' + (builtins.getAttr reason remediation) attrs; | |__________^ error[E029]: I/O error: /deep/thought:42: task panicked --> /nix/store/7xii7xcl0iliqxfq8hp577wdq5j0mikr-kp8vf3gzk1pff9r40j5p0y8kiwhkkqw1-nixpkgs-src/pkgs/stdenv/generic/check-meta.nix:255:41 | 255 | Package ‘${getName attrs}’ in ${pos_str meta} ${errormsg}, refusing to evaluate. | ^^^^^^^^^^^^^^^ Command exited with non-zero status 1 Benchmark: {"pkgsCross.aarch64-multiplatform.rocmPackages_5.hipblas.outPath":{"kbytes":"26613180","system":"22.35","user":"140.62"}} Change-Id: I587b57e9e49d1f3ecdc0fc9cf996d179a3548f34 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10223 Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-16 r/7024 docs(tvix/glue): fix doc-comment referenceFlorian Klink1-1/+1
This has been renamed to descend_to in cl/9373. Change-Id: Ia6201fb81c7d4fa953d311451cfff95373549a50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10045 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-04 r/6939 refactor(tvix/glue): move builtins into separate directoryFlorian Klink5-165/+175
Change-Id: I25b7197458dbfbde8623545dc0a0286eb2744f10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9911 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 r/6938 refactor(tvix/glue): move nix_path handling to helper in tvix-glueFlorian Klink2-2/+12
Change-Id: I2327560c4cf0d3f90e253e3c2f47cb29c762461e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9910 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-04 r/6937 refactor(tvix/glue): use add_derivation_builtins helperFlorian Klink1-7/+2
Change-Id: I284c82612f1c8c81a8b7711fe63c19778ff2fdf0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9909 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-11-04 r/6936 refactor(tvix): move tvix glue code into glue crateFlorian Klink12-0/+1510
There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-26T18·12+0000