about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2024-01-23 r/7443 feat(tvix/store/nar-info): Support async blob uploadConnor Brewster5-11/+85
Right now all blob uploads are performed synchronously, this means if a NAR contains many small files, the import time is dominated by round trip time to the blob service. For small files, we can buffer them in memory and upload them asynchronously to the blob service. Before returning we make sure to join all the uploads to make sure they complete successfully before responding OK. This reduces time to import a bash-interactive closure on my machine from 1m19s to 7s. Change-Id: Ica3695c159e6c8ad8769281ac20d037e3143e856 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10679 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-23 r/7442 feat(tvix/store/nar-bridge): Setup OpenTelemetryConnor Brewster6-34/+200
Sets up OpenTelemetry integration for nar-bridge. Right now it will export spans for HTTP server requests and all gRPC client requests. Having the spans available will make performance work significantly easier as it provides a high level overview of where time is being spent. In the future we can add application-specifc metrics and integrate logrus. Change-Id: Ie3860675d7ffc626a95673ba062c3c798d8bb2a7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10678 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-23 r/7441 docs(tvix/eval/tests): update commentFlorian Klink1-2/+1
There's no `expected_failures` feature, we run them unconditionally. Change-Id: Ibe1c93497e040d0d5b6cbfcaa043027814c191f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10681 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-01-22 r/7439 fix(tvix/glue/tests): resolve logic error in hasContextInAttrKeyssterni1-1/+2
Change-Id: I7aa09c10f803c2b79363fe3eb52c1127276a333b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10676 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-22 r/7438 docs(tvix): initial notes on a possible generic Nix lang test suitesterni1-0/+140
This kind of collects points to consider which should hopefully help in figuring out what such a lang test suite could or should look like exactly—which is something I currently struggle somewhat. Change-Id: If4f47546fe4b8046fb79718743fa9a72f9801876 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10657 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-22 r/7437 feat(tvix/castore): `process_entry` cannot process unsupported nodesRyan Lahfa1-1/+13
In the past, we had a `todo!` on unsupported node types, this returns a proper error that can be caught by the caller. Change-Id: Icba4c1dab33c0d670a97f162c9b358d1ed5855cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10675 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-21 r/7435 chore(tvix/store): Use BoxStream type aliasConnor Brewster15-67/+44
The BoxStream type alias is a more concise and easier to read than the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type. Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-20 r/7434 docs(tvix/eval): builtins.toXML is doneFlorian Klink1-1/+1
Our docs say it's still todo, but that's wrong, it landed in cl/7835 and cl/7962. Change-Id: Ice374ae47ab653146be18c02dce27145e7488397 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10674 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org>
2024-01-20 r/7433 feat(tvix/store): enable `name` customization in the storeRyan Lahfa3-15/+19
Sometimes, Nix lets someone customize the `name` in the store for a path, this is the case for `builtins.path` which takes a `name` argument, we leave it to the caller to choose the name, which can be the basename by default of the path. Change-Id: Icdbf71d1d8f2dca5716b99d20aac885aab905b80 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10653 Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 r/7432 refactor(tvix/store): `import_path` → `import_path_as_nar_ca`Ryan Lahfa5-118/+161
Add multiple additional helpers such as: - `path_to_name`: derive the basename of a given path - `derive_nar_ca_path_info`: derive the `PathInfo` for a content addressed NAR which isolates further the tree walking feature and the ingestion feature. Additionally, we don't `expect` anymore and propagate properly ingestion errors up. Change-Id: I60edb5b633911c58ade7e19f5002e6f75f90e262 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10574 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-20 r/7431 refactor(tvix/castore): break down `ingest_path`Ryan Lahfa1-70/+187
In one function that does the heavy lifting: `ingest_entries`, and three additional helpers: - `walk_path_for_ingestion` which perform the tree walking in a very naive way and can be replaced by the user - `leveled_entries_to_stream` which transforms a list of a list of entries ordered by their depth in the tree to a stream of entries in the bottom to top order (Merkle-compatible order I will say in the future). - `ingest_path` which calls the previous functions. Change-Id: I724b972d3c5bffc033f03363255eae448f017cef Reviewed-on: https://cl.tvl.fyi/c/depot/+/10573 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-20 r/7430 feat(tvix/castore): ingestion does DFS and invert itRyan Lahfa1-49/+69
To make use of the filtering feature, we need to revert the internal walker to a real DFS. We will therefore just invert the whole tree by storing all of its contents in a level-keyed vector. This is horribly expensive in memory, this is a compromise between CPU and memory, here is the fundamental reason for why: When you encounter a directory, it's either a leaf or not, i.e. it contains subdirectories or not. To know this fact, you can: - wait until you notice subdirectories under it, i.e. you need to store any intermediate nodes you see in the meantime -> memory penalty. - getdents or readdir on it to determine *NOW* its subdirectories -> CPU penalty and I/O penalty. This is an implementation of the first proposal, we pay memory. In practice, we are paying O(#nb of nodes) in memory. There's a smarter albeit much more complicated algorithm that pays only O(\sum_i #siblings(p_i)) nodes where (p_1, ..., p_n) is the path to a leaf. which means for: A / \ B C / / \ D E F We would never store D, E, F but only E, F at a given time. But we would still store B, C no matter what. Change-Id: I456ed1c3f0db493e018ba1182665d84bebe29c11 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10567 Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 r/7429 fix(nix-compat): accept SRI hashes of invalid lengthRyan Lahfa1-2/+36
In cl/10468, we accepted SRI hashes of invalid padding while checking their trailing bits. In this commit, we accept SRI hashes of invalid padding and invalid length, as Nix does. Real world example: `pkgs.javaPackages.openjfx11.deps` <https://github.com/NixOS/nixpkgs/blob/849e4dc5ff0ae70fb7a0df19dbbf633e408c0f32/pkgs/development/compilers/openjdk/openjfx/11.nix#L71> in nixpkgs. Change-Id: I834437e7b94dab9fbb030163f7a2741f52bbf03a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10668 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 r/7425 fix(tvix/glue/benches/eval): add impure_builtinsFlorian Klink1-1/+2
Impure builtins need to be explicitly added. This was missed from cl/10640. Change-Id: I1f8a84aa02f1cf8a26247960d6eb15895e09c610 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10667 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-19 r/7422 chore(3p/sources): Bump channels & overlayssterni13-50/+42
- Adjust to ecl 23.9.9 release - Regenerate go protos after protoc-gen-go update - Drop dhall fork which hasn't kept up with 1.42.* - Address new clippy warnings: - Variant naming of Error::ValidationError - Simplify .try_into().unwrap() - Drop unnecessary identity function - Test module must be last in file - Drop unused `pub use` - Update agenix to 0.15.0. Current master has a installCheckPhase that doesn't work with C++ Nix 2.3.*: https://github.com/ryantm/agenix/commit/a23aa271bec82d3e962bafb994595c1c4a62b133#commitcomment-137185861 Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-18 r/7413 refactor(tools): move crate2nix generate hereFlorian Klink2-10/+3
Having something running the depot crate2nix and formatting it with depotfmt is useful outside of tvix too. Change-Id: Iecc8f207da38cc6995747c5ea48d3911433fd416 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10658 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-18 r/7412 feat(tvix/castore): convert import error to `std::io::Error`Ryan Lahfa1-0/+6
So that we can just `map_err` easily in functions returning `std::io::Error` but calling functions returning `castore::import::Error`. Change-Id: Id181b95e8431c69e95f3a8cd569ca10306656e1d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10572 Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-18 r/7411 refactor(tvix/glue): add BuildService to TvixStoreIOFlorian Klink8-6/+25
TvixStoreIO triggers builds whenever IO into a not-yet-built store path is requested, if it knows how to build that path. Change-Id: If30e9db6be2f2a30cbc9d0576f357f3ecfa0d35a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10645 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-18 r/7410 feat(tvix/glue): use TvixStoreIO as derivation builtin stateRyan Lahfa7-104/+126
We propagate a `TvixStoreIO` as the `state` of our derivation-specific builtins in the glue crate. The evaluators `io_handle` itself is using a Rc<dyn EvalIO>. An earlier version of TvixStoreIO was also introducing generics over the different internal services themselves, but we opted for instead hardcoding this to Arc<dyn …> for the sake of less macro voodoo. Change-Id: I535c476f06b840858fa3070c4a237ece47f7a15b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10636 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-18 r/7409 refactor(tvix/eval): move Evaluation::{default,new_pure}() againFlorian Klink6-23/+22
Have a Evaluation::new() function that's used to set up the Evaluation struct initially - which is also used by both new_pure and new_impure internally. It's generic over the exact type of IO, making it easier to instantiate Evaluation with non-tvix-eval EvalIO implementations, that might not be in a Box. Change-Id: Ibf728da24aca59639c5b6df58d00ae98c99a63f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10640 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-01-18 r/7408 chore(tvix/shell): add `cargo-expand` to debug proc-macroRyan Lahfa1-0/+1
I lost a lot of hope and had to read the source code of `quote!`, `cargo expand` was invaluable in this adventure. We should keep it IMHO. Change-Id: Icfb4c80d413602f2bdc6deab0d595183825d88ad Reviewed-on: https://cl.tvl.fyi/c/depot/+/10635 Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-18 r/7407 refactor(tvix/eval): generalize EvalIO containerFlorian Klink8-48/+74
Don't restrict to a Box<dyn EvalIO>. There's still one or two places where we do restrict, this will be solved by b/262. Change-Id: Ic8d927d6ea81fa12d90b1e4352f35ffaafbd1adf Reviewed-on: https://cl.tvl.fyi/c/depot/+/10639 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-17 r/7406 fix(tvix/eval): catchable-aware `throw`Ryan Lahfa1-0/+4
`throw (throw "a")` should work and propagate the internal throw. Before this commit, it didn't work. Change-Id: Id5d46f74e484dba99e912ad9fa211f3bf1617bac Reviewed-on: https://cl.tvl.fyi/c/depot/+/10600 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 r/7405 fix(tvix/eval): catchable-aware `elem`Ryan Lahfa1-0/+4
`elem` did not catch the list being a catchable. This surfaced during Nixpkgs evaluation. Change-Id: Icf19b94e914e35a435c4412d769ee63ba59ab7b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10599 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-17 r/7404 feat(tvix/glue): introduce test suite for context stringsRyan Lahfa14-9/+377
This is an additional test suite on the top of the Nix ones for context strings matters. It already smoked out multiple mistakes and potential bugs and non-deterministic result from the evaluator. It uses a similar technology as the one in the tvix-eval albeit we instantiate a fully fledged evaluator with in-memory store. We copy the files instead of symlinking them because crates are built in isolation, so symlinks cannot work. Change-Id: I63ae225ce4f83c6e2c8ccd60d779c2f8eb9d08fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10619 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 r/7403 fix(tvix/eval): `getContext` merges underlying valuesRyan Lahfa4-1659/+1272
Previously, we were assembling very naively an attribute set composed of context we saw. But it was forgetting that `"${drv}${drv.drvPath}"` would contain 2 contexts with the same key, but with different values, one with `outputs = [ "out" ];` and `allOutputs = true;`. Following this reasoning and comparing with what Nix does, we ought to merge underlying values systematically. Hence, I bring `itertools` to perform a group by on the key and merge everything on the fly, it's not beautiful but it's the best I could find, notice that I don't use `group_by` but I talk about group by, that is, because `group_by` is a `group_by_consecutive`, see https://github.com/rust-itertools/itertools/issues/374. Initially, I tried to do it without a `into_grouping_map_by`, it was akin to assemble the final `NixAttrs` directly, it was less readable and harder to pull out because we don't have a lot of in-place mutable functions on our data structures. Change-Id: I9933c9bd88ffe04de50dda14f21879b60d8b8cd4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10620 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-17 r/7400 fix(tvix/eval): context-aware `dirOf`Ryan Lahfa1-1/+1
`dirOf` forgot to accepts contextful strings, e.g. derivations and propagates this context further. Change-Id: I6c05944a3ce5073e243e7676c9be56c48407d657 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10618 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-17 r/7399 fix(tvix/eval): context-aware… `hasContext`Ryan Lahfa1-1/+1
Yes, `hasContext e` should work where `e` is a contextful strings, otherwise, it is really useless. Change-Id: I5eb071fc257217d6e8a63fe519132ebd98186696 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10617 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 r/7398 feat(tvix/eval): move away from `test_generator` to `rstest`Ryan Lahfa4-39/+209
`test-generator` has not been updated in the past 2 years. `rstest` has not been updated in the past 5 months. This is an improvement in the maintenance state… I guess? We get also new features, it changes the name of the tests with numbers too. Change-Id: I5376104c7704f525dba7524da78daa09867cc669 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10623 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 r/7397 chore(third_party/crate2nix): run tests in /build/sourceFlorian Klink1-0/+10
Cherry-pick of https://github.com/nix-community/crate2nix/pull/328. This should fix rstest runs inside crate2nix-generated derivations. Change-Id: I9d393768f7f764e33c5938bd8fa14bd1bb0b72e1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10650 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-16 r/7395 fix(tvix/eval): lift VM ops over Catchableedef2-115/+102
We want to handle bottoms in a consistent fashion. Previously this was handled by repetitive is_catchable checks, which were not consistently present. Change-Id: I9614c479cc6297d1f64efba22b620a26e2a96802 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10485 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-01-16 r/7393 fix(tvix/cli/ci): prevent unnecessary dep on compared out/drv pathsterni1-1/+1
Change-Id: Ieab16fd5fad67a4ed0124f118aef8abbefd51cdb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10646 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-16 r/7389 refactor(nix-compat): use StorePathRef for hash derivation moduloFlorian Klink4-30/+34
Rather than passing strings around, use a StorePathRef. This makes things a bit more typesafe, and more aligned with what we want to do in b/264. Change-Id: Ib7080addf27e7f1a9c8da1d8aaa66744468e3b5a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10633 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-15 r/7388 chore(third_party/overlays): bump crate2nix to 0.13.0Florian Klink1-2/+1
We need to vendor in the package expression, as it's not possible to override cargoHash. Change-Id: Ib123647bb9b96d41f4630daa431d020f1cb8d4fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/10624 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-15 r/7387 feat(tvix/build): add CLI entrypointFlorian Klink4-0/+160
This starts a BuildService as a separate process, currently defaulting to the DummyBuildService. Change-Id: Ic206f00831641d3ffebaa44883b7dc053700b9ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/10631 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-15 r/7386 feat(tvix/build): add from_addr methodFlorian Klink5-0/+99
This allows constructing a BuildService from a URI, similar to how it's done in tvix-[ca]store. Change-Id: Ib962b329535c6c7e378ab7ac7f4dd254366497b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10630 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-15 r/7385 feat(tvix/build): add gRPC clientFlorian Klink4-1/+32
Change-Id: I0d917a9a308227b13d096def22945db917830d18 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10629 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-15 r/7384 feat(tvix/build): add GRPCBuildServiceWrapperFlorian Klink2-0/+39
This produces a gRPC BuildService server for anything implementing our BuildService trait. Change-Id: I59c690a432b5e1f59209fd67e2718cb8c935adf2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10628 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-15 r/7383 feat(tvix/build): add BuildService traitFlorian Klink6-0/+38
Also provide a dummy implementation that just fails on any build that's requested. Change-Id: I0df743a730c5331ec9ce6e97a966abe18ce067f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10627 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-15 r/7382 refactor(tvix/glue): have derivation_to_build_request consume inputsFlorian Klink1-58/+14
Determining the inputs might trigger additional builds/substitutions, so answering these lookups via a lambda in a lazy fashion gets complicated. You end up assembling the list of input nodes upfront, and the lambda will just be a dumb lookup into that preassembled list. Rather than doing that, simply have derivation_to_build_request leave the work of determining the inputs to the caller. Change-Id: I75880132916c76b930807c989090da298b6891bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10626 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-15 r/7381 refactor(tvix/glue/known_paths): drop some unused stuffFlorian Klink1-51/+3
This are leftovers from the "reference scanning" approach (which we didn't end up using). We still want a concept of known paths, so we can trace IO into storepaths back to the build recipe that'll produce it, so let's keep the rest of this struct around. Change-Id: I73d38e21e5b97950b8fc2a42176cae5f80d371c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10632 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-15 r/7380 feat(tvix/castore): implement Ord for node::NodeFlorian Klink1-0/+14
This allows assembling BTreeSets of node::Node. Change-Id: I97b83be5ffc3e891307a8ef2b5fc31e38b747a62 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10625 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-14 r/7379 fix(tvix/eval): catchable-aware builtinsRyan Lahfa6-42/+329
A bunch of operations in Tvix are not aware of catchable values and does not propagate them. In the meantime, as we wait for a better solution, we just offer this commit for moving the needle. Change-Id: Ic3f0e1550126b0847b597dfc1402c35e0eeef469 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10473 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-12 r/7377 feat(tvix/glue): input derivation context testsRyan Lahfa1-0/+32
`args` was not propagating context, here's a regression test for it. Change-Id: I8b6a3148508d40df0077128f0bafe68c098a03bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10610 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-12 r/7376 feat(tvix/glue/derivationStrict): support __structuredAttrsFlorian Klink6-197/+267
This adds support to handle the __structuredAttrs argument, which can be passed to builtins.derivationStrict. If __structuredAttrs is passed, and set to true, most of the arguments passed to builtins.derivationStrict are not simply coerced to a string and passed down to "environments", but instead kept in a more structured fashion. Inside ATerm, which is what's relevant as far as path calculation is concerned, a virtual `__json` environment variable is present, containing these structured values. Inside Builds, these structured values are not made available as an environment variable, but a JSON file (and source-able bash script). This will need to be respected once we start emitting BuildRequests, and for that we can probably just parse the `__json` key in Derivation.environment again - or keep this additionally in non-serialized form around during Evaluation. No matter what, this is left for a followup CL. The existing handle_derivation_parameters and populate_outputs helper function were removed, as __structuredAttrs causes quite a change in behaviour, and so handling both in the same place makes it more readable. There's some open questions w.r.t. string contexts for structured attrs itself. A TODO is left for this, but at least path calculation for individual structured attrs derivations are correct now. Part of b/366. Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 r/7375 feat(tvix/eval): make into_json publicFlorian Klink1-1/+1
Allow other crates (like tvix-glue) to look at a Value in JSON, which is used by the structured attrs feature. Change-Id: Iba02ace6e11a74c3f9b19dcbef4b008b76dec046 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10602 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-12 r/7374 docs(tvix/glue): misc reflowing of commentsRyan Lahfa1-7/+5
My OCD could not be stopped. Change-Id: I2bf504fe0865a5084ad02aee18e6180a8a3e19d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10609 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-12 r/7373 fix(tvix/glue/tvix_build): fn_input_drvs_to_output_nodesFlorian Klink1-47/+36
The Derivation input_derivations field contains a list of input derivations and (a subset of their) output names. This means, multiple nodes can be returned, so return a Vec. Also, update the name to better reflect the nodes are the nodes of the selected outputs, not a node representing the .drv file itself. Additionally, use a proto::node::Node (the naked enum), rather than proto::Node, which wraps this in an optional struct field until realizing the BuildRequest. Change-Id: Iec5620b5d7ac0462f2c76acac4abcaeea2de0aad Reviewed-on: https://cl.tvl.fyi/c/depot/+/10608 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-12 r/7372 refactor(tvix/glue/tvix_store_io): async store_path_to_nodeFlorian Klink1-17/+23
Provide a store_path_to_node_sync function which uses the runtime handle to block on the async function internally, but make store_path_to_node itself async, so it can call async functions internally. We'll use that later when triggering builds and waiting on their results. Change-Id: Idae9da7aa5b0878e0d3a2eba34ea2623e1ba84b2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10607 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 r/7371 refactor(tvix/store/pathinfoservice): make more genericFlorian Klink6-54/+51
We don't need Arcs in most of the cases, we're fine with some container. Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-22T01·59+0000