about summary refs log tree commit diff
path: root/tvix/glue
AgeCommit message (Collapse)AuthorFilesLines
2024-01-31 r/7460 fix(tvix): Represent strings as byte arraysAspen Smith3-40/+48
C++ nix uses C-style zero-terminated char pointers to represent strings internally - however, up to this point, tvix has used Rust `String` and `str` for string values. Since those are required to be valid utf-8, we haven't been able to properly represent all the string values that Nix supports. To fix that, this change converts the internal representation of the NixString struct from `Box<str>` to `BString`, from the `bstr` crate - this is a wrapper around a `Vec<u8>` with extra functions for treating that byte vector as a "morally string-like" value, which is basically exactly what we need. Since this changes a pretty fundamental assumption about a pretty core type, there are a *lot* of changes in a lot of places to make this work, but I've tried to keep the general philosophy and intent of most of the code in most places intact. Most notably, there's nothing that's been done to make the derivation stuff in //tvix/glue work with non-utf8 strings everywhere, instead opting to just convert to String/str when passing things into that - there *might* be something to be done there, but I don't know what the rules should be and I don't want to figure them out in this change. To deal with OS-native paths in a way that also works in WASM for tvixbolt, this also adds a dependency on the "os_str_bytes" crate. Fixes: b/189 Fixes: b/337 Change-Id: I5e6eb29c62f47dd91af954f5e12bfc3d186f5526 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10200 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-01-22 r/7439 fix(tvix/glue/tests): resolve logic error in hasContextInAttrKeyssterni1-1/+2
Change-Id: I7aa09c10f803c2b79363fe3eb52c1127276a333b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10676 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-20 r/7433 feat(tvix/store): enable `name` customization in the storeRyan Lahfa1-0/+1
Sometimes, Nix lets someone customize the `name` in the store for a path, this is the case for `builtins.path` which takes a `name` argument, we leave it to the caller to choose the name, which can be the basename by default of the path. Change-Id: Icdbf71d1d8f2dca5716b99d20aac885aab905b80 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10653 Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de>
2024-01-20 r/7432 refactor(tvix/store): `import_path` → `import_path_as_nar_ca`Ryan Lahfa1-1/+1
Add multiple additional helpers such as: - `path_to_name`: derive the basename of a given path - `derive_nar_ca_path_info`: derive the `PathInfo` for a content addressed NAR which isolates further the tree walking feature and the ingestion feature. Additionally, we don't `expect` anymore and propagate properly ingestion errors up. Change-Id: I60edb5b633911c58ade7e19f5002e6f75f90e262 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10574 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-20 r/7425 fix(tvix/glue/benches/eval): add impure_builtinsFlorian Klink1-1/+2
Impure builtins need to be explicitly added. This was missed from cl/10640. Change-Id: I1f8a84aa02f1cf8a26247960d6eb15895e09c610 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10667 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-19 r/7422 chore(3p/sources): Bump channels & overlayssterni1-2/+0
- Adjust to ecl 23.9.9 release - Regenerate go protos after protoc-gen-go update - Drop dhall fork which hasn't kept up with 1.42.* - Address new clippy warnings: - Variant naming of Error::ValidationError - Simplify .try_into().unwrap() - Drop unnecessary identity function - Test module must be last in file - Drop unused `pub use` - Update agenix to 0.15.0. Current master has a installCheckPhase that doesn't work with C++ Nix 2.3.*: https://github.com/ryantm/agenix/commit/a23aa271bec82d3e962bafb994595c1c4a62b133#commitcomment-137185861 Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-18 r/7411 refactor(tvix/glue): add BuildService to TvixStoreIOFlorian Klink4-6/+16
TvixStoreIO triggers builds whenever IO into a not-yet-built store path is requested, if it knows how to build that path. Change-Id: If30e9db6be2f2a30cbc9d0576f357f3ecfa0d35a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10645 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-18 r/7410 feat(tvix/glue): use TvixStoreIO as derivation builtin stateRyan Lahfa6-86/+106
We propagate a `TvixStoreIO` as the `state` of our derivation-specific builtins in the glue crate. The evaluators `io_handle` itself is using a Rc<dyn EvalIO>. An earlier version of TvixStoreIO was also introducing generics over the different internal services themselves, but we opted for instead hardcoding this to Arc<dyn …> for the sake of less macro voodoo. Change-Id: I535c476f06b840858fa3070c4a237ece47f7a15b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10636 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-18 r/7407 refactor(tvix/eval): generalize EvalIO containerFlorian Klink2-3/+6
Don't restrict to a Box<dyn EvalIO>. There's still one or two places where we do restrict, this will be solved by b/262. Change-Id: Ic8d927d6ea81fa12d90b1e4352f35ffaafbd1adf Reviewed-on: https://cl.tvl.fyi/c/depot/+/10639 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-17 r/7404 feat(tvix/glue): introduce test suite for context stringsRyan Lahfa11-0/+363
This is an additional test suite on the top of the Nix ones for context strings matters. It already smoked out multiple mistakes and potential bugs and non-deterministic result from the evaluator. It uses a similar technology as the one in the tvix-eval albeit we instantiate a fully fledged evaluator with in-memory store. We copy the files instead of symlinking them because crates are built in isolation, so symlinks cannot work. Change-Id: I63ae225ce4f83c6e2c8ccd60d779c2f8eb9d08fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10619 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-16 r/7389 refactor(nix-compat): use StorePathRef for hash derivation moduloFlorian Klink2-18/+17
Rather than passing strings around, use a StorePathRef. This makes things a bit more typesafe, and more aligned with what we want to do in b/264. Change-Id: Ib7080addf27e7f1a9c8da1d8aaa66744468e3b5a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10633 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-15 r/7382 refactor(tvix/glue): have derivation_to_build_request consume inputsFlorian Klink1-58/+14
Determining the inputs might trigger additional builds/substitutions, so answering these lookups via a lambda in a lazy fashion gets complicated. You end up assembling the list of input nodes upfront, and the lambda will just be a dumb lookup into that preassembled list. Rather than doing that, simply have derivation_to_build_request leave the work of determining the inputs to the caller. Change-Id: I75880132916c76b930807c989090da298b6891bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10626 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-15 r/7381 refactor(tvix/glue/known_paths): drop some unused stuffFlorian Klink1-51/+3
This are leftovers from the "reference scanning" approach (which we didn't end up using). We still want a concept of known paths, so we can trace IO into storepaths back to the build recipe that'll produce it, so let's keep the rest of this struct around. Change-Id: I73d38e21e5b97950b8fc2a42176cae5f80d371c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10632 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-14 r/7379 fix(tvix/eval): catchable-aware builtinsRyan Lahfa1-4/+23
A bunch of operations in Tvix are not aware of catchable values and does not propagate them. In the meantime, as we wait for a better solution, we just offer this commit for moving the needle. Change-Id: Ic3f0e1550126b0847b597dfc1402c35e0eeef469 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10473 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-12 r/7377 feat(tvix/glue): input derivation context testsRyan Lahfa1-0/+32
`args` was not propagating context, here's a regression test for it. Change-Id: I8b6a3148508d40df0077128f0bafe68c098a03bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10610 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-12 r/7376 feat(tvix/glue/derivationStrict): support __structuredAttrsFlorian Klink4-171/+220
This adds support to handle the __structuredAttrs argument, which can be passed to builtins.derivationStrict. If __structuredAttrs is passed, and set to true, most of the arguments passed to builtins.derivationStrict are not simply coerced to a string and passed down to "environments", but instead kept in a more structured fashion. Inside ATerm, which is what's relevant as far as path calculation is concerned, a virtual `__json` environment variable is present, containing these structured values. Inside Builds, these structured values are not made available as an environment variable, but a JSON file (and source-able bash script). This will need to be respected once we start emitting BuildRequests, and for that we can probably just parse the `__json` key in Derivation.environment again - or keep this additionally in non-serialized form around during Evaluation. No matter what, this is left for a followup CL. The existing handle_derivation_parameters and populate_outputs helper function were removed, as __structuredAttrs causes quite a change in behaviour, and so handling both in the same place makes it more readable. There's some open questions w.r.t. string contexts for structured attrs itself. A TODO is left for this, but at least path calculation for individual structured attrs derivations are correct now. Part of b/366. Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-12 r/7374 docs(tvix/glue): misc reflowing of commentsRyan Lahfa1-7/+5
My OCD could not be stopped. Change-Id: I2bf504fe0865a5084ad02aee18e6180a8a3e19d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10609 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-12 r/7373 fix(tvix/glue/tvix_build): fn_input_drvs_to_output_nodesFlorian Klink1-47/+36
The Derivation input_derivations field contains a list of input derivations and (a subset of their) output names. This means, multiple nodes can be returned, so return a Vec. Also, update the name to better reflect the nodes are the nodes of the selected outputs, not a node representing the .drv file itself. Additionally, use a proto::node::Node (the naked enum), rather than proto::Node, which wraps this in an optional struct field until realizing the BuildRequest. Change-Id: Iec5620b5d7ac0462f2c76acac4abcaeea2de0aad Reviewed-on: https://cl.tvl.fyi/c/depot/+/10608 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-12 r/7372 refactor(tvix/glue/tvix_store_io): async store_path_to_nodeFlorian Klink1-17/+23
Provide a store_path_to_node_sync function which uses the runtime handle to block on the async function internally, but make store_path_to_node itself async, so it can call async functions internally. We'll use that later when triggering builds and waiting on their results. Change-Id: Idae9da7aa5b0878e0d3a2eba34ea2623e1ba84b2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10607 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 r/7361 refactor(tvix/castore/directorysvc): AsRef traverse_toFlorian Klink1-2/+1
Change-Id: I641bd4ab3de591a013f03137f1e16295946315f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10579 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-09 r/7359 refactor(tvix): use AsRef<dyn …> instead of Deref<Target= …>Florian Klink1-16/+14
Removes some more needs for Arcs. Change-Id: I9a9f4b81641c271de260e9ffa98313a32944d760 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10578 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-08 r/7357 feat(tvix/glue): add test for output path calculationFlorian Klink1-0/+36
This test serves as a minimal reproducer for output path calculation. Derivations with the same name and output hash, but different build recipe should end up with the same outPath. However derivations with different name should end up with a different outPath. Change-Id: I555be59dd87ea675a0816188ed973f96c311e4e1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10416 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2024-01-06 r/7356 feat(tvix/build): add BuildRequest validationFlorian Klink1-2/+10
Change-Id: I8182e4c4a7e5694c1e6f1f56ce092751c22adf4c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10538 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-05 r/7353 fix(tvix/build): fix inconsistencies around pathsFlorian Klink1-7/+7
Change-Id: I76ce4e042f6fbe4bc897767b55ce4eb1f7269e5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10541 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-01-05 r/7352 refactor(tvix/glue): simplify TvixStoreIO furtherFlorian Klink1-45/+43
We don't need to spawn in all these places, we can just block_on directly, this is all IO bound. This also means, we don't need to clone any of the service handles (except preserving clone-ability of the BlobService). Change-Id: I7d90f4d6a263a98491caa071ada538a5197a5472 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10540 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-05 r/7351 refactor(tvix/glue): simplify store_path_to_[root]_nodeFlorian Klink1-41/+36
This was wrongly named, it returns a specific node at a subpath. Also, this code can be simplified a lot - we don't need to spawn additional tasks, and can get rid of some clones too. This is also where we need a certain build - so add some TODO to block / fetch here. Change-Id: Id26d7bd80f7a2095121e642b3f7716de78d6b6a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10539 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-05 r/7350 chore(tvix): bump test-case dep to 3.3.1Florian Klink1-1/+1
Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-01-03 r/7339 feat(tvix/eval): impl `unsafeDiscardStringContext`Ryan Lahfa1-0/+45
Change-Id: I7f0cc42cbebfe5cd27bf6d4f58a4af927b83646a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10423 Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-01-03 r/7338 feat(tvix/glue): contextful derivationRyan Lahfa4-214/+80
We calculate the input context by performing the union of context over all input of the derivation. Then, we just pass the rest of it to the remaining machinery. Finally, we re-emit an `outPath` and a `drvPath` containing the expected contexts. Change-Id: I74905fb258b5bee8b08d1208c9eb87f51b92a890 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10436 Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-01-03 r/7336 feat(tvix/glue): context-aware `toFile`Ryan Lahfa1-21/+14
This removes the reference tracking and uses instead the context for references and returns some. Change-Id: Ic359ca6b903b63f1a9c679c566004c617b792442 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10435 Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-01-03 r/7316 feat(tvix/glue): handle passAsFileFlorian Klink2-11/+168
This extends derivation_to_build_request to handle passAsFile the same way Nix does, and adds a unit test for it. I opted to making this function fallible (if passAsFile contains a non-existent env var), rather than pushing all of this into the Derivation validate function. Change-Id: I75b635f1f6c0c78d72b9a8fc7824f77e97b69951 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10522 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-03 r/7314 refactor(tvix/glue): create environment_vars Vec at the endFlorian Klink1-8/+7
Prevent code in this function from accessing the Vec, realize it only when returning the BuildRequest struct. Change-Id: I385c1adca552c9cb261cd70c5ca3ec45f0d7a9c3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10521 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-01-03 r/7313 test(tvix/glue): add test with passAsFileFlorian Klink1-0/+1
This already succeeds, as the output path calculation path does not need to be aware of any builder-specific custom handling for passAsFile. Change-Id: I9cc9e4e4351cdeaa3ec33ba58ee1569e7a368150 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10520 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-03 r/7312 feat(tvix/build/protos): add additional_files fieldFlorian Klink1-0/+4
Nix builds also have some other inputs, outside of inputs_dir, used in derivations setting `passAsFile` and `__structuredAttrs`. This extends our `BuildRequest` structure to accomodate these usecases. Change-Id: I8de6516eb467fa48a4961d88933ebd548f131049 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10519 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2024-01-03 r/7311 feat(tvix/glue): emit a warning in case of bad SRI hashesFlorian Klink3-3/+42
And include a test to ensure we show the warning. Change-Id: Ib6a436dbba2592b398b54e44f15a48d1aa345099 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10470 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7302 refactor(tvix/glue/tvix_store_io): use utils::import_pathFlorian Klink1-98/+25
This is doing the exact same thing. Change-Id: Iadc5e13dd192efc91cc3d36b2bdf4b8b99a312b2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10512 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7299 feat(nix-compat/store_path): build_ca_path may failFlorian Klink1-1/+6
Change-Id: Ia74ee870f38b7966501458bace541092256c3213 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10509 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-31 r/7289 refactor(tvix/eval): remove code and location from structFlorian Klink3-6/+6
Instead, it's passed in the evaluate/compile_only functions, which feels more naturally. It lets us set up the Evaluation struct long before we actually feed it with data to evaluate. Now that Evaluation::new() would be accepting an empty list of arguments, we can simply implement Default, making things a bit more idiomatic. Change-Id: I4369658634909a0c504fdffa18242a130daa0239 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10475 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-12-29 r/7283 feat(tvix/build): don't force outputs to be in store_dirFlorian Klink1-23/+16
There's no need to require builds to also put their outputs in store_dir. So rename store_dir to inputs_dir and make outputs consume paths, not basenames. Also document the possibility for builds to write to inputs_dir. Let's see if we want to make this a constraint later. Change-Id: Ib4df20afcdde2d771c269c422f04c7e95587cd0f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10447 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-12-26 r/7265 feat(tvix/build/protos): add some missing fieldsFlorian Klink2-12/+96
- directory in which the castore input nodes are mounted - working directory for the build command - scratch paths - network access y/n - whether a (static) /bin/sh should be provided Populate these fields appropriately, and extend the tests in tvix-glue with a FOD example. Change-Id: I4f9de1483d6696d74694a09784910c407acb0be0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10412 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-21 r/7244 docs(tvix/glue): fix docstringFlorian Klink1-1/+1
This is not an unclosed <html> tag. Change-Id: I2bd2426fc600de2d96dbab47743f1c7bd5fed35e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10398 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
2023-12-14 r/7219 test(tvix/glue): add nonexisting_path_without_import testFlorian Klink1-0/+18
This moves the failing example from cl/10285 into its separate test case. There were multiple complications: tvix-[ca]store was panicking in some places, rather than returning an error. This is now fixed. It needs to live in tvix-glue, so we actually have a "proper" EvalIO interface doing something. > toString ({ line = 42; col = 42; file = /deep/thought; }.file) Should not cause an error, because it shouldn't trigger an import, but leave the path as-is, and not care about it not being present. Change-Id: I76f70b3cb1f73a0fb05870375710fd9f67d5603c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10342 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-14 r/7218 fix(tvix/eval): remove incorrect imports when coercingsterni1-5/+14
The default behavior of string coercion in C++ Nix is to weakly coerce and import to store if necessary. There is a flag to make it strongly coerce (coerceMore) and a flag that controls whether path values have the corresponding file/directory imported into the store before returning the (store) path as a string (copyToStore). We need to implement our equivalent to the copyToStore (import_paths) flag for the benefit of weak coercions that don't import into the store (dirOf, baseNameOf, readFile, ...) and strong coercions that don't import into the store (toString). This makes coerce_to_string as well as CoercionKind weirder and more versatile, but prevents us from reimplementing parts of the coercion logic constantly as can be seen in the case of baseNameOf. Note that it is not possible to test this properly in //tvix/eval tests due to the lack of an appropriate EvalIO implementation being available. Tests should be added to //tvix/glue down the line. Change-Id: I8fb8ab99c7fe08e311d2ba1c36960746bf22f566 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10361 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-12-13 r/7213 feat(tvix/glue): add single-file import testsFlorian Klink1-0/+23
This ensures importing these paths also behave the same way as Nix. Change-Id: Icaa507bbe3d9867a301fc7a300c5d2b3f9feb911 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10355 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-12-13 r/7212 feat(tvix/glue): add some import testsFlorian Klink2-0/+91
This creates a directory with a .keep file inside, and uses "${path/to/there}" to coerce it to a string (and import it into the store), ensuring it calculates the same store paths as Nix does. Change-Id: Ie14ae075104ce278bc4f2cce93aab5762a2734d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10343 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 r/7210 fix(tvix/glue/tvix_store_io): don't unwrap ingest_path errorsFlorian Klink1-1/+1
Fixes b/344. Change-Id: I1446726e3be3a8fc20801d466a964c4d6b8cbc02 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10331 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-12 r/7197 feat(tvix/glue): make unimplemented-structuredAttrs catchableAdam Joseph1-1/+3
This commit adjusts the error produced by STRUCTURED_ATTRS so that it is catchable. This way we are able to enumerate the release packageset, and the enumeration process will simply treat the few derivations using structured attributes as being broken, rather than killing the whole eval session. Change-Id: I2e17638b8e3227f88543c3718aaf505deaec22ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/10306 Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-12-12 r/7170 refactor(tvix): use io::Result for EvalIOFlorian Klink2-10/+10
This is just a alias for Result<_, io::Error>, but shorter. Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-12 r/7161 feat(tvix/glue): add nixpkgs eval benchmarkFlorian Klink2-0/+73
This adds a criterion.rs-based testbench into tvix-glue. It can be invoked by running `cargo bench` from inside the `tvix-glue` crate. `target/criterion/report/index.html` contains nice graphs. It's able to diff against the previous run, so you can invoke `cargo bench` before and after a certain change to reason about the impact in evaluation performance. Currently, we need to create a bunch of Evaluator resources inside the benchmark loop itself, which is a bit annoying, as it leaks into the things we benchmark. This should become better with b/262. Fixes b/322. Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-12 r/7160 chore(tvix/glue): allow unused_variablesFlorian Klink1-0/+1
cl/9364 did introduce a warning here, which is visible when building in release mode - or invoking `cargo bench` in tvix-glue. Change-Id: Ia82082a58543f0fdd32866fdfcd37d0a5fdfda9c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10261 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-05T17·48+0000