about summary refs log tree commit diff
path: root/tvix/glue
AgeCommit message (Collapse)AuthorFilesLines
11 days r/8906 fix(tvix): use tracing::field::Empty for all indicatif.pb_showFlorian Klink2-2/+2
This prevents the field from getting printed to stderr, apparently. Change-Id: Ia9860e4ff37224003154db88ee5f83103060e626 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12756 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-11-04 r/8890 refactor(tvix/eval): remove Value::Json and related functionalityBob van der Linden1-6/+2
Currently Value::Json is used in combination with VMRequest::ToJson to recursively convert tvix Value to serde_json::Value. This functionality is used in builtins.toJSON as well as derivation __structuredAttrs. Both Value::Json and VMRequest::ToJson were removed in this commit. Related functionality in vm.rs is also removed: vm.rs does not know about JSON anymore. Recursively converting to serde_json now happens without going through the VM. Thrown errors that are part of the value of toJSON are now directly propagated as ErrorKind, were-as previously there was a split between CatchableErrorKind and ErrorKind, where eventually CatchableErrorKind would be converted to ErrorKind::Catchable. Change-Id: I066f064926c491e4c087a984f07af43d19124cfe Reviewed-on: https://cl.tvl.fyi/c/depot/+/12732 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-11-04 r/8888 test(tvix/glue): check context of __toString after toJSONBob van der Linden2-1/+2
Change-Id: I65e3a8e8284ffb4986970ea67ed8d6b216e1eb66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12730 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-11-02 r/8882 feat(tvix/glue): use InvalidHash for builtins.pathBob van der Linden1-10/+1
Previously such errors showed up as: error[E006]: expected value of type 'sha256', but found a 'not a sha256' Now they show up as: error[E041]: Invalid hash: invalid encoded digest length '31' for algo sha256 This is consistent with the errors of `builtins.fetchurl`. Change-Id: Id11b26fc7951778640cc4e41b3bf23203eaf07df Reviewed-on: https://cl.tvl.fyi/c/depot/+/12719 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-11-02 r/8881 feat(tvix/eval): introduce ErrorKind::InvalidHashBob van der Linden1-2/+1
The nixhash errors were wrapped in a generic TvixError. Now it has its own TvixError with unique error code. The nixhash error is passed along as a string. The errors looked like: error[E997]: invalid encoded digest length '51' for algo sha256 Now they look like: error[E041]: Invalid hash: invalid encoded digest length '51' for algo sha256 Change-Id: I5c420815538ba4c6567c95f5d44d60c4d48f43fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12718 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-24 r/8855 refactor(tvix/build): use stricter BuildRequest typeMarijan Petričević2-85/+80
Change-Id: Ifadd190e10ec22570ab3ccb4df54f64ae5ef0a44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12674 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-21 r/8850 fix(tvix/glue): don't use builtins.currentSystem in testFlorian Klink1-1/+1
This test will otherwise obviously fail on aarch64-linux, or anything not x86_64-linux, as it'll produce a different Derivation. Change-Id: Iedddcb6aeff05e49fc2e6da27f08111cdc9affec Reviewed-on: https://cl.tvl.fyi/c/depot/+/12677 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-10-19 r/8839 fix(tvix/glue/tvix_store_io): err from store_path_to_node with trace lvlFlorian Klink1-1/+1
When evaluating nixpkgs from the <nixpkgs> channel, store_path_to_node returns an error, falling back to regular filesystem access. This currently produces a warning in the CLI, which is misleading, so degrade to trace level. Change-Id: I4cb2297cc85a2c0e904a37343748f9051aa6d5c7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12665 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-10-19 r/8836 refactor(tvix/castore): add try_into_anonymous_node, rename to try_*Florian Klink1-3/+3
We have two places where we parse protos and want their names to be empty: - Receiving a root node in a nar-bridge NAR request - Processing the CalculateNAR gRPC call We don't have any place where we want to keep a name as bytes::Bytes around, yet we used the `into_name_bytes_and_node` method. It was also a bit wrongly named - it wasn't very clear the name was not validated, and that the function may fail. This moves the "splitting off the name as bytes::Bytes" part into a private helper, only leaving the `try_into_name_and_node` and `try_into_anonymous_node` methods around. Change-Id: I2c7fd9871d49ec67450d7efa6a30d96197fb319c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12664 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-10-18 r/8831 refactor(tvix/build): remove proto::BuildRequest::validateMarijan Petričević1-7/+8
Change-Id: I96fa98946bf6aff5eedcb220e2b6b3d90c204eec Reviewed-on: https://cl.tvl.fyi/c/depot/+/12633 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-16 r/8815 refactor(tvix/glue): merge builtins.{filterSource,path} codepathsFlorian Klink1-108/+82
This moves the implementation from builtins.path into a helper function, which we now call from both builtins. Most of the Value plumbing stays inside this helper. We also implemented handling of symlinks at the root, which was handled in builtins.filterSource, but not builtins.path - by peeking at the FileType using std::fs::metadata, instead of the EvalIO trait. For now, this is fine, as our filtered_ingest also goes via the filesystem directly. It ends up with the same semantics as before and in Nix - symlinks at the root are followed, except if they point to an invalid target. In the future, we should revisit this, and then maybe get both stat and lstat into EvalIO, though we will need to be very careful about the semantics for following symlink inside store paths. Change-Id: I6a941c0187db36165c2f7a338015e4e32d41b298 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12629 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-10-16 r/8814 fix(tvix/glue/builtins/import): Drop useless bstring conversionIlan Joselevich1-4/+1
In a previous refactoring CL this into_bstring method was accidentally kept, when we don't need it and can just to_str directly. Change-Id: Idd531d508b8fd530611b213d0164e7aaf0e87d80 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12631 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-10-15 r/8811 refactor(tvix/glue/builtins/import): refactorFlorian Klink3-161/+175
This removes all the intermediate helper functions and reorganizes the import code to only do the calculations where/when needed, and hopefully makes things easier to understand as well. Change-Id: I7e4c89c742bf8569b45e303523f7f801da7127ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/12627 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Jörg Thalheim <joerg@thalheim.io> Reviewed-by: edef <edef@edef.eu>
2024-10-15 r/8810 test(tvix/glue): improve comments and test cases a bitFlorian Klink1-14/+12
This makes it easier to understand what the specific test is testing. Change-Id: I34b2798841c6b9367849668451af2165dc78f997 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12626 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
2024-10-15 r/8809 test(tvix/glue): add builtins.storeDir regression testFlorian Klink3-0/+9
Change-Id: Ib87f41b3e0d634dd76e648acfe1b3026211f947a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12619 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Jörg Thalheim <joerg@thalheim.io> Tested-by: BuildkiteCI
2024-10-15 r/8808 fix(tvix/glue/import): builtins.storeDir fixesFlorian Klink3-39/+23
This didn't support store paths with a subpath joined to them, while Nix does. Use state.path_exists, which does. This also means we can drop the `store_path_exists` helper, which was only used here. Change-Id: I918ccb270f64acbdc41cb4d2a9c3c5871ce15002 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12618 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Jörg Thalheim <joerg@thalheim.io> Autosubmit: flokli <flokli@flokli.de>
2024-10-15 r/8807 refactor(nix-compat/store_path): use Path in from_absolute_path_fullFlorian Klink1-12/+4
These are not necessarily strings, and making it paths allows us to stop converting them to lossy strings. Change-Id: I11366c721dc5da1778aafe89092a1966b5a43178 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12617 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Jörg Thalheim <joerg@thalheim.io> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-13 r/8803 refactor(tvix/glue): remove use of lazy_staticVincent Ambo4-45/+64
This is now supported in the standard library via std::sync::LazyLock, but requires some manual shuffling around of code. Change-Id: Ibb3be8458b8a8912ea04c9360d64c5cf914254d4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12609 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-12 r/8796 refactor(tvix/eval): Make `strict` an EvalMode enumAspen Smith1-2/+2
Refactor the `strict` boolean passed into evaluation at the top-level to be a (two-variant, so far) EvalMode enum of Lazy and Strict. This is more explicit than a boolean, and if we ever add more EvalModes it's a simple extension of the enum. Change-Id: I3de50e74ec971011664f6cd0999d08b792118410 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12186 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-10-11 r/8789 refactor(tvix/glue/register_in_path_info_service): return only PathInfoFlorian Klink2-33/+28
The store path is already contained in the PathInfo, and the ca bits is already passed into the function, so known to the caller - there's no need to duplicate this. We can also avoid having two separate block_on in our import builtin - we already know the content hash before constructing, as we pass it in via ca_hash. There's still some room to unclutter some more of the code around importing - we still do NAR calculation twice in some cases, and some of the code might be share-able from other places producing PathInfo too. Log a TODO for this cleanup. Change-Id: I6a5fc427d15bc9293a396310143c7694dd2996c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12592 Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-11 r/8787 refactor(tvix/store): use strictly typed PathInfo structMarijan Petričević3-109/+69
This switches the PathInfoService trait from using the proto-derived PathInfo struct to a more restrictive struct, and updates all implementations to use it. It removes a lot of the previous conversion and checks, as invalid states became nonrepresentable, and validations are expressed on the type level. PathInfoService implementations consuming protobuf need to convert and do the verification internally, and can only return the strongly typed variant. The nix_compat::narinfo::NarInfo conversions for the proto PathInfo are removed, we only keep a version showing a NarInfo representation for the strong struct. Converting back to a PathInfo requires the root node now, but is otherwise trivial, so left to the users. Co-Authored-By: Florian Klink <flokli@flokli.de> Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-01 r/8743 feat(tvix/glue): wire up nix refscanningYureka2-12/+73
After this, attempting to build the nixpkgs still fails in the same way, because the references are not yet properly used by the code at `tvix/glue/src/tvix_store_io.rs`. Change-Id: I8a59ef8ef3c9a6f6aa7b05106dd9eef2e9ac0d0f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12532 Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-01 r/8742 feat(tvix/build): add refscanning interfaceYureka1-0/+4
This provides a generic interface to let the builder search for needles in the output, as described in the notes at `docs/src/build/index.md`. Change-Id: Ic2c5bd563e9aa2e766c157f2b13cdb19aede12f8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12531 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Brian Olsen <me@griff.name>
2024-10-01 r/8741 feat(castore/fs): optional refscanner for ingestYureka1-1/+2
Change-Id: Ieca06de4c2e2680d89fe05a380079fafa5454837 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12529 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-09-27 r/8724 refactor(tvix): move refscan module to castoreYureka3-342/+0
This is required to add the optional refscanner parameter to the ingest functions. Change-Id: Ib40a7287cf857eb55e31e0df309a79474fefb518 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12527 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-05 r/8653 chore(3p/sources): bump channels & overlays (2024-09-01)Vincent Ambo1-0/+3
Included changes: * users/aspen: explicitly use open-source nvidia driver This now has to be specified explicitly, otherwise evaluation fails with an error. * users/aspen: nixfmt -> nixfmt-classic * users/aspen: fixes for renamed packages & options * users/tazjin: fixes for renamed packages & options * 3p/overlays: remove cbtemulator patch (merged upstream) * tvix/shell: remove unnecessary patches (merged upstream) * 3p/rust-crates: mark libgit2_sys as broken * users/Profpatsch: mark git-db as broken * 3p/overlays: pick `mypaint` from stable channel * tvix: fix comments that clippy doesn't like anymore * tvix/glue: disable a misfiring clippy lint (applying its suggestion breaks code below) Change-Id: I6d3fc027694bbe7425a2d25dc53d65467a44f3b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12403 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi> Reviewed-by: Profpatsch <mail@profpatsch.de> Autosubmit: tazjin <tazjin@tvl.su>
2024-08-29 r/8610 chore(tvix): Migrate members to inherit deps from workspaceIlan Joselevich1-34/+32
From now on we will add the dependencies and their version in the root Cargo.toml and in order to enable the dependency for a workspace member we set `workspace = true` in the member's Cargo.toml. Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-29 r/8609 chore(tvix): Bump versions of all compatible deps via cargo upgradeIlan Joselevich1-15/+15
This updates all the dependencies and their "minimum" versions in Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s `cargo upgrade` command that will eventually be merged into `cargo update`. Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-20 r/8545 refactor(nix-compat/store_path): make StorePath generic on SFlorian Klink4-43/+45
Similar to how cl/12253 already did this for `Signature`, we apply the same logic to `StorePath`. `StorePathRef<'a>'` is now a `StorePath<&'a str>`, and there's less redundant code for the two different implementation. `.as_ref()` returns a `StorePathRef<'_>`, `.to_owned()` gives a `StorePath<String>` (for now). I briefly thought about only publicly exporting `StorePath<String>` as `StorePath`, but the diff is not too large and this will make it easier to gradually introduce more flexibility in which store paths to accept. Also, remove some silliness in `StorePath::from_absolute_path_full`, which now doesn't allocate anymore. Change-Id: Ife8843857a1a0a3a99177ca997649fd45b8198e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12258 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-19 r/8521 refactor(tvix/eval): remove use of imbl::OrdMapVincent Ambo1-1/+1
Removes imbl::OrdMap in favour of an Rc over the standard library's BTreeMap, which allows us to drop the imbl dependency completely. In my local tests this is actually slightly faster for `hello` and `firefox`. Change-Id: Ic9597ead4e98bf9530f290c6a94a3c5c3efd0acc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12201 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI
2024-08-19 r/8518 docs(tvix/glue/tvix_io): update doc-commentsFlorian Klink1-4/+2
Since we switched from reference scanning to string context, this only handles the `__corepkgs__` hack. Update the docstrings. Change-Id: Ie857c8c99ae1cdb4697323ec738f88be0580df3e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12246 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-08-17 r/8509 chore(tvix/glue): drop some explicit allow(clippy::mutable_key_type)Florian Klink2-3/+0
This is covered by clippy.toml these days. Change-Id: I2330af5781844d5f9d975793d770efcea48d371b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12223 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 r/8507 refactor(tvix/castore): add into_nodes(), implement consuming proto convFlorian Klink1-8/+3
Provide a into_nodes() function on a Directory, which consumes self and returns owned PathComponent and Node. Use it to provide a proper conversion from Directory to the proto variant that doesn't clone. There's no need for the one taking only &Directory, we don't use it anywhere, and once someone needs that they might as well clone Directory before converting it. Update all other users of the `.nodes()` function to use `.into_nodes()` where applicable, and avoid some more cloning there. Change-Id: Id4577b9eb173c012e225337458898d3937112bcb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12218 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 r/8506 refactor(tvix/castore): add PathComponent type for checked componentsFlorian Klink1-5/+11
This encodes a verified component on the type level. Internally, it contains a bytes::Bytes. The castore Path/PathBuf component() and file_name() methods now return this type, the old ones returning bytes were renamed to component_bytes() and component_file_name() respectively. We can drop the directory_reject_invalid_name test - it's not possible anymore to pass an invalid name to Directories::add. Invalid names in the Directory proto are still being tested to be rejected in the validate_invalid_names tests. Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8505 refactor(tvix/castore): drop {Directory,File,Symlink}NodeFlorian Klink5-33/+46
Add a `SymlinkTarget` type to represent validated symlink targets. With this, no invalid states are representable, so we can make `Node` be just an enum of all three kind of types, and allow access to these fields directly. Change-Id: I20bdd480c8d5e64a827649f303c97023b7e390f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12216 Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8504 refactor(tvix/castore): remove `name` from NodesFlorian Klink5-102/+99
Nodes only have names if they're contained inside a Directory, or if they're a root node and have something else possibly giving them a name externally. This removes all `name` fields in the three different Nodes, and instead maintains it inside a BTreeMap inside the Directory. It also removes the NamedNode trait (they don't have a get_name()), as well as Node::rename(self, name), and all [Partial]Ord implementations for Node (as they don't have names to use for sorting). The `nodes()`, `directories()`, `files()` iterators inside a `Directory` now return a tuple of Name and Node, as does the RootNodesProvider. The different {Directory,File,Symlink}Node struct constructors got simpler, and the {Directory,File}Node ones became infallible - as there's no more possibility to represent invalid state. The proto structs stayed the same - there's now from_name_and_node and into_name_and_node to convert back and forth between the two `Node` structs. Some further cleanups: The error types for Node validation were renamed. Everything related to names is now in the DirectoryError (not yet happy about the naming) There's some leftover cleanups to do: - There should be a from_(sorted_)iter and into_iter in Directory, so we can construct and deconstruct in one go. That should also enable us to implement conversions from and to the proto representation that moves, rather than clones. - The BuildRequest and PathInfo structs are still proto-based, so we still do a bunch of conversions back and forth there (and have some ugly expect there). There's not much point for error handling here, this will be moved to stricter types in a followup CL. Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-16 r/8502 feat(tvix/glue): Add refscanner pattern and AsyncReadBrian Olsen2-22/+244
This splits the existing ReferenceScanner into a ReferenceScanner and ReferencePattern as well as adds an AsyncRead implementation that can do a scan while you read from it. The reason to split the scanner in two is that generating the pattern is expensive and when ingesting build results with multiple outputs you want to do several independant scans that look for the same pattern. The reader is for scanning files without having to load the entire file into memory. Change-Id: I993f5a32308c12d9035840f8e04fe82e8dc1d962 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12052 Autosubmit: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-15 r/8494 chore(tvix): Retain original passthru in Rust buildsBrian Olsen1-2/+2
When using the runTests feature of crate2nix the derivation that runs the tests is put into passthru.test but all default.nix files for Rust crates in Tvix threw that away. This commit retains passthru so that you can get access to the test derivation. Change-Id: I8b7b7db57a49069348f08c12c00a3b1a41a0c05b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12215 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink5-9/+7
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka5-97/+79
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-12 r/8482 feat(tvix/cli): Add derivation file dumping functionalityIlan Joselevich2-1/+22
Provides a derivation file dumping functionality for tvix-cli that can be used when passing the --drv-dumpdir CLI arg to tvix-cli. This will dump all the known derivation files into the specified directory, making it easier to debug derivation divergences between Tvix generated drvs and the drvs generated by Nix. Supersedes: https://cl.tvl.fyi/c/depot/+/11265 Change-Id: I0e10b26eba22032b84ac543af0d4150ad87aed3e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12192 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-08 r/8461 feat(tvix): Jemalloc -> MiMallocIlan Joselevich2-7/+3
Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-07-31 r/8434 fix(tvix/glue): coerce path argument to builtins.path to a pathAspen Smith1-3/+8
The "path" key in the arguments to builtins.path supports any path-coercible type (a string, a path...). Coerce it to a path in the argument rather than just requiring it already be one and throwing an error if it's not. This is... annoying to test, since it requires a file with known contents that's available in the build sandbox. But it works! Trust me! Fixes: b/412 Change-Id: I3c8e339bf344a208d5ed5990193942651f318745 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12053 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-07-30 r/8425 test(tvix/glue): Add a benchmark for firefox outPathAspen Smith1-0/+6
This is nice to test too - it's similar to hello, but runs for a lot longer (like 7.5 seconds on my laptop) which means we get even better stats for stuff. Change-Id: I7935818f10a6d846d446e685b9263a72d7e2aabd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12061 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-07-22 r/8399 refactor(tvix): move service addrs into shared clap structYureka5-8/+22
Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka1-1/+1
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-20 r/8380 refactor(tvix/store): use composition in tvix_store crateYureka4-4/+4
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-07 r/8357 refactor(tvix): point SSL_CERT_FILE to /dev/nullFlorian Klink1-2/+2
reqwest wants to be able to read a file of trust roots when constructed, but as it doesn't actually do any HTTPS connections inside the nix build, an empty list of trust roots is totally sufficient. Thankfully /dev/null provides such a file. Change-Id: I9bd1619b2c9f8ff2a6640d2ac410d4de5b20c2ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/11961 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-07-06 r/8351 refactor(tvix/eval): Builderize EvaluationAspen Smith5-49/+61
Make constructing of a new Evaluation use the builder pattern rather than setting public mutable fields. This is currently a pure refactor (no functionality has changed) but has a few advantages: - We've encapsulated the internals of the fields in Evaluation, meaning we can change them without too much breakage of clients - We have type safety that prevents us from ever changing the fields of an Evaluation after it's built (which matters more in a world where we reuse Evaluations). More importantly, this paves the road for doing different things with the construction of an Evaluation - notably, sharing certain things like the GlobalsMap across subsequent evaluations in eg the REPL. Fixes: b/262 Change-Id: I4a27116faac14cdd144fc7c992d14ae095a1aca4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11956 Tested-by: BuildkiteCI Autosubmit: aspen <root@gws.fyi> Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 r/8337 chore(3p/sources): bump to OpenSSH vulnerability hotfixVincent Ambo1-2/+2
See https://github.com/NixOS/nixpkgs/pull/323753 for details. Changes: * git: temporarily comment out dottime patch (it doesn't apply, but it's not critical) * third-party/cgit: use an older git version where dottime patch still applies * 3p/crate2nix: remove crate2nix patches included in latest release * tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs) * tvix: regenerate Cargo.nix * tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function * tvix/nix-compat: remove redundant trait bounds * tvix/glue: use clone_into() to set drv.{builder,system} * tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327 * toold/depotfmt: expose treefmt config as passthru * tools/crate2nix: undo some more hacks in the crate2nix-check drv Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c Co-Authored-By: Florian Klink <flokli@flokli.de> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>