about summary refs log tree commit diff
path: root/tvix/castore/src
AgeCommit message (Collapse)AuthorFilesLines
2024-10-11 r/8787 refactor(tvix/store): use strictly typed PathInfo structMarijan Petričević1-0/+29
This switches the PathInfoService trait from using the proto-derived PathInfo struct to a more restrictive struct, and updates all implementations to use it. It removes a lot of the previous conversion and checks, as invalid states became nonrepresentable, and validations are expressed on the type level. PathInfoService implementations consuming protobuf need to convert and do the verification internally, and can only return the strongly typed variant. The nix_compat::narinfo::NarInfo conversions for the proto PathInfo are removed, we only keep a version showing a NarInfo representation for the strong struct. Converting back to a PathInfo requires the root node now, but is otherwise trivial, so left to the users. Co-Authored-By: Florian Klink <flokli@flokli.de> Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-10 r/8785 feat(tvix/composition): allow urls as anonymous storesYureka3-83/+135
This allows specifying an url in place of a named reference to another composition entry, if the castore crate has been compiled with the xp-store-composition feature. Example: `--directory-service-addr cache://?near=memory://&far=memory://` This would be equivalent to the instantiation via toml file: ```toml [memory1] type = "memory" [memory2] type = "memory" [default] type = "cache" near = "memory1" far = "memory2" ``` Note that each anonymous url causes a distinct instance to be created. Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-10 r/8784 feat(tvix/dirsvc/Cache): support building from urlYureka1-5/+6
Change-Id: I80121319795319bb977427efeca3666c6b87a1b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12147 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-03 r/8752 chore(tvix/[ca]store): Drop sled support completely in favor of redbIlan Joselevich4-283/+0
Over the past couple of months we've been using redb instead of sled as the default filesystem-based database in PS and DS. I am confident that we can get rid of sled completely now, and just keep redb. Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567 Reviewed-by: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-10-01 r/8748 test(tvix/castore/refscan): add empty pattern regression testYureka1-0/+16
Change-Id: I165261170edaabfc56f6ac0a6baae388332b3f73 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12559 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-10-01 r/8747 fix(tvix/castore/refscan): don't panic on empty patternsYureka1-1/+9
Previously, the overlap calculation would underflow when the pattern is empty. Change-Id: I1f6bf49fafc4b8183a3a5e5e491a5a5bfc41ca97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12558 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-10-01 r/8741 feat(castore/fs): optional refscanner for ingestYureka2-17/+52
Change-Id: Ieca06de4c2e2680d89fe05a380079fafa5454837 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12529 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-01 r/8740 feat(tvix/castore/refscan): share the scanner between readersYureka1-39/+30
This changes the only actual state the ReferenceScanner has to use atomic bools, so it no longer requires a mutable borrow for .scan(). This allows passing an immutable borrow of a reference scanner to multiple threads which might be ingesting blobs in parallel, and using them in the ReferenceReader or calling .scan() there. Change-Id: Id5c30bcebb06bf15eae8c4451d70eb806cab722e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12528 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-28 r/8726 chore(tvix/castore/fuse): impl Layer for TvixStoreFsYureka1-1/+12
Allows using a TvixStoreFs in overlay filesystems Change-Id: I10e63c92776b1c783947d92070f360865ef2883c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12539 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-28 r/8725 chore(tvix/castore/fuse): update fuse-backend-rsYureka1-0/+2
This release includes support for overlay filesystems Change-Id: I946cbf346df045209afaea2d720bb57fc2f2659f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12538 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-09-27 r/8724 refactor(tvix): move refscan module to castoreYureka2-0/+340
This is required to add the optional refscanner parameter to the ingest functions. Change-Id: Ib40a7287cf857eb55e31e0df309a79474fefb518 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12527 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-26 r/8717 fix(tvix/castore): don't return object_store::has with INFO levelFlorian Klink1-1/+1
This otherwise spams the logs quite a bit, for example when uploading to nar-bridge with the new check from cl/12497. Change-Id: Idc2bcc513caea6fae38ae04489e4e3ee7be64bce Reviewed-on: https://cl.tvl.fyi/c/depot/+/12510 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-24 r/8711 fix(tvix/nar-bridge): Remove name check for root node in nar generationsinavir2-13/+24
Nar-bridge tried to parse the name of the protobuf node encoded in the URL into a PathComponent but this name was empty, leading to an error when the user tried to retrieve the nar file. This was an oversight from the conversion to stricter types (some of the CLs in the serious containing cl/12217). We need a version converting a protobuf without a name to our stricter types, but an empty PathComponent cannot be constructed. So we need a into_name_and_node() version that returns the name as Bytes, not PathComponent. Change-Id: I2996cdd2e0107133e502748947298f512f1cc521 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12504 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-19 r/8704 fix(tvix/castore/import): check small blobs firstFlorian Klink1-0/+13
ConcurrentBlobUploader buffers small blobs in memory, and then uploads them to the BlobService in the background. In these cases, we know the hash of the whole blob, so we could check if it exists first before, uploading it. We were however not, and this caused rate limiting issues in GCS, as it has an update limit of one write per second on the same key, which we ran into especially frequently with the empty blob. This reduces the amount of writes of the same blob considerably. In the future, we might be able to drop this, as our chunked blob uploading protocol gets smarter and covers these cases. Change-Id: Icf482df815812f80a0b65cec0426f8e686308abb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12497 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-09-07 r/8662 feat(tvix/castore): expose ValidatedDirectoryGraph typeYureka1-1/+1
Change-Id: Id994258e74f75df8790159ed3180ce360874d0de Reviewed-on: https://cl.tvl.fyi/c/depot/+/12438 Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-21 r/8548 fix(tvix/castore): u32 -> u64 in DirectoryError::SizeOverflow messageBen Webb1-1/+1
Fix a discrepancy in the error message for DirectoryError::SizeOverflow. The message indicates that the SizeOverflow error occurs when total size exceeds u32::MAX, but that's not true. All size fields within the castore's internal Directory ADT are u64, and the SizeOverflow error is only returned after a call to the checked_add implementation on u64. See tvix/castore/nodes/directory.rs +111 and tvix/castore/nodes/directory.rs +88 as of this commit. Change-Id: I74d161ea8927362e1cb601ba163489aa96fb91b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12259 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-18 r/8514 refactor(tvix/castore): have SymlinkTarget-specific errorsFlorian Klink5-26/+172
Don't use ValidateNodeError, but SymlinkTargetError. Also, add checks for too long symlink targets. Change-Id: I4b533325d494232ff9d0b3f4f695f5a1a0a36199 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12230 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-18 r/8513 refactor(tvix/castore): have PathComponent-specific errorsFlorian Klink6-76/+246
Don't use DirectoryError, but PathComponentError. Also add checks for too long path components. Change-Id: Ia9deb9dd0351138baadb2e9c9454c3e019d5a45e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12229 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
2024-08-18 r/8512 feat(tvix/castore/proto): add owned conv to castore::DirectoryFlorian Klink2-65/+84
Replace the hand-rolled code comparing names with a try_fold. Also, make it slightly stricter here, detecting duplicates in the same fields earlier. Change-Id: I9c560838ece88c3b8b339249a8ecbf3b05969538 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12226 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-08-18 r/8511 feat(tvix/castore): add Directory::try_from_iter()Florian Klink4-160/+201
This provides a batched variant to construct a Directory, which reuses the previously calculated size. Checking and inserting code is factored out into a check_insert_node function, taking the current size as a parameter and returning the new size. Change-Id: Ia6c2970a0c12181b7c40e63cf7ce8c93298ea37c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12225 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-08-17 r/8508 refactor(tvix/castore): add `name` back to edge weightsFlorian Klink1-11/+22
Make this a proper struct with named fields. We apparently never access B3Digest in there, so it can be removed. Change-Id: Ifc07310393e1afb0a835778eae137a19b54070b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12224 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-17 r/8507 refactor(tvix/castore): add into_nodes(), implement consuming proto convFlorian Klink4-21/+21
Provide a into_nodes() function on a Directory, which consumes self and returns owned PathComponent and Node. Use it to provide a proper conversion from Directory to the proto variant that doesn't clone. There's no need for the one taking only &Directory, we don't use it anywhere, and once someone needs that they might as well clone Directory before converting it. Update all other users of the `.nodes()` function to use `.into_nodes()` where applicable, and avoid some more cloning there. Change-Id: Id4577b9eb173c012e225337458898d3937112bcb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12218 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 r/8506 refactor(tvix/castore): add PathComponent type for checked componentsFlorian Klink17-146/+249
This encodes a verified component on the type level. Internally, it contains a bytes::Bytes. The castore Path/PathBuf component() and file_name() methods now return this type, the old ones returning bytes were renamed to component_bytes() and component_file_name() respectively. We can drop the directory_reject_invalid_name test - it's not possible anymore to pass an invalid name to Directories::add. Invalid names in the Directory proto are still being tested to be rejected in the validate_invalid_names tests. Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8505 refactor(tvix/castore): drop {Directory,File,Symlink}NodeFlorian Klink18-368/+447
Add a `SymlinkTarget` type to represent validated symlink targets. With this, no invalid states are representable, so we can make `Node` be just an enum of all three kind of types, and allow access to these fields directly. Change-Id: I20bdd480c8d5e64a827649f303c97023b7e390f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12216 Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8504 refactor(tvix/castore): remove `name` from NodesFlorian Klink24-699/+504
Nodes only have names if they're contained inside a Directory, or if they're a root node and have something else possibly giving them a name externally. This removes all `name` fields in the three different Nodes, and instead maintains it inside a BTreeMap inside the Directory. It also removes the NamedNode trait (they don't have a get_name()), as well as Node::rename(self, name), and all [Partial]Ord implementations for Node (as they don't have names to use for sorting). The `nodes()`, `directories()`, `files()` iterators inside a `Directory` now return a tuple of Name and Node, as does the RootNodesProvider. The different {Directory,File,Symlink}Node struct constructors got simpler, and the {Directory,File}Node ones became infallible - as there's no more possibility to represent invalid state. The proto structs stayed the same - there's now from_name_and_node and into_name_and_node to convert back and forth between the two `Node` structs. Some further cleanups: The error types for Node validation were renamed. Everything related to names is now in the DirectoryError (not yet happy about the naming) There's some leftover cleanups to do: - There should be a from_(sorted_)iter and into_iter in Directory, so we can construct and deconstruct in one go. That should also enable us to implement conversions from and to the proto representation that moves, rather than clones. - The BuildRequest and PathInfo structs are still proto-based, so we still do a bunch of conversions back and forth there (and have some ugly expect there). There's not much point for error handling here, this will be moved to stricter types in a followup CL. Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink24-584/+588
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka32-1061/+1131
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-01 r/8437 feat(tvix/castore): add RedbDirectoryServiceIlan Joselevich5-0/+340
This provides a DirectoryService implementation which uses redb (https://github.com/cberner/redb) as the database. It provides both in-memory and persistent on-filesystem implementations. Change-Id: Id8f7c812e2cf401cccd1c382b19907b17a6887bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12038 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-01 r/8435 chore(3p/sources): Bump channels & overlays (2024-07-28)Aspen Smith1-2/+10
* Treewide: re-run depotfmt * //third_party/nixpkgs:html5validator: build with Python 3.11, dependency openstackdocstheme doesn't support 3.12 * //users/sterni/machines/ingeborg: adapt to poorly handled fcgiwrap module API change: https://github.com/NixOS/nixpkgs/pull/318599 * //tvix/*-go: regenerate protobuf files * //third_party/nixpkgs:treefmt: Remove patch for merged pull request * //users/flokli/ipu6-softisp: rebase, drop upstreamed kernel patches Change-Id: Ie4e0df007c287e8cd6207683a9a25838aa5bd39a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11971 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-25 r/8413 test(tvix/castore/dirsvc): check for a pitfall with deduplicated dirsYureka2-5/+49
Change-Id: I3cff2c2e8b2c8a2fd8d8074647d0d99a1db8e693 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12034 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-25 r/8412 fix(tvix/castore/GRPCDirectorySvc): fix a bug in the get_recursive fnYureka1-3/+9
When retrieving a closure with get_recursive, the following could happen in the GRPC client: - The first reference to the deduplicated directory is added to expected_directory_digests - The deduplicated directory is obtained removed from expected_directory_digests - The second reference to the deduplicated directory is added to expected_directory_digests - The deduplicated directory has already been sent, but is still in the expected_directory_digests. It looks to the GRPC client like the closure is incomplete and the stream ended prematurely. Change-Id: Ic62bca12e7f8fb85af5fa4dacd199f0f3b8eea8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12033 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-23 r/8411 refactor(directoryservice): get rid of NaiveSeeker helperYureka3-319/+10
Use the ChunkedReader in CombinedBlobService instead which also supports seeking. Change-Id: I681331a80763172c27e55362b7044fe81aaa323b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12031 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-23 r/8407 fix(tvix/directoryservice): clarify get_recursive not foundYureka4-13/+41
Change-Id: I47f21b4b7db9304eadb04094d41bf04d443fcc3b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12025 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8405 feat(tvix/store): add redb PathInfoServiceIlan Joselevich1-0/+36
This provides a PathInfoService implementation using redb (https://github.com/cberner/redb) as the underlying storage engine. Both an in-memory variant, as well as a filesystem one is provided, similar how it's done with the sled implementation. Supersedes: https://cl.tvl.fyi/c/depot/+/11692 Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8397 feat(tvix/composition): improve error message for unknown tagYureka1-2/+2
Change-Id: I61a31488de17725ae4311f5f5bf8e02edb752cf9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12017 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8392 test(tvix/composition): add recursion and concurrent testsYureka1-0/+63
Change-Id: Ic10773b08d940b45c3067bd514c3080cadac9606 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12003 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8391 docs(tvix/composition): add comment about stackYureka1-0/+3
Change-Id: I3888b5034c53728e2f9cfe24213f8854eb38bfe1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12005 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8390 feat(tvix/composition): downcast boxed error if possibleYureka1-4/+7
We still have the unique store name to identify which instantiation caused the error. For recursion errors, the full chain is still retained inside the CompositionError. Change-Id: Iaddcece445a5df331e578d7c69d710db3d5f8dcd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12002 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 r/8388 fix(tvix/composition): include typeid in recursion checkYureka1-5/+13
Change-Id: Icc279d41a4980d4b57acbb4243bbd509039b753f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12000 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka3-3/+9
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-20 r/8380 refactor(tvix/store): use composition in tvix_store crateYureka1-0/+7
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-20 r/8375 feat(tvix/castore/protos): return root node after validationFlorian Klink1-2/+4
This allows avoiding a `.node.unwrap()`` after validation. Change-Id: Ieef1ffebab16cdca94c979ca6831a7ab4f6007da Reviewed-on: https://cl.tvl.fyi/c/depot/+/11989 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-07-19 r/8372 feat(tvix): different service types in one compositionYureka11-46/+64
This will be necessary for the PathInfoService composition, as some PathInfoService implementations require a BlobService & DirectoryService to ingest into. Using the Extend trait for creating compositions allows extending the same composition with configs of various types e.g. BlobStore, DirectoryStore Generics are moved from the Composition struct to the functions.The storage of the InstantiatonStates uses the TypeId in the key and a Box<dyn Any> in the value, which is downcasted to InstantiatonState<T>. Change-Id: I2af11f26c535029adfb1c62905e0e7c4aaed7b51 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11980 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-07-19 r/8371 test(tvix/castore/ObjectStoreBlobService): test correct chunkingYureka1-6/+23
Change-Id: I4a8f5cda2705560a6d52a2eb2be0dde35b784c8f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11982 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-07-19 r/8370 fix(tvix/castore/blob/object_store): empty chunk list for single chunkYureka1-0/+9
Change-Id: Ie3609ac6184ecead11c4e03073433c26ce129848 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11981 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-18 r/8368 refactor(tvix): use composition & registry for from_addrYureka14-202/+295
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-07-18 r/8367 feat(tvix/castore): add sled to composition registryYureka2-1/+51
Change-Id: I03fa8dfabcee14c5f657380f86bb1a7aa00e08ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/11977 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-18 r/8366 feat(tvix/composition): allow creating a blank CompositionContextYureka1-6/+13
this is useful when oneshot-instantiating a store from a single config Change-Id: I08538fdee1d0bb26b3ae2da7d3b2339b2e93bc0a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11975 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-18 r/8365 feat(tvix/castore): add composition moduleYureka13-46/+666
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748 Signed-off-by: Yureka <tvl@yuka.dev> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-29 r/8329 fix(tvix/castore/object_store): make query pairs object_store optsFlorian Klink2-4/+10
We previously called ObjectStoreBlobService::parse_url, which passes an empty list of options when constructing the ObjectStore. This is most likely not what we want. The more reasonable thing to do is pass along the query string (pairs) as options to `object_store::parse_url_opts`, and remove them from the plain URL we pass to object_store itself. Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI