about summary refs log tree commit diff
path: root/tvix/castore/src/directoryservice
AgeCommit message (Collapse)AuthorFilesLines
2024-10-10 r/8785 feat(tvix/composition): allow urls as anonymous storesYureka1-1/+11
This allows specifying an url in place of a named reference to another composition entry, if the castore crate has been compiled with the xp-store-composition feature. Example: `--directory-service-addr cache://?near=memory://&far=memory://` This would be equivalent to the instantiation via toml file: ```toml [memory1] type = "memory" [memory2] type = "memory" [default] type = "cache" near = "memory1" far = "memory2" ``` Note that each anonymous url causes a distinct instance to be created. Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-10 r/8784 feat(tvix/dirsvc/Cache): support building from urlYureka1-5/+6
Change-Id: I80121319795319bb977427efeca3666c6b87a1b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12147 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-03 r/8752 chore(tvix/[ca]store): Drop sled support completely in favor of redbIlan Joselevich4-283/+0
Over the past couple of months we've been using redb instead of sled as the default filesystem-based database in PS and DS. I am confident that we can get rid of sled completely now, and just keep redb. Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567 Reviewed-by: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-09-07 r/8662 feat(tvix/castore): expose ValidatedDirectoryGraph typeYureka1-1/+1
Change-Id: Id994258e74f75df8790159ed3180ce360874d0de Reviewed-on: https://cl.tvl.fyi/c/depot/+/12438 Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-18 r/8511 feat(tvix/castore): add Directory::try_from_iter()Florian Klink2-23/+18
This provides a batched variant to construct a Directory, which reuses the previously calculated size. Checking and inserting code is factored out into a check_insert_node function, taking the current size as a parameter and returning the new size. Change-Id: Ia6c2970a0c12181b7c40e63cf7ce8c93298ea37c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12225 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-08-17 r/8508 refactor(tvix/castore): add `name` back to edge weightsFlorian Klink1-11/+22
Make this a proper struct with named fields. We apparently never access B3Digest in there, so it can be removed. Change-Id: Ifc07310393e1afb0a835778eae137a19b54070b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12224 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-17 r/8507 refactor(tvix/castore): add into_nodes(), implement consuming proto convFlorian Klink1-1/+1
Provide a into_nodes() function on a Directory, which consumes self and returns owned PathComponent and Node. Use it to provide a proper conversion from Directory to the proto variant that doesn't clone. There's no need for the one taking only &Directory, we don't use it anywhere, and once someone needs that they might as well clone Directory before converting it. Update all other users of the `.nodes()` function to use `.into_nodes()` where applicable, and avoid some more cloning there. Change-Id: Id4577b9eb173c012e225337458898d3937112bcb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12218 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 r/8506 refactor(tvix/castore): add PathComponent type for checked componentsFlorian Klink3-9/+7
This encodes a verified component on the type level. Internally, it contains a bytes::Bytes. The castore Path/PathBuf component() and file_name() methods now return this type, the old ones returning bytes were renamed to component_bytes() and component_file_name() respectively. We can drop the directory_reject_invalid_name test - it's not possible anymore to pass an invalid name to Directories::add. Invalid names in the Directory proto are still being tested to be rejected in the validate_invalid_names tests. Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8505 refactor(tvix/castore): drop {Directory,File,Symlink}NodeFlorian Klink6-83/+99
Add a `SymlinkTarget` type to represent validated symlink targets. With this, no invalid states are representable, so we can make `Node` be just an enum of all three kind of types, and allow access to these fields directly. Change-Id: I20bdd480c8d5e64a827649f303c97023b7e390f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12216 Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8504 refactor(tvix/castore): remove `name` from NodesFlorian Klink6-47/+50
Nodes only have names if they're contained inside a Directory, or if they're a root node and have something else possibly giving them a name externally. This removes all `name` fields in the three different Nodes, and instead maintains it inside a BTreeMap inside the Directory. It also removes the NamedNode trait (they don't have a get_name()), as well as Node::rename(self, name), and all [Partial]Ord implementations for Node (as they don't have names to use for sorting). The `nodes()`, `directories()`, `files()` iterators inside a `Directory` now return a tuple of Name and Node, as does the RootNodesProvider. The different {Directory,File,Symlink}Node struct constructors got simpler, and the {Directory,File}Node ones became infallible - as there's no more possibility to represent invalid state. The proto structs stayed the same - there's now from_name_and_node and into_name_and_node to convert back and forth between the two `Node` structs. Some further cleanups: The error types for Node validation were renamed. Everything related to names is now in the DirectoryError (not yet happy about the naming) There's some leftover cleanups to do: - There should be a from_(sorted_)iter and into_iter in Directory, so we can construct and deconstruct in one go. That should also enable us to implement conversions from and to the proto representation that moves, rather than clones. - The BuildRequest and PathInfo structs are still proto-based, so we still do a bunch of conversions back and forth there (and have some ugly expect there). There's not much point for error handling here, this will be moved to stricter types in a followup CL. Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink6-481/+17
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka14-294/+650
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-01 r/8437 feat(tvix/castore): add RedbDirectoryServiceIlan Joselevich4-0/+329
This provides a DirectoryService implementation which uses redb (https://github.com/cberner/redb) as the database. It provides both in-memory and persistent on-filesystem implementations. Change-Id: Id8f7c812e2cf401cccd1c382b19907b17a6887bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12038 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-01 r/8435 chore(3p/sources): Bump channels & overlays (2024-07-28)Aspen Smith1-2/+10
* Treewide: re-run depotfmt * //third_party/nixpkgs:html5validator: build with Python 3.11, dependency openstackdocstheme doesn't support 3.12 * //users/sterni/machines/ingeborg: adapt to poorly handled fcgiwrap module API change: https://github.com/NixOS/nixpkgs/pull/318599 * //tvix/*-go: regenerate protobuf files * //third_party/nixpkgs:treefmt: Remove patch for merged pull request * //users/flokli/ipu6-softisp: rebase, drop upstreamed kernel patches Change-Id: Ie4e0df007c287e8cd6207683a9a25838aa5bd39a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11971 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-25 r/8413 test(tvix/castore/dirsvc): check for a pitfall with deduplicated dirsYureka1-5/+34
Change-Id: I3cff2c2e8b2c8a2fd8d8074647d0d99a1db8e693 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12034 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-25 r/8412 fix(tvix/castore/GRPCDirectorySvc): fix a bug in the get_recursive fnYureka1-3/+9
When retrieving a closure with get_recursive, the following could happen in the GRPC client: - The first reference to the deduplicated directory is added to expected_directory_digests - The deduplicated directory is obtained removed from expected_directory_digests - The second reference to the deduplicated directory is added to expected_directory_digests - The deduplicated directory has already been sent, but is still in the expected_directory_digests. It looks to the GRPC client like the closure is incomplete and the stream ended prematurely. Change-Id: Ic62bca12e7f8fb85af5fa4dacd199f0f3b8eea8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12033 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-23 r/8407 fix(tvix/directoryservice): clarify get_recursive not foundYureka4-13/+41
Change-Id: I47f21b4b7db9304eadb04094d41bf04d443fcc3b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12025 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka1-1/+2
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-19 r/8372 feat(tvix): different service types in one compositionYureka6-6/+6
This will be necessary for the PathInfoService composition, as some PathInfoService implementations require a BlobService & DirectoryService to ingest into. Using the Extend trait for creating compositions allows extending the same composition with configs of various types e.g. BlobStore, DirectoryStore Generics are moved from the Composition struct to the functions.The storage of the InstantiatonStates uses the TypeId in the key and a Box<dyn Any> in the value, which is downcasted to InstantiatonState<T>. Change-Id: I2af11f26c535029adfb1c62905e0e7c4aaed7b51 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11980 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-07-18 r/8368 refactor(tvix): use composition & registry for from_addrYureka7-95/+123
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-07-18 r/8367 feat(tvix/castore): add sled to composition registryYureka2-1/+51
Change-Id: I03fa8dfabcee14c5f657380f86bb1a7aa00e08ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/11977 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-18 r/8365 feat(tvix/castore): add composition moduleYureka6-42/+170
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748 Signed-off-by: Yureka <tvl@yuka.dev> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-29 r/8329 fix(tvix/castore/object_store): make query pairs object_store optsFlorian Klink1-2/+5
We previously called ObjectStoreBlobService::parse_url, which passes an empty list of options when constructing the ObjectStore. This is most likely not what we want. The more reasonable thing to do is pass along the query string (pairs) as options to `object_store::parse_url_opts`, and remove them from the plain URL we pass to object_store itself. Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8300 refactor(tvix/*store): remove some trait boundsFlorian Klink1-8/+2
We don't need to require these things for these impl blocks yet. Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser2-10/+24
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8298 feat(tvix/castore): instrument tokio task with current spanSimon Hauser1-8/+11
By default tokio::spawn does not instrument the spawned task with the current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do this manually for all tokio::spawn functions in functions that are instrumented. Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-16 r/8284 feat(castore/directory): add cache combinatorYureka2-0/+144
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 r/8283 refactor(castore/directory/objectstore): use new order validation logicYureka1-14/+10
Use the new helper introduced in CL 11708 instead of rolling our own. Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 r/8282 refactor(castore/directory): separate order logic from ClosureValidatorYureka7-323/+628
ClosureValidator was previously only suitable for a very narrow use case: Validating incoming uploads, which are in leaves-to-root order. This is because the ordering validation was hard-wired into the add() function. This - Re-name ClosureValidator to DirectoryGraph, which is more suitable since it actually stores the Directory structs and is drained in the end. - Move the ordering-related logic to a separate OrderValidator, which can be used independently. - re-write DirectoryGraph to be a general purpose validator which can accept the input in both orders and can be drained in both orders as well. This means the DirectoryGraph and OrderValidator can now serve multiple new purposes: - Validating the incoming closure on the client while downloading. - Validating the incoming closure downloaded in a caching layer from the `far` cache, and re-order it for insertion into the `near` cache. Change-Id: I2b4b226348416912d7a31935bec050e53d911b70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-05-16 r/8153 fix(castore/directory/objectstore): fix responses for deduplicated dirsYureka1-7/+13
Using remove_node messed up the extraction of nodes from the graph. Use into_nodes_edges() instead, to remove the nodes without cloning. Change-Id: Id76c7935d082d6f26192cc3cd490483594f1d1e2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11684 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-05-16 r/8150 feat(tvix/store): add ObjectStoreDirectoryServiceYureka5-9/+323
Change-Id: I1636012be2e8ee3ae64f7bc62fd28bfe0cb2bca5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11668 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-13 r/8142 feat(tvix/castore/directory/get_recursive): add 'static to BoxStreamFlorian Klink5-6/+6
Necessary to directly use this in the GRPC DirectoryService wrapper directly. Change-Id: Ic6a0038a40dc30071d145af5035345fcd93288ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/11634 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 r/8141 feat(tvix/castore/directory/traverse_directory): simplifyFlorian Klink1-48/+43
Use try_stream! rather than stream!, and a bit more map_err and ok_err to make things a bit more concise. Once we have proper error types here, and impl Froms, a lot of the error mapping would disappear entirely. Change-Id: I5240a6b0ff7818b94c151322774242b2c142e33b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11633 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-13 r/8140 feat(tvix/castore/directory/traverse): simplify codeFlorian Klink1-53/+42
Replace the loop manually driving the iterator with a for … in, and some of the match with ok_or_else. Change-Id: I6d7b3ef1bf1c7aa128bd6adef09390b54f79479e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11632 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8114 refactor(tvix/castore/directoryservice/memory): use tokio RwLockFlorian Klink1-3/+4
We don't want to block here, and this also means there's no poisoning to deal with. Change-Id: Ic375571970c48beace0005ae2c012135086a4d67 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8101 fix(tvix/castore/directory/sled): use spawn_blockingFlorian Klink1-39/+60
This does IO, which might take a longer amount of time than what we want to be blocking the normal executor. Use spawn_blocking instead. I didn't add it for the constructors, as we only call these once. Change-Id: I96231fcff8d10abe90cafde25a099a2db6ea9414 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11617 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-03 r/8070 test(tvix/[ca]store): enable bigtable only with "integration" featureFlorian Klink2-4/+4
The emulator and bigtable client are quite big. Remove them from the default //tvix:shell. Put the tests behind a `integration` feature flag, and add a variant with that enabled to CI, and drop the bigtable tools from //tvix:shell. Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-02 r/8066 feat(tvix/castore/directory/traverse): use castore PathsFlorian Klink1-52/+14
This switches from using std::path::Path to using castore paths. We can drop some error handling in descend_to, as absolute (or redundant) paths are not representable. We however now need to convert from a std::path::Path to our representation, and decide to accept .. canonicalization, as paths in EvalIO might contain this. Dealing .. to hop into another store path, if we encounter this, should be dealt with in a previous step. Change-Id: I5e94693808420c5d56587c68731252b54755bf93 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11575 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-25 r/8008 fix(tvix): typoedef1-1/+1
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-20 r/7985 feat(tvix/castore): Fix build warnings in release modeConnor Brewster1-0/+1
Fixes some build warnings that only happen when building in release mode which disables `debug_assertions`. Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de>
2024-04-15 r/7928 fix(tvix/castore/directory): fix graph traversalFlorian Klink1-90/+102
Use a proper graph library to ensure all nodes are reachable from the root. We had a bit of that handrolled during add(), as well as later, which had an annoying bug: Redundant nodes were omitted during insert, but when returning the list during finalize, we did not properly account they need to be introduced before their parents are sent. We now simply populate a petgraph DiGraph during insert (skipping inserting nodes we already saw), and use petgraph's DfsPostOrder to traverse the graph during finalize. If the number of returned indices equals the total number of nodes in the graph, all nodes are reachable from the root, we can consume the graph and return the nodes as a vec, in the same order as the traversal (and insertion). Providing a regression test for the initial bug is challenging, as the current code uses a bunch of HashSets. I manually tested ingesting a full NixOS closure using this mechanism (via gRPC, which exposes this problem, as it validates twice), and it now works. Change-Id: Ic1d5e3e981f2993cc08c5c6b60ad895e578326dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/11418 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-13 r/7906 feat(tvix/*store/bigtable): limit retries connecting to cbtemulatorFlorian Klink1-1/+3
This kept retrying indefinitely if the socket didn't appear. Change-Id: I4d4ef61df73cef6abda698501432f370abc8a82c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11406 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-12 r/7886 feat(tvix/castore/directory/grpc): instrument functionsFlorian Klink1-1/+3
Change-Id: I9cc0a6a32184773597556ab5f9250257aa18ca4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11399 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-09 r/7881 feat(tvix/castore/directory): add bigtable backendFlorian Klink5-1/+410
This adds a Directory service using https://cloud.google.com/bigtable/docs/ as a K/V store. Directory (closures) are put in individual keys. We don't do any bucketed upload of directory closures (yet), as castore/ fs does query individually, does not request recursively (and buffers). This will be addressed by store composition at some point. Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-04-07 r/7865 refactor(tvix/castore/directoryservice/from_addr): migrate to rstestFlorian Klink1-18/+19
Change-Id: I52b14652822766421d66e95bc646ed7baecc705f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11369 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Tested-by: BuildkiteCI
2024-04-07 r/7864 refactor(tvix/castore): migrate closure_validator to rstestFlorian Klink1-13/+14
Change-Id: I6c594d2e670a681484b858c3e04bc25b9e5a2077 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11368 Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-03-28 r/7797 feat(tvix/castore/*service/tests): add objectstore to tests, sortFlorian Klink1-1/+1
Change-Id: If3a45d2f8008b75524ead704b05320364cc60d92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11282 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7796 feat(tvix/castore/directoryservice): log more TODOsFlorian Klink1-0/+3
We need to define behaviours and add tests for these. Change-Id: Id5825fafbf47897d8de42503ea6006eb131b1082 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11281 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-28 r/7795 refactor(tvix/castore/*): drop utils.rs and grpc directorysvc testsFlorian Klink3-164/+10
This drops pretty much all of castore/utils.rs. There were only two things left in there, both a bit messy and only used for tests: Some `gen_*_service()` helper functions. These can be expressed by `from_addr("memory://")`. The other thing was some plumbing code to test the gRPC layer, by exposing a in-memory implementation via gRPC, and then connecting to that channel via a gRPC client again. Previous CLs moved the connection setup code to {directory,blob}service::tests::utils, close to where we exercise them, the new rstest-based tests. The tests interacting directly on the gRPC types are removed, all scenarios that were in there show now be covered through the rstest ones on the trait level. Change-Id: I450ccccf983b4c62145a25d81c36a40846664814 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11223 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7788 feat(tvix/store): add rstest-based PathInfoService testsFlorian Klink1-1/+1
This introduces rstest-based tests. We also add fixtures for creating some BlobService / DirectoryService out of thin air. To test a PathInfoService, we don't really care too much about its internal storage - ensuring they work is up to the castore tests. Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>