about summary refs log tree commit diff
path: root/tvix/castore
AgeCommit message (Collapse)AuthorFilesLines
2024-02-20 r/7570 docs(tvix/castore/directorysvc): K/V is not necessarily flatFlorian Klink2-0/+10
Some implementations of DirectoryService might not allow retrieval of intermediate Directory nodes, that are not at the "root". Think about an object store implementation. The client is doing a get_recursive anyways to reduce the number of roundtrips. By documenting the fact we don't need to support looking up intermediate Directory messages, we can just batch all directories into the same object, keyed by the root. Change-Id: I019d720186d03c4125cec9191e93d20586a20963 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10988 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2024-02-19 r/7558 feat(tvix/castore): Compile fix for DarwinPeter Kolloch1-0/+3
Towards https://b.tvl.fyi/issues/264 Change-Id: If8fa912ae3fb2987b761f649ab738529ebf3b2e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10970 Autosubmit: Peter Kolloch <info@eigenvalue.net> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-18 r/7547 fix(tvix/castore): don't emit ret as INFOFlorian Klink1-1/+2
This otherwise gets a bit spammy. Change-Id: I288350a600d79a394c239f253424ad55bc3cefc5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10954 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-17 r/7535 feat(tvix/castore/fs): make allow_other configurableFlorian Klink2-3/+8
Also add a cli argument to the `tvix-store` binary. Change-Id: Id07d7fedb60d6060543b195f3a810a46137f9ad5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10945 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-10 r/7494 feat(tvix/castore/blobsvc): add Chunked{Blob,Reader}Florian Klink2-0/+489
These provide seekable access into a Blob for which we have more granular chunking information. There's no support for verified streaming in here yet, this simply produces a stream of readers for each chunk, skipping irrelevant chunks and data from the first chunk at the beginning. A seek simply does produce a new reader using the same process. Change-Id: I37f76b752adce027586770475435f3990a6dee0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10731 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-06 r/7479 docs(tvix/castore/blobstore): reorganize docsFlorian Klink3-121/+251
docs/verified-streaming.md explained how CDC and verified streaming can work together, but didn't really highlight enough how chunking in general also helps with seeking. In addition, a lot of the thoughts w.r.t. the BlobStore protocol, both gRPC and Rust traits, as well as why there's no support for seeking directly in gRPC, as well as how clients should behave w.r.t. chunked fetching was missing, or mixed together with the verified streaming bits. While there is no verified streaming version yet, a chunked one is coming soon, and documenting this a bit better is gonna make it easier to understand, as well as provide some lookout on where this is heading. Change-Id: Ib11b8ccf2ef82f9f3a43b36103df0ad64a9b68ce Reviewed-on: https://cl.tvl.fyi/c/depot/+/10733 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-02 r/7471 fix(tvix/castore/grpc/svc_wrapper): expose chunks() over gRPCFlorian Klink1-3/+6
The Stat() method was just always signalling no granular chunks are available. However, as we now have a .chunks() method, we can expose it over gRPC. Change-Id: I74f0890ae083f301bb0cec62f1ea4a95463ac590 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10736 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-02 r/7470 feat(tvix/castore/blobsvc): validate StatBlobResponseFlorian Klink2-0/+29
All chunks must have valid blake3 digests. It is allowed to send an empty list, if no more granular chunking is available. Change-Id: I7ecb53579cdf40fd938bb68a85685751b4d3626f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10726 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-02-02 r/7469 refactor(tvix/castore/grpc/blobsvc): inline stream_mapperFlorian Klink1-12/+3
This can be written without the additional function. Change-Id: Ib11c5d5254d3e44c8fa9661414835b0622eb1ac4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10735 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-02 r/7468 docs(tvix/castore/blobsvc): fix doc comments on traitFlorian Klink1-10/+14
The readers implement AsyncRead/AsyncSeek, not their sync counterparts. Also update expectations around chunks. Change-Id: Ic266688039d80d16d33f651b96ce2bcdedecfa00 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10734 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-02 r/7466 feat(tvix/castore/docs/verified-streaming): clarify replyFlorian Klink1-1/+1
"given chunksize" is misleading here. It's up to the backend to decide if it does chunking at all, and how it chunks. Change-Id: I4f130ca9ac34db79f18ef1d6475295806ac7f9a4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10728 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-02-02 r/7465 refactor(tvix/castore/blobsvc/combinator): compact trait boundsFlorian Klink1-1/+2
BlobService already implies Send and Sync, we don't need to explicitly list it here. Change-Id: I58a4c5912be61a60acd961565979aa01d94ee0f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10727 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-01-22 r/7437 feat(tvix/castore): `process_entry` cannot process unsupported nodesRyan Lahfa1-1/+13
In the past, we had a `todo!` on unsupported node types, this returns a proper error that can be caught by the caller. Change-Id: Icba4c1dab33c0d670a97f162c9b358d1ed5855cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10675 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-21 r/7435 chore(tvix/store): Use BoxStream type aliasConnor Brewster7-24/+17
The BoxStream type alias is a more concise and easier to read than the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type. Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-20 r/7431 refactor(tvix/castore): break down `ingest_path`Ryan Lahfa1-70/+187
In one function that does the heavy lifting: `ingest_entries`, and three additional helpers: - `walk_path_for_ingestion` which perform the tree walking in a very naive way and can be replaced by the user - `leveled_entries_to_stream` which transforms a list of a list of entries ordered by their depth in the tree to a stream of entries in the bottom to top order (Merkle-compatible order I will say in the future). - `ingest_path` which calls the previous functions. Change-Id: I724b972d3c5bffc033f03363255eae448f017cef Reviewed-on: https://cl.tvl.fyi/c/depot/+/10573 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-20 r/7430 feat(tvix/castore): ingestion does DFS and invert itRyan Lahfa1-49/+69
To make use of the filtering feature, we need to revert the internal walker to a real DFS. We will therefore just invert the whole tree by storing all of its contents in a level-keyed vector. This is horribly expensive in memory, this is a compromise between CPU and memory, here is the fundamental reason for why: When you encounter a directory, it's either a leaf or not, i.e. it contains subdirectories or not. To know this fact, you can: - wait until you notice subdirectories under it, i.e. you need to store any intermediate nodes you see in the meantime -> memory penalty. - getdents or readdir on it to determine *NOW* its subdirectories -> CPU penalty and I/O penalty. This is an implementation of the first proposal, we pay memory. In practice, we are paying O(#nb of nodes) in memory. There's a smarter albeit much more complicated algorithm that pays only O(\sum_i #siblings(p_i)) nodes where (p_1, ..., p_n) is the path to a leaf. which means for: A / \ B C / / \ D E F We would never store D, E, F but only E, F at a given time. But we would still store B, C no matter what. Change-Id: I456ed1c3f0db493e018ba1182665d84bebe29c11 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10567 Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de>
2024-01-19 r/7422 chore(3p/sources): Bump channels & overlayssterni1-3/+1
- Adjust to ecl 23.9.9 release - Regenerate go protos after protoc-gen-go update - Drop dhall fork which hasn't kept up with 1.42.* - Address new clippy warnings: - Variant naming of Error::ValidationError - Simplify .try_into().unwrap() - Drop unnecessary identity function - Test module must be last in file - Drop unused `pub use` - Update agenix to 0.15.0. Current master has a installCheckPhase that doesn't work with C++ Nix 2.3.*: https://github.com/ryantm/agenix/commit/a23aa271bec82d3e962bafb994595c1c4a62b133#commitcomment-137185861 Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2024-01-18 r/7412 feat(tvix/castore): convert import error to `std::io::Error`Ryan Lahfa1-0/+6
So that we can just `map_err` easily in functions returning `std::io::Error` but calling functions returning `castore::import::Error`. Change-Id: Id181b95e8431c69e95f3a8cd569ca10306656e1d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10572 Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-15 r/7380 feat(tvix/castore): implement Ord for node::NodeFlorian Klink1-0/+14
This allows assembling BTreeSets of node::Node. Change-Id: I97b83be5ffc3e891307a8ef2b5fc31e38b747a62 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10625 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-09 r/7366 feat(tvix/castore): implement CombinedBlobServiceFlorian Klink2-0/+137
First attempt on composition of BlobServices. Change-Id: I6e70248007edfd322a503fd40c1c4b4300cbc30c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10587 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-09 r/7365 feat(tvix/castore/blobsvc): add chunks methodFlorian Klink2-2/+47
This adds support to retrieve a list of chunks for a given blob to the BlobService interface. While theoretically all chunk-awareness could be kept private inside each BlobService reader, we'd not be able to resolve individual chunks from different Blobservices - and due to this, not able to substitute chunks we already have in a more local store. This function allows asking a BlobService for the list of chunks, leaving any actual fetching up to the caller (be it through individual calls to open_read), or asking another store for it. Change-Id: I1d33c591195ed494be3aec71a8c804743cbe0dca Reviewed-on: https://cl.tvl.fyi/c/depot/+/10586 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-09 r/7364 refactor(tvix/castore): do clone inside a scopeFlorian Klink1-6/+5
Make it clear this is only used inside the scope. Change-Id: Ie94f88d7f0fb58cd4bf9c2f1176000b272e6f2e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10585 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 r/7363 feat(tvix/castore/grpc): instrument some more functionsFlorian Klink1-0/+2
Change-Id: Icedb148c88c5f4a3b2242ed12df1dd8692af94fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10584 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 r/7361 refactor(tvix/castore/directorysvc): AsRef traverse_toFlorian Klink1-14/+12
Change-Id: I641bd4ab3de591a013f03137f1e16295946315f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10579 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-01-09 r/7360 refactor(tvix/castore/blobservice/grpc): remove fn pointer hackFlorian Klink1-15/+8
It looks like the workaround isn't necessary anymore. Change-Id: Ifbcef1d631b3f369cac3db25a2c793480043f697 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10583 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 r/7359 refactor(tvix): use AsRef<dyn …> instead of Deref<Target= …>Florian Klink5-36/+29
Removes some more needs for Arcs. Change-Id: I9a9f4b81641c271de260e9ffa98313a32944d760 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10578 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 r/7358 feat(tvix/castore/blobsvc/grpc): rm VecDec, fix docstringFlorian Klink1-21/+9
The docstrings were not updated once we made the BlobService trait async. There's no more need to turn things into a sync reader. Also, rearrange the stream manipulation a bit, and remove the need to create a new VecDeque for each element in the stream. bytes::Bytes implements the Buf trait. Fixes b/289. Change-Id: Id2bbedca5876b462e630c144b74cc289c3916c4d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10582 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-06 r/7354 chore(tvix/castore): fix the docstring for `process_entry`Ryan Lahfa1-15/+15
It was a `//` not a `///`. Change-Id: Iee3e8c116d73b5dd8a41c027153714415a66695f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10566 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-05 r/7350 chore(tvix): bump test-case dep to 3.3.1Florian Klink1-1/+1
Change-Id: I643548d95a5fab84563c7cbe51ca2ce640c186a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10537 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-01-05 r/7347 fix(tvix/castore): validate Option<Node>Florian Klink1-1/+15
Extend our validation function to also check for the None case. Change-Id: Ib75f880646d7fb3d66588f1988e61ec18be816a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10534 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-05 r/7346 refactor(tvix/castore): relax trait bounds for DSFlorian Klink2-5/+6
Make this an `AsRef<dyn DirectoryService>`. This helps dropping some Clone requirements. Unfortunately, we can't thread this through to TvixStoreIO just yet. Change-Id: I3f07eb28d6c793d3313fe21506ada84d5a8aa3ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/10533 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7307 refactor(tvix/castore/tests): let gen_*_service return BoxesFlorian Klink3-15/+22
Only convert to and reuse an Arc<…> where needed. Change-Id: I2c1bc69cca5a4a3ebd3bdb33d6e28e1f5fb86cb9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10514 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7305 refactor(tvix/castore/blobsvc/grpc/wrapper): don't require Arc<_>Florian Klink3-14/+13
Change-Id: I9655f5588c7dc98427de6af47d74b4ab7ce22071 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10516 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7304 refactor(tvix/castore/directorysvc/grpc/wrapper): no Arc<_>Florian Klink3-54/+48
We can also drop the Clone requirement. Because the trait is async since some time, there's no need to clone before moving into an async closure, allowing us to simplify the code a bit. Change-Id: I9b0a0e10077d8c548d218207b908bfd92c5b8de0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10515 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-01 r/7303 refactor(tvix/castore): make directorysvc more genericFlorian Klink1-4/+7
This works on Box<dyn DirectoryService> too. Change-Id: Ib869f0f4d963ef4dbaeab22db03ff6afb71ede04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10513 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-01 r/7300 feat(tvix/castore/import): generalize ingest_pathFlorian Klink1-8/+16
We don't actually care if it's an Arc<dyn BlobService>, or something else, as long as we can Deref to a BlobService and clone. Change-Id: I0852aaf723f51c5e6b820be8db1199d17309ab08 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10510 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-31 r/7294 refactor(tvix/castore/directorysvc): return Box, not ArcFlorian Klink1-6/+5
While we currently mostly use it in an Arc, as we need to clone it inside PathInfoService, there might be other usecases not requiring it to be Clone. Change-Id: Ia05bb370340792a048e2036be30e285ef1e63870 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10483 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-31 r/7293 refactor(tvix/castore/blobsvc): return Box, not ArcFlorian Klink1-7/+6
While we currently mostly use it in an Arc, as we need to clone it inside PathInfoService, there might be other usecases not requiring it to be Clone. Change-Id: I7bd337cd2e4c2d4154b385461eefa62c9b78345d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10482 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-29 r/7270 feat(tvix/castore/fs): borrow some matchesFlorian Klink1-4/+4
We only do things with the reference, so we don't need to locally borrow it. Change-Id: I6073f7ec7aff717ae3069e28a00b1cb408a50ceb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10455 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-29 r/7267 feat(tvix/castore/fs): instrument FuseDaemon functionsFlorian Klink1-2/+4
Change-Id: I696b7ab6b4c08004db147c0fda7312bbebaa0eec Reviewed-on: https://cl.tvl.fyi/c/depot/+/10451 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-24 r/7261 chore(tvix): switch to upstream futures 0.3.30Florian Klink1-1/+1
The bugs have been fixed, https://github.com/rust-lang/futures-rs/pull/2801 and https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended up in that release. Change-Id: Iefd990d2d1719b884504093343e54e9c5258e2e2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10414 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-12-24 r/7259 refactor(tvix/castore/fs/tests): drop unused argsFlorian Klink1-39/+20
There's no need to pass in an unused directory service into the populate_blob_* method, and considering we have one or two invocation of each of these, we don't really gain much from having all these functions follow the same structure, at least for now. Also, update some function names to better describe what they're doing. Change-Id: I92f680745c157fb0a602b07342f8838bfad23ecd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10411 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-24 r/7258 refactor(tvix/castore): add RootNode impl for BTreeMap, mv fs testsFlorian Klink3-2/+1183
cl/10378 did already move store/fs to castore/fs, but we kept the tests in tvix-store, as they were populating a PathInfoService to make nodes appear in the mount root. Update these tests to now just insert root nodes into a BTreeMap, and ensure we can use that as a RootNodes too. Change-Id: Iad7d1ee4f9423eb6e3a1da33f433842c9ae0de1f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10410 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-12-22 r/7256 chore(tvix): move store/fs to castore/fsFlorian Klink9-0/+1362
With the recent introduction of the RootNodes trait, there's nothing in the fs module pulling in tvix-store dependencies, so it can live in tvix-castore. This allows other crates to make use of TvixStoreFS, without having to pull in tvix-store. For example, a tvix-build using a fuse mountpoint at /nix/store doesn't need a PathInfoService to hold the root nodes that should be present, but just a list. tvix-store now has a pathinfoservice/fs module, which contains the necessary glue logic to implement the RootNodes trait for a PathInfoService. To satisfy Rust orphan rules for trait implementations, we had to add a small wrapper struct. It's mostly hidden away by the make_fs helper function returning a TvixStoreFs. It can't be entirely private, as its still leaking into the concrete type of TvixStoreFS. tvix-store still has `fuse` and `virtiofs` features, but they now simply enable these features in the `tvix-castore` crate they depend on. The tests for the fuse functionality stay in tvix-store for now, as they populate the root nodes through a PathInfoService. Once above mentioned "list of root nodes" implementation exists, we might want to shuffle this around one more time. Fixes b/341. Change-Id: I989f664827a5a361b23b34368d242d10c157c756 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-12-21 r/7252 docs(tvix/castore/protos): remove referenceFlorian Klink1-1/+1
This is not gonna end up as a interlinked docstring. Change-Id: I2b0ca106aa75bae0156c0b411da5931da60c725d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10406 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-12-21 r/7247 docs(tvix/castore): fix referenceFlorian Klink1-3/+4
Change-Id: I00b1d56d58c4d3779b57ab0056cff1c7e6053b9b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10401 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-12-17 r/7228 feat(tvix/castore): simple filesystem blob serviceRyan Lahfa4-1/+209
The simple filesystem `BlobService` enable a user to write blob store on an existing filesystem using a prefix-style layout in the provided root directory, e.g. the two first bytes of the blake3 hashes are used as directories prefixes. Change-Id: I3451a688a6f39027b9c6517d853b95a87adb3a52 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10071 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-12-16 r/7227 refactor(tvix/castore): drop is_closed() from impl DirectoryPutterFlorian Klink3-20/+33
This is only used in the gRPC version (GRPCPutter), during the test automation. So define it as a method there, behind #[cfg(test)], and remove from the trait. Change-Id: Idf170884e3a10be0e96c75d946d9c431171e5e88 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10340 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-13 r/7216 docs(tvix/castore/directorysvc): update commentFlorian Klink1-3/+3
This comment didn't make a lot of sense before. Change-Id: Ie057a133ca4b1a099ed3c885e32316b0d87c5eb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10339 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-12-13 r/7215 docs(tvix/castore): document expectations about DirectoryServiceFlorian Klink1-1/+7
Namely, all trait implementations should reject invalid data being fed, and detect invalid data being returned. b/355 tracks writing some more tests for this, to ensure we're compliant with this. Change-Id: I3b05752932837ce208785efb21ffc21508b4b33a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10338 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-09T09·24+0000