about summary refs log tree commit diff
path: root/tvix/store/src/proto
AgeCommit message (Collapse)AuthorFilesLines
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink11-1398/+51
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 r/6623 refactor(tvix/store): Asyncify PathInfoService and DirectoryServiceConnor Brewster2-13/+15
We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-18 r/6608 chore(tvix/store/protos): drop unused fields for nowFlorian Klink1-4/+0
This wasn't removed yet, and no code is using/populating it so far. It's confusing, let's update it to the state of things now, and re- introduce it once we get there. Change-Id: I68f5ba17a8eee604d8ccd82749da7c8be094cb99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9351 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-18 r/6606 refactor(tvix/store/blobsvc): make BlobStore asyncFlorian Klink4-196/+27
We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-17 r/6598 feat(tvix/store): improve error message formattingFlorian Klink1-1/+2
WrongSorting(Vec<u8>) actually encodes the name of a node, so if we can, we want this to display it as a string (and fall back to the base64 encoding if we can't). Before: > rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: [108, 111, 99, 97, 108, 101] is not sorted After: > rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: locale is not sorted Change-Id: I68420c53a89cb1aa96e4bdce414366cebcb7915f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9350 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-17 r/6597 chore(tvix/store): allow non_snake_caseFlorian Klink1-1/+1
Some of the tonic-generated types create new warnings. Change-Id: Ib538bfd7d90d857d80a129fbbdf3a38c89cc04ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9349 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-05 r/6555 feat(tvix/store/pathinfosvc): provide listingFlorian Klink2-2/+32
This provides an additional method in the PathInfoService trait, as well as an RPC method on the gRPC layer to list all PathInfo objects in a PathInfoService. Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-02 r/6546 refactor(tvix/store): implement rename for node::NodeFlorian Klink1-0/+11
This returns a node with a new name. Change-Id: Iebcab537f8dd63d826b9841d4d0181fcb941afdd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9211 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-08-20 r/6500 refactor(tvix/store): cargo clippyFlorian Klink1-1/+1
Change-Id: I3a80560d036e7ed08036b5e9f0974080d1a30ded Reviewed-on: https://cl.tvl.fyi/c/depot/+/9096 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-07-22 r/6439 feat(tvix/store/proto): use Bytes instead of Vec<u8>Florian Klink9-60/+69
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-07-21 r/6437 feat(tvix/store/digests): use bytes::Bytes instead of Vec<u8>Florian Klink4-17/+23
This will save us some copies, because a clone will simply create an additional pointer to the same data. Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-07-21 r/6436 refactor(tvix/store): use bytes for node names and symlink targetsFlorian Klink7-107/+108
Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-07-21 r/6433 feat(tvix/nix-compat): have StorePath accept bytesFlorian Klink2-4/+6
The primary constructor for this is now from_bytes, from_string is simply calling .as_bytes() on the string, passing it along. The InvalidName error now contains a Vec<u8>, to encode the invalid name (which might not be a string anymore). from_absolute_path now accepts a &[u8] (even though we might want to make this a OSString of some sort). StorePath::validate_name has been degraded to a pub(crate) function. It's still used in src/derivation, even though it probably shouldn't at all - that cleanup is left for cl/8412 though. Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 r/6432 feat(tvix/nix-compat): fold NameError into ErrorFlorian Klink1-3/+3
This being a nested error makes things more complicated than necessary. Also, this caused BuildStorePathError to only hold NameError, so refactor these utility functions to either return Error, or BuildStorePathError. Change-Id: I046fb403780cc5135df8b8833a291fc2a90fd913 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8972 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6279 refactor(tvix/store/pathinfosvc): use Arc<dyn …>Florian Klink2-9/+12
This removes the use of generics, like previously done with Blob and Directory services. Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6278 refactor(tvix/store/blobsvc): drop Result<_,_> around open_writeFlorian Klink1-4/+1
We never returned Err here anyways, and we can still return an error during the first (or subsequent) write(s). Change-Id: I4b4cd3d35f6ea008e9ffe2f7b71bfc9187309e2f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8750 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6276 feat(tvix/store): increase blob chunk sizeRyan Lahfa1-2/+69
From 64 bytes to 100 KBytes. We need to provide a custom wrapper with a different Default instance. Change-Id: Id7c6c437b8183b355a9e388f98cef1622b363f64 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8748 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-06-12 r/6273 refactor(tvix/store): use Arc instead of BoxFlorian Klink2-8/+8
This allows us to blob services without closing them before putting them in a box. We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers require Sync. Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6272 refactor(tvix/store): use Box<dyn DirectoryService>Florian Klink2-11/+8
Once we support configuring services at runtime, we don't know what DirectoryService we're using at compile time. This also means, we can't explicitly use the is_closed method from GRPCPutter, without making it part of the DirectoryPutter itself. Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink2-31/+17
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6270 refactor(tvix/src/nar): drop NARCalculationServiceFlorian Klink2-16/+30
There's only one way to calculate NAR files, by walking through them. Things like caching such replies should be done closer to where we use these, composing NARCalculationService doesn't actually give us much. Instead, expose two functions, `nar::calculate_size_and_sha256` and `nar::writer_nar`, the latter writing NAR to a writer, the former using write_nar to only keeping the NAR size and digest. Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6269 feat(tvix/store): eliminate generics in BlobStoreFlorian Klink2-13/+7
To construct various stores at runtime, we need to eliminate associated types from the BlobService trait, and return Box<dyn …> instead of specific types. This also means we can't consume self in the close() method, so everything we write to is put in an Option<>, and during the first close we take from there. Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6178 refactor(tvix/store/blobsvc): move from Vec<u8> to B3DigestFlorian Klink1-18/+6
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-23 r/6177 refactor(tvix/store/directorysvc): move from Vec<u8> to B3DigestFlorian Klink5-29/+35
This introduces a new struct, B3Digest, which internally holds a Vec<u8>, but only allows construction with 32 bytes. It also implements display, which will print the base64 representation. This should reduce some boilerplate when parsing Vec<u8>. Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-18 r/6164 fix(tvix/store/proto/grpc_blobsvc): clippyFlorian Klink1-1/+1
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-17 r/6148 fix(tvix/store/src/proto/mod.rs): fix error stringFlorian Klink1-1/+1
This was missed while renaming NixPath to StorePath. Change-Id: Ibcc929c43b111e4370e8222c1dd86d403548367f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8577 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-16 r/6146 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directlyFlorian Klink1-7/+12
We currently only support querying by the output hash digest. This makes the interface a bit simpler. Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-11 r/6133 refactor(tvix/store): remove ChunkServiceFlorian Klink5-318/+270
Whether chunking is involved or not, is an implementation detail of each Blobstore. Consumers of a whole blob shouldn't need to worry about that. It currently is not visible in the gRPC interface either. It shouldn't bleed into everything. Let the BlobService trait provide `open_read` and `open_write` methods, which return handles providing io::Read or io::Write, and leave the details up to the implementation. This means, our custom BlobReader module can go away, and all the chunking bits in there, too. In the future, we might still want to add more chunking-aware syncing, but as a syncing strategy some stores can expose, not as a fundamental protocol component. This currently needs "SyncReadIntoAsyncRead", taken and vendored in from https://github.com/tokio-rs/tokio/pull/5669. It provides a AsyncRead for a sync Read, which is necessary to connect our (sync) BlobReader interface to a GRPC server implementation. As an alternative, we could also make the BlobReader itself async, and let consumers of the trait (EvalIO) deal with the async-ness, but this is less of a change for now. In terms of vendoring, I initially tried to move our tokio crate to these commits, but ended up in version incompatibilities, so let's vendor it in for now. Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-09 r/6088 refactor(nix-compat): Properly encapsulate store path constructionJohn Ericson1-3/+3
Before there was code scattered about (e.g. text hashing module and derivation output computation) constructing store paths from low level building blocks --- there was some duplication and it was easy to make nonsense store paths. Now, we have roughly the same "safe-ish" ways of constructing them as C++ Nix, and only those are exposed: - Make text hashed content-addressed store paths - Make other content-addressed store paths - Make input-addressed fixed output hashes Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-04-07 r/6077 docs(tvix/store): update comment regarding reachability checkFlorian Klink1-9/+6
This might be more understandable. Change-Id: I91371d304a9a316a2934ce41929998eaef67e11c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8416 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-04-07 r/6076 docs(tvix/store/grpc_directorysvc_wrapper): add TODOFlorian Klink1-0/+3
The logic validating connectivity of Directory nodes should be moved to SimplePutter, and this use whatever DirectoryPutter the store comes with. Change-Id: Id68a86a96cc49ff73920017839788859ea9c5161 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8358 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-30 r/6060 refactor(tvix/nix-compat): move build_store_path out of derivationFlorian Klink2-7/+7
This doesn't have anything to do with ATerms, we just happen to be using the aterm representation of a Derivation as contents. Moving this into store_path/utils.rs makes these things much cleaner - Have a build_store_path_from_references function, and a build_store_path_from_fingerprint helper function that makes use of it. build_store_path_from_references is invoked from the derivation module which can be used to calculate the derivation path. In the derivation module, we also invoke build_store_path_from_fingerprint during the output path calculation. Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-27 r/6046 feat(tvix/store/directorysvc): add DirectoryService::get_recursive()Florian Klink1-94/+36
This moves the recursive BFS traversal of Directory closures from the GRPCDirectoryServiceWrapper out into a a DirectoryTraverser struct implementing Iterator. It is then used from various implementors of DirectoryService in the `get_recursive()` method. This allows distinguishing between recursive requests and non-recursive requests in the gRPC client trait implementation. Change-Id: I50bfd4a0d9eb11832847329b78c587ec7c9dc7b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8351 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-27 r/6041 docs(tvix/store): fix typo in commentFlorian Klink1-1/+1
We query the blob service for detailled blob info, not the chunk service. Change-Id: I85a6a57b1dae74a950f734be7d4455c5c35ae355 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8348 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-17 r/6019 refactor(tvix/store/nar): pass in &proto::node::NodeFlorian Klink1-1/+1
Passing in a &proto::node::Node into all this allows us consumers to keep ownership of the proto::node::Node. Change-Id: I44882a86c46826b06a8a8a0b24c18adfc7052662 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8316 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-17 r/6018 chore(tvix/store/grpcblobsvc): clippy lintFlorian Klink1-1/+1
Change-Id: Idb78e0417a962599cdfdef5e7346f7fa41e3fa1b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8320 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-16 r/6015 refactor(tvix/store/chunksvc): use [u8; 32] instead of Vec<u8>Florian Klink1-9/+16
Change-Id: Ie2b94aa5d69ff2c61fb77e13ae844f81f6270273 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8314 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-16 r/6014 refactor(tvix/store/directorysvc): use [u8; 32] instead of Vec<u8>Florian Klink4-39/+57
Also, simplify the trait interface, only allowing lookups of Directory objects by their digest. Change-Id: I6eec28a8cb0557bed9b69df8b8ff99a5e0f8fe35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8313 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 r/5959 refactor(tvix/store): use read_all_and_chunk in gRPC blobserviceFlorian Klink1-35/+38
This was the last piece of code using BlobWriter. We can also use `read_all_and_chunk`, it's just requires a bit more plumbing: - The data coming from the client (stream) needs to be mapped (we extract the .data field). - The stream needs to be turned into an (async) reader - The reader needs to be made sync, and that code using the sync reader needs to be in a `task::spawn_blocking`. Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-11 r/5952 refactor(tvix/store): factor out hash update into functionFlorian Klink1-6/+6
We're using this in a bunch of places. Let's move it into a helper function. Change-Id: I118fba35f6d343704520ba37280e4ca52a61da44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8251 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5949 refactor(tvix/store): rename C to DS, client to directory_serviceFlorian Klink1-9/+11
This matches the semantics in other gRPC wrappers. Change-Id: I505c901aa77a58deab216316d08542877b0b2c14 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8247 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5946 feat(tvix/store): clippyFlorian Klink1-2/+2
warning: the following explicit lifetimes could be elided: 'set --> store/src/proto/mod.rs:201:1 | 201 | / fn update_if_lt_prev<'set, 'n>( 202 | | prev_name: &'set mut &'n str, 203 | | name: &'n str, 204 | | ) -> Result<(), ValidateDirectoryError> { | |_______________________________________^ | = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#needless_lifetimes = note: `#[warn(clippy::needless_lifetimes)]` on by default Change-Id: I053d370a34e5c6721ef86502c7f225c3fbfb8746 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8244 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-10 r/5934 feat(tvix/store): add new_temporary for all Sled servicesFlorian Klink3-32/+21
This provides a service using /dev/shm, that's deleted once the reference is dropped. Refactor all tests to use these, which allows getting rid of most TempDir usage in the tests. The only place where we still use TempDir is in the importer tests, which work on a filesystem path. Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5933 fix(tvix/store/proto/grpc_blobservice_wrapper): buffer recv dataFlorian Klink1-7/+14
While we don't want to keep all of the data in memory, we want to feed a reasonably-enough buffer to the chunking function, to prevent unnecessarily trying to chunk over and over again. Change-Id: I5bbe2d55e8c1c63f8f7ce343889d374b528b559e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8160 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5929 refactor(tvix/store/tests): move fixtures into separate moduleFlorian Klink3-47/+3
Change-Id: I362dbf0899e4dc42114fd2e6a8fa7f537e9ea138 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8156 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5928 refactor(tvix/store/tests): move gen_*_service() into helperFlorian Klink3-43/+24
This allows hiding to tests what exact implementation we're using, when testing things that do something with a store, but don't care what's used for underlying storage. Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5927 refactor(tvix/store): move blob splitting into a BlobWriter structFlorian Klink1-76/+11
This will moves the chunking-as-we-receive logic that so far only lived in grpc_blobservice_wrapper.rs into a generic BlobWriter. Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5926 feat(tvix/store): do not buffer blob dataFlorian Klink2-73/+139
Use the FastCDC::cut function to ask fastcd for cutting points as we receive the data. Make sure to keep the last chunk in the temporary buffer, as we might not actually cut at the end. Also, use rayon to calculate the blake3 hash if the input data is > 128KiB. Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5925 feat(tvix/store): bump fastcdc, use v2020 versionFlorian Klink1-1/+1
This switches away from the less canonical "ronomon" version to the implementation as described in the [paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et al., in 2020. That version uses 64-bit hash values and tends to be faster than both the ronomon and v2016 versions, and produces the same chunking as the 2016 version. As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's the recommended choice. The crate also gained support for streaming version of chunkers: https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be useful. Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5921 feat(tvix/store): add tests for GRPCPathInfoServiceWrapperFlorian Klink2-0/+108
Change-Id: I22de33c28c132fc9710ddecd0e9158ee3b3ef94b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8108 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-22T19·08+0000