about summary refs log tree commit diff
path: root/tvix/store
AgeCommit message (Collapse)AuthorFilesLines
2023-05-24 r/6187 feat(tvix/store/directorysvc): GRPCDirectoryService::from_clientFlorian Klink1-0/+11
Change-Id: Ifa274a380683b01f0f24cd9ff1f50d22f13b38bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/8612 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6186 feat(tvix/store/blobsvc): add constructorsFlorian Klink1-1/+23
Change-Id: I55e06bf4e8a11dc2caf92c597558f1b820b42566 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8610 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6185 fix(tvix/store/directorysvc): stop renaming spansFlorian Klink1-2/+2
We have code.namespace as a field, set to tvix_store::directoryservice::sled, so there's no need to repeat the name here. Change-Id: Ic1aa8a2b24de439c6a189966bd773e9acf49d1e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8623 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 r/6184 feat(tvix/store/nar): add GRPCNARCalculationServiceFlorian Klink2-0/+71
This asks a remote tvix-store for the nar size and digest of a given root node. Change-Id: If9f916d9bfc5f8dc3166e2c6c1671c0f0124d1c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8611 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-23 r/6181 refactor(tvix/store/nar): clippyFlorian Klink1-1/+1
Change-Id: I0d864fd1b9248fe4d5c88e3d8fa396ea759fea09 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8616 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6180 refactor(tvix/store/directorysvc): clippyFlorian Klink1-1/+1
Change-Id: I30f7e98fe79ba22d218b4aabaef88c84b6085d82 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8617 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6179 refactor(tvix/store/digests): clippyFlorian Klink1-1/+1
useless conversion to the same type: `std::vec::Vec<u8>` Change-Id: Idcb16679bb2a3350784965d8d9bbb593c760634e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8615 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6178 refactor(tvix/store/blobsvc): move from Vec<u8> to B3DigestFlorian Klink9-95/+79
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-23 r/6177 refactor(tvix/store/directorysvc): move from Vec<u8> to B3DigestFlorian Klink17-165/+199
This introduces a new struct, B3Digest, which internally holds a Vec<u8>, but only allows construction with 32 bytes. It also implements display, which will print the base64 representation. This should reduce some boilerplate when parsing Vec<u8>. Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-18 r/6166 fix(tvix/store): fix timing sensitivity in gRPC directorysvc testFlorian Klink1-24/+45
One test in here assumed the server was fast, but when very busy, a computer running these tests might not be fast. Treat both cases that can occur separately. Change-Id: Iba168ad39f2458c4fb8873211df33beeaff7c6c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8595 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-18 r/6165 refactor(tvix/store): bump CalculateNARResponse.nar_size to u64Florian Klink5-9/+9
Change-Id: I5f41981820363a5eb9982c3d19830916207c62cc Reviewed-on: https://cl.tvl.fyi/c/depot/+/8591 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 r/6164 fix(tvix/store/proto/grpc_blobsvc): clippyFlorian Klink1-1/+1
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-18 r/6163 refactor(tvix/store/blobsvc/sled): clippyFlorian Klink1-5/+5
Change-Id: Icfea8ceb6b827a645c6f10efae741f6fa2114fad Reviewed-on: https://cl.tvl.fyi/c/depot/+/8590 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 r/6162 refactor(tvix/store/directorysvc/traverse): clippy, use NamedNodeFlorian Klink1-17/+6
Also, get rid of the explicit byte comparison here, but unwrap like in the rest of the codebase. We'll deal with this in a generic manner in b/267 and b/189. Change-Id: Ie5f3d27ab35b7e88d67a2796c29cdd7bc7df71f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8589 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 r/6150 feat(tvix/store/directorysvc): add traverse_toFlorian Klink2-0/+238
This walks from a node further down until it reaches the requested path. Change-Id: I2f9a15a8601db4d06c95d7b47cd6153264e203e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8568 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-17 r/6149 refactor(tvix/store): rename import::{import_path -> ingest_path}Florian Klink3-12/+13
This distinguishes it better from the EvalIO::import_path method. Also update the docstring to explain what it does (and what it doesn't). Change-Id: I32a8b2869fa67a894df28532b22bf170961a2abf Reviewed-on: https://cl.tvl.fyi/c/depot/+/8578 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-17 r/6148 fix(tvix/store/src/proto/mod.rs): fix error stringFlorian Klink1-1/+1
This was missed while renaming NixPath to StorePath. Change-Id: Ibcc929c43b111e4370e8222c1dd86d403548367f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8577 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-16 r/6146 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directlyFlorian Klink5-72/+46
We currently only support querying by the output hash digest. This makes the interface a bit simpler. Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-14 r/6138 feat(tvix/store): add GRPCBlobServiceFlorian Klink2-0/+198
This connects to a (remote) tvix-store BlobService over gRPC. Change-Id: If31f706738a5c3445886c117feca8b61f3203e9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8552 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-11 r/6133 refactor(tvix/store): remove ChunkServiceFlorian Klink24-1281/+555
Whether chunking is involved or not, is an implementation detail of each Blobstore. Consumers of a whole blob shouldn't need to worry about that. It currently is not visible in the gRPC interface either. It shouldn't bleed into everything. Let the BlobService trait provide `open_read` and `open_write` methods, which return handles providing io::Read or io::Write, and leave the details up to the implementation. This means, our custom BlobReader module can go away, and all the chunking bits in there, too. In the future, we might still want to add more chunking-aware syncing, but as a syncing strategy some stores can expose, not as a fundamental protocol component. This currently needs "SyncReadIntoAsyncRead", taken and vendored in from https://github.com/tokio-rs/tokio/pull/5669. It provides a AsyncRead for a sync Read, which is necessary to connect our (sync) BlobReader interface to a GRPC server implementation. As an alternative, we could also make the BlobReader itself async, and let consumers of the trait (EvalIO) deal with the async-ness, but this is less of a change for now. In terms of vendoring, I initially tried to move our tokio crate to these commits, but ended up in version incompatibilities, so let's vendor it in for now. Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-10 r/6131 chore(tvix): bump cargo dependenciesFlorian Klink1-3/+3
Change-Id: I6b872a33885f4e29082c554062a60317db754188 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8550 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Johannes Kirschbauer <hsjobeki+github@gmail.com>
2023-05-10 r/6130 fix(tvix/store/nar/renderer): handle digest errorFlorian Klink1-3/+7
Change-Id: I183580732e1dd33ed079a2593097ec790def0a55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8554 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-09 r/6088 refactor(nix-compat): Properly encapsulate store path constructionJohn Ericson1-3/+3
Before there was code scattered about (e.g. text hashing module and derivation output computation) constructing store paths from low level building blocks --- there was some duplication and it was easy to make nonsense store paths. Now, we have roughly the same "safe-ish" ways of constructing them as C++ Nix, and only those are exposed: - Make text hashed content-addressed store paths - Make other content-addressed store paths - Make input-addressed fixed output hashes Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-04-07 r/6077 docs(tvix/store): update comment regarding reachability checkFlorian Klink1-9/+6
This might be more understandable. Change-Id: I91371d304a9a316a2934ce41929998eaef67e11c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8416 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-04-07 r/6076 docs(tvix/store/grpc_directorysvc_wrapper): add TODOFlorian Klink1-0/+3
The logic validating connectivity of Directory nodes should be moved to SimplePutter, and this use whatever DirectoryPutter the store comes with. Change-Id: Id68a86a96cc49ff73920017839788859ea9c5161 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8358 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-04-07 r/6074 refactor(tvix/store/import): use DirectoryPutter in import.rsFlorian Klink1-5/+7
This should allow import_path to communicate to a gRPC remote store, that actually verifies the Directory nodes are interconnected. Change-Id: Ic5d28c33518f50dedec15f1732d81579a3afaff1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8357 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-04-07 r/6073 feat(tvix/store/directorysvc): add put_multiple_startFlorian Klink5-12/+281
This provides a handle to upload multiple proto::Directory as part of the same closure. Change-Id: I9213dde257a260c8622239918ea541064b270484 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8356 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-31 r/6065 refactor(tvix/store/directorysvc): move DirectoryTraverser to utilsFlorian Klink2-105/+108
Change-Id: Ie60a660e0fda7c80a6c7de20404c1965fe0e0d63 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8355 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-30 r/6060 refactor(tvix/nix-compat): move build_store_path out of derivationFlorian Klink2-7/+7
This doesn't have anything to do with ATerms, we just happen to be using the aterm representation of a Derivation as contents. Moving this into store_path/utils.rs makes these things much cleaner - Have a build_store_path_from_references function, and a build_store_path_from_fingerprint helper function that makes use of it. build_store_path_from_references is invoked from the derivation module which can be used to calculate the derivation path. In the derivation module, we also invoke build_store_path_from_fingerprint during the output path calculation. Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-27 r/6050 docs(tvix/store/directorysvc/grpc): fix commentFlorian Klink1-1/+1
We do compare for equality. This comment probably was when I tried to compare the `Result<T, E>`, and as `E` doesn't derive PartialEq it was annoying. Change-Id: I18bb19528c76af91c9d24d88d55dd46d0c092d20 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8354 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 r/6046 feat(tvix/store/directorysvc): add DirectoryService::get_recursive()Florian Klink5-101/+320
This moves the recursive BFS traversal of Directory closures from the GRPCDirectoryServiceWrapper out into a a DirectoryTraverser struct implementing Iterator. It is then used from various implementors of DirectoryService in the `get_recursive()` method. This allows distinguishing between recursive requests and non-recursive requests in the gRPC client trait implementation. Change-Id: I50bfd4a0d9eb11832847329b78c587ec7c9dc7b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8351 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-27 r/6044 feat(tvix/store/directorysvc): add gRPC clientFlorian Klink3-1/+184
This provides a GRPCDirectoryService struct implementing DirectoryService, allowing a client to Directory objects from a (remote) tvix-store. Remote in this case is anything outside the current process, be it another process, or an endpoint on the network. To keep the sync interface in the `DirectoryService` trait, a handle to some tokio runtime needs to be passed into the constructor, and the two methods use `self.tokio_handle.spawn` to start an async function, and `self.tokio_handle.block_on` to wait for its completion. The client handle, called `grpc_client` itself is easy to clone, and treats concurrent requests internally. This means, even though we keep the `DirectoryService` trait sync, there's nothing preventing it from being used concurrently, let's say from multiple threads. There's still two limitations for now: 1) The trait doesn't make use of the `recursive` request, which currently leads to a N+1 query problem. This can be fixed by `GRPCDirectoryService` having a reference to another `DirectoryService` acting as the local side. I want to wait for general store composition code to pop up before manually coding this here. 2) It's currently only possible to put() leaf directory nodes, as the request normally requires uploading a whole closure. We might want to add another batch function to upload a whole closure, and/or do this batching in certain cases. This still needs some more thinking. Change-Id: I7ffec791610b72c0960cf5307cefbb12ec946dc9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8336 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 r/6043 feat(tvix/store/pathinfosvc): add gRPC clientFlorian Klink3-0/+80
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-27 r/6042 docs(tvix/store/blobreader): fix docstringFlorian Klink1-1/+1
Change-Id: Ia8ce1dd1663663903b9f60feada12e6a16fb4af2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8353 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-27 r/6041 docs(tvix/store): fix typo in commentFlorian Klink1-1/+1
We query the blob service for detailled blob info, not the chunk service. Change-Id: I85a6a57b1dae74a950f734be7d4455c5c35ae355 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8348 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-21 r/6029 chore(3p/sources): Bump channels & overlayssterni8-20/+46
* //users/wpcarro/emacs: use top level (ELPA) version of eglot, as it was removed from MELPA: https://github.com/melpa/melpa/commit/dc2ead17a8e5d5df59fc3729c8f0435cfcbf55ef * //3p/nixpkgs:tdlib: 1.8.11 -> 1.8.12 * //tvix/store: regenerate protos after buf update Change-Id: I782a8d91fda5ed461788055a3721104e8c032207 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8327 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-19 r/6028 feat(tvix/store/bin): print store pathFlorian Klink1-0/+38
After ingestion of the contents into the store, this will use the NonCachingNARCalculationService to create a NAR stream or the contents of the path, and use our Derivation output path calculation machinery to determine the output path (using recursive hashing strategy). In a real-world scenario, we obviously want to cache these calculations, but this should be sufficient to tinker around with it. Change-Id: I9b2e69384414f0be1bdcb5a99a4bfd46e8db9932 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8317 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-17 r/6019 refactor(tvix/store/nar): pass in &proto::node::NodeFlorian Klink5-21/+29
Passing in a &proto::node::Node into all this allows us consumers to keep ownership of the proto::node::Node. Change-Id: I44882a86c46826b06a8a8a0b24c18adfc7052662 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8316 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-17 r/6018 chore(tvix/store/grpcblobsvc): clippy lintFlorian Klink1-1/+1
Change-Id: Idb78e0417a962599cdfdef5e7346f7fa41e3fa1b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8320 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-17 r/6017 chore(tvix/store/chunkservice): clippy lintFlorian Klink3-5/+5
Change-Id: I2073408fe6e87ffadbeb35ef2f9aaa0316c54966 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8319 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-17 r/6016 chore(tvix/store/blobreader): clippy lintFlorian Klink1-1/+4
Change-Id: I4c85425782878295277abe6dc91a1b570768f7fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/8318 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-16 r/6015 refactor(tvix/store/chunksvc): use [u8; 32] instead of Vec<u8>Florian Klink6-54/+70
Change-Id: Ie2b94aa5d69ff2c61fb77e13ae844f81f6270273 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8314 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-16 r/6014 refactor(tvix/store/directorysvc): use [u8; 32] instead of Vec<u8>Florian Klink12-130/+128
Also, simplify the trait interface, only allowing lookups of Directory objects by their digest. Change-Id: I6eec28a8cb0557bed9b69df8b8ff99a5e0f8fe35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8313 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 r/6007 chore(tvix/store): drop separate bin packageFlorian Klink2-5/+2
Apparently, having multiple packages with the same path is a bad thing: ``` The bin target `tvix-store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)` has the same output filename as the lib target `tvix_store` in package `tvix-store-bin v0.1.0 (/home/flokli/tvl/tvix/store)`. Colliding filename is: /home/flokli/tvl/tvix/target/doc/tvix_store/index.html The output filenames should be unique. This is a known bug where multiple crates with the same name use the same path; see <https://github.com/rust-lang/cargo/issues/6313>. ``` Change-Id: Ic785c0349070783baf5e8fd23f5fb60603a3c995 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8308 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 r/5960 feat(tvix/store): drop BlobWriterFlorian Klink2-142/+0
All code initially using this has been replaced by the simpler and more performant implementation with StreamCDC and read_all_and_chunk. Change-Id: I08889e9a6984de91c5debcf2b612cb68ae5072d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8265 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-13 r/5959 refactor(tvix/store): use read_all_and_chunk in gRPC blobserviceFlorian Klink2-35/+39
This was the last piece of code using BlobWriter. We can also use `read_all_and_chunk`, it's just requires a bit more plumbing: - The data coming from the client (stream) needs to be mapped (we extract the .data field). - The stream needs to be turned into an (async) reader - The reader needs to be made sync, and that code using the sync reader needs to be in a `task::spawn_blocking`. Change-Id: I4e374e1a9f47d5a0933f59a8f5c121185a5f3e95 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8260 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-13 r/5958 refactor(tvix/store): add read_all_and_chunk methodFlorian Klink3-43/+46
This moves the logic from src/import.rs that - reads over the contents of a file - chunks them up and uploads individual chunks - keeps track of the uploaded chunks in a BlobMeta structure - returns the hash of the blob and the BlobMeta structure … into a generic read_all_and_chunk function in src/chunkservice/util.rs. It will work on anything implementing io::Read, not just files, which will help us in a bit. Change-Id: I53bf628114b73ee2e515bdae29974571ea2b6f6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8259 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-03-13 r/5957 refactor(tvix/store): use update_hasher in blobwriterFlorian Klink1-7/+3
Make use of the helper function here as well. Change-Id: Ia0afd84eb3903bb897ee6aee884dc291f3e4371c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8258 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-12 r/5956 chore(tvix/store): allow(unused_mut)Florian Klink1-0/+1
When building tvix-store without default features, this variable doesn't need to be mutable. Silence the warning. Change-Id: Iec61be0064c0cef276a29ef22e5c4af3b052efe8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8267 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-11 r/5953 refactor(tvix/store): bump fastcdc depFlorian Klink2-2/+2
This removes the use of Box::new, switching fastcdc to version 3.0.2 with https://github.com/nlfiedler/fastcdc-rs/issues/25 fixed. Change-Id: I64f388b9e0a7f358e25a8bb7ca0e4df1d3bb01c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8249 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-02T17·44+0000