about summary refs log tree commit diff
path: root/tvix/store
AgeCommit message (Collapse)AuthorFilesLines
2023-06-10 r/6261 chore(tvix/store): add fuse featureFlorian Klink1-1/+10
This brings in fuse (via the `fuser` crate), and adds pkg-config and libfuse to the dev shell, so `cargo build` can link against it. Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-09 r/6253 docs(src/tests/nar_renderer): fix commentFlorian Klink1-2/+1
This testcase tests a missing blob fails the rendering, the comment has been copied from elsewhere. Change-Id: I48fa3fa454e12506590fa14a3591d156bafa8b5e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8722 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-09 r/6252 docs(tvix/store): drop nar-bridge referencesFlorian Klink1-1/+1
The CLs did bitrot a bit, they're based on an older version of the protocol, and it's unclear if they'll be a separate Go Binary, or just another HTTP handler inside tvix-store itself, considering we now have way more NAR juggling code than before. Change-Id: I3632035cda8d75a8ff23b3132312f0f086d9e02f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8732 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-09 r/6251 docs(tvix/store): update invocation in READMEFlorian Klink1-1/+1
This has moved to a `daemon` subcommand. Change-Id: Iae9778d8a59e6bf84555119fabfd62db3917bb62 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8731 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-09 r/6245 docs(tvix/store/docs): fix grammarFlorian Klink1-1/+1
Change-Id: I1d7d306e5d997a7ba47a83d613edc19a4fba1ed4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8721 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-30 r/6223 feat(tvix/store/bin): use sled for daemon, grpc for import cmdFlorian Klink1-8/+23
This now creates different store client, depending on the cli subcommand. The `import` command will connect to the gRPC service, and the `daemon` command will use the sled implementation. It might make sense to define some URI syntax to make this configurable by the user, via the CLI. Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 r/6221 feat(tvix/store/bin/import): process all path imports concurrentlyFlorian Klink1-36/+47
Change-Id: I3e1428a4725fc2e552e8f37bc0550121117fcef6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8633 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 r/6220 refactor(tvix/store): drop Clone from BS and DS trait boundFlorian Klink1-6/+2
We don't need to be able to clone these services in here. Change-Id: Ifb69450f7ebdc8364cbf9cdfb6464f8560440e4c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8645 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-25 r/6206 refactor(tvix/store/bin): instantiating TvixStoreIO onceFlorian Klink1-9/+11
Instead of instantiating it once in every loop iteration, put it in an Arc, and clone that before passing it to the spawned task. Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 r/6202 refactor(tvix/eval): stop borrowing &mut selfFlorian Klink1-6/+6
This does undo cl/8571. Change-Id: Ib14b4e7404f906e346304b6113860ae811afc94a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8631 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-25 r/6200 refactor(tvix/store): drop mut self borrow in ingest_pathFlorian Klink2-7/+6
With traverse_to not requiring a &mut anymore, we can drop the &mut self in all these function signatures. Change-Id: I22105376b625cb281c39e92d3206df8a6ce97a88 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8629 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-25 r/6199 refactor(tvix/store/directorysvc): don't borrow self mutableFlorian Klink1-1/+1
Change-Id: I97f183e1ef3b1209a8f05e05e152f70d1f7a9596 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8628 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 r/6198 refactor(tvix/store/bin): use std::ioFlorian Klink1-3/+4
Change-Id: I3f3dc9732d90790d92268c04c75eccbe92e7e05b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8634 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-25 r/6197 fix(tvix/store/bin): use spawn_blocking to call import_pathFlorian Klink1-10/+14
This operation is blocking, so it should be run inside a blocking tokio task. Tokio panics if it detects a blocking operation inside a non- blocking task, so cl/8619 would cause it to panic (as the GRPC clients use spawn_blocking under the hood). As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614), we create a new instance of TvixStoreIO inside each loop iteration. Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6192 feat(tvix/store/pathinfosvc): derive CloneFlorian Klink1-0/+1
This allows closing a TvixStoreIO if it contains a SledPathInfoService. Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-24 r/6191 feat(tvix/store): implement TvixStoreIOFlorian Klink4-39/+383
This providesEvalIO, asking given PathInfoService, DirectoryService and BlobService. Change-Id: I32f210f5a7aa8173ad9a7d53e8a5ac03619f527a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8561 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 r/6188 feat(tvix/store/pathinfosvc): GRPCPathInfoService::from_clientFlorian Klink1-0/+11
Change-Id: I4a4f6f713dd76bf95f393c094c0eb9b6d15c5436 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8613 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6187 feat(tvix/store/directorysvc): GRPCDirectoryService::from_clientFlorian Klink1-0/+11
Change-Id: Ifa274a380683b01f0f24cd9ff1f50d22f13b38bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/8612 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6186 feat(tvix/store/blobsvc): add constructorsFlorian Klink1-1/+23
Change-Id: I55e06bf4e8a11dc2caf92c597558f1b820b42566 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8610 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6185 fix(tvix/store/directorysvc): stop renaming spansFlorian Klink1-2/+2
We have code.namespace as a field, set to tvix_store::directoryservice::sled, so there's no need to repeat the name here. Change-Id: Ic1aa8a2b24de439c6a189966bd773e9acf49d1e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8623 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 r/6184 feat(tvix/store/nar): add GRPCNARCalculationServiceFlorian Klink2-0/+71
This asks a remote tvix-store for the nar size and digest of a given root node. Change-Id: If9f916d9bfc5f8dc3166e2c6c1671c0f0124d1c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8611 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-23 r/6181 refactor(tvix/store/nar): clippyFlorian Klink1-1/+1
Change-Id: I0d864fd1b9248fe4d5c88e3d8fa396ea759fea09 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8616 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6180 refactor(tvix/store/directorysvc): clippyFlorian Klink1-1/+1
Change-Id: I30f7e98fe79ba22d218b4aabaef88c84b6085d82 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8617 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6179 refactor(tvix/store/digests): clippyFlorian Klink1-1/+1
useless conversion to the same type: `std::vec::Vec<u8>` Change-Id: Idcb16679bb2a3350784965d8d9bbb593c760634e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8615 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6178 refactor(tvix/store/blobsvc): move from Vec<u8> to B3DigestFlorian Klink9-95/+79
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-23 r/6177 refactor(tvix/store/directorysvc): move from Vec<u8> to B3DigestFlorian Klink17-165/+199
This introduces a new struct, B3Digest, which internally holds a Vec<u8>, but only allows construction with 32 bytes. It also implements display, which will print the base64 representation. This should reduce some boilerplate when parsing Vec<u8>. Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-18 r/6166 fix(tvix/store): fix timing sensitivity in gRPC directorysvc testFlorian Klink1-24/+45
One test in here assumed the server was fast, but when very busy, a computer running these tests might not be fast. Treat both cases that can occur separately. Change-Id: Iba168ad39f2458c4fb8873211df33beeaff7c6c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8595 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-18 r/6165 refactor(tvix/store): bump CalculateNARResponse.nar_size to u64Florian Klink5-9/+9
Change-Id: I5f41981820363a5eb9982c3d19830916207c62cc Reviewed-on: https://cl.tvl.fyi/c/depot/+/8591 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 r/6164 fix(tvix/store/proto/grpc_blobsvc): clippyFlorian Klink1-1/+1
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-18 r/6163 refactor(tvix/store/blobsvc/sled): clippyFlorian Klink1-5/+5
Change-Id: Icfea8ceb6b827a645c6f10efae741f6fa2114fad Reviewed-on: https://cl.tvl.fyi/c/depot/+/8590 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 r/6162 refactor(tvix/store/directorysvc/traverse): clippy, use NamedNodeFlorian Klink1-17/+6
Also, get rid of the explicit byte comparison here, but unwrap like in the rest of the codebase. We'll deal with this in a generic manner in b/267 and b/189. Change-Id: Ie5f3d27ab35b7e88d67a2796c29cdd7bc7df71f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8589 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 r/6150 feat(tvix/store/directorysvc): add traverse_toFlorian Klink2-0/+238
This walks from a node further down until it reaches the requested path. Change-Id: I2f9a15a8601db4d06c95d7b47cd6153264e203e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8568 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-17 r/6149 refactor(tvix/store): rename import::{import_path -> ingest_path}Florian Klink3-12/+13
This distinguishes it better from the EvalIO::import_path method. Also update the docstring to explain what it does (and what it doesn't). Change-Id: I32a8b2869fa67a894df28532b22bf170961a2abf Reviewed-on: https://cl.tvl.fyi/c/depot/+/8578 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-17 r/6148 fix(tvix/store/src/proto/mod.rs): fix error stringFlorian Klink1-1/+1
This was missed while renaming NixPath to StorePath. Change-Id: Ibcc929c43b111e4370e8222c1dd86d403548367f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8577 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-16 r/6146 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directlyFlorian Klink5-72/+46
We currently only support querying by the output hash digest. This makes the interface a bit simpler. Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-14 r/6138 feat(tvix/store): add GRPCBlobServiceFlorian Klink2-0/+198
This connects to a (remote) tvix-store BlobService over gRPC. Change-Id: If31f706738a5c3445886c117feca8b61f3203e9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8552 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-11 r/6133 refactor(tvix/store): remove ChunkServiceFlorian Klink24-1281/+555
Whether chunking is involved or not, is an implementation detail of each Blobstore. Consumers of a whole blob shouldn't need to worry about that. It currently is not visible in the gRPC interface either. It shouldn't bleed into everything. Let the BlobService trait provide `open_read` and `open_write` methods, which return handles providing io::Read or io::Write, and leave the details up to the implementation. This means, our custom BlobReader module can go away, and all the chunking bits in there, too. In the future, we might still want to add more chunking-aware syncing, but as a syncing strategy some stores can expose, not as a fundamental protocol component. This currently needs "SyncReadIntoAsyncRead", taken and vendored in from https://github.com/tokio-rs/tokio/pull/5669. It provides a AsyncRead for a sync Read, which is necessary to connect our (sync) BlobReader interface to a GRPC server implementation. As an alternative, we could also make the BlobReader itself async, and let consumers of the trait (EvalIO) deal with the async-ness, but this is less of a change for now. In terms of vendoring, I initially tried to move our tokio crate to these commits, but ended up in version incompatibilities, so let's vendor it in for now. Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-10 r/6131 chore(tvix): bump cargo dependenciesFlorian Klink1-3/+3
Change-Id: I6b872a33885f4e29082c554062a60317db754188 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8550 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Johannes Kirschbauer <hsjobeki+github@gmail.com>
2023-05-10 r/6130 fix(tvix/store/nar/renderer): handle digest errorFlorian Klink1-3/+7
Change-Id: I183580732e1dd33ed079a2593097ec790def0a55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8554 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-09 r/6088 refactor(nix-compat): Properly encapsulate store path constructionJohn Ericson1-3/+3
Before there was code scattered about (e.g. text hashing module and derivation output computation) constructing store paths from low level building blocks --- there was some duplication and it was easy to make nonsense store paths. Now, we have roughly the same "safe-ish" ways of constructing them as C++ Nix, and only those are exposed: - Make text hashed content-addressed store paths - Make other content-addressed store paths - Make input-addressed fixed output hashes Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-04-07 r/6077 docs(tvix/store): update comment regarding reachability checkFlorian Klink1-9/+6
This might be more understandable. Change-Id: I91371d304a9a316a2934ce41929998eaef67e11c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8416 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-04-07 r/6076 docs(tvix/store/grpc_directorysvc_wrapper): add TODOFlorian Klink1-0/+3
The logic validating connectivity of Directory nodes should be moved to SimplePutter, and this use whatever DirectoryPutter the store comes with. Change-Id: Id68a86a96cc49ff73920017839788859ea9c5161 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8358 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-04-07 r/6074 refactor(tvix/store/import): use DirectoryPutter in import.rsFlorian Klink1-5/+7
This should allow import_path to communicate to a gRPC remote store, that actually verifies the Directory nodes are interconnected. Change-Id: Ic5d28c33518f50dedec15f1732d81579a3afaff1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8357 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-04-07 r/6073 feat(tvix/store/directorysvc): add put_multiple_startFlorian Klink5-12/+281
This provides a handle to upload multiple proto::Directory as part of the same closure. Change-Id: I9213dde257a260c8622239918ea541064b270484 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8356 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-03-31 r/6065 refactor(tvix/store/directorysvc): move DirectoryTraverser to utilsFlorian Klink2-105/+108
Change-Id: Ie60a660e0fda7c80a6c7de20404c1965fe0e0d63 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8355 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-30 r/6060 refactor(tvix/nix-compat): move build_store_path out of derivationFlorian Klink2-7/+7
This doesn't have anything to do with ATerms, we just happen to be using the aterm representation of a Derivation as contents. Moving this into store_path/utils.rs makes these things much cleaner - Have a build_store_path_from_references function, and a build_store_path_from_fingerprint helper function that makes use of it. build_store_path_from_references is invoked from the derivation module which can be used to calculate the derivation path. In the derivation module, we also invoke build_store_path_from_fingerprint during the output path calculation. Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-27 r/6050 docs(tvix/store/directorysvc/grpc): fix commentFlorian Klink1-1/+1
We do compare for equality. This comment probably was when I tried to compare the `Result<T, E>`, and as `E` doesn't derive PartialEq it was annoying. Change-Id: I18bb19528c76af91c9d24d88d55dd46d0c092d20 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8354 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 r/6046 feat(tvix/store/directorysvc): add DirectoryService::get_recursive()Florian Klink5-101/+320
This moves the recursive BFS traversal of Directory closures from the GRPCDirectoryServiceWrapper out into a a DirectoryTraverser struct implementing Iterator. It is then used from various implementors of DirectoryService in the `get_recursive()` method. This allows distinguishing between recursive requests and non-recursive requests in the gRPC client trait implementation. Change-Id: I50bfd4a0d9eb11832847329b78c587ec7c9dc7b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8351 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-27 r/6044 feat(tvix/store/directorysvc): add gRPC clientFlorian Klink3-1/+184
This provides a GRPCDirectoryService struct implementing DirectoryService, allowing a client to Directory objects from a (remote) tvix-store. Remote in this case is anything outside the current process, be it another process, or an endpoint on the network. To keep the sync interface in the `DirectoryService` trait, a handle to some tokio runtime needs to be passed into the constructor, and the two methods use `self.tokio_handle.spawn` to start an async function, and `self.tokio_handle.block_on` to wait for its completion. The client handle, called `grpc_client` itself is easy to clone, and treats concurrent requests internally. This means, even though we keep the `DirectoryService` trait sync, there's nothing preventing it from being used concurrently, let's say from multiple threads. There's still two limitations for now: 1) The trait doesn't make use of the `recursive` request, which currently leads to a N+1 query problem. This can be fixed by `GRPCDirectoryService` having a reference to another `DirectoryService` acting as the local side. I want to wait for general store composition code to pop up before manually coding this here. 2) It's currently only possible to put() leaf directory nodes, as the request normally requires uploading a whole closure. We might want to add another batch function to upload a whole closure, and/or do this batching in certain cases. This still needs some more thinking. Change-Id: I7ffec791610b72c0960cf5307cefbb12ec946dc9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8336 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 r/6043 feat(tvix/store/pathinfosvc): add gRPC clientFlorian Klink3-0/+80
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-28T10·39+0000