about summary refs log tree commit diff
path: root/tvix/store/src
AgeCommit message (Collapse)AuthorFilesLines
2023-06-19 r/6335 docs(tvix/store): correct some cargo doc warningsFlorian Klink5-5/+5
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-19 r/6333 chore(tvix/store/pathinfosvc): clippyFlorian Klink2-11/+8
Change-Id: Ied4bed08e989791f832922da8776d2104035e28a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8812 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-19 r/6332 chore(tvix/store/directorysvc): clippyFlorian Klink2-10/+7
Change-Id: Idf45aaa0f6211ac35a9a41d0f3f60dfbe1009398 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8811 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-19 r/6331 chore(tvix/store/blobsvc): clippyFlorian Klink2-10/+7
Change-Id: Ie384bdd27e1e9282ceda83edc74ffaad387f352b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8810 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-14 r/6306 fix(tvix/store/blobsvc): drop stray println! debug statementFlorian Klink1-1/+0
Change-Id: Ica073820ea8240d42ce8b979309f881af18a8cde Reviewed-on: https://cl.tvl.fyi/c/depot/+/8781 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-14 r/6305 refactor(tvix/store/blobsvc): make from_addr not asyncFlorian Klink2-4/+4
A previous iteration of this code did actually connect (in the gRPC client), which was why we had this function async. However, as the connection there is now lazy too, we can drop the asyncness in this function. Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 r/6304 refactor(tvix/store/pathinfosvc): add from_addrFlorian Klink6-18/+552
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-14 r/6303 feat(tvix/store/directorysvc): add from_addrFlorian Klink6-21/+287
Add --directory-service-addr arg to tvix-store CLI. Change-Id: Iea1e6f08f27f7157b21ccf397297c68358bd78a0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8743 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 r/6302 feat(tvix/store/blobsvc): add from_addrFlorian Klink6-18/+431
This allows constructing blob stores with a URL syntax at runtime, by passing the --blob-service-addr arg. We probably still want to have some builder pattern here, to allow additional schemes to be registered. Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6282 chore(tvix/store/pathinfosvc): remove GRPCPathInfoService::newFlorian Klink1-12/+0
There's very little reason to instantiate a GRPCPathInfoService in a context where we are not already in a tokio context. Change-Id: Ib81d649387717cb98de8a8039f92472f727b10c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8755 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6281 chore(tvix/store/directorysvc): remove GRPCDirectoryService::newFlorian Klink1-14/+1
The only place where we did use new was also already where we've been in a tokio context, so just using from_client is easier. Change-Id: I39dbc18f6aaa3abc342409be623395647f968530 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8754 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6280 chore(tvix/store/blobsvc): remove GRPCBlobService::newFlorian Klink1-12/+1
There's very little reason to instantiate a GRPCBlobService in a context where we are not already in a tokio context. Change-Id: Ic6e18809a9f2a76f1c098ed330118d8dcfba5137 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8753 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-12 r/6279 refactor(tvix/store/pathinfosvc): use Arc<dyn …>Florian Klink7-26/+30
This removes the use of generics, like previously done with Blob and Directory services. Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6278 refactor(tvix/store/blobsvc): drop Result<_,_> around open_writeFlorian Klink7-19/+15
We never returned Err here anyways, and we can still return an error during the first (or subsequent) write(s). Change-Id: I4b4cd3d35f6ea008e9ffe2f7b71bfc9187309e2f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8750 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6277 docs(tvix/store/blobservice): fix docstringsFlorian Klink1-2/+3
These are outdated. Change-Id: I1a1ae130a55847f57a48d5e244e7e029c1ecae7b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8749 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6276 feat(tvix/store): increase blob chunk sizeRyan Lahfa1-2/+69
From 64 bytes to 100 KBytes. We need to provide a custom wrapper with a different Default instance. Change-Id: Id7c6c437b8183b355a9e388f98cef1622b363f64 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8748 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-06-12 r/6275 refactor(tvix/store/fuse): use Arc<dyn …> instead of genericsFlorian Klink2-11/+17
Change-Id: I5685379bd6f89d17da6843d31bef4c1fc4dc0a18 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8745 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6274 refactor(tvix/store/blobsvc/sled): cargo clippyFlorian Klink1-1/+1
Change-Id: I00b8b567509d5e0847270f36dadb5dcb534b9b73 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8739 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-12 r/6273 refactor(tvix/store): use Arc instead of BoxFlorian Klink12-97/+126
This allows us to blob services without closing them before putting them in a box. We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers require Sync. Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6272 refactor(tvix/store): use Box<dyn DirectoryService>Florian Klink17-123/+135
Once we support configuring services at runtime, we don't know what DirectoryService we're using at compile time. This also means, we can't explicitly use the is_closed method from GRPCPutter, without making it part of the DirectoryPutter itself. Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink11-66/+145
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6270 refactor(tvix/src/nar): drop NARCalculationServiceFlorian Klink10-335/+256
There's only one way to calculate NAR files, by walking through them. Things like caching such replies should be done closer to where we use these, composing NARCalculationService doesn't actually give us much. Instead, expose two functions, `nar::calculate_size_and_sha256` and `nar::writer_nar`, the latter writing NAR to a writer, the former using write_nar to only keeping the NAR size and digest. Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6269 feat(tvix/store): eliminate generics in BlobStoreFlorian Klink14-141/+228
To construct various stores at runtime, we need to eliminate associated types from the BlobService trait, and return Box<dyn …> instead of specific types. This also means we can't consume self in the close() method, so everything we write to is put in an Option<>, and during the first close we take from there. Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-10 r/6263 feat(tvix/store): add mount command to entrypoint and fuse modFlorian Klink3-0/+55
`tvix-store mount PATH` will mount the tvix-store to the given path. Change-Id: Icb82a6b3cb8a22eec856c375a28ae5580403833f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-09 r/6253 docs(src/tests/nar_renderer): fix commentFlorian Klink1-2/+1
This testcase tests a missing blob fails the rendering, the comment has been copied from elsewhere. Change-Id: I48fa3fa454e12506590fa14a3591d156bafa8b5e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8722 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-30 r/6223 feat(tvix/store/bin): use sled for daemon, grpc for import cmdFlorian Klink1-8/+23
This now creates different store client, depending on the cli subcommand. The `import` command will connect to the gRPC service, and the `daemon` command will use the sled implementation. It might make sense to define some URI syntax to make this configurable by the user, via the CLI. Change-Id: I72351fcf0e83a013b6aa67a90b64c108cbb01ffd Reviewed-on: https://cl.tvl.fyi/c/depot/+/8619 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 r/6221 feat(tvix/store/bin/import): process all path imports concurrentlyFlorian Klink1-36/+47
Change-Id: I3e1428a4725fc2e552e8f37bc0550121117fcef6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8633 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-30 r/6220 refactor(tvix/store): drop Clone from BS and DS trait boundFlorian Klink1-6/+2
We don't need to be able to clone these services in here. Change-Id: Ifb69450f7ebdc8364cbf9cdfb6464f8560440e4c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8645 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-25 r/6206 refactor(tvix/store/bin): instantiating TvixStoreIO onceFlorian Klink1-9/+11
Instead of instantiating it once in every loop iteration, put it in an Arc, and clone that before passing it to the spawned task. Change-Id: I5d9c838f27048726166fa50206d1edd5ed6849b5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8632 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 r/6202 refactor(tvix/eval): stop borrowing &mut selfFlorian Klink1-6/+6
This does undo cl/8571. Change-Id: Ib14b4e7404f906e346304b6113860ae811afc94a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8631 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-25 r/6200 refactor(tvix/store): drop mut self borrow in ingest_pathFlorian Klink2-7/+6
With traverse_to not requiring a &mut anymore, we can drop the &mut self in all these function signatures. Change-Id: I22105376b625cb281c39e92d3206df8a6ce97a88 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8629 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-25 r/6199 refactor(tvix/store/directorysvc): don't borrow self mutableFlorian Klink1-1/+1
Change-Id: I97f183e1ef3b1209a8f05e05e152f70d1f7a9596 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8628 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-25 r/6198 refactor(tvix/store/bin): use std::ioFlorian Klink1-3/+4
Change-Id: I3f3dc9732d90790d92268c04c75eccbe92e7e05b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8634 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-25 r/6197 fix(tvix/store/bin): use spawn_blocking to call import_pathFlorian Klink1-10/+14
This operation is blocking, so it should be run inside a blocking tokio task. Tokio panics if it detects a blocking operation inside a non- blocking task, so cl/8619 would cause it to panic (as the GRPC clients use spawn_blocking under the hood). As spawn_blocking moves, and we can't clone `TvixStoreIO` (see cl/8614), we create a new instance of TvixStoreIO inside each loop iteration. Change-Id: I0c6548b3d4ac42d180d4c92314af8fd2b16510da Reviewed-on: https://cl.tvl.fyi/c/depot/+/8618 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6192 feat(tvix/store/pathinfosvc): derive CloneFlorian Klink1-0/+1
This allows closing a TvixStoreIO if it contains a SledPathInfoService. Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-24 r/6191 feat(tvix/store): implement TvixStoreIOFlorian Klink3-39/+380
This providesEvalIO, asking given PathInfoService, DirectoryService and BlobService. Change-Id: I32f210f5a7aa8173ad9a7d53e8a5ac03619f527a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8561 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 r/6188 feat(tvix/store/pathinfosvc): GRPCPathInfoService::from_clientFlorian Klink1-0/+11
Change-Id: I4a4f6f713dd76bf95f393c094c0eb9b6d15c5436 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8613 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6187 feat(tvix/store/directorysvc): GRPCDirectoryService::from_clientFlorian Klink1-0/+11
Change-Id: Ifa274a380683b01f0f24cd9ff1f50d22f13b38bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/8612 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6186 feat(tvix/store/blobsvc): add constructorsFlorian Klink1-1/+23
Change-Id: I55e06bf4e8a11dc2caf92c597558f1b820b42566 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8610 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6185 fix(tvix/store/directorysvc): stop renaming spansFlorian Klink1-2/+2
We have code.namespace as a field, set to tvix_store::directoryservice::sled, so there's no need to repeat the name here. Change-Id: Ic1aa8a2b24de439c6a189966bd773e9acf49d1e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8623 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 r/6184 feat(tvix/store/nar): add GRPCNARCalculationServiceFlorian Klink2-0/+71
This asks a remote tvix-store for the nar size and digest of a given root node. Change-Id: If9f916d9bfc5f8dc3166e2c6c1671c0f0124d1c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8611 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-23 r/6181 refactor(tvix/store/nar): clippyFlorian Klink1-1/+1
Change-Id: I0d864fd1b9248fe4d5c88e3d8fa396ea759fea09 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8616 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6180 refactor(tvix/store/directorysvc): clippyFlorian Klink1-1/+1
Change-Id: I30f7e98fe79ba22d218b4aabaef88c84b6085d82 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8617 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6179 refactor(tvix/store/digests): clippyFlorian Klink1-1/+1
useless conversion to the same type: `std::vec::Vec<u8>` Change-Id: Idcb16679bb2a3350784965d8d9bbb593c760634e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8615 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6178 refactor(tvix/store/blobsvc): move from Vec<u8> to B3DigestFlorian Klink9-95/+79
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-23 r/6177 refactor(tvix/store/directorysvc): move from Vec<u8> to B3DigestFlorian Klink17-165/+199
This introduces a new struct, B3Digest, which internally holds a Vec<u8>, but only allows construction with 32 bytes. It also implements display, which will print the base64 representation. This should reduce some boilerplate when parsing Vec<u8>. Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-18 r/6166 fix(tvix/store): fix timing sensitivity in gRPC directorysvc testFlorian Klink1-24/+45
One test in here assumed the server was fast, but when very busy, a computer running these tests might not be fast. Treat both cases that can occur separately. Change-Id: Iba168ad39f2458c4fb8873211df33beeaff7c6c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8595 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-18 r/6165 refactor(tvix/store): bump CalculateNARResponse.nar_size to u64Florian Klink1-1/+1
Change-Id: I5f41981820363a5eb9982c3d19830916207c62cc Reviewed-on: https://cl.tvl.fyi/c/depot/+/8591 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 r/6164 fix(tvix/store/proto/grpc_blobsvc): clippyFlorian Klink1-1/+1
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-18 r/6163 refactor(tvix/store/blobsvc/sled): clippyFlorian Klink1-5/+5
Change-Id: Icfea8ceb6b827a645c6f10efae741f6fa2114fad Reviewed-on: https://cl.tvl.fyi/c/depot/+/8590 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-21T20·24+0000