about summary refs log tree commit diff
path: root/tvix/store/src/pathinfoservice
AgeCommit message (Collapse)AuthorFilesLines
2023-11-19 r/7038 feat(tvix/store/pathinfoservice): implement NixHTTPPathInfoServiceFlorian Klink3-7/+240
NixHTTPPathInfoService acts as a bridge in between the Nix HTTP Binary cache protocol provided by Nix binary caches such as cache.nixos.org, and the Tvix Store Model. It implements the [PathInfoService] trait in an interesting way: Every [PathInfoService::get] fetches the .narinfo and referred NAR file, inserting components into a [BlobService] and [DirectoryService], then returning a [PathInfo] struct with the root. Due to this being quite a costly operation, clients are expected to layer this service with store composition, so they're only ingested once. The client is expected to be (indirectly) using the same [BlobService] and [DirectoryService], so able to fetch referred Directories and Blobs. [PathInfoService::put] and [PathInfoService::nar] are not implemented and return an error if called. This behaves very similar to the nar-bridge-pathinfo code in nar-bridge, except it's now in Rust. Change-Id: Ia03d4fed9d0657965d100299af97cd917a03f2f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10069 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-15 r/7022 refactor(tvix/castore/tonic): make async, support wait-connect=?Florian Klink2-13/+10
This moves the sync `channel::from_url` to a async `tonic::channel_from_url`. It now allows connecting non-lazily if `wait- connect=1` is set in the URL params. Also, make the pingpong tests for blobsvc and directorysvc use the wait- connect=1 codepath. Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7019 refactor(tvix/store/pathinfosvc/from_addr): use test_caseFlorian Klink1-93/+46
All we do is constructing some strings, and checking if from_addr succeeds or not. This can be written in a much more concise way using test_case. Use lazy_static to provide temporary directories. Also add some more grpc-related test cases. Change-Id: Ia310dd01f617f7628f1e7e21304ac70da2ab3534 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10027 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-15 r/7018 refactor(tvix/store/pathinfosvc): inline SledPathInfoSvc::from_urlFlorian Klink2-136/+81
Change-Id: I0d905228df086a422bb30322add7236ca41e807b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10026 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7017 refactor(tvix/store/pathinfosvc): inline GRPCPathInfoSvc::from_urlFlorian Klink2-28/+27
Change-Id: Ib53b5525ae13c276e61b7f564673b7c6144ffc0e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10025 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7016 feat(tvix/castore/src/channel): move from_url testsFlorian Klink1-73/+1
These gRPC PathInfoService tests were actually not too useful in here, what we're mostly testing is the channel construction, so move it to there. Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-15 r/7015 refactor(tvix/store/pathinfosvc): inline MemoryPathInfoSvc::from_urlFlorian Klink2-88/+47
Change-Id: If27eb518d372f4004b7b38fc765a42957f2a6b50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10023 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7014 refactor(tvix/store): remove from_url from PathInfoService traitFlorian Klink4-31/+13
We don't gain much from making this part of the trait, it's still up to `tvix_store::pathinfoservice::from_addr` to do most of the construction. Move it out of the trait and into the specific *Service impls directly. This allows further refactorings in followup CLs. Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-27 r/6888 refactor(nix-compat/store_path): make digest and name privateedef2-2/+5
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-14 r/6802 refactor(tvix/*store): helper for channel creation from urlFlorian Klink1-41/+4
This moves the repetitive code to parse a URL and create a channel connected to it into `tvix_castore::channel::from_url`. Part of b/308 Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6801 refactor(tvix/*store): reorganize from_urlFlorian Klink1-14/+9
Move the channel creation depending on the string-based URL into its own block. Change-Id: I546b769acd2296b548eb966b62c495f910266df5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6800 refactor(tvix/*store): remove some grpc_client let bindingsFlorian Klink1-20/+17
We had to have these all while the traits where sync, and there was a lot of spawning and moving. Most of this can now be removed in favor of some inline `.clone()`. Change-Id: Id5466c32a403100bc3347866b3172e06a792e311 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 r/6743 test(tvix/store): use tokio-retry for exp backoffFlorian Klink1-17/+15
Rather than using this loop, use exponential backoff while waiting for the socket path to be created. Change-Id: I3056b1525784cd712b1d81f84876c9ca0be10dc6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9569 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-08 r/6741 refactor(tvix/store): simplify test_valid_unix_path_ping_pongFlorian Klink1-43/+31
We don't need to spawn two tokio runtimes anymore, and can do the URL parsing at once, too. Change-Id: I7885a894bb1250cd087d4e1893e3e73b631331da Reviewed-on: https://cl.tvl.fyi/c/depot/+/9567 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink5-63/+70
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6625 docs(tvix/store/pathinfosvc): fix copypasta errorFlorian Klink1-1/+0
This is not a simple get and put returning digest. You currently can get by output hash, and a put gives you back the (possibly modified) PathInfo message. Change-Id: I43174d87a764417b8620c4d5bdac058cc25b0373 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9371 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-21 r/6623 refactor(tvix/store): Asyncify PathInfoService and DirectoryServiceConnor Brewster4-126/+102
We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-05 r/6558 refactor(tvix/store): use tokio::task::JoinHandleFlorian Klink1-21/+19
This makes the inside code a bit less verbose. I wasn't able to describe the type of the async move closure itself, which would allow us to remove the JoinHandle<_> type annotation entirely. Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-09-05 r/6555 feat(tvix/store/pathinfosvc): provide listingFlorian Klink4-2/+103
This provides an additional method in the PathInfoService trait, as well as an RPC method on the gRPC layer to list all PathInfo objects in a PathInfoService. Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-07-22 r/6439 feat(tvix/store/proto): use Bytes instead of Vec<u8>Florian Klink1-1/+2
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-19 r/6335 docs(tvix/store): correct some cargo doc warningsFlorian Klink2-2/+2
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-19 r/6333 chore(tvix/store/pathinfosvc): clippyFlorian Klink2-11/+8
Change-Id: Ied4bed08e989791f832922da8776d2104035e28a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8812 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-14 r/6304 refactor(tvix/store/pathinfosvc): add from_addrFlorian Klink5-1/+524
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-12 r/6282 chore(tvix/store/pathinfosvc): remove GRPCPathInfoService::newFlorian Klink1-12/+0
There's very little reason to instantiate a GRPCPathInfoService in a context where we are not already in a tokio context. Change-Id: Ib81d649387717cb98de8a8039f92472f727b10c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8755 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6279 refactor(tvix/store/pathinfosvc): use Arc<dyn …>Florian Klink1-1/+1
This removes the use of generics, like previously done with Blob and Directory services. Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6273 refactor(tvix/store): use Arc instead of BoxFlorian Klink2-15/+23
This allows us to blob services without closing them before putting them in a box. We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers require Sync. Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6272 refactor(tvix/store): use Box<dyn DirectoryService>Florian Klink2-23/+18
Once we support configuring services at runtime, we don't know what DirectoryService we're using at compile time. This also means, we can't explicitly use the is_closed method from GRPCPutter, without making it part of the DirectoryPutter itself. Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink4-14/+102
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6270 refactor(tvix/src/nar): drop NARCalculationServiceFlorian Klink1-0/+2
There's only one way to calculate NAR files, by walking through them. Things like caching such replies should be done closer to where we use these, composing NARCalculationService doesn't actually give us much. Instead, expose two functions, `nar::calculate_size_and_sha256` and `nar::writer_nar`, the latter writing NAR to a writer, the former using write_nar to only keeping the NAR size and digest. Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6192 feat(tvix/store/pathinfosvc): derive CloneFlorian Klink1-0/+1
This allows closing a TvixStoreIO if it contains a SledPathInfoService. Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-24 r/6188 feat(tvix/store/pathinfosvc): GRPCPathInfoService::from_clientFlorian Klink1-0/+11
Change-Id: I4a4f6f713dd76bf95f393c094c0eb9b6d15c5436 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8613 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-16 r/6146 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directlyFlorian Klink4-65/+34
We currently only support querying by the output hash digest. This makes the interface a bit simpler. Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 r/6043 feat(tvix/store/pathinfosvc): add gRPC clientFlorian Klink2-0/+73
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-10 r/5940 refactor(tvix/store): impl Default for MemoryPathInfoServiceFlorian Klink1-8/+1
Change-Id: I07f83f8aaa6312cc37ce7e7ee2b713e60df23782 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8234 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-10 r/5934 feat(tvix/store): add new_temporary for all Sled servicesFlorian Klink1-0/+7
This provides a service using /dev/shm, that's deleted once the reference is dropped. Refactor all tests to use these, which allows getting rid of most TempDir usage in the tests. The only place where we still use TempDir is in the importer tests, which work on a filesystem path. Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5919 feat(tvix/store): add pathinfoserviceFlorian Klink3-0/+164
This adds a PathInfoService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-18T19·45+0000