about summary refs log tree commit diff
path: root/tvix/store/src/pathinfoservice/mod.rs
AgeCommit message (Collapse)AuthorFilesLines
2024-05-11 r/8103 refactor(tvix/store): drop calculate_nar from PathInfoServiceFlorian Klink1-16/+0
This shouldn't be part of the PathInfoService trait. Pretty much none of the PathInfoServices do implement it, and requiring them to implement it means they also cannot make use of this calculation already being done by other PathInfoServices. Move it out into its own NarCalculationService trait, defined somewhere at tvix_store::nar, and have everyone who wants to trigger nar calculation use nar_calculation_service directly, which now is an additional field in TvixStoreIO for example. It being moved outside the PathInfoService trait doesn't prohibit specific implementations to implement it (like the GRPC client for the `PathInfoService` does. This is currently wired together in a bit of a hacky fashion - as of now, everything uses the naive implementation that traverses blob and directoryservice, rather than composing it properly. I want to leave that up to a later CL, dealing with other parts of store composition too. Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-09 r/7884 feat(tvix/store): add bigtable pathinfoservice backendFlorian Klink1-0/+5
Put behind the "cloud" backend, like in the `tvix-castore` crate. Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7788 feat(tvix/store): add rstest-based PathInfoService testsFlorian Klink1-0/+3
This introduces rstest-based tests. We also add fixtures for creating some BlobService / DirectoryService out of thin air. To test a PathInfoService, we don't really care too much about its internal storage - ensuring they work is up to the castore tests. Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-27 r/7786 feat(tvix/store): AsRef<dyn PathInfoService> impl PathInfoServiceFlorian Klink1-0/+25
Change-Id: I7fc06ae97a50d04b8c36292b3457c112242a7a70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11270 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-01-21 r/7435 chore(tvix/store): Use BoxStream type aliasConnor Brewster1-3/+2
The BoxStream type alias is a more concise and easier to read than the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type. Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-12-22 r/7256 chore(tvix): move store/fs to castore/fsFlorian Klink1-0/+6
With the recent introduction of the RootNodes trait, there's nothing in the fs module pulling in tvix-store dependencies, so it can live in tvix-castore. This allows other crates to make use of TvixStoreFS, without having to pull in tvix-store. For example, a tvix-build using a fuse mountpoint at /nix/store doesn't need a PathInfoService to hold the root nodes that should be present, but just a list. tvix-store now has a pathinfoservice/fs module, which contains the necessary glue logic to implement the RootNodes trait for a PathInfoService. To satisfy Rust orphan rules for trait implementations, we had to add a small wrapper struct. It's mostly hidden away by the make_fs helper function returning a TvixStoreFs. It can't be entirely private, as its still leaking into the concrete type of TvixStoreFS. tvix-store still has `fuse` and `virtiofs` features, but they now simply enable these features in the `tvix-castore` crate they depend on. The tests for the fuse functionality stay in tvix-store for now, as they populate the root nodes through a PathInfoService. Once above mentioned "list of root nodes" implementation exists, we might want to shuffle this around one more time. Fixes b/341. Change-Id: I989f664827a5a361b23b34368d242d10c157c756 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-11-19 r/7038 feat(tvix/store/pathinfoservice): implement NixHTTPPathInfoServiceFlorian Klink1-0/+2
NixHTTPPathInfoService acts as a bridge in between the Nix HTTP Binary cache protocol provided by Nix binary caches such as cache.nixos.org, and the Tvix Store Model. It implements the [PathInfoService] trait in an interesting way: Every [PathInfoService::get] fetches the .narinfo and referred NAR file, inserting components into a [BlobService] and [DirectoryService], then returning a [PathInfo] struct with the root. Due to this being quite a costly operation, clients are expected to layer this service with store composition, so they're only ingested once. The client is expected to be (indirectly) using the same [BlobService] and [DirectoryService], so able to fetch referred Directories and Blobs. [PathInfoService::put] and [PathInfoService::nar] are not implemented and return an error if called. This behaves very similar to the nar-bridge-pathinfo code in nar-bridge, except it's now in Rust. Change-Id: Ia03d4fed9d0657965d100299af97cd917a03f2f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10069 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-15 r/7014 refactor(tvix/store): remove from_url from PathInfoService traitFlorian Klink1-17/+1
We don't gain much from making this part of the trait, it's still up to `tvix_store::pathinfoservice::from_addr` to do most of the construction. Move it out of the trait and into the specific *Service impls directly. This allows further refactorings in followup CLs. Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink1-7/+12
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6625 docs(tvix/store/pathinfosvc): fix copypasta errorFlorian Klink1-1/+0
This is not a simple get and put returning digest. You currently can get by output hash, and a put gives you back the (possibly modified) PathInfo message. Change-Id: I43174d87a764417b8620c4d5bdac058cc25b0373 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9371 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-09-21 r/6623 refactor(tvix/store): Asyncify PathInfoService and DirectoryServiceConnor Brewster1-4/+15
We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-05 r/6555 feat(tvix/store/pathinfosvc): provide listingFlorian Klink1-0/+4
This provides an additional method in the PathInfoService trait, as well as an RPC method on the gRPC layer to list all PathInfo objects in a PathInfoService. Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-14 r/6304 refactor(tvix/store/pathinfosvc): add from_addrFlorian Klink1-0/+17
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-12 r/6279 refactor(tvix/store/pathinfosvc): use Arc<dyn …>Florian Klink1-1/+1
This removes the use of generics, like previously done with Blob and Directory services. Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink1-1/+4
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6270 refactor(tvix/src/nar): drop NARCalculationServiceFlorian Klink1-0/+2
There's only one way to calculate NAR files, by walking through them. Things like caching such replies should be done closer to where we use these, composing NARCalculationService doesn't actually give us much. Instead, expose two functions, `nar::calculate_size_and_sha256` and `nar::writer_nar`, the latter writing NAR to a writer, the former using write_nar to only keeping the NAR size and digest. Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-16 r/6146 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directlyFlorian Klink1-5/+2
We currently only support querying by the output hash digest. This makes the interface a bit simpler. Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 r/6043 feat(tvix/store/pathinfosvc): add gRPC clientFlorian Klink1-0/+2
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-10 r/5919 feat(tvix/store): add pathinfoserviceFlorian Klink1-0/+21
This adds a PathInfoService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-12T13·20+0000