about summary refs log tree commit diff
path: root/tvix/store/src/pathinfoservice/grpc.rs
AgeCommit message (Collapse)AuthorFilesLines
2024-06-20 r/8300 refactor(tvix/*store): remove some trait boundsFlorian Klink1-8/+2
We don't need to require these things for these impl blocks yet. Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser1-7/+25
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-17 r/8288 feat(tvix/store): display progress for NAR calculationFlorian Klink1-2/+7
This is currently still taking a noticeable amount of time, so make sure we show it is happening. Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847 Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-20 r/8157 refactor(tvix/store/pathinfo): test with PathInfoService directlyFlorian Klink1-0/+1
Since cl/…, a PathInfoService doesn't need to implement `calculate_nar` anymore, so most of them don't actually have a handle to a {Blob,Directory}Service anymore. This means, we can simplify the construction of them for test cases a lot. Change-Id: I100e9e1c9b00a049b4d6136c57aad4cdb04461c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11691 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-12 r/8117 feat(tvix/store/pathinfo/*): add more instrumentationFlorian Klink1-2/+2
Add instrumentation to the get() and put() implementations of all PathInfoService. Use the nixbase32 representation of the digest, not the base64 one. Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8103 refactor(tvix/store): drop calculate_nar from PathInfoServiceFlorian Klink1-25/+31
This shouldn't be part of the PathInfoService trait. Pretty much none of the PathInfoServices do implement it, and requiring them to implement it means they also cannot make use of this calculation already being done by other PathInfoServices. Move it out into its own NarCalculationService trait, defined somewhere at tvix_store::nar, and have everyone who wants to trigger nar calculation use nar_calculation_service directly, which now is an additional field in TvixStoreIO for example. It being moved outside the PathInfoService trait doesn't prohibit specific implementations to implement it (like the GRPC client for the `PathInfoService` does. This is currently wired together in a bit of a hacky fashion - as of now, everything uses the naive implementation that traverses blob and directoryservice, rather than composing it properly. I want to leave that up to a later CL, dealing with other parts of store composition too. Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8099 refactor(tvix/store/pathinfo/grpc): simplify pingpongFlorian Klink1-75/+6
We already have the same code in make_grpc_path_info_service_client. Change-Id: Ibcd60831af8a061a2c3bb2f960f52a43d06cf6fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11615 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-19 r/7965 chore(tvix/store): migrate import.rs and tests/pathinfo.rs to rstestFlorian Klink1-1/+1
Also, rename the DUMMY_NAME constant in the fixtures to DUMMY_PATH, which aligns more with the ToString representation and from_bytes conversions we have on StorePath[Ref]. Change-Id: I39763c9dfa84c5d86f2fd0171b3a4d36fd72f267 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11464 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-12 r/7887 feat(tvix/store/pathinfo/grpc): instrument functionsFlorian Klink1-0/+6
Change-Id: Idb7d7144be1917fbaf83e9fd76c5b2ebb3df98d2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11400 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-28 r/7792 refactor(tvix/pathinfo/grpc): use rstest fixturesFlorian Klink1-6/+12
Change-Id: Ib114a4d141ca829520aed34600678d613994f875 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11277 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7787 refactor(tvix/store/tests/fixtures): use [u8; 20]Florian Klink1-1/+1
This makes the fixture more use-able when interacting with the trait, the Bytes are only useful for the gRPC version. Change-Id: Iaaea1adc6df18491f236a28c4343f5b4ee5fcfd3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11271 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-01-21 r/7435 chore(tvix/store): Use BoxStream type aliasConnor Brewster1-3/+2
The BoxStream type alias is a more concise and easier to read than the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type. Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-12 r/7371 refactor(tvix/store/pathinfoservice): make more genericFlorian Klink1-2/+2
We don't need Arcs in most of the cases, we're fine with some container. Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-05 r/7348 feat(tvix/store/pathinfosvc): call validateFlorian Klink1-1/+9
Change-Id: I557ed67a04112ba3e363303637d1988309250a93 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10535 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7307 refactor(tvix/castore/tests): let gen_*_service return BoxesFlorian Klink1-2/+2
Only convert to and reuse an Arc<…> where needed. Change-Id: I2c1bc69cca5a4a3ebd3bdb33d6e28e1f5fb86cb9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10514 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-22 r/7255 refactor(tvix/store): remove Arc<> from PathInfoService::from_addrFlorian Klink1-3/+2
This makes PathInfoService::from_addr return a Box<dyn PathInfoService>, rather than an Arc<dyn …>, and leaves it up to the consumers to rewrap it into an Arc where needed. This allows us to drop the Arc for the tvix-store daemon subcommand. Change-Id: Ic83aa2ade6c51912281bd17c7eef7252e152b2d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10409 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-11-15 r/7022 refactor(tvix/castore/tonic): make async, support wait-connect=?Florian Klink1-1/+3
This moves the sync `channel::from_url` to a async `tonic::channel_from_url`. It now allows connecting non-lazily if `wait- connect=1` is set in the URL params. Also, make the pingpong tests for blobsvc and directorysvc use the wait- connect=1 codepath. Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7017 refactor(tvix/store/pathinfosvc): inline GRPCPathInfoSvc::from_urlFlorian Klink1-23/+7
Change-Id: Ib53b5525ae13c276e61b7f564673b7c6144ffc0e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10025 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7016 feat(tvix/castore/src/channel): move from_url testsFlorian Klink1-73/+1
These gRPC PathInfoService tests were actually not too useful in here, what we're mostly testing is the channel construction, so move it to there. Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-15 r/7014 refactor(tvix/store): remove from_url from PathInfoService traitFlorian Klink1-4/+4
We don't gain much from making this part of the trait, it's still up to `tvix_store::pathinfoservice::from_addr` to do most of the construction. Move it out of the trait and into the specific *Service impls directly. This allows further refactorings in followup CLs. Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6802 refactor(tvix/*store): helper for channel creation from urlFlorian Klink1-41/+4
This moves the repetitive code to parse a URL and create a channel connected to it into `tvix_castore::channel::from_url`. Part of b/308 Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6801 refactor(tvix/*store): reorganize from_urlFlorian Klink1-14/+9
Move the channel creation depending on the string-based URL into its own block. Change-Id: I546b769acd2296b548eb966b62c495f910266df5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6800 refactor(tvix/*store): remove some grpc_client let bindingsFlorian Klink1-20/+17
We had to have these all while the traits where sync, and there was a lot of spawning and moving. Most of this can now be removed in favor of some inline `.clone()`. Change-Id: Id5466c32a403100bc3347866b3172e06a792e311 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 r/6743 test(tvix/store): use tokio-retry for exp backoffFlorian Klink1-17/+15
Rather than using this loop, use exponential backoff while waiting for the socket path to be created. Change-Id: I3056b1525784cd712b1d81f84876c9ca0be10dc6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9569 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-08 r/6741 refactor(tvix/store): simplify test_valid_unix_path_ping_pongFlorian Klink1-43/+31
We don't need to spawn two tokio runtimes anymore, and can do the URL parsing at once, too. Change-Id: I7885a894bb1250cd087d4e1893e3e73b631331da Reviewed-on: https://cl.tvl.fyi/c/depot/+/9567 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink1-24/+21
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 r/6623 refactor(tvix/store): Asyncify PathInfoService and DirectoryServiceConnor Brewster1-110/+66
We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-05 r/6558 refactor(tvix/store): use tokio::task::JoinHandleFlorian Klink1-21/+19
This makes the inside code a bit less verbose. I wasn't able to describe the type of the async move closure itself, which would allow us to remove the JoinHandle<_> type annotation entirely. Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-09-05 r/6555 feat(tvix/store/pathinfosvc): provide listingFlorian Klink1-2/+62
This provides an additional method in the PathInfoService trait, as well as an RPC method on the gRPC layer to list all PathInfo objects in a PathInfoService. Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-07-22 r/6439 feat(tvix/store/proto): use Bytes instead of Vec<u8>Florian Klink1-1/+2
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-19 r/6335 docs(tvix/store): correct some cargo doc warningsFlorian Klink1-1/+1
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-14 r/6304 refactor(tvix/store/pathinfosvc): add from_addrFlorian Klink1-1/+231
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-12 r/6282 chore(tvix/store/pathinfosvc): remove GRPCPathInfoService::newFlorian Klink1-12/+0
There's very little reason to instantiate a GRPCPathInfoService in a context where we are not already in a tokio context. Change-Id: Ib81d649387717cb98de8a8039f92472f727b10c1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8755 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink1-0/+32
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 r/6188 feat(tvix/store/pathinfosvc): GRPCPathInfoService::from_clientFlorian Klink1-0/+11
Change-Id: I4a4f6f713dd76bf95f393c094c0eb9b6d15c5436 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8613 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-16 r/6146 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directlyFlorian Klink1-5/+4
We currently only support querying by the output hash digest. This makes the interface a bit simpler. Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-27 r/6043 feat(tvix/store/pathinfosvc): add gRPC clientFlorian Klink1-0/+71
Change-Id: Ie8e205c691bd11db99fcf097357c1e49161c6e19 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8349 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-05T02·13+0000