about summary refs log tree commit diff
path: root/tvix/store/src/pathinfoservice/sled.rs
AgeCommit message (Collapse)AuthorFilesLines
2024-07-20 r/8380 refactor(tvix/store): use composition in tvix_store crateYureka1-0/+68
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-06-10 r/8242 feat(tvix/tracing): introduce common tvix-tracing crateSimon Hauser1-2/+1
Introduce a new common crate that contains tracing boilerplate which then can be used in the cli, tvix-store and tvix-build crates. It has otlp as an optional feature, which is currently only used by tvix-store. Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-05-12 r/8117 feat(tvix/store/pathinfo/*): add more instrumentationFlorian Klink1-2/+2
Add instrumentation to the get() and put() implementations of all PathInfoService. Use the nixbase32 representation of the digest, not the base64 one. Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8105 refactor(tvix/store/pathinfo/sled): drop {blob,directory}_serviceFlorian Klink1-29/+8
These are not used anymore. Change-Id: I9c348391c9600e9319f171faf3eda7175ebf7076 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11621 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8103 refactor(tvix/store): drop calculate_nar from PathInfoServiceFlorian Klink1-12/+2
This shouldn't be part of the PathInfoService trait. Pretty much none of the PathInfoServices do implement it, and requiring them to implement it means they also cannot make use of this calculation already being done by other PathInfoServices. Move it out into its own NarCalculationService trait, defined somewhere at tvix_store::nar, and have everyone who wants to trigger nar calculation use nar_calculation_service directly, which now is an additional field in TvixStoreIO for example. It being moved outside the PathInfoService trait doesn't prohibit specific implementations to implement it (like the GRPC client for the `PathInfoService` does. This is currently wired together in a bit of a hacky fashion - as of now, everything uses the naive implementation that traverses blob and directoryservice, rather than composing it properly. I want to leave that up to a later CL, dealing with other parts of store composition too. Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-11 r/8102 fix(tvix/store/pathinfo/sled): use spawn_blockingFlorian Klink1-22/+45
This does IO, which might take a longer amount of time than what we want to be blocking the normal executor. Use spawn_blocking instead. I didn't add it for the constructors, as we only call these once. Change-Id: I9a1063099bac9582ca9681043c58c1edc780c5ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/11618 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-03 r/8071 refactor(tvix/store/pathinfo/sled): cleanup, add instrumentationFlorian Klink1-57/+42
Write this a bit more compact, by using map_err(|e| …) and ?. Ideally we'd get rid of the error mapping entirely, by using proper error types, but that's left for a followup. Change-Id: I68dc72b162ac89c5ff82d8c2bc26e1c808a0affd Reviewed-on: https://cl.tvl.fyi/c/depot/+/11584 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-01-21 r/7435 chore(tvix/store): Use BoxStream type aliasConnor Brewster1-3/+4
The BoxStream type alias is a more concise and easier to read than the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type. Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-12 r/7371 refactor(tvix/store/pathinfoservice): make more genericFlorian Klink1-19/+16
We don't need Arcs in most of the cases, we're fine with some container. Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-11 r/7159 refactor(tvix/*store/sled): make ::new() more genericFlorian Klink1-3/+3
We don't really require the Path to be a PathBuf, we don't even require it to be a Path, we only need it to be AsRef<Path>>. This removes some conversion in the from_addr cases, which can just reuse `url.path()` (a `&str`). Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-11 r/7158 chore(tvix): drop sled compression supportFlorian Klink1-1/+3
It's been a while since the last sled release, and that one binds to a pretty old version of zstd, requiring workarounds like cl/10090. Upstream sled main branch currently has zstd halfway patched out (it's a no-op, but the feature flag and options are still there), and it's in that state for a year. Rather than maintaining our own fork of sled, let's just stop using the compression feature in sled, dropping the version pin to zstd that way, removing the need for cl/10090. This doesn't mean we won't reintroduce per-blob compression - but we probably just won't let sled take care of the compression, but do it ourselves - which is necessary for more chunked blob storage anyways. Even though we do drop the feature flag, we still need to explicitly use use_compression(false). Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-15 r/7018 refactor(tvix/store/pathinfosvc): inline SledPathInfoSvc::from_urlFlorian Klink1-131/+0
Change-Id: I0d905228df086a422bb30322add7236ca41e807b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10026 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7014 refactor(tvix/store): remove from_url from PathInfoService traitFlorian Klink1-5/+4
We don't gain much from making this part of the trait, it's still up to `tvix_store::pathinfoservice::from_addr` to do most of the construction. Move it out of the trait and into the specific *Service impls directly. This allows further refactorings in followup CLs. Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-27 r/6888 refactor(nix-compat/store_path): make digest and name privateedef1-1/+4
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink1-13/+16
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-21 r/6623 refactor(tvix/store): Asyncify PathInfoService and DirectoryServiceConnor Brewster1-7/+11
We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-05 r/6555 feat(tvix/store/pathinfosvc): provide listingFlorian Klink1-0/+25
This provides an additional method in the PathInfoService trait, as well as an RPC method on the gRPC layer to list all PathInfo objects in a PathInfoService. Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-19 r/6335 docs(tvix/store): correct some cargo doc warningsFlorian Klink1-1/+1
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-19 r/6333 chore(tvix/store/pathinfosvc): clippyFlorian Klink1-8/+6
Change-Id: Ied4bed08e989791f832922da8776d2104035e28a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8812 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-14 r/6304 refactor(tvix/store/pathinfosvc): add from_addrFlorian Klink1-0/+134
Change-Id: I24e822351a837fce2aed568a647d009099ef32ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/8747 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-12 r/6273 refactor(tvix/store): use Arc instead of BoxFlorian Klink1-9/+13
This allows us to blob services without closing them before putting them in a box. We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers require Sync. Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6272 refactor(tvix/store): use Box<dyn DirectoryService>Florian Klink1-12/+8
Once we support configuring services at runtime, we don't know what DirectoryService we're using at compile time. This also means, we can't explicitly use the is_closed method from GRPCPutter, without making it part of the DirectoryPutter itself. Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink1-9/+38
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 r/6192 feat(tvix/store/pathinfosvc): derive CloneFlorian Klink1-0/+1
This allows closing a TvixStoreIO if it contains a SledPathInfoService. Change-Id: Ife451eda331bafdb1af91f45a94cccd13f2f67c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8620 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-16 r/6146 refactor(tvix/store/pathinfosvc): drop ByWhat, use digest directlyFlorian Klink1-32/+19
We currently only support querying by the output hash digest. This makes the interface a bit simpler. Change-Id: I80b285373f1923e85cb0e404c4b15d51a7f259ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8570 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 r/5934 feat(tvix/store): add new_temporary for all Sled servicesFlorian Klink1-0/+7
This provides a service using /dev/shm, that's deleted once the reference is dropped. Refactor all tests to use these, which allows getting rid of most TempDir usage in the tests. The only place where we still use TempDir is in the importer tests, which work on a filesystem path. Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5919 feat(tvix/store): add pathinfoserviceFlorian Klink1-0/+81
This adds a PathInfoService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-26T11·27+0000