about summary refs log tree commit diff
path: root/tvix/castore/src/blobservice
AgeCommit message (Collapse)AuthorFilesLines
2024-03-24 r/7778 feat(tvix/castore): AsRef<dyn BlobService> impl BlobServiceFlorian Klink1-0/+22
This allows us to use containers around BlobServices as BlobServices too. Change-Id: I3c7feb074f42b4e07c550fb8dfa63cf81d448ab5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11249 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-20 r/7751 feat(tvix/castore/blob): document missing objectstore+*:// URLFlorian Klink1-0/+1
Change-Id: I3cbc75e636efd467ddda7fc3c3c8f19ad42204ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/11206 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-20 r/7750 refactor(tvix/castore/blob): drop simplefsFlorian Klink3-208/+1
This functionality is provided by the object store backend too (using `objectstore+file://$some_path`). This backend also supports content-defined chunking and compresses chunks with zstd. Change-Id: I5968c713112c400d23897c59db06b6c713c9d8cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/11205 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-20 r/7749 refactor(tvix/castore): introduce "cloud" feature flagFlorian Klink1-5/+17
This controls whether tvix-castore has support for various cloud backends or not. Use this to control the set of feature flags for the object_store backend, and only enable the aws, azure and gcp ones if it's set. In the future this can be used to enable/disable other cloud backends too. Without feature flags, `object_store` already supports the `InMemory` and `LocalFilesystem` backends, and we also want to unconditionally enable the `http` one. Make sure at least the construction of these services is covered in the tests. Similarly, the tvix-store crate, which provides the tvix-store CLI has a `cloud` feature flag too (defaulting to enabled). Change-Id: I9fb9c87b740e7dc83f8ff7a0862905d036d513f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11204 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-03-20 r/7745 refactor(tvix/castore/blob/from_addr): use match guardsFlorian Klink1-53/+59
This will allow feature-flagging some of the backends. Change-Id: Idffbf8b3fd154f5a3d938225c3871feffea8ff8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11200 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-03-18 r/7730 refactor(tvix/castore/blobsvc): use B3Digest Display implFlorian Klink2-10/+2
We don't need to use BASE64 here on our own, B3Digest has a Display impl. This will also make sure the `b3:` digest is present in field values. Change-Id: I0ce6ee0f7e7e99fb9b16872953a1b742e99be291 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11192 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-03-18 r/7724 feat(tvix/blobservice/object_store) more loggingFlorian Klink1-1/+3
Have derive_{blob,chunk}_path emit trace-level events for both the values they're called with, as well as the return value. With RUST_LOG in place, it doesn't get lost in other unrelated noise. Change-Id: Id2451e3657324eff482841eb26a22d19e22bde30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11136 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-18 r/7722 feat(tvix/castore/blobsvc/grpc): read data in chunksFlorian Klink1-26/+54
Whenever this encounters an open_read(), it'll first check for more granular chunking. If there's more granular chunking data available, a ChunkedReader is constructed (which supports seeking backwards). This currently is still a bit stupid, and doesn't compose, as `ChunkedReader` uses `self` as the `BlobService` to ask for the individual chunks. In store composition future, we might want to compose this differently, essentially constructing `ChunkedReader` with another `BlobService` representing the entire hierarchy, so there's a chance to locally cache things, and do less requests. Change-Id: I22e0df4d6245f666d083b4f0b7114d3ac41d1dce Reviewed-on: https://cl.tvl.fyi/c/depot/+/11185 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 r/7720 refactor(tvix/castore/src/blobservice): remove useless else caseFlorian Klink1-4/+3
Change-Id: I09000371a1d8ff212ab46050d1a480509c6ffe70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11183 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-11 r/7685 feat(tvix/castore/blobsvc/from_addr): support object_storeFlorian Klink1-1/+21
The object_store crate supports a ton of different stores, with different schemes. For now, use a objectstore+ scheme prefix to enable these. Change-Id: I946f76e32a0fb0867ef59060217894cda5b959b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11080 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-03-11 r/7684 feat(tvix/castore/blobsvc): add object storage implementationFlorian Klink3-0/+561
This uses the `object_store` crate to expose a tvix-castore BlobService backed by object storage. It's using FastCDC to chunk blobs into smaller chunks when writing to it. These are exposed at the .chunks() method. Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-03-03 r/7648 refactor(tvix/*/from_addr): improve test debuggabilityFlorian Klink1-2/+6
If there's an unexpected test failure, print it out, rather than just saying something is false even though it should be true. Use .expect() for this, which displays the error if it failed. We can't use expect_err(), as our stores are not display'able, so use an assertion with a message there. Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de>
2024-03-03 r/7631 refactor(tvix/castore/blobsvc/chunked_reader): refactor, documentFlorian Klink1-131/+139
The public-consumable thing here is ChunkedReader, not ChunkedBlob. ChunkedBlob is a helper that can be used to get a new AsyncRead, but not AsyncSeek. It is used internally by ChunkedReader whenever the client seeks. Make this more obvious, by extending the documentation, and putting ChunkedReader at the top of this file. Also make ChunkedBlob and its methods private, and give ChunkedReader a more useful constructor (from_chunks, instead of from_chunked_blob). Change-Id: I2399867591df923faa73927b924e7c116ad98dc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11079 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-02 r/7627 feat(tvix/castore/blobsvc): BlobReader for more trivial typesFlorian Klink1-0/+2
Change-Id: I80e4f26c41a504fa4c6a013c2a1e76de613ba294 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11078 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-02 r/7626 fix(tvix/castore/blobwriter): don't require Sync + 'staticFlorian Klink1-1/+1
There's no reason for these two. Change-Id: Ie6f238bbb0b17971c9877b11b61ea7ebca573c13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11075 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-02-10 r/7494 feat(tvix/castore/blobsvc): add Chunked{Blob,Reader}Florian Klink2-0/+489
These provide seekable access into a Blob for which we have more granular chunking information. There's no support for verified streaming in here yet, this simply produces a stream of readers for each chunk, skipping irrelevant chunks and data from the first chunk at the beginning. A seek simply does produce a new reader using the same process. Change-Id: I37f76b752adce027586770475435f3990a6dee0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10731 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-02 r/7470 feat(tvix/castore/blobsvc): validate StatBlobResponseFlorian Klink1-0/+4
All chunks must have valid blake3 digests. It is allowed to send an empty list, if no more granular chunking is available. Change-Id: I7ecb53579cdf40fd938bb68a85685751b4d3626f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10726 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-02-02 r/7468 docs(tvix/castore/blobsvc): fix doc comments on traitFlorian Klink1-10/+14
The readers implement AsyncRead/AsyncSeek, not their sync counterparts. Also update expectations around chunks. Change-Id: Ic266688039d80d16d33f651b96ce2bcdedecfa00 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10734 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-02-02 r/7465 refactor(tvix/castore/blobsvc/combinator): compact trait boundsFlorian Klink1-1/+2
BlobService already implies Send and Sync, we don't need to explicitly list it here. Change-Id: I58a4c5912be61a60acd961565979aa01d94ee0f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10727 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-01-09 r/7366 feat(tvix/castore): implement CombinedBlobServiceFlorian Klink2-0/+137
First attempt on composition of BlobServices. Change-Id: I6e70248007edfd322a503fd40c1c4b4300cbc30c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10587 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-01-09 r/7365 feat(tvix/castore/blobsvc): add chunks methodFlorian Klink2-2/+47
This adds support to retrieve a list of chunks for a given blob to the BlobService interface. While theoretically all chunk-awareness could be kept private inside each BlobService reader, we'd not be able to resolve individual chunks from different Blobservices - and due to this, not able to substitute chunks we already have in a more local store. This function allows asking a BlobService for the list of chunks, leaving any actual fetching up to the caller (be it through individual calls to open_read), or asking another store for it. Change-Id: I1d33c591195ed494be3aec71a8c804743cbe0dca Reviewed-on: https://cl.tvl.fyi/c/depot/+/10586 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-09 r/7363 feat(tvix/castore/grpc): instrument some more functionsFlorian Klink1-0/+2
Change-Id: Icedb148c88c5f4a3b2242ed12df1dd8692af94fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10584 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 r/7360 refactor(tvix/castore/blobservice/grpc): remove fn pointer hackFlorian Klink1-15/+8
It looks like the workaround isn't necessary anymore. Change-Id: Ifbcef1d631b3f369cac3db25a2c793480043f697 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10583 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-09 r/7358 feat(tvix/castore/blobsvc/grpc): rm VecDec, fix docstringFlorian Klink1-21/+9
The docstrings were not updated once we made the BlobService trait async. There's no more need to turn things into a sync reader. Also, rearrange the stream manipulation a bit, and remove the need to create a new VecDeque for each element in the stream. bytes::Bytes implements the Buf trait. Fixes b/289. Change-Id: Id2bbedca5876b462e630c144b74cc289c3916c4d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10582 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7305 refactor(tvix/castore/blobsvc/grpc/wrapper): don't require Arc<_>Florian Klink1-3/+2
Change-Id: I9655f5588c7dc98427de6af47d74b4ab7ce22071 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10516 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-31 r/7293 refactor(tvix/castore/blobsvc): return Box, not ArcFlorian Klink1-7/+6
While we currently mostly use it in an Arc, as we need to clone it inside PathInfoService, there might be other usecases not requiring it to be Clone. Change-Id: I7bd337cd2e4c2d4154b385461eefa62c9b78345d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10482 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-17 r/7228 feat(tvix/castore): simple filesystem blob serviceRyan Lahfa3-1/+208
The simple filesystem `BlobService` enable a user to write blob store on an existing filesystem using a prefix-style layout in the provided root directory, e.g. the two first bytes of the blake3 hashes are used as directories prefixes. Change-Id: I3451a688a6f39027b9c6517d853b95a87adb3a52 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10071 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-12-12 r/7207 refactor(tvix/castore/blobservice): use io::Result in traitFlorian Klink4-39/+35
For all these calls, the caller has enough context about what it did, so it should be fine to use io::Result here. We pretty much only constructed crate::Error::StorageError before anyways, so this conveys *more* information. Change-Id: I5cabb3769c9c2314bab926d34dda748fda9d3ccc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10328 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-12-11 r/7159 refactor(tvix/*store/sled): make ::new() more genericFlorian Klink2-4/+3
We don't really require the Path to be a PathBuf, we don't even require it to be a Path, we only need it to be AsRef<Path>>. This removes some conversion in the from_addr cases, which can just reuse `url.path()` (a `&str`). Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-11 r/7158 chore(tvix): drop sled compression supportFlorian Klink2-2/+4
It's been a while since the last sled release, and that one binds to a pretty old version of zstd, requiring workarounds like cl/10090. Upstream sled main branch currently has zstd halfway patched out (it's a no-op, but the feature flag and options are still there), and it's in that state for a year. Rather than maintaining our own fork of sled, let's just stop using the compression feature in sled, dropping the version pin to zstd that way, removing the need for cl/10090. This doesn't mean we won't reintroduce per-blob compression - but we probably just won't let sled take care of the compression, but do it ourselves - which is necessary for more chunked blob storage anyways. Even though we do drop the feature flag, we still need to explicitly use use_compression(false). Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-12-09 r/7140 refactor(tvix/castore): address clippyFlorian Klink1-10/+7
We match to destructure a single pattern. Change-Id: I564a3510b4860e90b3315a9639effc48ee88b483 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10233 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-19 r/7037 refactor(tvix/castore/blobservice): rm AsyncBufRead from BlobReaderFlorian Klink1-4/+1
There's no need to already require this to be buffered here. Change-Id: Ib9a11b194e0754d87ab8d2ef0b8cb0f4edc01229 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10074 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-18 r/7027 feat(tvix/castore): fix tracing instrument in `MemoryBlobService`Ryan Lahfa1-2/+3
Change-Id: Iedba57e8b3e1a44f14f5baa1e981275d4b02eb56 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10070 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-15 r/7022 refactor(tvix/castore/tonic): make async, support wait-connect=?Florian Klink2-13/+13
This moves the sync `channel::from_url` to a async `tonic::channel_from_url`. It now allows connecting non-lazily if `wait- connect=1` is set in the URL params. Also, make the pingpong tests for blobsvc and directorysvc use the wait- connect=1 codepath. Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7020 refactor(tvix/castore/blobsvc): remove BlobService::from_urlFlorian Klink5-241/+98
Make blobservice::from_addr use the more specific constructors. Change-Id: Id9637e279d6910ce6d92ff0086a984be5c65a8c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10028 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6972 fix(tvix/castore): B3Digest::{to_vec -> as_slice}edef1-4/+4
Not a single call site actually makes use of the Vec. Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-05 r/6954 feat(tvix/castore): extend blobstore protos for verified streamingFlorian Klink1-0/+1
This pdates the proto docstrings a bit, especially w.r.t. verified streaming. It also adds send_chunks, send_bao fields to StatBlobRequest (renamed from BlobMeta) Change-Id: I590cc8646d86b73bca9f38a9b6d9ea15e4df5cb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9951 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-14 r/6802 refactor(tvix/*store): helper for channel creation from urlFlorian Klink1-44/+5
This moves the repetitive code to parse a URL and create a channel connected to it into `tvix_castore::channel::from_url`. Part of b/308 Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6801 refactor(tvix/*store): reorganize from_urlFlorian Klink1-10/+9
Move the channel creation depending on the string-based URL into its own block. Change-Id: I546b769acd2296b548eb966b62c495f910266df5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6800 refactor(tvix/*store): remove some grpc_client let bindingsFlorian Klink1-4/+3
We had to have these all while the traits where sync, and there was a lot of spawning and moving. Most of this can now be removed in favor of some inline `.clone()`. Change-Id: Id5466c32a403100bc3347866b3172e06a792e311 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 r/6742 test(tvix/castore): use tokio-retry for exp backoffFlorian Klink1-15/+13
Rather than using this loop, use exponential backoff while waiting for the socket path to be created. Change-Id: I18706a64ce06f8916a07892dfbcd409ac5b3bff1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9568 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-08 r/6740 refactor(tvix/castore): simplify test_valid_unix_path_ping_pongFlorian Klink1-65/+47
We don't need to spawn two tokio runtimes anymore, and can do the URL parsing at once, too. Change-Id: I38ab96978cb7f8c31ded2726262e0b1366655094 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9566 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink7-0/+1478
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-23T12·01+0000