about summary refs log tree commit diff
path: root/tvix/store/src/blobservice
AgeCommit message (Collapse)AuthorFilesLines
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink7-1478/+0
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 r/6608 chore(tvix/store/protos): drop unused fields for nowFlorian Klink1-1/+0
This wasn't removed yet, and no code is using/populating it so far. It's confusing, let's update it to the state of things now, and re- introduce it once we get there. Change-Id: I68f5ba17a8eee604d8ccd82749da7c8be094cb99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9351 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-18 r/6606 refactor(tvix/store/blobsvc): make BlobStore asyncFlorian Klink7-400/+638
We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-12 r/6581 fix(tvix/store/blobservice/seeker): fix debug assertFlorian Klink1-2/+9
We were asserting absolute_offset > self.pos, but that's not true for both being zero. Ramp up the tracing bits a bit, so we actually can see this in the debug logs. Change-Id: I21693bcafab227549b19cd6f1215d2f8dee77ecc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9292 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-09-05 r/6558 refactor(tvix/store): use tokio::task::JoinHandleFlorian Klink1-12/+11
This makes the inside code a bit less verbose. I wasn't able to describe the type of the async move closure itself, which would allow us to remove the JoinHandle<_> type annotation entirely. Change-Id: I06193982a0c7010bd72d3ffa4f760bea1b097632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9268 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-09-03 r/6548 docs(tvix/store): address cargo doc warningsFlorian Klink2-2/+2
Fix some broken link references. Change-Id: I69c9b2b62af35bb777e4df1a01ba3181a368be47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9214 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-08-20 r/6500 refactor(tvix/store): cargo clippyFlorian Klink2-15/+7
Change-Id: I3a80560d036e7ed08036b5e9f0974080d1a30ded Reviewed-on: https://cl.tvl.fyi/c/depot/+/9096 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-07-22 r/6439 feat(tvix/store/proto): use Bytes instead of Vec<u8>Florian Klink1-2/+2
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-07-21 r/6437 feat(tvix/store/digests): use bytes::Bytes instead of Vec<u8>Florian Klink3-7/+5
This will save us some copies, because a clone will simply create an additional pointer to the same data. Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-07-21 r/6435 feat(tvix/store/fuse): implement open explicitlyFlorian Klink1-1/+11
This "reverts" commit 9f600de22671ee1f88e6fb9e53a5a385b434871b (the initial revert of f5e291cf8328096d790f5416cf1968cb9164220a). Now with BlobService returning a BlobReader that implements io::Seek, we can actually just call blob_reader.seek(io::SeekFrom::Start(offset as u64)). This means, we currently will fail to seek backwards inside a file. Change-Id: I9c19448df6831a3537252f99210374f2126ecfc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8886 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 r/6434 feat(tvix/store/blobservice): implement seekFlorian Klink6-11/+240
For memory and sled, it's trivial, as we already have a Cursor<Vec<u8>>. For gRPC, we simply reject going backwards, and skip n bytes for now. Once the gRPC protocol gets support for offsets and verified streaming, this can be improved. Change-Id: I734066a514aed287ea3db64bfb1680911ac1eeb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8885 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 r/6370 feat(tvix/store/blobsvc): add more blobservice testsFlorian Klink2-0/+89
Change-Id: I3e27dfb4ce3e52974d7614814abb7b5ae4a37f8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8782 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-30 r/6369 docs(tvix/store/blobservice): update docstringFlorian Klink2-2/+2
Change-Id: I6ae693a66530e1ecca57723a97d56b309fa0651a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8881 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-30 r/6367 fix(tvix/store/blobservice): write into hasher from b, not bufFlorian Klink2-2/+2
buf contains everything written so far, whereas b is the slice passed in the current write() call. If we copy from &buf, we end up with the wrong hash, because we keep writing the wrong data to the hash function. Change-Id: I768d4645934a6a7d75b9c8eeba35f8f3be5edd26 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8880 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-19 r/6335 docs(tvix/store): correct some cargo doc warningsFlorian Klink1-1/+1
Change-Id: I5053e3f7dcea01e75baa933e4986396583ff22e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8831 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-19 r/6331 chore(tvix/store/blobsvc): clippyFlorian Klink2-10/+7
Change-Id: Ie384bdd27e1e9282ceda83edc74ffaad387f352b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8810 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-06-14 r/6306 fix(tvix/store/blobsvc): drop stray println! debug statementFlorian Klink1-1/+0
Change-Id: Ica073820ea8240d42ce8b979309f881af18a8cde Reviewed-on: https://cl.tvl.fyi/c/depot/+/8781 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-14 r/6305 refactor(tvix/store/blobsvc): make from_addr not asyncFlorian Klink1-1/+1
A previous iteration of this code did actually connect (in the gRPC client), which was why we had this function async. However, as the connection there is now lazy too, we can drop the asyncness in this function. Change-Id: Idd5bd953a6a1c2334066ee672cfb87fcb74f9f94 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8780 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-14 r/6302 feat(tvix/store/blobsvc): add from_addrFlorian Klink5-1/+402
This allows constructing blob stores with a URL syntax at runtime, by passing the --blob-service-addr arg. We probably still want to have some builder pattern here, to allow additional schemes to be registered. Change-Id: Ie588ff7a7c6fb64c9474dfbd2e4bc5f168dfd778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8742 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6280 chore(tvix/store/blobsvc): remove GRPCBlobService::newFlorian Klink1-12/+1
There's very little reason to instantiate a GRPCBlobService in a context where we are not already in a tokio context. Change-Id: Ic6e18809a9f2a76f1c098ed330118d8dcfba5137 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8753 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-12 r/6278 refactor(tvix/store/blobsvc): drop Result<_,_> around open_writeFlorian Klink4-11/+10
We never returned Err here anyways, and we can still return an error during the first (or subsequent) write(s). Change-Id: I4b4cd3d35f6ea008e9ffe2f7b71bfc9187309e2f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8750 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6277 docs(tvix/store/blobservice): fix docstringsFlorian Klink1-2/+3
These are outdated. Change-Id: I1a1ae130a55847f57a48d5e244e7e029c1ecae7b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8749 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6274 refactor(tvix/store/blobsvc/sled): cargo clippyFlorian Klink1-1/+1
Change-Id: I00b8b567509d5e0847270f36dadb5dcb534b9b73 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8739 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-12 r/6269 feat(tvix/store): eliminate generics in BlobStoreFlorian Klink4-89/+192
To construct various stores at runtime, we need to eliminate associated types from the BlobService trait, and return Box<dyn …> instead of specific types. This also means we can't consume self in the close() method, so everything we write to is put in an Option<>, and during the first close we take from there. Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-24 r/6186 feat(tvix/store/blobsvc): add constructorsFlorian Klink1-1/+23
Change-Id: I55e06bf4e8a11dc2caf92c597558f1b820b42566 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8610 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-23 r/6178 refactor(tvix/store/blobsvc): move from Vec<u8> to B3DigestFlorian Klink4-47/+38
Change-Id: I809bab75221f81b6023cfe75c2fe9e589c1e9192 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8605 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-18 r/6163 refactor(tvix/store/blobsvc/sled): clippyFlorian Klink1-5/+5
Change-Id: Icfea8ceb6b827a645c6f10efae741f6fa2114fad Reviewed-on: https://cl.tvl.fyi/c/depot/+/8590 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-14 r/6138 feat(tvix/store): add GRPCBlobServiceFlorian Klink2-0/+198
This connects to a (remote) tvix-store BlobService over gRPC. Change-Id: If31f706738a5c3445886c117feca8b61f3203e9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8552 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-11 r/6133 refactor(tvix/store): remove ChunkServiceFlorian Klink3-77/+150
Whether chunking is involved or not, is an implementation detail of each Blobstore. Consumers of a whole blob shouldn't need to worry about that. It currently is not visible in the gRPC interface either. It shouldn't bleed into everything. Let the BlobService trait provide `open_read` and `open_write` methods, which return handles providing io::Read or io::Write, and leave the details up to the implementation. This means, our custom BlobReader module can go away, and all the chunking bits in there, too. In the future, we might still want to add more chunking-aware syncing, but as a syncing strategy some stores can expose, not as a fundamental protocol component. This currently needs "SyncReadIntoAsyncRead", taken and vendored in from https://github.com/tokio-rs/tokio/pull/5669. It provides a AsyncRead for a sync Read, which is necessary to connect our (sync) BlobReader interface to a GRPC server implementation. As an alternative, we could also make the BlobReader itself async, and let consumers of the trait (EvalIO) deal with the async-ness, but this is less of a change for now. In terms of vendoring, I initially tried to move our tokio crate to these commits, but ended up in version incompatibilities, so let's vendor it in for now. Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 r/5943 refactor(tvix/store): impl Default for MemoryBlobServiceFlorian Klink1-9/+1
Change-Id: I78a4cd86cb364b970e99393579808e773db5ceb2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8237 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-10 r/5934 feat(tvix/store): add new_temporary for all Sled servicesFlorian Klink1-0/+7
This provides a service using /dev/shm, that's deleted once the reference is dropped. Refactor all tests to use these, which allows getting rid of most TempDir usage in the tests. The only place where we still use TempDir is in the importer tests, which work on a filesystem path. Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-06 r/5894 feat(tvix/store): add blobserviceFlorian Klink3-0/+145
This adds a BlobService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-10T09·15+0000