about summary refs log tree commit diff
path: root/tvix/castore/src
AgeCommit message (Collapse)AuthorFilesLines
2024-08-01 r/8437 feat(tvix/castore): add RedbDirectoryServiceIlan Joselevich5-0/+340
This provides a DirectoryService implementation which uses redb (https://github.com/cberner/redb) as the database. It provides both in-memory and persistent on-filesystem implementations. Change-Id: Id8f7c812e2cf401cccd1c382b19907b17a6887bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12038 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-01 r/8435 chore(3p/sources): Bump channels & overlays (2024-07-28)Aspen Smith1-2/+10
* Treewide: re-run depotfmt * //third_party/nixpkgs:html5validator: build with Python 3.11, dependency openstackdocstheme doesn't support 3.12 * //users/sterni/machines/ingeborg: adapt to poorly handled fcgiwrap module API change: https://github.com/NixOS/nixpkgs/pull/318599 * //tvix/*-go: regenerate protobuf files * //third_party/nixpkgs:treefmt: Remove patch for merged pull request * //users/flokli/ipu6-softisp: rebase, drop upstreamed kernel patches Change-Id: Ie4e0df007c287e8cd6207683a9a25838aa5bd39a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11971 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-25 r/8413 test(tvix/castore/dirsvc): check for a pitfall with deduplicated dirsYureka2-5/+49
Change-Id: I3cff2c2e8b2c8a2fd8d8074647d0d99a1db8e693 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12034 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-25 r/8412 fix(tvix/castore/GRPCDirectorySvc): fix a bug in the get_recursive fnYureka1-3/+9
When retrieving a closure with get_recursive, the following could happen in the GRPC client: - The first reference to the deduplicated directory is added to expected_directory_digests - The deduplicated directory is obtained removed from expected_directory_digests - The second reference to the deduplicated directory is added to expected_directory_digests - The deduplicated directory has already been sent, but is still in the expected_directory_digests. It looks to the GRPC client like the closure is incomplete and the stream ended prematurely. Change-Id: Ic62bca12e7f8fb85af5fa4dacd199f0f3b8eea8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12033 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-23 r/8411 refactor(directoryservice): get rid of NaiveSeeker helperYureka3-319/+10
Use the ChunkedReader in CombinedBlobService instead which also supports seeking. Change-Id: I681331a80763172c27e55362b7044fe81aaa323b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12031 Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-23 r/8407 fix(tvix/directoryservice): clarify get_recursive not foundYureka4-13/+41
Change-Id: I47f21b4b7db9304eadb04094d41bf04d443fcc3b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12025 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 r/8405 feat(tvix/store): add redb PathInfoServiceIlan Joselevich1-0/+36
This provides a PathInfoService implementation using redb (https://github.com/cberner/redb) as the underlying storage engine. Both an in-memory variant, as well as a filesystem one is provided, similar how it's done with the sled implementation. Supersedes: https://cl.tvl.fyi/c/depot/+/11692 Change-Id: I744619c51bf2efd0fb63659b12a27cbe0b2fd6fc Signed-off-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-22 r/8397 feat(tvix/composition): improve error message for unknown tagYureka1-2/+2
Change-Id: I61a31488de17725ae4311f5f5bf8e02edb752cf9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12017 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8392 test(tvix/composition): add recursion and concurrent testsYureka1-0/+63
Change-Id: Ic10773b08d940b45c3067bd514c3080cadac9606 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12003 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8391 docs(tvix/composition): add comment about stackYureka1-0/+3
Change-Id: I3888b5034c53728e2f9cfe24213f8854eb38bfe1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12005 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8390 feat(tvix/composition): downcast boxed error if possibleYureka1-4/+7
We still have the unique store name to identify which instantiation caused the error. For recursion errors, the full chain is still retained inside the CompositionError. Change-Id: Iaddcece445a5df331e578d7c69d710db3d5f8dcd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12002 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 r/8388 fix(tvix/composition): include typeid in recursion checkYureka1-5/+13
Change-Id: Icc279d41a4980d4b57acbb4243bbd509039b753f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12000 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-21 r/8385 chore(tvix): upgrade to tonic 0.12 / hyper 1.0Yureka3-3/+9
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-20 r/8380 refactor(tvix/store): use composition in tvix_store crateYureka1-0/+7
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-20 r/8375 feat(tvix/castore/protos): return root node after validationFlorian Klink1-2/+4
This allows avoiding a `.node.unwrap()`` after validation. Change-Id: Ieef1ffebab16cdca94c979ca6831a7ab4f6007da Reviewed-on: https://cl.tvl.fyi/c/depot/+/11989 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-07-19 r/8372 feat(tvix): different service types in one compositionYureka11-46/+64
This will be necessary for the PathInfoService composition, as some PathInfoService implementations require a BlobService & DirectoryService to ingest into. Using the Extend trait for creating compositions allows extending the same composition with configs of various types e.g. BlobStore, DirectoryStore Generics are moved from the Composition struct to the functions.The storage of the InstantiatonStates uses the TypeId in the key and a Box<dyn Any> in the value, which is downcasted to InstantiatonState<T>. Change-Id: I2af11f26c535029adfb1c62905e0e7c4aaed7b51 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11980 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-07-19 r/8371 test(tvix/castore/ObjectStoreBlobService): test correct chunkingYureka1-6/+23
Change-Id: I4a8f5cda2705560a6d52a2eb2be0dde35b784c8f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11982 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-07-19 r/8370 fix(tvix/castore/blob/object_store): empty chunk list for single chunkYureka1-0/+9
Change-Id: Ie3609ac6184ecead11c4e03073433c26ce129848 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11981 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-18 r/8368 refactor(tvix): use composition & registry for from_addrYureka14-202/+295
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev>
2024-07-18 r/8367 feat(tvix/castore): add sled to composition registryYureka2-1/+51
Change-Id: I03fa8dfabcee14c5f657380f86bb1a7aa00e08ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/11977 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-18 r/8366 feat(tvix/composition): allow creating a blank CompositionContextYureka1-6/+13
this is useful when oneshot-instantiating a store from a single config Change-Id: I08538fdee1d0bb26b3ae2da7d3b2339b2e93bc0a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11975 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev> Tested-by: BuildkiteCI
2024-07-18 r/8365 feat(tvix/castore): add composition moduleYureka13-46/+666
Change-Id: I0868f3278db85ae5fe030089ee9033837bc08748 Signed-off-by: Yureka <tvl@yuka.dev> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11853 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-29 r/8329 fix(tvix/castore/object_store): make query pairs object_store optsFlorian Klink2-4/+10
We previously called ObjectStoreBlobService::parse_url, which passes an empty list of options when constructing the ObjectStore. This is most likely not what we want. The more reasonable thing to do is pass along the query string (pairs) as options to `object_store::parse_url_opts`, and remove them from the plain URL we pass to object_store itself. Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-29 r/8328 refactor(tvix/castore/blobsvc): inline things a bitFlorian Klink1-5/+5
Get rid of the `let grpc_client` and `let resp` in some cases. Change-Id: Idc1c0f566a3b1b48da62e6f1977b07620656b16c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11884 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-29 r/8327 docs(tvix/castore): drop wrong commentFlorian Klink1-1/+0
This doesnt panic anymore. Change-Id: I1987f8b0640b1e5fe33abcbf46164449c766fd0d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11883 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-06-20 r/8300 refactor(tvix/*store): remove some trait boundsFlorian Klink2-16/+4
We don't need to require these things for these impl blocks yet. Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI
2024-06-20 r/8299 feat(tvix/tracing): gRPC trace context propagationSimon Hauser4-21/+48
This introduces optional helper function in tvix/tracing for trace propagation and uses these helper in the `tvix-store`. The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now accept a generic client, meaning the client can be generated with either `::new` or `::with_interceptor`. This was tested and validated by starting a `tvix-store daemon` and `tvix-store import`. Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-20 r/8298 feat(tvix/castore): instrument tokio task with current spanSimon Hauser3-18/+27
By default tokio::spawn does not instrument the spawned task with the current spawn (https://github.com/tokio-rs/tokio/discussions/6008), do this manually for all tokio::spawn functions in functions that are instrumented. Change-Id: I83dd8145b3a62421454aff57d34180cebbee8304 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11864 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 r/8289 feat(tvix/tvix-store): improve progress barsFlorian Klink2-22/+36
Don't show an empty spinner for daemon commands. Move the bar to the right, so the text is better aligned between spinner progress and bar progress styles. Generally, push progress bars a bit more down to the place where we can track progress. This includes adding one in the upload_blob span. Introduce another progress style template for transfers, which interprets the counter as bytes (not just a plain integer), and also a data rate. Use it for here and in the fetching code, and also make the progress bar itself a bit less wide. Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-16 r/8285 fix(tvix/store/bin): fix shutdown behaviour for FUSEFlorian Klink2-43/+57
Both umounts happening from another process, as well as tvix-store itself calling umount() on FuseDaemon will cause the FUSE worker threads to terminate. So far there was no nice way to wait on these threads to be terminated from multiple places, causing the `tvix-store mount` command to only be terminated if interrupted via ctrl-c, not via an external umount. Update FuseDaemon to use a ThreadPool, which gives us a join primitive over all threads, that can also be called from multiple places. Await on a join() from there to end the program, not the ctrl-c signal handler as it was before. Using FuseDaemon from multiple tasks requires Arc<>-ing both the ThreadPool as well as the inner FuseSession (which also needs to be inside a Mutex if we want to unmount), but now we can clone FuseDaemon around and use it in two places. We could probably also have used an Option and drop the FuseSession after the first umount, but this looks cleaner. Change-Id: Id635ef59b560c111db52ad0b3ca3d12bc7ae28ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/11825 Reviewed-by: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI
2024-06-16 r/8284 feat(castore/directory): add cache combinatorYureka2-0/+144
Change-Id: Ie8850a40e378f6cc0637a85e526fe5b5ed09fcd7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11710 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-06-16 r/8283 refactor(castore/directory/objectstore): use new order validation logicYureka1-14/+10
Use the new helper introduced in CL 11708 instead of rolling our own. Change-Id: I292a9bc8baf73a6c75efe784031bcda1835bb645 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11709 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-06-16 r/8282 refactor(castore/directory): separate order logic from ClosureValidatorYureka8-328/+640
ClosureValidator was previously only suitable for a very narrow use case: Validating incoming uploads, which are in leaves-to-root order. This is because the ordering validation was hard-wired into the add() function. This - Re-name ClosureValidator to DirectoryGraph, which is more suitable since it actually stores the Directory structs and is drained in the end. - Move the ordering-related logic to a separate OrderValidator, which can be used independently. - re-write DirectoryGraph to be a general purpose validator which can accept the input in both orders and can be drained in both orders as well. This means the DirectoryGraph and OrderValidator can now serve multiple new purposes: - Validating the incoming closure on the client while downloading. - Validating the incoming closure downloaded in a caching layer from the `far` cache, and re-order it for insertion into the `near` cache. Change-Id: I2b4b226348416912d7a31935bec050e53d911b70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11708 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-06-06 r/8220 feat(tvix/store/bin): add progress bar infrastructureFlorian Klink2-2/+19
This adds the tracing-indicatif crate, and configures it as a layer in our tracing_subscriber pipeline to emit progress for every span that's configured so. It also moves from using std::io::stderr to write logs to using their writer, to avoid clobbering output. Progress bar styles are defined in a lazy_static, moving this into a general tracing is left for later. This adds some usage of this to the `imports` and `copy` commands. The output can still be improved a bit - we should probably split each task up into a smaller (instrumented) helper functions, so we can create a progress bar for each task. Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-26 r/8172 refactor(tvix/castore): move src/fs/test into fuse modFlorian Klink3-8/+9
These tests only interact with the FUSE layer, and import super::fuse to do its work. However, this only works if the `fuse` feature is enabled, which we don't do if we enable the `virtiofs` feature only, causing the tests to fail: ``` ❯ cargo test --no-default-features --features virtiofs Compiling tvix-castore v0.1.0 (/home/flokli/dev/nixos/code.tvl.fyi-submit2/tvix/castore) error[E0432]: unresolved import `super::fuse` --> castore/src/fs/tests.rs:14:13 | 14 | use super::{fuse::FuseDaemon, TvixStoreFs}; | ^^^^ could not find `fuse` in `super` ``` We move src/fs/tests.rs to src/fs/fuse/tests.rs (and src/fs/fuse.rs to src/fs/fuse/mod.rs) to better structure this, which will automatically cause both tests and code to only be built if we have the `fuse` feature enabled. Change-Id: I8fbbad3e4457e326bdfd171aa5c43d25d3187b5b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11715 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-20 r/8158 refactor(tvix/castore): extract concurrent blob uploaderConnor Brewster3-97/+190
The archive ingester has a mechanism for concurrently uploading small blobs to the blob service in order to hide round trip latency with the blob service when ingesting many small blobs. Other ingestion sources like NARs also need a similar mechanism, this extracts the concurrent blob uploading mechanism into its own struct to make it more reusable. Change-Id: I05020419ff4b9ad5829fbfb5cd08d36db983b8c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11693 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-16 r/8153 fix(castore/directory/objectstore): fix responses for deduplicated dirsYureka1-7/+13
Using remove_node messed up the extraction of nodes from the graph. Use into_nodes_edges() instead, to remove the nodes without cloning. Change-Id: Id76c7935d082d6f26192cc3cd490483594f1d1e2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11684 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-05-16 r/8150 feat(tvix/store): add ObjectStoreDirectoryServiceYureka5-9/+323
Change-Id: I1636012be2e8ee3ae64f7bc62fd28bfe0cb2bca5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11668 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-13 r/8143 fix(tvix/castore/grpc): don't use explicit channelFlorian Klink1-47/+36
We can just use the `BoxStream` directly, or a `once` with the single `Directory`. In the recursive case, we also did not properly close the channel after the first error. Change-Id: Ifad56d307fc7861107b6d3cffd28d35631d526e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11635 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-05-13 r/8142 feat(tvix/castore/directory/get_recursive): add 'static to BoxStreamFlorian Klink5-6/+6
Necessary to directly use this in the GRPC DirectoryService wrapper directly. Change-Id: Ic6a0038a40dc30071d145af5035345fcd93288ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/11634 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 r/8141 feat(tvix/castore/directory/traverse_directory): simplifyFlorian Klink1-48/+43
Use try_stream! rather than stream!, and a bit more map_err and ok_err to make things a bit more concise. Once we have proper error types here, and impl Froms, a lot of the error mapping would disappear entirely. Change-Id: I5240a6b0ff7818b94c151322774242b2c142e33b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11633 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-13 r/8140 feat(tvix/castore/directory/traverse): simplify codeFlorian Klink1-53/+42
Replace the loop manually driving the iterator with a for … in, and some of the match with ok_or_else. Change-Id: I6d7b3ef1bf1c7aa128bd6adef09390b54f79479e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11632 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8115 refactor(tvix/castore/error): drop From<PoisonError<T>> for ErrorFlorian Klink1-7/+0
We don't produce these erorrs anymore, no need to provide a conversion to it. Change-Id: I37933e436ad15c5d90b3ac270c4ef5742980513d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11614 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8114 refactor(tvix/castore/directoryservice/memory): use tokio RwLockFlorian Klink1-3/+4
We don't want to block here, and this also means there's no poisoning to deal with. Change-Id: Ic375571970c48beace0005ae2c012135086a4d67 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8113 refactor(tvix/castore/blobservice/memory): use parking_lot RwLockFlorian Klink1-19/+9
This one doesn't require us to deal with poisoning, is upgradeable and the right thing to use when locking access to data, not IO resources. Change-Id: I78634953a73404500d28f51f1d93a87e215c8149 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8101 fix(tvix/castore/directory/sled): use spawn_blockingFlorian Klink1-39/+60
This does IO, which might take a longer amount of time than what we want to be blocking the normal executor. Use spawn_blocking instead. I didn't add it for the constructors, as we only call these once. Change-Id: I96231fcff8d10abe90cafde25a099a2db6ea9414 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11617 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8100 chore(tvix/castore/blob): remove sled implementationFlorian Klink4-195/+1
This never did any chunking, and sled (rightfully) performs really bad if values get too large. We switched the default to using the objectstore backend with the local filesystem a while ago, no need to keep this footgun around anymore. Change-Id: I2c12672f2ea6a22e40d0cbf9161560baddd73d4a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11616 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 r/8079 test(tvix-castore/import): add tests for ingest_entriesFlorian Klink1-3/+138
Change-Id: Ia7906533868fd948509419e0d64b64582575a7fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11591 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 r/8078 fix(tvix/castore/import): assert end of streamFlorian Klink1-0/+5
Once we break out with the root node, there may be no more elements in the stream. Change-Id: I6f5fc5662095aa2b2a56bcad506d25520d9ad00c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11592 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-06 r/8077 fix(tvix/castore/import): deal with entry.path() not having a parentFlorian Klink1-7/+9
We got away with not properly dealing with this for the archive case, where everything is contained inside a toplevel dir, but NARs can encode a single file/symlink. Properly break if the IngestionEntry path has the ROOT as parent, and only create filling directories in the other case. Change-Id: Ib378d0d1040de7c3fe310912a0b0488c55afee83 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11590 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-27T02·05+0000