about summary refs log tree commit diff
path: root/tvix/castore
AgeCommit message (Collapse)AuthorFilesLines
2024-06-08 r/8227 docs(tvix): fix some typos across various documentsBen Webb3-6/+6
Fix some typos found while reading various documents, mostly those relating to the castore. Here is a summary of the edits. - fix broken link between documents in the store and castore directories - clarify expression in castore's data model document that indicates that the *name* of each child node of a directory must be unique across all three lists of children - add missing closing parenthesis in castore's data model document - replace "how" with "what" in the phrase "unclear how a ... would even look like" in castore's why-not-git-trees document - remove unnecessary articles in castore's blobstore chunking document - add missing "y" to "optionall" in eval's compilation of bindings document Change-Id: I1997ea91bb4e9c40abcd81e0cde9405968580ba6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11763 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-06 r/8220 feat(tvix/store/bin): add progress bar infrastructureFlorian Klink3-2/+21
This adds the tracing-indicatif crate, and configures it as a layer in our tracing_subscriber pipeline to emit progress for every span that's configured so. It also moves from using std::io::stderr to write logs to using their writer, to avoid clobbering output. Progress bar styles are defined in a lazy_static, moving this into a general tracing is left for later. This adds some usage of this to the `imports` and `copy` commands. The output can still be improved a bit - we should probably split each task up into a smaller (instrumented) helper functions, so we can create a progress bar for each task. Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-06-03 r/8211 feat(tvix): add CI targets for checking crate features powersetIlan Joselevich2-15/+20
Closes: https://b.tvl.fyi/issues/401 With this change all crate features (and their combinations) will be built and tested in CI. From now on, when adding/removing a Cargo feature for a crate, you will want to add it to the features power set that gets tested in CI. For each crate there's a default.nix with a `mkFeaturePowerset` invocation, modify the list to include/remove the feature. Note that you don't want to add "collection" features, such as `fs` for tvix-[ca]store or `default`. Change-Id: I966dde1413d057770787da3296cce9c1924570e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11717 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-26 r/8172 refactor(tvix/castore): move src/fs/test into fuse modFlorian Klink3-8/+9
These tests only interact with the FUSE layer, and import super::fuse to do its work. However, this only works if the `fuse` feature is enabled, which we don't do if we enable the `virtiofs` feature only, causing the tests to fail: ``` ❯ cargo test --no-default-features --features virtiofs Compiling tvix-castore v0.1.0 (/home/flokli/dev/nixos/code.tvl.fyi-submit2/tvix/castore) error[E0432]: unresolved import `super::fuse` --> castore/src/fs/tests.rs:14:13 | 14 | use super::{fuse::FuseDaemon, TvixStoreFs}; | ^^^^ could not find `fuse` in `super` ``` We move src/fs/tests.rs to src/fs/fuse/tests.rs (and src/fs/fuse.rs to src/fs/fuse/mod.rs) to better structure this, which will automatically cause both tests and code to only be built if we have the `fuse` feature enabled. Change-Id: I8fbbad3e4457e326bdfd171aa5c43d25d3187b5b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11715 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-20 r/8158 refactor(tvix/castore): extract concurrent blob uploaderConnor Brewster3-97/+190
The archive ingester has a mechanism for concurrently uploading small blobs to the blob service in order to hide round trip latency with the blob service when ingesting many small blobs. Other ingestion sources like NARs also need a similar mechanism, this extracts the concurrent blob uploading mechanism into its own struct to make it more reusable. Change-Id: I05020419ff4b9ad5829fbfb5cd08d36db983b8c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11693 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-16 r/8153 fix(castore/directory/objectstore): fix responses for deduplicated dirsYureka1-7/+13
Using remove_node messed up the extraction of nodes from the graph. Use into_nodes_edges() instead, to remove the nodes without cloning. Change-Id: Id76c7935d082d6f26192cc3cd490483594f1d1e2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11684 Tested-by: BuildkiteCI Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de>
2024-05-16 r/8150 feat(tvix/store): add ObjectStoreDirectoryServiceYureka6-10/+325
Change-Id: I1636012be2e8ee3ae64f7bc62fd28bfe0cb2bca5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11668 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-13 r/8143 fix(tvix/castore/grpc): don't use explicit channelFlorian Klink1-47/+36
We can just use the `BoxStream` directly, or a `once` with the single `Directory`. In the recursive case, we also did not properly close the channel after the first error. Change-Id: Ifad56d307fc7861107b6d3cffd28d35631d526e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11635 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-05-13 r/8142 feat(tvix/castore/directory/get_recursive): add 'static to BoxStreamFlorian Klink5-6/+6
Necessary to directly use this in the GRPC DirectoryService wrapper directly. Change-Id: Ic6a0038a40dc30071d145af5035345fcd93288ae Reviewed-on: https://cl.tvl.fyi/c/depot/+/11634 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 r/8141 feat(tvix/castore/directory/traverse_directory): simplifyFlorian Klink1-48/+43
Use try_stream! rather than stream!, and a bit more map_err and ok_err to make things a bit more concise. Once we have proper error types here, and impl Froms, a lot of the error mapping would disappear entirely. Change-Id: I5240a6b0ff7818b94c151322774242b2c142e33b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11633 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-13 r/8140 feat(tvix/castore/directory/traverse): simplify codeFlorian Klink1-53/+42
Replace the loop manually driving the iterator with a for … in, and some of the match with ok_or_else. Change-Id: I6d7b3ef1bf1c7aa128bd6adef09390b54f79479e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11632 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 r/8139 chore(tvix/clippy): configure clippy allow block_in_conditionsFlorian Klink1-0/+3
… in Cargo.toml. This gets an imperative `cargo clippy` run to pick up that config, so `-A clippy::blocks_in_conditions` doesn't need to be explicitly specified anymore. Change-Id: I32b6cc50c77c22cba0d816d0db508c2f94b2c383 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11659 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-05-11 r/8115 refactor(tvix/castore/error): drop From<PoisonError<T>> for ErrorFlorian Klink1-7/+0
We don't produce these erorrs anymore, no need to provide a conversion to it. Change-Id: I37933e436ad15c5d90b3ac270c4ef5742980513d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11614 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8114 refactor(tvix/castore/directoryservice/memory): use tokio RwLockFlorian Klink1-3/+4
We don't want to block here, and this also means there's no poisoning to deal with. Change-Id: Ic375571970c48beace0005ae2c012135086a4d67 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8113 refactor(tvix/castore/blobservice/memory): use parking_lot RwLockFlorian Klink1-19/+9
This one doesn't require us to deal with poisoning, is upgradeable and the right thing to use when locking access to data, not IO resources. Change-Id: I78634953a73404500d28f51f1d93a87e215c8149 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 r/8101 fix(tvix/castore/directory/sled): use spawn_blockingFlorian Klink1-39/+60
This does IO, which might take a longer amount of time than what we want to be blocking the normal executor. Use spawn_blocking instead. I didn't add it for the constructors, as we only call these once. Change-Id: I96231fcff8d10abe90cafde25a099a2db6ea9414 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11617 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-11 r/8100 chore(tvix/castore/blob): remove sled implementationFlorian Klink4-195/+1
This never did any chunking, and sled (rightfully) performs really bad if values get too large. We switched the default to using the objectstore backend with the local filesystem a while ago, no need to keep this footgun around anymore. Change-Id: I2c12672f2ea6a22e40d0cbf9161560baddd73d4a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11616 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 r/8079 test(tvix-castore/import): add tests for ingest_entriesFlorian Klink1-3/+138
Change-Id: Ia7906533868fd948509419e0d64b64582575a7fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11591 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 r/8078 fix(tvix/castore/import): assert end of streamFlorian Klink1-0/+5
Once we break out with the root node, there may be no more elements in the stream. Change-Id: I6f5fc5662095aa2b2a56bcad506d25520d9ad00c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11592 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-06 r/8077 fix(tvix/castore/import): deal with entry.path() not having a parentFlorian Klink1-7/+9
We got away with not properly dealing with this for the archive case, where everything is contained inside a toplevel dir, but NARs can encode a single file/symlink. Properly break if the IngestionEntry path has the ROOT as parent, and only create filling directories in the other case. Change-Id: Ib378d0d1040de7c3fe310912a0b0488c55afee83 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11590 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-05 r/8076 feat(tvix-castore/import) have IngestionEntry.path() return &PathFlorian Klink2-3/+4
There's no need for this to be a &PathBuf. Change-Id: I2d4126d57cfd8ddaad5dd327943b70b83d45c749 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11589 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-04 r/8073 refactor(tvix/*store): use DS: DirectoryServiceFlorian Klink3-4/+4
We implement DirectoryService for Arc<DirectoryService> and Box<DirectoryService>, this is sufficient. Change-Id: I0a5a81cbc4782764406b5bca57f908ace6090737 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11586 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-03 r/8070 test(tvix/[ca]store): enable bigtable only with "integration" featureFlorian Klink4-7/+22
The emulator and bigtable client are quite big. Remove them from the default //tvix:shell. Put the tests behind a `integration` feature flag, and add a variant with that enabled to CI, and drop the bigtable tools from //tvix:shell. Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-02 r/8067 refactor(tvix/castore/import): use crate Path[Buf] in IngestionEntryFlorian Klink4-69/+74
This explicitly splits ingestion-method-specific path types from the castore types. Change-Id: Ia3b16105fadb8d52927a4ed79dc4b34efdf4311b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11563 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-02 r/8066 feat(tvix/castore/directory/traverse): use castore PathsFlorian Klink1-52/+14
This switches from using std::path::Path to using castore paths. We can drop some error handling in descend_to, as absolute (or redundant) paths are not representable. We however now need to convert from a std::path::Path to our representation, and decide to accept .. canonicalization, as paths in EvalIO might contain this. Dealing .. to hop into another store path, if we encounter this, should be dealt with in a previous step. Change-Id: I5e94693808420c5d56587c68731252b54755bf93 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11575 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-05-02 r/8065 feat(tvix/castore/path): implement PathBuf::from_host_pathFlorian Klink1-1/+105
This allows converting from std::path::Path to castore PathBufs. A flag is present to control .. canonicalization, and the usual caveats about platform-specific differences apply. Currently only added for unix, we'll carefully consider other platforms on a case-by-case basis. Change-Id: If289a92f75a2e5c3eec132b6a91a28d225fc1989 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11577 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-01 r/8063 feat(tvix/castore/path): impl AsRef<Path> for PathFlorian Klink1-0/+6
This allows using both Path and PathBuf in a function argument taking `impl AsRef<Path>`. Change-Id: Ibd3ba6fac538069d2fe729d1ef399fdef301668f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11574 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-01 r/8062 feat(tvix/castore): expose PathBuf tooFlorian Klink1-1/+1
Change-Id: I1538725766a73436eba9a92dcb4f3275ace2fa4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11573 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-01 r/8061 fix(tvix/castore/path): join, push -> try_{join,push}edef1-7/+7
These are fallible methods, so they should be named accordingly. Change-Id: I6dc271c42989dd6500173488190f65381835d6fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/11572 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8060 feat(tvix/castore/path): add PathBuf::{new, with_capacity, push}edef1-12/+37
Change-Id: Ia64f4bda80e91adbdb804f4f26cef5ace8f5406a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11571 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8059 feat(tvix/castore/path): more conversionsedef1-2/+37
Change-Id: I3ee510b444848316df520dc8ca445d0f3c7d607f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11567 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8058 feat(tvix/castore/path): single-component paths are children of ROOTedef1-11/+21
The empty path (Path::ROOT) is explicitly a valid path, and "foo" is simply a child of "". The root itself is the only path without a parent. Change-Id: Iff00dc8aed89eaf98702b664c0df658bd5a1d88a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11569 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-01 r/8057 test(tvix/castore/path): test the null byte caseedef1-0/+1
Change-Id: Ia4ceaf56f6cb4d8ed1ad935c77b3898d711da73a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11568 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8056 feat(tvix/castore/path): use proto::validate_node_nameedef2-6/+13
Use the shared code for validating node names, since that is what path components represent. Change-Id: I12109c1306b224718faa66cf1f2874c78c1436a7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11566 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8055 chore(tvix/castore/path): drop now-duplicate testsedef1-24/+1
Since PathBuf doesn't have inherent methods anymore, these just forward to Path itself. Change-Id: I30f44adc9994337c367bad985ada0e8fcb98dd6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11570 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8052 feat(tvix/castore/path): implement Debug + Displayedef1-3/+33
We implement Debug explicitly, so that we don't just see raw integers. Change-Id: I11213094728f3e0c674562ee71c092a950041632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11565 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-01 r/8051 feat(tvix/castore/path): make Path unsized, repr-equivalent to [u8]edef1-51/+57
Change-Id: Ia1e6a10dba0bcda44e06e4cab7d4e52ca88b5859 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11564 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 r/8050 feat(tvix/castore/path): add Path{Buf}Florian Klink2-0/+248
This contains Path and PathBuf, representing platform-independent paths representable by the castore model. These are always relative, and platform-independent, which distinguishes them from the ones provided in the standard library. A subsequent CL will move IngestionEntry (and more) to use them. Change-Id: Ib85857f4159ebc2f3c00192c95d4e5b54ffd4fcf Reviewed-on: https://cl.tvl.fyi/c/depot/+/11558 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-04-30 r/8048 refactor(tvix/castore/import): restructure error typesFlorian Klink4-65/+119
Have ingest_entries return an Error type with only three kinds: - Error while uploading a specific Directory - Error while finalizing the directory upload - Error from the producer Move all ingestion method-specific errors to the individual implementations. Change-Id: I2a015cb7ebc96d084cbe2b809f40d1b53a15daf3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11557 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-30 r/8047 refactor(tvix/castore): remove IngestionEntry::UnknownFlorian Klink2-10/+1
We shouldn't try to represent non-representable things in the ingestion entries (only to throw an error). It's cleaner to throw the error directly in the part producing the stream. Change-Id: I6b6f6d8c2f677425210142a39f1829ddeefec812 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11556 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: firefly <firefly@firefly.nu>
2024-04-30 r/8046 refactor(tvix/castore/import): move upload_blob_at_path into fs modFlorian Klink2-28/+27
This is only useful for when we have access to a filesystem, so it shouldn't be in the root. Change-Id: I9923aaed1aef9d3a1e8fad41f58821d51c2eb34b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11555 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: firefly <firefly@firefly.nu> Tested-by: BuildkiteCI
2024-04-30 r/8045 fix(tvix/castore/import): symlink targets are Vec<u8>Florian Klink3-4/+9
These can be arbitrary bytes in theory. Some of our libraries might be more strict, or inconsistent w.r.t. their representation of path separators. Change-Id: I7981b74fc7d3dd79f5589cf2ef52ced7b71dd003 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11551 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-04-30 r/8044 docs(tvix/castore): fix tvix_castore::import sub-mod docstringsFlorian Klink2-2/+4
The one for `fs` was wrong, and ended up being attached to ingest_path, and the one for `archive` was missing entirely. Change-Id: I8a4c32fb5293badb1ea0764c278a88e4ca33c018 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11552 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-04-30 r/8038 test(tvix): Fix tvix tests on macOSAlice Carroll1-6/+8
Prior to this, some tests would not build or would fail in an obscure way. Change-Id: I68587cc7592492ebfd71ca02fc7ccc9ff7c0196f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11544 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-28 r/8025 chore(3p/sources): Bump channels & overlayssterni1-0/+1
- //tvix: address new clippy lints - //users/tazjin: Satisfy gonic module's new need for a playlist folder. - //users/aspen/games: adjust for changed location of df's default init.txt and d_init.txt. Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-04-25 r/8008 fix(tvix): typoedef1-1/+1
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-24 r/8002 refactor(tvix/castore): add separate Error enum for archivesConnor Brewster2-33/+34
The `Error` enum for the `imports` crate has both filesystem and archive specific errors and was starting to get messy. This adds a separate `Error` enum for archive-specific errors and then keeps a single `Archive` variant in the top-level import `Error` for all archive errors. Change-Id: I4cd0746c864e5ec50b1aa68c0630ef9cd05176c7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11498 Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com> Reviewed-by: flokli <flokli@flokli.de>
2024-04-23 r/8001 feat(tvix/castore): upload blobs concurrently when ingesting archivesConnor Brewster1-10/+90
Ingesting tarballs with a lot of small files is very slow because of the round trip time to the `BlobService`. To mitigate this, small blobs can be buffered into memory and uploaded concurrently in the background. Change-Id: I3376d11bb941ae35377a089b96849294c9c139e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11497 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-04-23 r/8000 refactor(tvix/castore): switch to `ingest_entries` for tarball ingestionConnor Brewster3-128/+239
With `ingest_entries` being more generalized, we can now use it for ingesting the directory entries generated from tarballs. Change-Id: Ie1f7a915c456045762e05fcc9af45771f121eb43 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11489 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-20 r/7987 refactor(tvix/castore): ingest filesystem entries in parallelFlorian Klink2-55/+41
Rather than carrying around an Future in the IngestionEntry::Regular, simply carry the plain B3Digest. Code reading through a non-seekable data stream has no choice but to read and upload blobs immediately, and code seeking through something seekable (like a filesystem) probably knows better what concurrency to pick when ingesting, rather than the consuming side. (Our only) one of these seekable source implementations is now doing exactly that. We produce a stream of futures, and then use [StreamExt::buffered] to process more than one, concurrently. We still keep the same order, to avoid shuffling things and violating the stream order. This also cleans up walk_path_for_ingestion in castore/import, as well as ingest_dir_entries in glue/tvix_store_io. Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-09T21·54+0000