diff options
| author | Florian Klink <flokli@flokli.de> | 2024-04-20T16·32+0300 |
|---|---|---|
| committer | clbot <clbot@tvl.fyi> | 2024-04-20T18·54+0000 |
| commit | 5fc403587ffb38635e767697f1c18e7e3559aea6 (patch) | |
| tree | 8779c1714af0145ca580de482a41e63821cd59f2 /tvix/store/src/bin/tvix-store.rs | |
| parent | 01239a4f6f871733231c01d6126c3ffedcc504b7 (diff) | |
refactor(tvix/castore): ingest filesystem entries in parallel r/7987
Rather than carrying around an Future in the IngestionEntry::Regular, simply carry the plain B3Digest. Code reading through a non-seekable data stream has no choice but to read and upload blobs immediately, and code seeking through something seekable (like a filesystem) probably knows better what concurrency to pick when ingesting, rather than the consuming side. (Our only) one of these seekable source implementations is now doing exactly that. We produce a stream of futures, and then use [StreamExt::buffered] to process more than one, concurrently. We still keep the same order, to avoid shuffling things and violating the stream order. This also cleans up walk_path_for_ingestion in castore/import, as well as ingest_dir_entries in glue/tvix_store_io. Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
Diffstat (limited to 'tvix/store/src/bin/tvix-store.rs')
0 files changed, 0 insertions, 0 deletions