about summary refs log tree commit diff
path: root/tvix/store/protos
diff options
context:
space:
mode:
authorFlorian Klink <flokli@flokli.de>2024-04-20T16·32+0300
committerclbot <clbot@tvl.fyi>2024-04-20T18·54+0000
commit5fc403587ffb38635e767697f1c18e7e3559aea6 (patch)
tree8779c1714af0145ca580de482a41e63821cd59f2 /tvix/store/protos
parent01239a4f6f871733231c01d6126c3ffedcc504b7 (diff)
refactor(tvix/castore): ingest filesystem entries in parallel r/7987
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.

Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.

(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.

We still keep the same order, to avoid shuffling things and violating
the stream order.

This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.

Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Diffstat (limited to 'tvix/store/protos')
0 files changed, 0 insertions, 0 deletions