feat(tvix/store): use rayon to upload chunks concurrently r/5939

Look at the data that's written to us, and upload all chunks but the
rest in parallel, using rayon. This required moving `upload_chunk`
outside the struct, and accepting a ChunkService to use for upload
(which it was previously getting from `self.chunk_service`).

This doesn't speed up things too much for now, because things are still
mostly linear.

Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>

1 files changed, 4 insertions, 0 deletions

diff --git a/tvix/Cargo.nix b/tvix/Cargo.nix
index 20751f596a3f..7167a3cc1370 100644
--- a/tvix/Cargo.nix
+++ b/tvix/Cargo.nix
@@ -8331,6 +8331,10 @@ rec {
             packageId = "prost";
           }
           {
+            name = "rayon";
+            packageId = "rayon";
+          }
+          {
             name = "sha2";
             packageId = "sha2 0.10.6";
           }