diff options
| author | Florian Klink <flokli@flokli.de> | 2023-03-01T17·30+0100 |
|---|---|---|
| committer | flokli <flokli@flokli.de> | 2023-03-10T11·53+0000 |
| commit | 510927e43a950e727c31e100f1e88f0d8a80b6b9 (patch) | |
| tree | 918fcd3fcee62b0bafc9637c8116faa8680eb935 /tvix/store/src/import.rs | |
| parent | 2ef60282b61a61496e642021f0ab8eab7569bbaa (diff) | |
feat(tvix/store): use rayon to upload chunks concurrently r/5939
Look at the data that's written to us, and upload all chunks but the rest in parallel, using rayon. This required moving `upload_chunk` outside the struct, and accepting a ChunkService to use for upload (which it was previously getting from `self.chunk_service`). This doesn't speed up things too much for now, because things are still mostly linear. Change-Id: Id785b5705c3392214d2da1a5b6a182bcf5048c8d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8195 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Diffstat (limited to 'tvix/store/src/import.rs')
| -rw-r--r-- | tvix/store/src/import.rs | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/tvix/store/src/import.rs b/tvix/store/src/import.rs index e71798a6eb4c..5449634cc9f5 100644 --- a/tvix/store/src/import.rs +++ b/tvix/store/src/import.rs @@ -58,7 +58,7 @@ impl From<super::Error> for Error { // // It assumes the caller adds returned nodes to the directories it assembles. #[instrument(skip_all, fields(entry.file_type=?&entry.file_type(),entry.path=?entry.path()))] -fn process_entry<BS: BlobService, CS: ChunkService, DS: DirectoryService>( +fn process_entry<BS: BlobService, CS: ChunkService + std::marker::Sync, DS: DirectoryService>( blob_service: &mut BS, chunk_service: &mut CS, directory_service: &mut DS, @@ -167,7 +167,7 @@ fn process_entry<BS: BlobService, CS: ChunkService, DS: DirectoryService>( #[instrument(skip(blob_service, chunk_service, directory_service), fields(path=?p))] pub fn import_path< BS: BlobService, - CS: ChunkService, + CS: ChunkService + std::marker::Sync, DS: DirectoryService, P: AsRef<Path> + Debug, >( |