about summary refs log tree commit diff
path: root/tvix/store/src/proto/grpc_blobservice_wrapper.rs
diff options
context:
space:
mode:
authorFlorian Klink <flokli@flokli.de>2023-02-17T17·15+0100
committerflokli <flokli@flokli.de>2023-03-10T10·58+0000
commita40d2dcdcd453e44d53d44b4c1471f3b503c7cd6 (patch)
tree66d63a4ae80b107227deaef191ce148efd775109 /tvix/store/src/proto/grpc_blobservice_wrapper.rs
parentfbabcf0420bcc913971d529c11d58f6f888f1002 (diff)
feat(tvix/store): bump fastcdc, use v2020 version r/5925
This switches away from the less canonical "ronomon" version to the
implementation as described in the
[paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et
al., in 2020.

That version uses 64-bit hash values and tends to be faster than both
the ronomon and v2016 versions, and produces the same chunking as the
2016 version.

As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's
the recommended choice.

The crate also gained support for streaming version of chunkers:
https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be
useful.

Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Diffstat (limited to 'tvix/store/src/proto/grpc_blobservice_wrapper.rs')
-rw-r--r--tvix/store/src/proto/grpc_blobservice_wrapper.rs2
1 files changed, 1 insertions, 1 deletions
diff --git a/tvix/store/src/proto/grpc_blobservice_wrapper.rs b/tvix/store/src/proto/grpc_blobservice_wrapper.rs
index 6204fc14005b..2516b5d3f933 100644
--- a/tvix/store/src/proto/grpc_blobservice_wrapper.rs
+++ b/tvix/store/src/proto/grpc_blobservice_wrapper.rs
@@ -157,7 +157,7 @@ impl<
 
         // initialize a new chunker
         // TODO: play with chunking sizes
-        let chunker = fastcdc::FastCDC::new(
+        let chunker = fastcdc::v2020::FastCDC::new(
             &blob_contents,
             64 * 1024 / 4, // min
             64 * 1024,     // avg