From c92ef2df64f4013e72037cefb548f68d158488cc Mon Sep 17 00:00:00 2001 From: Florian Klink Date: Wed, 20 Mar 2024 20:21:51 +0200 Subject: feat(tvix/castore/directory): add ClosureValidator This can be used to validate a Directory closure (connected DAG of Directories), and their insertion order. Directories need to be inserted (via `add`), in an order from the leaves to the root. During insertion, we validate as much as we can at that time: - individual validation of Directory messages - validation of insertion order (no upload of not-yet-known Directories) - validation of size fields of referred Directories Internally it keeps all received Directories (and their sizes) in a HashMap, keyed by digest. Once all Directories have been inserted, a drain() function can be called to get a (deduplicated and) validated list of directories, in from-leaves-to-root order (to be stored somewhere). While assembling that list, a check for graph connectivity is performed too, to ensure there's no separate components being sent (and only one root). It adds a test suite for these cases, which is much nicer to test than where we previously had these checks (only in the gRPC server wrapper). Followup CLs will move the existing putters to use this. Change-Id: Ie88c832924c170a24626e9e3e91d868497b5d7a4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11220 Tested-by: BuildkiteCI Reviewed-by: raitobezarius Autosubmit: flokli --- tvix/castore/src/directoryservice/mod.rs | 2 ++ 1 file changed, 2 insertions(+) (limited to 'tvix/castore/src/directoryservice/mod.rs') diff --git a/tvix/castore/src/directoryservice/mod.rs b/tvix/castore/src/directoryservice/mod.rs index ca8e60e87304..b6d1c2fcab13 100644 --- a/tvix/castore/src/directoryservice/mod.rs +++ b/tvix/castore/src/directoryservice/mod.rs @@ -2,6 +2,7 @@ use crate::{proto, B3Digest, Error}; use futures::stream::BoxStream; use tonic::async_trait; +mod closure_validator; mod from_addr; mod grpc; mod memory; @@ -9,6 +10,7 @@ mod sled; mod traverse; mod utils; +pub use self::closure_validator::ClosureValidator; pub use self::from_addr::from_addr; pub use self::grpc::GRPCDirectoryService; pub use self::memory::MemoryDirectoryService; -- cgit 1.4.1