depot - monorepo for the virus lounge

Age	Commit message (Collapse)	Author	Files	Lines
2024-01-20	r/7431 refactor(tvix/castore): break down `ingest_path`	Ryan Lahfa	1	-70/+187
	In one function that does the heavy lifting: `ingest_entries`, and three additional helpers: - `walk_path_for_ingestion` which perform the tree walking in a very naive way and can be replaced by the user - `leveled_entries_to_stream` which transforms a list of a list of entries ordered by their depth in the tree to a stream of entries in the bottom to top order (Merkle-compatible order I will say in the future). - `ingest_path` which calls the previous functions. Change-Id: I724b972d3c5bffc033f03363255eae448f017cef Reviewed-on: https://cl.tvl.fyi/c/depot/+/10573 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: raitobezarius <tvl@lahfa.xyz>
2024-01-20	r/7430 feat(tvix/castore): ingestion does DFS and invert it	Ryan Lahfa	1	-49/+69
	To make use of the filtering feature, we need to revert the internal walker to a real DFS. We will therefore just invert the whole tree by storing all of its contents in a level-keyed vector. This is horribly expensive in memory, this is a compromise between CPU and memory, here is the fundamental reason for why: When you encounter a directory, it's either a leaf or not, i.e. it contains subdirectories or not. To know this fact, you can: - wait until you notice subdirectories under it, i.e. you need to store any intermediate nodes you see in the meantime -> memory penalty. - getdents or readdir on it to determine NOW its subdirectories -> CPU penalty and I/O penalty. This is an implementation of the first proposal, we pay memory. In practice, we are paying O(#nb of nodes) in memory. There's a smarter albeit much more complicated algorithm that pays only O(\sum_i #siblings(p_i)) nodes where (p_1, ..., p_n) is the path to a leaf. which means for: A / \ B C / / \ D E F We would never store D, E, F but only E, F at a given time. But we would still store B, C no matter what. Change-Id: I456ed1c3f0db493e018ba1182665d84bebe29c11 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10567 Tested-by: BuildkiteCI Autosubmit: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de>
2024-01-20	r/7429 fix(nix-compat): accept SRI hashes of invalid length	Ryan Lahfa	1	-2/+36
	In cl/10468, we accepted SRI hashes of invalid padding while checking their trailing bits. In this commit, we accept SRI hashes of invalid padding and invalid length, as Nix does. Real world example: `pkgs.javaPackages.openjfx11.deps` <https://github.com/NixOS/nixpkgs/blob/849e4dc5ff0ae70fb7a0df19dbbf633e408c0f32/pkgs/development/compilers/openjdk/openjfx/11.nix#L71> in nixpkgs. Change-Id: I834437e7b94dab9fbb030163f7a2741f52bbf03a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10668 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-20	r/7428