about summary refs log tree commit diff
path: root/tvix/tools/crunch-v2
AgeCommit message (Collapse)AuthorFilesLines
2024-08-25 r/8585 feat(nix-compat): Add NixDeserialize and NixRead traitsBrian Olsen3-7/+14
Add a trait for deserializing a type from a daemon worker connection. This adds the NixDeserialize trait which is kind of like the serde Deserialize trait in that individual types are meant to implement it and it can potentially be derived in the future. The NixDeserialize trait takes something that implements NixRead as input so that you can among other things mock the reader. Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990 Autosubmit: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-24 r/8577 chore(tvix/tools/crunch-v2): bump depsFlorian Klink2-20/+24
h2: - https://rustsec.org/advisories/RUSTSEC-2024-0332.html - https://rustsec.org/advisories/RUSTSEC-2024-0003.html mio: - https://rustsec.org/advisories/RUSTSEC-2024-0019.html shlex: - https://rustsec.org/advisories/RUSTSEC-2024-0006.html zerocopy: - https://rustsec.org/advisories/RUSTSEC-2024-0006.html Change-Id: Ia5f6f96378d5f7e77e178dc054f47bda8b766a30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12340 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-23 r/8568 feat(tvix/tools/crunch-v2): Add crate2nix-checkIlan Joselevich1-1/+3
Change-Id: I922caae2ec3f87a7f3a5dcc8e9d366f0e71ce900 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12300 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-21 r/8547 feat(nix-compat/nix_http): init parse_nar[info]_strFlorian Klink2-1/+53
This moves the URL component parsing code we had in nar-bridge to nix-compat. We change the function signature to return an Option, not a Result<_, StatusCode>. This allows returning more appropriate error codes, as we can ok_or(…) at the callsite, which we now do: on an upload to an invalid path, we now return "unauthorized", while on a GET/HEAD, we return "not found". This also adds support to parse compression suffixes. While not supported in nar-bridge, other users of nix-compat might very well want to parse these paths. Also fix the error message when parsing NAR urls, it mentioned 32, not 52, which is a copypasta error from the narinfo URL parsing code. Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8499 chore(tvix/tools/crunch-v2): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: I11651ea56e52988ee7cef2e02d0cbf61062f5c7b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12213 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-08-16 r/8495 chore(tvix/tools/crunch-v2): update Cargo.{lock,nix}Florian Klink2-46/+245
These were out of date. Change-Id: Ideaf888c2851bb9ec36ae01b11d93165b62cd7a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12209 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-09 r/8362 chore(tvix/tools/crunch-v2): Regenerate Cargo.nix and filter srcIlan Joselevich3-68/+104
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: I90245bad27f6fee01455557bbccc865301a4f1a0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11968 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-08 r/8095 refactor(nix-compat/nar/reader): reuse prev_name allocationedef1-1/+1
We reuse the prev_name allocation for Entry, instead of allocating and returning a separate Vec. We encode the `prev_name: None` case as an empty vector, since we don't allow empty names anyway, and the sorting is equivalent. Change-Id: I975b37ff873805f5ff099bc82128706891052247 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607 Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-08 r/8089 chore(tvix/tools/crunch-v2): upgrade tokioedef2-5/+20
This drops the Sized requirement on AsyncReadExt. Change-Id: I2a89c708922ca528cb493aefd0613f477f648e83 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11608 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-01-27 r/7453 feat(tvix/tools/crunch-v2): add CLI argsFlorian Klink6-33/+884
Use clap derive to make the input and output files configurable, as well as the chunk size parameters. Change-Id: I02b29126f3bd2c13ba2c6e7e0aa4ff048ff803ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/10691 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-01-27 r/7452 feat(tvix/tools/crunch-v2): initedef12-0/+15022
This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format. Currently, produced chunks are not preserved, and this purely serves as a way of measuring compression/deduplication ratios for various chunking and compression parameters. Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>