about summary refs log tree commit diff
path: root/tvix/tools
AgeCommit message (Collapse)AuthorFilesLines
2024-07-09 r/8364 chore(tvix/tools/weave): Regenerate Cargo.nix and filter srcIlan Joselevich3-42/+101
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: I79b062eddb13f59c4656a62735df1e527fc91a44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11970 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 r/8363 chore(tvix/tools/turbofetch): Regenerate Cargo.nix and filter srcIlan Joselevich3-37/+28
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: I48cabe514e0cf1dd1f00870bcb7958a2a49663d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11969 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-09 r/8362 chore(tvix/tools/crunch-v2): Regenerate Cargo.nix and filter srcIlan Joselevich3-68/+104
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: I90245bad27f6fee01455557bbccc865301a4f1a0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11968 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 r/8361 chore(tvix/tools/narinfo2parquet): Regenerate Cargo.nix and filter srcIlan Joselevich3-38/+31
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: Id5225e73ec5894bc324f363660472a7efdd803cf Reviewed-on: https://cl.tvl.fyi/c/depot/+/11967 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-01 r/8331 chore(tvix): bump to data-encoding 2.6.0Florian Klink1-1/+1
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-05-11 r/8108 chore(tvix/narinfo2parquet): bump dependenciesFlorian Klink2-27/+135
Get rid of the RUSTSEC warnings. Also upgrade tokio to drop the Sized requirement, similar to cl/11608. Change-Id: Idc6fece23c79eb30fd1dfc5fe64fa2e4c08cd412 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11624 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-08 r/8095 refactor(nix-compat/nar/reader): reuse prev_name allocationedef1-1/+1
We reuse the prev_name allocation for Entry, instead of allocating and returning a separate Vec. We encode the `prev_name: None` case as an empty vector, since we don't allow empty names anyway, and the sorting is equivalent. Change-Id: I975b37ff873805f5ff099bc82128706891052247 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607 Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-08 r/8089 chore(tvix/tools/crunch-v2): upgrade tokioedef2-5/+20
This drops the Sized requirement on AsyncReadExt. Change-Id: I2a89c708922ca528cb493aefd0613f477f648e83 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11608 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-03-03 r/7644 fix(tvix/tools/weave): handle sliced arrays correctlyedef1-8/+12
The start/end offsets are not necessarily coterminous with the underlying values array, so even if the stride is fixed, we still we need to slice the chunks down to match the start/end offsets. This bug shouldn't affect the correctness of any existing code, since we're always working with unsliced arrays read directly from Parquet. Change-Id: I2f7ddc4e66d4d3b2317a44bd436a35bff36bac79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11081 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-27 r/7617 feat(tvix/tools/weave): initedef9-0/+11471
Scalable tracing GC for the cache.nixos.org dataset. Change-Id: I6c7852796f28e1a1c7607384ffb55f44407e1185 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10765 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-19 r/7555 chore(tvix/tools/narinfo2parquet): polars 0.34.2 -> 0.36.2edef4-704/+1419
Change-Id: Ib17a7bbe560000c3fef3ba6182819f710d707909 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10967 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-07 r/7481 chore(tvix/tools/turbofetch): bump magic-buffer to 0.1.1Florian Klink4-16/+25
This contains https://github.com/sklose/magic-buffer/pull/4, so we don't have to impl Send ourselves. Change-Id: If046596e13345ad4fec22209440e65859e44d540 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10748 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-01-27 r/7453 feat(tvix/tools/crunch-v2): add CLI argsFlorian Klink6-33/+884
Use clap derive to make the input and output files configurable, as well as the chunk size parameters. Change-Id: I02b29126f3bd2c13ba2c6e7e0aa4ff048ff803ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/10691 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-01-27 r/7452 feat(tvix/tools/crunch-v2): initedef12-0/+15022
This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format. Currently, produced chunks are not preserved, and this purely serves as a way of measuring compression/deduplication ratios for various chunking and compression parameters. Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-12-24 r/7262 chore(tvix/turbofetch): switch to futures 0.3.30Florian Klink3-79/+79
The bugs have been fixed, https://github.com/rust-lang/futures-rs/pull/2801 and https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended up in that release. Change-Id: I301c0ffc951f04a5b3b7267e922771c837a3f5a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10415 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-27 r/7076 chore(3p/sources): bump channels & overlays (2023-11-27)Vincent Ambo1-2/+2
* picked avrdude from stable channel * removed override for texlive, as the upstream fix is merged * picked awscli2 from stable channel * bump tdlib to 1.8.21 (new minimum for telega.el) * tvix/turbofetch: switch to nixpkgs-native mechanism for CARGO_MANIFEST_LINKS (whatever that is) Change-Id: Ic695721b5ca750b89d21cab7a257e1db682b23c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10083 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-11-19 r/7042 feat(tvix/nix-compat): add narinfo::Signature::verifyFlorian Klink2-41/+588
This adds support for verifying signatures found in NARInfo files. This still needs to be hooked together with the nix+http[s] backend. Change-Id: Ic1c8ddbdecfb05cefca2492808388b0f7f3f2637 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10081 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-10 r/6988 feat(tvix/tools/narinfo2parquet): initedef6-0/+9703
Convert turbofetch output to queryable Parquet. Change-Id: I076f5a431f8aab8cfe7d973bdc9fe019cebde111 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9989 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6979 feat(tvix/tools/turbofetch): initedef9-0/+8624
Change-Id: I2efa6f94f57e812c52371256a4e62d1d54ff5057 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9925 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-12T04·44+0000