about summary refs log tree commit diff
path: root/tvix/tools
AgeCommit message (Collapse)AuthorFilesLines
2024-10-12 r/8801 chore(tvix/tools): move narinfo2parquet to //users/edefVincent Ambo6-11345/+0
This is not a core Tvix tool, it's a tool that uses a Tvix component. Change-Id: I81d2b2374da23489df0097dcabb8295c82652fc1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12606 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2024-10-12 r/8800 chore(tvix/tools): move weave to //users/edefVincent Ambo9-11508/+0
This is not a core Tvix tool, it's a tool that uses a Tvix component. Change-Id: I705f2c4ab87f1512e005007c933e16b84ed4279f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12605 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2024-10-12 r/8799 chore(tvix/tools): move turbofetch to //users/edefVincent Ambo9-9375/+0
This was introduced in cl/9925 without any commit message, but this is clearly not relevant to Tvix itself (it even says so in a comment in Cargo.toml). Change-Id: I84f12d5145c3f53c9df23863f887bad913856c50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12604 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: edef <edef@edef.eu>
2024-10-12 r/8798 chore(tvix/tools): move crunch-v2 to //users/edefVincent Ambo12-15454/+0
This is not a core Tvix tool, it's some sort of one-off analysis thing. Change-Id: I05fcbed45abad27d6b5cfd49db1727249dad3971 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12603 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-10-12 r/8794 chore(tvix/tools/crunch-v2): bump crate dependenciesFlorian Klink2-1974/+1266
Change-Id: Ib67e0121cebf6b2f049490125c7ded9637955928 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12599 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-12 r/8793 chore(tvix/tools/weave): bump crate dependenciesFlorian Klink2-1208/+903
Change-Id: I31e5f3d46321265d662d41aa79aa4178e121818f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12598 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-10-12 r/8792 chore(tvix/tools/turbofetch): bump crate dependenciesFlorian Klink2-685/+1354
Change-Id: I81988571344829708881021527b83e7d99b598c3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12597 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-25 r/8585 feat(nix-compat): Add NixDeserialize and NixRead traitsBrian Olsen7-19/+40
Add a trait for deserializing a type from a daemon worker connection. This adds the NixDeserialize trait which is kind of like the serde Deserialize trait in that individual types are meant to implement it and it can potentially be derived in the future. The NixDeserialize trait takes something that implements NixRead as input so that you can among other things mock the reader. Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990 Autosubmit: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-24 r/8578 chore(tvix/tools/turbofetch): bump depsFlorian Klink2-34/+53
h2: - https://rustsec.org/advisories/RUSTSEC-2024-0332.html - https://rustsec.org/advisories/RUSTSEC-2024-0003.html mio: - https://rustsec.org/advisories/RUSTSEC-2024-0019.html shlex: - https://rustsec.org/advisories/RUSTSEC-2024-0006.html zerocopy: - https://rustsec.org/advisories/RUSTSEC-2024-0006.html Change-Id: I7ce3cab410b9c2b6c28474ca81c3fa5b5283f3fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/12341 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-24 r/8577 chore(tvix/tools/crunch-v2): bump depsFlorian Klink2-20/+24
h2: - https://rustsec.org/advisories/RUSTSEC-2024-0332.html - https://rustsec.org/advisories/RUSTSEC-2024-0003.html mio: - https://rustsec.org/advisories/RUSTSEC-2024-0019.html shlex: - https://rustsec.org/advisories/RUSTSEC-2024-0006.html zerocopy: - https://rustsec.org/advisories/RUSTSEC-2024-0006.html Change-Id: Ia5f6f96378d5f7e77e178dc054f47bda8b766a30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12340 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-24 r/8576 chore(tvix/tools/weave): bump mioFlorian Klink2-4/+4
https: //rustsec.org/advisories/RUSTSEC-2024-0019.html Change-Id: I03543105b35da6407a4930893b1514b379a89fbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12339 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-23 r/8571 feat(tvix/tools/weave): Add crate2nix-checkIlan Joselevich1-1/+3
Change-Id: I4e02f56678e6efa6e2656b2f7398b6ad92581c57 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12303 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 r/8570 feat(tvix/tools/turbofetch): Add crate2nix-checkIlan Joselevich1-1/+3
Change-Id: Ie615d824032c85af6f8f40419bc22d7e3f38bb4c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12302 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 r/8569 feat(tvix/tools/narinfo2parquet): Add crate2nix-checkIlan Joselevich1-1/+3
Change-Id: I22bc6e7711e5e2c0a76d4eea4dd6edaf724e4cef Reviewed-on: https://cl.tvl.fyi/c/depot/+/12301 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 r/8568 feat(tvix/tools/crunch-v2): Add crate2nix-checkIlan Joselevich1-1/+3
Change-Id: I922caae2ec3f87a7f3a5dcc8e9d366f0e71ce900 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12300 Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de>
2024-08-21 r/8547 feat(nix-compat/nix_http): init parse_nar[info]_strFlorian Klink6-1/+305
This moves the URL component parsing code we had in nar-bridge to nix-compat. We change the function signature to return an Option, not a Result<_, StatusCode>. This allows returning more appropriate error codes, as we can ok_or(…) at the callsite, which we now do: on an upload to an invalid path, we now return "unauthorized", while on a GET/HEAD, we return "not found". This also adds support to parse compression suffixes. While not supported in nar-bridge, other users of nix-compat might very well want to parse these paths. Also fix the error message when parsing NAR urls, it mentioned 32, not 52, which is a copypasta error from the narinfo URL parsing code. Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-19 r/8540 feat(nix-compat/narinfo/signature): generalize name fieldFlorian Klink1-1/+1
Requiring `name` to be a `&str` means it'll get annoying to pass around `Signature`, but being able to pass them around in an owned fashion is kinda a requirement for a stronger typed `PathInfo` struct, where we want to have full ownership. Rework the `Signature` struct to become generic over the type of the `name` field. This means, it becomes possible to have owned versions of it. We don't want to impose `String` or `SmolStr` for example, but want to leave it up to the nix-compat user to decide. Provide a type alias for the existing `&str` variant (`SignatureRef`), and use it where we previously used the non-generic `Signature` one. Add some tests to ensure it's possible to *use* `Signature` with both `String` and `SmolStr` (but only pull in `smol_str` as dev dependency for the tests). Also, add some more docstrings, these were a bit sparse. Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-16 r/8499 chore(tvix/tools/crunch-v2): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: I11651ea56e52988ee7cef2e02d0cbf61062f5c7b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12213 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-08-16 r/8498 chore(tvix/tools/narinfo2parquet): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: Iad02bcf312af8b6bf450ab3c4c6de31d9c02ab87 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12212 Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-16 r/8497 chore(tvix/tools/weave): bump curve25519-dalekFlorian Klink2-30/+4
Fix for RUSTSEC-2024-0344. Change-Id: I414c1424941af1352f4d3b4e5908083ab0cf4e9d Reviewed-on: https://cl.tvl.fyi/c/depot/+/12211 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-16 r/8495 chore(tvix/tools/crunch-v2): update Cargo.{lock,nix}Florian Klink2-46/+245
These were out of date. Change-Id: Ideaf888c2851bb9ec36ae01b11d93165b62cd7a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12209 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-15 r/8493 chore(tvix/tools/narinfo2parquet): update Cargo.{lock,nix}Florian Klink2-39/+209
These were out of date. Change-Id: I89df37f088ad6c53b676d965f0dc1023f40481f4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12208 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-15 r/8492 chore(tvix/tools/turbofetch): update Cargo.nixFlorian Klink1-39/+100
This was out of date. Change-Id: Icea6ee865d389d56bc46941f6049721db293ef7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12207 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Autosubmit: flokli <flokli@flokli.de>
2024-08-15 r/8491 chore(tvix/tools/wave): update Cargo.lock and Cargo.nixFlorian Klink2-39/+209
This apparently was out of date. Change-Id: I9768d5e62f200169d7815ad85aa0f12eadc56a8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12206 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-27 r/8417 feat(tvix/tools/narinfo2parquet): preserve the deriveredef1-1/+20
Change-Id: Idbada585b87eef81cad5d40cb7592a7890704695 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12037 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-27 r/8414 refactor(tvix/nix-compat): introduce CaHash::algo_strFlorian Klink1-21/+2
This is the `{fixed,fixed:r,text}:{sha*,md5}` prefix used in various string representations. Factor that code out, and use it in the two places it can be used in. Change-Id: Ic9555fa9e1884198d435e55c7f630b8d3ba2a032 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12041 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-07-09 r/8364 chore(tvix/tools/weave): Regenerate Cargo.nix and filter srcIlan Joselevich3-42/+101
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: I79b062eddb13f59c4656a62735df1e527fc91a44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11970 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 r/8363 chore(tvix/tools/turbofetch): Regenerate Cargo.nix and filter srcIlan Joselevich3-37/+28
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: I48cabe514e0cf1dd1f00870bcb7958a2a49663d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11969 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-09 r/8362 chore(tvix/tools/crunch-v2): Regenerate Cargo.nix and filter srcIlan Joselevich3-68/+104
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: I90245bad27f6fee01455557bbccc865301a4f1a0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11968 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-09 r/8361 chore(tvix/tools/narinfo2parquet): Regenerate Cargo.nix and filter srcIlan Joselevich3-38/+31
The Cargo.lock also got changed because of crate2nix using --freezed. Change-Id: Id5225e73ec5894bc324f363660472a7efdd803cf Reviewed-on: https://cl.tvl.fyi/c/depot/+/11967 Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-01 r/8331 chore(tvix): bump to data-encoding 2.6.0Florian Klink1-1/+1
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-05-11 r/8108 chore(tvix/narinfo2parquet): bump dependenciesFlorian Klink2-27/+135
Get rid of the RUSTSEC warnings. Also upgrade tokio to drop the Sized requirement, similar to cl/11608. Change-Id: Idc6fece23c79eb30fd1dfc5fe64fa2e4c08cd412 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11624 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-05-08 r/8095 refactor(nix-compat/nar/reader): reuse prev_name allocationedef1-1/+1
We reuse the prev_name allocation for Entry, instead of allocating and returning a separate Vec. We encode the `prev_name: None` case as an empty vector, since we don't allow empty names anyway, and the sorting is equivalent. Change-Id: I975b37ff873805f5ff099bc82128706891052247 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607 Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-08 r/8089 chore(tvix/tools/crunch-v2): upgrade tokioedef2-5/+20
This drops the Sized requirement on AsyncReadExt. Change-Id: I2a89c708922ca528cb493aefd0613f477f648e83 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11608 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-03-03 r/7644 fix(tvix/tools/weave): handle sliced arrays correctlyedef1-8/+12
The start/end offsets are not necessarily coterminous with the underlying values array, so even if the stride is fixed, we still we need to slice the chunks down to match the start/end offsets. This bug shouldn't affect the correctness of any existing code, since we're always working with unsliced arrays read directly from Parquet. Change-Id: I2f7ddc4e66d4d3b2317a44bd436a35bff36bac79 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11081 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-27 r/7617 feat(tvix/tools/weave): initedef9-0/+11471
Scalable tracing GC for the cache.nixos.org dataset. Change-Id: I6c7852796f28e1a1c7607384ffb55f44407e1185 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10765 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-19 r/7555 chore(tvix/tools/narinfo2parquet): polars 0.34.2 -> 0.36.2edef4-704/+1419
Change-Id: Ib17a7bbe560000c3fef3ba6182819f710d707909 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10967 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-02-07 r/7481 chore(tvix/tools/turbofetch): bump magic-buffer to 0.1.1Florian Klink4-16/+25
This contains https://github.com/sklose/magic-buffer/pull/4, so we don't have to impl Send ourselves. Change-Id: If046596e13345ad4fec22209440e65859e44d540 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10748 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-01-27 r/7453 feat(tvix/tools/crunch-v2): add CLI argsFlorian Klink6-33/+884
Use clap derive to make the input and output files configurable, as well as the chunk size parameters. Change-Id: I02b29126f3bd2c13ba2c6e7e0aa4ff048ff803ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/10691 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-01-27 r/7452 feat(tvix/tools/crunch-v2): initedef12-0/+15022
This is a tool for ingesting subsets of cache.nixos.org into its own flattened castore format. Currently, produced chunks are not preserved, and this purely serves as a way of measuring compression/deduplication ratios for various chunking and compression parameters. Change-Id: I3983af02a66f7837d76874ee0fc8b2fab62ac17e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10486 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-12-24 r/7262 chore(tvix/turbofetch): switch to futures 0.3.30Florian Klink3-79/+79
The bugs have been fixed, https://github.com/rust-lang/futures-rs/pull/2801 and https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended up in that release. Change-Id: I301c0ffc951f04a5b3b7267e922771c837a3f5a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10415 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-27 r/7076 chore(3p/sources): bump channels & overlays (2023-11-27)Vincent Ambo1-2/+2
* picked avrdude from stable channel * removed override for texlive, as the upstream fix is merged * picked awscli2 from stable channel * bump tdlib to 1.8.21 (new minimum for telega.el) * tvix/turbofetch: switch to nixpkgs-native mechanism for CARGO_MANIFEST_LINKS (whatever that is) Change-Id: Ic695721b5ca750b89d21cab7a257e1db682b23c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10083 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-11-19 r/7042 feat(tvix/nix-compat): add narinfo::Signature::verifyFlorian Klink2-41/+588
This adds support for verifying signatures found in NARInfo files. This still needs to be hooked together with the nix+http[s] backend. Change-Id: Ic1c8ddbdecfb05cefca2492808388b0f7f3f2637 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10081 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-10 r/6988 feat(tvix/tools/narinfo2parquet): initedef6-0/+9703
Convert turbofetch output to queryable Parquet. Change-Id: I076f5a431f8aab8cfe7d973bdc9fe019cebde111 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9989 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6979 feat(tvix/tools/turbofetch): initedef9-0/+8624
Change-Id: I2efa6f94f57e812c52371256a4e62d1d54ff5057 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9925 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-22T18·42+0000