about summary refs log tree commit diff
path: root/users/edef
AgeCommit message (Collapse)AuthorFilesLines
2024-10-19 r/8846 feat(users/edef/weave): use FxHashSet and dedupe earlyedef4-4/+33
Deduping early saves a fair bit of memory, but the extra hashing is costly. We switch to FxHash, since we don't need a DoS-proof hash, but we do need it to be *fast*. Change-Id: Ic6b7010874c417862baa9b882593208c8dd1d5e6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12648 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-19 r/8845 feat(users/edef/weave): ingest roots in Parquet formatedef1-13/+15
Parsing of store-paths.xz is now handled by //users/edef/fetchroots. Change-Id: I78be5aada0c0a321ed79d80c9b615e5f997ac3e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12670 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-19 r/8844 refactor(users/edef/weave/swizzle): use polars streamingedef5-112/+70
This vastly reduces the memory requirements, so we can run in ~40G RAM. Change-Id: I4952a780df294bd852a8b4682ba2fd59b9bae675 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12667 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-19 r/8843 refactor(users/edef/weave): simplify OwningRef handlingedef2-32/+12
We switch to working with the underlying Arc<Bytes<u8>> type, since Buffer<u8> is already essentially an OwningRef-esque subslice. Because we're now working with an exposed Arc directly, we don't need to have our own `unsafe impl StableAddress` any more. Change-Id: I9ce2edc6899177145e15b72aa5380f708a62173c Reviewed-on: https://cl.tvl.fyi/c/depot/+/12668 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-19 r/8842 refactor(users/edef/weave): use swap rather than CASedef1-3/+5
We only care about ordering on the same variable, and we rely on the release barrier provided by rayon's thread joining. The comparison failing is always an error path, and we're indifferent about which thread it fails on. Change-Id: I592a7eaae5b6935cf424c6576a49786f39909fb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12666 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-19 r/8841 refactor(users/edef/weave): simplify DoubleEndedIterator boundedef1-1/+1
Thanks, Clippy! Change-Id: I116b478124b72e070e94150ee850532752f64d60 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12669 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-10-17 r/8829 chore(users/edef/crunch-v2): bump crate dependenciesFlorian Klink2-25/+25
Fixes rustls 0.20.9: RUSTSEC-2024-0336 (CVE-2024-32650, GHSA-6g7w-8wpp-frhj) Change-Id: I1be307876465f47c28af05c4b515bf6a7b6713ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/12654 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2024-10-17 r/8828 fix(users/edef/weave): use safer_owning_refedef4-33/+35
owning_ref has serious unsoundness. Change-Id: Ie760697cd6399e6bc75f1ad17c9bb74adc077a35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12656 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: edef <edef@edef.eu> Reviewed-by: flokli <flokli@flokli.de>
2024-10-17 r/8827 chore(users/edef/turbofetch): bump crate dependenciesFlorian Klink2-20/+20
Fixes rustls 0.20.9: RUSTSEC-2024-0336 (CVE-2024-32650, GHSA-6g7w-8wpp-frhj) Change-Id: Id38f867a85320e3a3ca870dbf224c4fd4c5abacb Reviewed-on: https://cl.tvl.fyi/c/depot/+/12655 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu>
2024-10-17 r/8818 docs(users/edef/fetchroots): add a README for other userszimbatm1-0/+36
Change-Id: Ibce9121fbd209e713ce04271e285f70e2af9867e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12646 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Reviewed-by: flokli <flokli@flokli.de>
2024-10-17 r/8817 chore(users/edef/fetchroots): wire up the buildzimbatm4-4/+12634
Change-Id: I6830b885394001882b54fddf310bbd38d253679a Reviewed-on: https://cl.tvl.fyi/c/depot/+/12645 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-10-17 r/8816 feat(users/edef/fetchroots): initedef3-0/+3620
Change-Id: I6861661bcd99652a1875ccde421e78e87c45054c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11094 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2024-10-12 r/8801 chore(tvix/tools): move narinfo2parquet to //users/edefVincent Ambo6-0/+11617
This is not a core Tvix tool, it's a tool that uses a Tvix component. Change-Id: I81d2b2374da23489df0097dcabb8295c82652fc1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12606 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2024-10-12 r/8800 chore(tvix/tools): move weave to //users/edefVincent Ambo9-0/+11508
This is not a core Tvix tool, it's a tool that uses a Tvix component. Change-Id: I705f2c4ab87f1512e005007c933e16b84ed4279f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12605 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2024-10-12 r/8799 chore(tvix/tools): move turbofetch to //users/edefVincent Ambo9-0/+9375
This was introduced in cl/9925 without any commit message, but this is clearly not relevant to Tvix itself (it even says so in a comment in Cargo.toml). Change-Id: I84f12d5145c3f53c9df23863f887bad913856c50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12604 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: edef <edef@edef.eu>
2024-10-12 r/8798 chore(tvix/tools): move crunch-v2 to //users/edefVincent Ambo12-0/+15454
This is not a core Tvix tool, it's some sort of one-off analysis thing. Change-Id: I05fcbed45abad27d6b5cfd49db1727249dad3971 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12603 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-01-11 r/5646 feat(users/edef/refscan): AArch64 supportedef1-0/+60
Change-Id: I5062078739f0bf9f70c6789a9f2eafceff65d76e Reviewed-on: https://cl.tvl.fyi/c/depot/+/7690 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-11 r/5645 fix(users/edef/refscan): quiet the camel-case type lintedef1-0/+1
Change-Id: I6677c6dd48d66a44970f5af459449dbc9aecfe5a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7689 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-01-10 r/5638 docs(users/edef/refscan): add copyright and license informationedef9-0/+514
reUSE.software compliant :) Change-Id: I349f1958a11ab7ee98d67c90d02f1c799e9d30fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/7688 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-01-09 r/5637 refactor(users/edef/refscan): switch to intrinsicsedef3-25/+41
packed_simd is deprecated, but we don't need very much SIMD: * _mm256_set1_epi8 / vpbroadcastb (splat) * _mm256_cmpgt_epi8 / vpcmpgtb (comparison) * _mm256_movemask_epi8 / vpmovmskb (compress to bitmask) This also simplifies the code by only vectorising the bare minimum, since we just get a bitmask and operate in scalar mode as soon as possible. We don't need nightly Rust anymore: we're using only stable intrinsics. Change-Id: Id410b5fef2549f3c97f48049f722f1e643e68553 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7687 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-01-09 r/5636 feat(users/edef/refscan): high-performance Nix reference scanneredef7-0/+154
Research-grade code, treat with care. Change-Id: I99804df93e64101ef24928238ef0a8a02b59c2aa Reviewed-on: https://cl.tvl.fyi/c/depot/+/7686 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2022-09-19 r/4922 chore(gerrit): migrate OWNERS files to code-owners styleLuke Granger-Brown1-3/+3
Change-Id: Iacc521dfdd4b4a2d5cef3920cf8189bcce35a488
2022-01-31 r/3723 style: format entire depot with nixpkgs-fmtVincent Ambo1-3/+4
This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2020-08-05 r/1592 fix(edef/keys): add readTree headeredef1-0/+2
Change-Id: I5e1e0f58377201b591730a49f44678a33243b1d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1653 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1591 chore(edef/users/depot-scan): move to a subdiredef3-3/+1
Change-Id: If202b466aa0874048e89e0dc270ff14883510131 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1652 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1590 fix(users/edef/depot-scan): properly trace readFile/readDiredef3-2/+19
Change-Id: I493e7d20504d34f83a53db50b69e86668874756e Reviewed-on: https://cl.tvl.fyi/c/depot/+/1651 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1589 fix(users/edef/depot-scan): properly depend on pl scriptKane York2-2/+2
Change-Id: I83c75ab75dae9c904e127ad6160c7641d4523e40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1647 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1588 feat(edef): add SSH CAedef1-0/+5
Change-Id: If6839735d2730fa49363fa46e6fa059dd19bfaf7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1649 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-05 r/1587 feat(edef/depot-scan): initedef2-0/+23
A small script that parses `nix-instantiate -vv` output to track the files read while instantiating a depot build. Change-Id: I5acf31d55f39c1d1acf9cdead03d33e2c8abab6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/1646 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2020-08-03 r/1560 chore(3p/loxy): Move //users/edef/loxy to //third_partyedef1-31/+0
Change-Id: Ia451d190d4e415ef875c51ebeca12c7c5dc72342 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1590 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: tazjin <mail@tazj.in>
2020-08-03 r/1559 chore(nix/fetchGoModule): Move //users/edef/fetchGoModule to //nixedef2-25/+1
Change-Id: Ie7f52367ef91c0c8017a36c2a6c7edb06f10e57d Reviewed-on: https://cl.tvl.fyi/c/depot/+/1589 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2020-08-03 r/1556 feat(edef/loxy): initedef1-0/+31
loxy(8) is a logging IRC proxy. Change-Id: I78fd798ec32636cf38f877c364abc87ef9f14836 Co-authored-by: V <v@anomalous.eu> Reviewed-on: https://cl.tvl.fyi/c/depot/+/1587 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-03 r/1555 feat(edef/fetchGoModule): initedef1-0/+24
Change-Id: I160a4264b9ade35d7cfcc1609ab395eab164d4d8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/1586 Tested-by: BuildkiteCI Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-03 r/1554 feat(edef): take ownership of //users/edefedef1-0/+3
Change-Id: Ic3bd9bf4ef354688bdd06961a348eb127dab97bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/1588 Tested-by: BuildkiteCI Reviewed-by: ericvolp12 <ericvolp12@gmail.com> Reviewed-by: lukegb <lukegb@tvl.fyi>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-22T13·24+0000