about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2023-11-03 r/6930 refactor(tvix/eval): more efficiently intersect attributesVincent Ambo1-9/+70
builtins.intersectAttrs is used a _lot_ in nixpkgs eval, for whatever reason. We previously had a very inefficient implementation that would allocate for each comparison. It stuck out like a sore thumb in perf analysis. This moves to a custom algorithm with two iterators, one for the left and one for the right side, advancing them along the (borrowed) map keys until a match is found and allocation is required. I've not made any effort to reduce the verbosity of this code, I don't think it's worth it. On my machine this reduces the mean runtime of evaluating `nixpkgs.emacs.outPath` by ~8%. Change-Id: Ie506d82cb8d5f45909628f771a6b73e0eca16b27 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9898 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-03 r/6929 fix(tvix): fix `cargo test` for `nix_oracle`Florian Klink1-0/+1
As described in https://b.tvl.fyi/issues/313, the `nix_oracle` tests currently fail if run without Nix 2.3 in `$PATH`. This is not a problem for people running Nix 2.3 on their systems, and neither is it for the tests running inside Nix derivations itself (as we hardcode Nix 2.3 there), but for interactive `cargo test` runs, either using `mg shell //tvix:shell`, or `nix-shell` (from the tvix workspace) it currently is broken. Adding Nix 2.3 to the shell fixes that. Change-Id: Idbe01aa92d63eb1a3dab5797b9be5eba1e8d7f7a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9907 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-03 r/6928 chore(third_party/nixpkgs/nix_2_3): switch to 2.3-maintenanceFlorian Klink1-6/+7
The comment about what fixes have been applied was outdated (more was cherry-picked), and all our backports have been upstreamed to the 2.3- maintenance branch. Switch to there, which will also give us support for zstd, which was merged separately. Fixes https://b.tvl.fyi/issues/328 Change-Id: Ice25ebaaf8250a97f032a40a4dd0c7fad2ebda28 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9906 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-03 r/6927 docs(users/tazjin): add graph explaining tvix-eval Value structureVincent Ambo1-0/+98
Change-Id: I436aeba6da6a0d5096452529d7dd0be11a466e8f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9905 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-11-02 r/6926 fix(tvix/eval/benches): use black_box properlyFlorian Klink1-6/+3
The purpose of black_box is to actually prevent the compiler from being able to optimize computation of the benchmarked function away. To accomplish this, we need to actually *use* black_box to blackbox the input data away, rather than the return type. Change-Id: I5438982f57509fbf7b85034346a2739d76aef1fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/9902 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6925 feat(tazjin/khamovnik): add some perf analysis toolingVincent Ambo1-0/+3
Change-Id: Ia6db1213bb62cfba475ee9d1654763d56c98fc7c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9897 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6924 chore(tazjin/nixos): remove some unused softwareVincent Ambo1-4/+0
Change-Id: I87e4b709906333ff9316792effb1082973dc918c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9896 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-11-02 r/6923 fix(tazjin/khamovnik): enable services.throttledVincent Ambo1-0/+3
This can apparently work around some of the CPU throttling bugs on ~modern~ computers. Change-Id: I807ece85d3eba53857a1cb1e73a33f7924538e96 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9895 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-02 r/6922 chore(third_party/crate2nix): apply patches to fix crossFlorian Klink4-6/+70
Together with temporarily pointing nixpkgs past b10994c38c61038970a19fa60bfbec21a61755cc, this now fixes cross- compilation for tvix. This incorporates the changes proposed in https://cl.tvl.fyi/9888 and https://cl.tvl.fyi/9889, but by fixing it in crate2nix, and using the (re-)generated version of it. Changes were sent upstream at https://github.com/nix-community/crate2nix/pull/309, this pulls in a minimal patch for now. Change-Id: I70bb6f003bbc3e89de9c4eb4985ea4708ac3a9fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9890 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-02 r/6921 docs(tvix/castore): add notes on verified streamingFlorian Klink1-0/+121
These are still a bit WIP, but should already explain the idea better than not having any documentation about how we intend to implement this at all. Let's put this into the repo, as a working document. Co-Authored-By: edef <edef@edef.eu> Change-Id: Ic0c5195cdf0a33c8382af1b5cff9064fe6ef9ffb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9894 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6920 chore(tvix/castore): move data model docs to hereFlorian Klink3-15/+20
These describe the castore data model, so it should live in the castore crate. Also, some minor edits to //tvix/store/docs/api.md, to honor the move of the castore bits to tvix-castore. Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-10-31 r/6919 chore(tvix): regenerate Cargo.nixFlorian Klink1-6/+22
Change-Id: I96e01f938a46d12a94da85968caaf190d041b9ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9887 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-31 r/6918 chore(third_party/overlay): use crate2nix upstreamFlorian Klink1-16/+2
Apply the patch that didn't land using `patches`. Change-Id: Id87675cd44123eea8cc2175efe04f61421f26f37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9886 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-31 r/6917 chore(3p/sources): spooky channel and overlay bump (halloween '23)Vincent Ambo1-12/+12
Change-Id: I88542b467c0ff22e4238a9df5b5645cc6aeb4cc0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9885 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-30 r/6916 chore(3p/sources): bump channels & overlays (2023-10-29)Vincent Ambo3-24/+30
* //3p/nixpkgs: use py3status from stable for //users/grfn * //3p/nixpkgs: use electrum from stable for //users/tazjin * //3p/overlays: update crate2nix vendor hash not sure why this hash is not stable if the package isn't updated, but whatever Change-Id: Ia7a13ac35b9c5534eefa53eff66e47ea1c32c62a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9830 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-30 r/6915 chore(tools/depotfmt): use Go version from buildGoVincent Ambo3-10/+13
This is required because Go 1.18 is actually being deleted. I've applied the formatting breakage that it introduces (such as breaking comment formatting), because I can't be bothered to try and work around broken Go stuff. Change-Id: Ica7cee0d01228845d6a766079fef36df99a3da96 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9832 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-30 r/6914 feat(users/flokli/nixos): add direnv supportFlorian Klink2-6/+7
Expose `deps` separately, add a direnv with PATH_add for it to bring tooling into $PATH. Change-Id: I432cd2b082cad89e08bef78dc4653e10e137cd6b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9842 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-30 r/6913 feat(users/flokli/nixos): use lazydeps for shellFlorian Klink1-2/+7
Avoid having to re-enter the shell whenever the config is changed. Change-Id: Ib9f6bb4075e29acaeb4863d64c017695ca85b60b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9841 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6912 feat(users/flokli/archeology): add awscli, htop, kitty terminfoFlorian Klink1-1/+8
Change-Id: Ib7ae1871a5d0b16a68c79b68e7e79fd302da79bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9840 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6911 feat(users/flokli): use nix-copy-closure instead of nix copyFlorian Klink1-1/+1
nix copy seems to stall on the EC2 box for unknown reasons. Change-Id: I30639a52758814968d3b54d716522fb88db80cfe Reviewed-on: https://cl.tvl.fyi/c/depot/+/9839 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6910 feat(users/flokli): expose readTree targets to CIFlorian Klink1-1/+1
Change-Id: I3ea801b47267f4c985c2ab5cb1b79b2659894307 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9838 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6909 refactor(users/flokli): move common stuff to `archeology` profileFlorian Klink3-29/+24
Change-Id: I8470c0a2416c0c397e009affb44f8c7a852cd526 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9837 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-30 r/6908 feat(third_party/nixpkgs): allow setting crossSystemFlorian Klink2-2/+4
This allows cross-compiling depot targets to other architectures, so `nix-build --argstr crossSystem aarch64-linux -A tvix.nar-bridge` will emit a cross-compiled aarch64-linux binary. There's still some other cross-related issues in crate2nix to sort out for crate2nix builds, but this CL can already land. Co-Authored-By: raitobezarius <tvl@lahfa.xyz> Change-Id: I467d49d125dd707a4142bfde18eea4a1c3afaf70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9755 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-30 r/6907 feat(users/flokli): add archeology-ec2Florian Klink4-3/+70
This add the EC2 box config to the repo. Change-Id: Id7a888a2cfbf1454cd9f9465018df377e14b4e9f Reviewed-on: https://cl.tvl.fyi/c/depot/+/9836 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 r/6906 chore(third_party/gerrit-queue): move to tvl overlayFlorian Klink21-1585/+20
Bump to a version including https://github.com/flokli/gerrit-queue/pull/15 Change-Id: Ie316498ca2c608e5489901c5705ce5f2dc047f29 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9808 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-10-29 r/6905 feat(users/flokli/nixos): add deploy scriptFlorian Klink1-1/+16
This adds a deploy-archeology script. I tried getting morph to work first, but passing it a depot.ops.nixos.nixosFor seems to be very hard - the NixOS module system doesn't like the arguments it's called with. Replace morph with a 3 line bash script, which assumes your ssh_config contains config for an `archeology` host. Change-Id: I2bf694c60ded39c201efbbb899f3b5512aa4d0f2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9835 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-29 r/6904 refactor(users/flokli): drop some optional args and complexityFlorian Klink3-9/+7
Change-Id: Ifdcac829ede4ec469a7ce1b608e78bae11f2766b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9834 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-29 r/6903 feat(users/flokli/nixos): init archeologyFlorian Klink5-0/+88
Change-Id: Ic31cb8030179ff37b1cc3d3d9241e2582cfe3e5e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9833 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de>
2023-10-28 r/6902 fix(nix-compat/narinfo): don't allocate errors earlyedef1-11/+12
We also switch the MissingField error to &'static str, since we only parse a fixed set of fields. Together, this makes the performance impact of error handling negligible in batch happy-path parsing. Change-Id: I2bd0ef2f5b35fcaced56b32d238eca75ac199ef1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9867 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-28 r/6901 fix(nix-compat/narinfo): unwrap in the benchmarkedef1-2/+6
We primarily want to measure the speed of the happy path. Change-Id: Iad0146dde86fc262e2a4b8295bde4eb297b8bf30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9866 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: edef <edef@edef.eu>
2023-10-28 r/6900 refactor(tvix/nix-compat/narinfo): return errorsFlorian Klink1-45/+127
This provides more info about where a NARInfo failed to parse, rather than just returning None and leaving a library user to manually debug. Change-Id: I9a28ddd8e5712101483ebe686fdc474c7bbc8e4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9831 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-28 r/6899 refactor(nix-compat/nixbase32): clean up encodeedef1-16/+15
Change-Id: I63da5bd47cd9033e1cc13fbe4b4864514a5343cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9865 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6898 fix(nix-compat/nar/writer): `use ByteSlice` closer to call siteedef2-2/+2
Otherwise, it'll generate unused import warnings in release mode. Change-Id: I0399d2617a8a8b973a2b8c18f4a0a613d70c48f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9864 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6897 feat(tvix/nix-compat): add narinfo parsing and serialisationedef8-76/+492
Change-Id: I72c63414794642ca8d85c3f635f49db888420c40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9852 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6896 refactor(nix-compat/store_path): use nixbase32::decode_fixededef1-4/+2
Change-Id: I81471ee57920aa8fa889fb00c7903cdc570af9c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6895 feat(nix-compat/store_path): add StorePathRefedef1-22/+72
Change-Id: I0d888a55d93e5c22e77cb0264d09757656f731d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9862 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6894 feat(tvix/castore): carry name in ValidateNodeError::InvalidNameedef2-14/+14
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6893 refactor(tvix): turn nullary enum variants into unit variantsedef6-30/+30
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6892 refactor(nix-compat/store_path): speed up validate_name fast pathedef1-12/+39
Change-Id: Ie50b29145804777f7644952c65cb42519a8565e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9859 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6891 feat(nix-compat/store_path): validate_name takes AsRef<[u8]>edef2-3/+4
Change-Id: I8819e2a7b63008a68f4f82035a08b960ac480dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9858 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6890 feat(nix-compat/store_path): validate_name over borrowed dataedef2-6/+10
Change-Id: Ifeb6231f48d4ad267a7acd398b4b3b687ee4d560 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9857 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6889 refactor(nix-compat/store_path): don't materialise fingerprintedef1-8/+9
Change-Id: I6a88531ded05c0dfb9232a0343a465fa02fb6989 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9856 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6888 refactor(nix-compat/store_path): make digest and name privateedef6-21/+27
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6887 fix(tvix/nix-compat): validate store path name lengthedef1-2/+13
Change-Id: I89ac0ad147a1872c021ab4235ca46ef3f51d0446 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9854 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6886 fix(tvix/nix-compat): don't box CAHash::Textedef3-14/+9
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6885 feat(tvix/nix-compat): add fixed-length base32 decodingedef1-1/+34
Change-Id: Iea9d51157c1e38e1734dc1ba752eba9b2372c61b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9850 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6884 feat(tvix/nix-compat): use AsRef<[u8]> for base32 decode inputedef1-3/+5
Change-Id: If2275dda62b852aedb64b4f9915f7dc4173cabd2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9851 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6883 refactor(tvix): condense long bytestringsedef11-93/+67
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6882 feat(tvix/nix-compat): decode base32 with a lookup tableedef1-22/+42
This also takes input validation out of the loop, leaving the loop backedge as the sole branch in the hot path. Change-Id: Id08e6fb9cf5b074780efa09a7ad389352a601bcc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9847 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6881 refactor(tvix/nix-compat): clean up base32 decodingedef1-23/+18
Change-Id: I8591a3e1075f2281b7dc49f7fa9e0027062a78a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9846 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-08T02·12+0000