about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2023-11-03 r/6931 chore(tvix/eval): add a marker for sorted borrowed attrs iterationVincent Ambo1-0/+6
Similar to `into_iter_sorted`, add a marker function for call sites that want *borrowed* sorted iteration. Change-Id: I7c6f14e1ac43fdb14b861b3da183eb5d12bba139 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9899 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-03 r/6930 refactor(tvix/eval): more efficiently intersect attributesVincent Ambo1-9/+70
builtins.intersectAttrs is used a _lot_ in nixpkgs eval, for whatever reason. We previously had a very inefficient implementation that would allocate for each comparison. It stuck out like a sore thumb in perf analysis. This moves to a custom algorithm with two iterators, one for the left and one for the right side, advancing them along the (borrowed) map keys until a match is found and allocation is required. I've not made any effort to reduce the verbosity of this code, I don't think it's worth it. On my machine this reduces the mean runtime of evaluating `nixpkgs.emacs.outPath` by ~8%. Change-Id: Ie506d82cb8d5f45909628f771a6b73e0eca16b27 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9898 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-03 r/6929 fix(tvix): fix `cargo test` for `nix_oracle`Florian Klink1-0/+1
As described in https://b.tvl.fyi/issues/313, the `nix_oracle` tests currently fail if run without Nix 2.3 in `$PATH`. This is not a problem for people running Nix 2.3 on their systems, and neither is it for the tests running inside Nix derivations itself (as we hardcode Nix 2.3 there), but for interactive `cargo test` runs, either using `mg shell //tvix:shell`, or `nix-shell` (from the tvix workspace) it currently is broken. Adding Nix 2.3 to the shell fixes that. Change-Id: Idbe01aa92d63eb1a3dab5797b9be5eba1e8d7f7a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9907 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-02 r/6926 fix(tvix/eval/benches): use black_box properlyFlorian Klink1-6/+3
The purpose of black_box is to actually prevent the compiler from being able to optimize computation of the benchmarked function away. To accomplish this, we need to actually *use* black_box to blackbox the input data away, rather than the return type. Change-Id: I5438982f57509fbf7b85034346a2739d76aef1fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/9902 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6922 chore(third_party/crate2nix): apply patches to fix crossFlorian Klink1-5/+7
Together with temporarily pointing nixpkgs past b10994c38c61038970a19fa60bfbec21a61755cc, this now fixes cross- compilation for tvix. This incorporates the changes proposed in https://cl.tvl.fyi/9888 and https://cl.tvl.fyi/9889, but by fixing it in crate2nix, and using the (re-)generated version of it. Changes were sent upstream at https://github.com/nix-community/crate2nix/pull/309, this pulls in a minimal patch for now. Change-Id: I70bb6f003bbc3e89de9c4eb4985ea4708ac3a9fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9890 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-02 r/6921 docs(tvix/castore): add notes on verified streamingFlorian Klink1-0/+121
These are still a bit WIP, but should already explain the idea better than not having any documentation about how we intend to implement this at all. Let's put this into the repo, as a working document. Co-Authored-By: edef <edef@edef.eu> Change-Id: Ic0c5195cdf0a33c8382af1b5cff9064fe6ef9ffb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9894 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6920 chore(tvix/castore): move data model docs to hereFlorian Klink3-15/+20
These describe the castore data model, so it should live in the castore crate. Also, some minor edits to //tvix/store/docs/api.md, to honor the move of the castore bits to tvix-castore. Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-10-31 r/6919 chore(tvix): regenerate Cargo.nixFlorian Klink1-6/+22
Change-Id: I96e01f938a46d12a94da85968caaf190d041b9ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9887 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-28 r/6902 fix(nix-compat/narinfo): don't allocate errors earlyedef1-11/+12
We also switch the MissingField error to &'static str, since we only parse a fixed set of fields. Together, this makes the performance impact of error handling negligible in batch happy-path parsing. Change-Id: I2bd0ef2f5b35fcaced56b32d238eca75ac199ef1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9867 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-28 r/6901 fix(nix-compat/narinfo): unwrap in the benchmarkedef1-2/+6
We primarily want to measure the speed of the happy path. Change-Id: Iad0146dde86fc262e2a4b8295bde4eb297b8bf30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9866 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: edef <edef@edef.eu>
2023-10-28 r/6900 refactor(tvix/nix-compat/narinfo): return errorsFlorian Klink1-45/+127
This provides more info about where a NARInfo failed to parse, rather than just returning None and leaving a library user to manually debug. Change-Id: I9a28ddd8e5712101483ebe686fdc474c7bbc8e4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9831 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-28 r/6899 refactor(nix-compat/nixbase32): clean up encodeedef1-16/+15
Change-Id: I63da5bd47cd9033e1cc13fbe4b4864514a5343cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9865 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6898 fix(nix-compat/nar/writer): `use ByteSlice` closer to call siteedef2-2/+2
Otherwise, it'll generate unused import warnings in release mode. Change-Id: I0399d2617a8a8b973a2b8c18f4a0a613d70c48f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9864 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6897 feat(tvix/nix-compat): add narinfo parsing and serialisationedef8-76/+492
Change-Id: I72c63414794642ca8d85c3f635f49db888420c40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9852 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6896 refactor(nix-compat/store_path): use nixbase32::decode_fixededef1-4/+2
Change-Id: I81471ee57920aa8fa889fb00c7903cdc570af9c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6895 feat(nix-compat/store_path): add StorePathRefedef1-22/+72
Change-Id: I0d888a55d93e5c22e77cb0264d09757656f731d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9862 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6894 feat(tvix/castore): carry name in ValidateNodeError::InvalidNameedef2-14/+14
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6893 refactor(tvix): turn nullary enum variants into unit variantsedef6-30/+30
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6892 refactor(nix-compat/store_path): speed up validate_name fast pathedef1-12/+39
Change-Id: Ie50b29145804777f7644952c65cb42519a8565e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9859 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6891 feat(nix-compat/store_path): validate_name takes AsRef<[u8]>edef2-3/+4
Change-Id: I8819e2a7b63008a68f4f82035a08b960ac480dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9858 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6890 feat(nix-compat/store_path): validate_name over borrowed dataedef2-6/+10
Change-Id: Ifeb6231f48d4ad267a7acd398b4b3b687ee4d560 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9857 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6889 refactor(nix-compat/store_path): don't materialise fingerprintedef1-8/+9
Change-Id: I6a88531ded05c0dfb9232a0343a465fa02fb6989 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9856 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6888 refactor(nix-compat/store_path): make digest and name privateedef6-21/+27
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6887 fix(tvix/nix-compat): validate store path name lengthedef1-2/+13
Change-Id: I89ac0ad147a1872c021ab4235ca46ef3f51d0446 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9854 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6886 fix(tvix/nix-compat): don't box CAHash::Textedef3-14/+9
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6885 feat(tvix/nix-compat): add fixed-length base32 decodingedef1-1/+34
Change-Id: Iea9d51157c1e38e1734dc1ba752eba9b2372c61b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9850 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6884 feat(tvix/nix-compat): use AsRef<[u8]> for base32 decode inputedef1-3/+5
Change-Id: If2275dda62b852aedb64b4f9915f7dc4173cabd2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9851 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6883 refactor(tvix): condense long bytestringsedef11-93/+67
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6882 feat(tvix/nix-compat): decode base32 with a lookup tableedef1-22/+42
This also takes input validation out of the loop, leaving the loop backedge as the sole branch in the hot path. Change-Id: Id08e6fb9cf5b074780efa09a7ad389352a601bcc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9847 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6881 refactor(tvix/nix-compat): clean up base32 decodingedef1-23/+18
Change-Id: I8591a3e1075f2281b7dc49f7fa9e0027062a78a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9846 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6880 refactor(tvix/nix-compat): simplify encode_lenedef1-6/+5
Change-Id: Idfefd3b5b0d1ffbd4e9208e634a2d9989965a291 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9845 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-25 r/6878 chore(tvix): move buildkite.yml from views/tvix to tvix/Florian Klink1-0/+9
This should remove the ambiguity on where buildkite.yml should show up at least. We might still want to move workspace.josh into tvix/ too at some point, but that's blocked on https://github.com/josh-project/josh/issues/1287 This commit doesn't cause any change in views/tvix, with a previous version of this commit added to my HEAD, I ran josh-filter to verify it still points to ce0df88561fee8a4d6052ca9f2a1792f899a202a. Change-Id: Ic03e92260dc7725bc2fcf01b7c72a097d92c028e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9828 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-23 r/6876 fix(tvix/cli/derivation): fix populate_output_configurationFlorian Klink1-302/+81
The `if let` wasn't matching `outputHashAlgo` being unset, and didn't populate it in that case. Port the remaining commented-out testcases over to nix-lang based tests. Change-Id: I140b5643b9ed9d29f9522ec65d98d0b12262d728 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9825 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-23 r/6875 refactor(tvix/cli): rename `a` to `algo_str`Florian Klink1-2/+3
Change-Id: I65d307126647841b50db008c21c155441a6b6cb5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9818 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-23 r/6874 test(tvix/cli/derivation): add test cases for output path calcFlorian Klink4-0/+49
Get some test coverage with some calls to builtins.derivation. The expected output paths were calculated with Nix. Change-Id: I2ce205ea5244e8ef939d9cacb033283fc6f15d17 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9817 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 r/6873 refactor(tvix/cli): move evaluator instantiation to helperFlorian Klink1-30/+29
Have a private `eval` function in the test module that returns an EvaluationResult, and migrate the existing tests over to use it, rather than repeating itself. Change-Id: I879987700c8507248c644ef03b62a8cb8e308139 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9816 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 r/6872 fix(nix-compat/src/derivation/validate): remove breakFlorian Klink1-2/+31
If the output was fixed, we broke out of the for loop too early, before actually validating individual outputs. Change-Id: I2259697dfa2a157764358f6d326a1f7f6610647c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9815 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 r/6871 refactor(tvix/nix-compat): rename NixHashWithMode -> CAHashFlorian Klink10-209/+222
This specific struct is only used to represent content-addressed paths (in case a Derivation has a fixed-output hash, for example). Rename `Output`'s `hash_with_mode` to `ca_hash`. We now also include `CAHash::Text`, and update the `validate` function of the `Output` struct to reject text hashes there. This allows cleaning up the various output path calculation functions inside nix-compat/src/store_path/utils.rs, as they can now match on the type. `make_type` is renamed to `make_references_string`, `build_regular_ca_path` is renamed to `build_ca_path`, and `build_text_path` has a disclaimer added, because you might not actually want to use it. Change-Id: I674d065f2ed5c804012ddfed56e161ac49d23931 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9814 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-22 r/6870 feat(tvix/store/protos): add CA fieldFlorian Klink4-3/+67
This adds support to represent the `CA` field found in some .narinfo files. As `deriver`, it's also a subfield of the `narinfo` field. Extending nix-compat with a more accessible data structure that can take care of formatting, as well as writing validation functions in Rust + Golang, and integrating it into nar-bridge is something for a followup CL. Change-Id: I71e9c30957bcd03051a491aa54d7baac25b6dd2d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9795 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-22 r/6869 refactor(tvix/store): move deriver field into narinfo submessageFlorian Klink11-90/+92
This information is present in the .narinfo files, it should have gone there. Change-Id: Ib43d0cf30c2795bf1fe77c46646174353ade0458 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9794 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-18 r/6857 test(tvix/nix-compat/nar/writer/sync): add testsFlorian Klink3-0/+138
Change-Id: I04c556e7b57f20dd33f71b0379e6104e1702a8ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/9785 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-18 r/6856 feat(tvix/nix-compat/nar/reader): add testsFlorian Klink5-0/+122
Change-Id: If0abaf74a3b7d201af8972e54e3e34b20b8c0bc4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9764 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-18 r/6855 fix(tvix/nix-compat/nar/wire): correct Node::Some doc commentedef1-1/+2
It is *eventually* followed by a Node, but there is some stuff in between. Change-Id: Ie7c7b462828bd3e066f4a7e774895f30b82763ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/9768 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6854 refactor(nix-compat/nar/writer): use clone_intoedef2-4/+2
Equivalent to the existing code, but a little less cryptic. Change-Id: Ib9b2f9aedddc84d0e79840bba4cce01f92d9bc56 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9766 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6853 feat(tvix/nix-compat): NAR readeredef8-4/+530
Change-Id: I50d51baf62c0419eaf17f0dc262f728aaff9794d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9688 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6852 docs(tvix/nix-compat/nar): document the wire formatedef1-2/+66
Change-Id: I6c8e23bad27fa6ada1b8973482b4d99190cf050d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9767 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-18 r/6851 refactor(tvix/nix-compat): use hash_with_mode's nix_hash_stringFlorian Klink1-7/+1
HashWithMode already provides a to_nix_hash_string() method, giving us exactly the string we need here. Change-Id: Id2635bf3ea6c2514faf3c26b297866d774f4ff4a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9799 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-17 r/6850 docs(tvix/castore/directorysvc): document drop without put and closeFlorian Klink1-0/+4
As seen in cl/9761, we only use put() and close() in some cases. Make sure this is documented, so we don't end up having implementations not able to deal with that. Change-Id: I406ad23f2f4c9604eca731d898ae41a4663fb846 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9786 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-17 r/6849 chore(tvix/nar-bridge): update to latest go mod renamesFlorian Klink18-29/+29
- code.tvl.fyi/tvix/castore/protos -> code.tvl.fyi/tvix/castore-go - code.tvl.fyi/tvix/store/protos -> code.tvl.fyi/tvix/store-go See cl/9791, cl/9792 for context. Change-Id: I44614c6ed40b9f52d9dcdea8e61fe2c3c830ce78 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9793 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-17 r/6848 chore(tvix/store-go): rename go moduleFlorian Klink12-16/+16
`code.tvl.fyi/tvix/store/protos` now points to a directory that only contains the `.proto` files, while all golang tooling and .pb.go files live in tvix/store-go. As discussed in https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount of people currently using this is still small, so rename the go.mod now, while it doesn't yet hurt. Also, use code.tvl.fyi/tvix/castore-go instead of code.tvl.fyi/tvix/ castore/protos, to make use of cl/9791. Change-Id: I9ea89957d7c29dfae4c893b9aae8ac8a0bad2d8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9792 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-14T22·18+0000