about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2023-11-05 r/6954 feat(tvix/castore): extend blobstore protos for verified streamingFlorian Klink5-91/+264
This pdates the proto docstrings a bit, especially w.r.t. verified streaming. It also adds send_chunks, send_bao fields to StatBlobRequest (renamed from BlobMeta) Change-Id: I590cc8646d86b73bca9f38a9b6d9ea15e4df5cb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9951 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 r/6953 fix(tvix/*store-go): fix depot checksFlorian Klink2-2/+2
This wasn't doing anything, because $(regenerate) was a directory: /nix/store/cxfxvz76zh9vb7x26h3cx98gkv234jz2-pb-go-check: line 2: /nix/store/my1nd1qvg5iis38rfyn1pm2c7ib7myn5-regenerate: Is a directory Change-Id: I6ebed1d7b84dceb885c1f25527ce62d973146819 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9959 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6952 chore(tvix/store-go): regenerate proto bindingsFlorian Klink1-25/+223
As surfaced in cl/9959, we didn't actually spot having to run the regeneration in CI. Change-Id: Iabdb79339d2a220a6cb368f9c5c70166f1ce9eb3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9960 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6951 feat(tvix/store/nar): add read_nar methodFlorian Klink2-0/+338
This adds a function that can ingest a NAR file into a given blob_service and directory_service, returning the root node. Passing around the directory_putter is a bit annoying, but at least for this linear "reading through a NAR" case, we don't need it to be clone or sync. Change-Id: Ia26d596f5dffc6dbd69aa39a82b59bba41685a9c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9915 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-05 r/6950 refactor(tvix/nix-compat): remove unused importsFlorian Klink1-2/+1
Change-Id: I64523df2344233d9e424812d94b1c7ed2ecb1a74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9956 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 r/6949 refactor(tvix/nix-compat): use matches! macroFlorian Klink1-5/+1
Flagged by ``#[warn(clippy::match_like_matches_macro)]`.` Change-Id: If07109e5ec01b05df898119f9a577196dfe11b37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9955 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 r/6948 refactor(tvix/nix-compat): drop useless try_into().expect()Florian Klink1-1/+1
This already has the right type. Change-Id: I8f5850a41f9e97f1bc5f2a45ca05cf7439665c9d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9954 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-05 r/6947 refactor(tvix/store/tests/nar_renderer): use castorepb consistentlyFlorian Klink1-10/+8
Change-Id: I40f33044c838a7cfb9b3665fdea8e70e6886343e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9953 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-05 r/6946 feat(tvix/castore): bump [Directory,File]Node size to u64Florian Klink16-57/+55
Having more than 4GiB files is quite possible (think about the NixOS graphical installer, and an uncompressed iso of it). No wire format changes. Change-Id: Ia78a07e4c554e91b93c5b9f8533266e4bd7f22b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9950 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-04 r/6940 fix(nix-compat/nixbase32): mark encode_len/decode_len constedef2-10/+3
Change-Id: Ib688bbb37cd54cfcd01e5cb3a8c376414ee8311e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9926 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-04 r/6939 refactor(tvix/glue): move builtins into separate directoryFlorian Klink7-166/+177
Change-Id: I25b7197458dbfbde8623545dc0a0286eb2744f10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9911 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 r/6938 refactor(tvix/glue): move nix_path handling to helper in tvix-glueFlorian Klink3-11/+14
Change-Id: I2327560c4cf0d3f90e253e3c2f47cb29c762461e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9910 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-04 r/6937 refactor(tvix/glue): use add_derivation_builtins helperFlorian Klink1-7/+2
Change-Id: I284c82612f1c8c81a8b7711fe63c19778ff2fdf0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9909 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-11-04 r/6936 refactor(tvix): move tvix glue code into glue crateFlorian Klink17-22/+154
There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 r/6932 refactor(tvix/eval): delay allocation when comparing attr valuesVincent Ambo1-4/+4
Delays allocation (through cloning) of the values to be compared until *after* the keys have been compared. Change-Id: I7d68c27d7a0fbcdcc387db7c092bce50ca4b94ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/9900 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-03 r/6931 chore(tvix/eval): add a marker for sorted borrowed attrs iterationVincent Ambo1-0/+6
Similar to `into_iter_sorted`, add a marker function for call sites that want *borrowed* sorted iteration. Change-Id: I7c6f14e1ac43fdb14b861b3da183eb5d12bba139 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9899 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-03 r/6930 refactor(tvix/eval): more efficiently intersect attributesVincent Ambo1-9/+70
builtins.intersectAttrs is used a _lot_ in nixpkgs eval, for whatever reason. We previously had a very inefficient implementation that would allocate for each comparison. It stuck out like a sore thumb in perf analysis. This moves to a custom algorithm with two iterators, one for the left and one for the right side, advancing them along the (borrowed) map keys until a match is found and allocation is required. I've not made any effort to reduce the verbosity of this code, I don't think it's worth it. On my machine this reduces the mean runtime of evaluating `nixpkgs.emacs.outPath` by ~8%. Change-Id: Ie506d82cb8d5f45909628f771a6b73e0eca16b27 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9898 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-03 r/6929 fix(tvix): fix `cargo test` for `nix_oracle`Florian Klink1-0/+1
As described in https://b.tvl.fyi/issues/313, the `nix_oracle` tests currently fail if run without Nix 2.3 in `$PATH`. This is not a problem for people running Nix 2.3 on their systems, and neither is it for the tests running inside Nix derivations itself (as we hardcode Nix 2.3 there), but for interactive `cargo test` runs, either using `mg shell //tvix:shell`, or `nix-shell` (from the tvix workspace) it currently is broken. Adding Nix 2.3 to the shell fixes that. Change-Id: Idbe01aa92d63eb1a3dab5797b9be5eba1e8d7f7a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9907 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-02 r/6926 fix(tvix/eval/benches): use black_box properlyFlorian Klink1-6/+3
The purpose of black_box is to actually prevent the compiler from being able to optimize computation of the benchmarked function away. To accomplish this, we need to actually *use* black_box to blackbox the input data away, rather than the return type. Change-Id: I5438982f57509fbf7b85034346a2739d76aef1fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/9902 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6922 chore(third_party/crate2nix): apply patches to fix crossFlorian Klink1-5/+7
Together with temporarily pointing nixpkgs past b10994c38c61038970a19fa60bfbec21a61755cc, this now fixes cross- compilation for tvix. This incorporates the changes proposed in https://cl.tvl.fyi/9888 and https://cl.tvl.fyi/9889, but by fixing it in crate2nix, and using the (re-)generated version of it. Changes were sent upstream at https://github.com/nix-community/crate2nix/pull/309, this pulls in a minimal patch for now. Change-Id: I70bb6f003bbc3e89de9c4eb4985ea4708ac3a9fd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9890 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-02 r/6921 docs(tvix/castore): add notes on verified streamingFlorian Klink1-0/+121
These are still a bit WIP, but should already explain the idea better than not having any documentation about how we intend to implement this at all. Let's put this into the repo, as a working document. Co-Authored-By: edef <edef@edef.eu> Change-Id: Ic0c5195cdf0a33c8382af1b5cff9064fe6ef9ffb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9894 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6920 chore(tvix/castore): move data model docs to hereFlorian Klink3-15/+20
These describe the castore data model, so it should live in the castore crate. Also, some minor edits to //tvix/store/docs/api.md, to honor the move of the castore bits to tvix-castore. Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-10-31 r/6919 chore(tvix): regenerate Cargo.nixFlorian Klink1-6/+22
Change-Id: I96e01f938a46d12a94da85968caaf190d041b9ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9887 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-28 r/6902 fix(nix-compat/narinfo): don't allocate errors earlyedef1-11/+12
We also switch the MissingField error to &'static str, since we only parse a fixed set of fields. Together, this makes the performance impact of error handling negligible in batch happy-path parsing. Change-Id: I2bd0ef2f5b35fcaced56b32d238eca75ac199ef1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9867 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-28 r/6901 fix(nix-compat/narinfo): unwrap in the benchmarkedef1-2/+6
We primarily want to measure the speed of the happy path. Change-Id: Iad0146dde86fc262e2a4b8295bde4eb297b8bf30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9866 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: edef <edef@edef.eu>
2023-10-28 r/6900 refactor(tvix/nix-compat/narinfo): return errorsFlorian Klink1-45/+127
This provides more info about where a NARInfo failed to parse, rather than just returning None and leaving a library user to manually debug. Change-Id: I9a28ddd8e5712101483ebe686fdc474c7bbc8e4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9831 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-28 r/6899 refactor(nix-compat/nixbase32): clean up encodeedef1-16/+15
Change-Id: I63da5bd47cd9033e1cc13fbe4b4864514a5343cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/9865 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6898 fix(nix-compat/nar/writer): `use ByteSlice` closer to call siteedef2-2/+2
Otherwise, it'll generate unused import warnings in release mode. Change-Id: I0399d2617a8a8b973a2b8c18f4a0a613d70c48f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9864 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6897 feat(tvix/nix-compat): add narinfo parsing and serialisationedef8-76/+492
Change-Id: I72c63414794642ca8d85c3f635f49db888420c40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9852 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6896 refactor(nix-compat/store_path): use nixbase32::decode_fixededef1-4/+2
Change-Id: I81471ee57920aa8fa889fb00c7903cdc570af9c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9863 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6895 feat(nix-compat/store_path): add StorePathRefedef1-22/+72
Change-Id: I0d888a55d93e5c22e77cb0264d09757656f731d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9862 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6894 feat(tvix/castore): carry name in ValidateNodeError::InvalidNameedef2-14/+14
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6893 refactor(tvix): turn nullary enum variants into unit variantsedef6-30/+30
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6892 refactor(nix-compat/store_path): speed up validate_name fast pathedef1-12/+39
Change-Id: Ie50b29145804777f7644952c65cb42519a8565e3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9859 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6891 feat(nix-compat/store_path): validate_name takes AsRef<[u8]>edef2-3/+4
Change-Id: I8819e2a7b63008a68f4f82035a08b960ac480dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9858 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6890 feat(nix-compat/store_path): validate_name over borrowed dataedef2-6/+10
Change-Id: Ifeb6231f48d4ad267a7acd398b4b3b687ee4d560 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9857 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6889 refactor(nix-compat/store_path): don't materialise fingerprintedef1-8/+9
Change-Id: I6a88531ded05c0dfb9232a0343a465fa02fb6989 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9856 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6888 refactor(nix-compat/store_path): make digest and name privateedef6-21/+27
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6887 fix(tvix/nix-compat): validate store path name lengthedef1-2/+13
Change-Id: I89ac0ad147a1872c021ab4235ca46ef3f51d0446 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9854 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6886 fix(tvix/nix-compat): don't box CAHash::Textedef3-14/+9
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6885 feat(tvix/nix-compat): add fixed-length base32 decodingedef1-1/+34
Change-Id: Iea9d51157c1e38e1734dc1ba752eba9b2372c61b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9850 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6884 feat(tvix/nix-compat): use AsRef<[u8]> for base32 decode inputedef1-3/+5
Change-Id: If2275dda62b852aedb64b4f9915f7dc4173cabd2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9851 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6883 refactor(tvix): condense long bytestringsedef11-93/+67
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6882 feat(tvix/nix-compat): decode base32 with a lookup tableedef1-22/+42
This also takes input validation out of the loop, leaving the loop backedge as the sole branch in the hot path. Change-Id: Id08e6fb9cf5b074780efa09a7ad389352a601bcc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9847 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6881 refactor(tvix/nix-compat): clean up base32 decodingedef1-23/+18
Change-Id: I8591a3e1075f2281b7dc49f7fa9e0027062a78a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9846 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6880 refactor(tvix/nix-compat): simplify encode_lenedef1-6/+5
Change-Id: Idfefd3b5b0d1ffbd4e9208e634a2d9989965a291 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9845 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-25 r/6878 chore(tvix): move buildkite.yml from views/tvix to tvix/Florian Klink1-0/+9
This should remove the ambiguity on where buildkite.yml should show up at least. We might still want to move workspace.josh into tvix/ too at some point, but that's blocked on https://github.com/josh-project/josh/issues/1287 This commit doesn't cause any change in views/tvix, with a previous version of this commit added to my HEAD, I ran josh-filter to verify it still points to ce0df88561fee8a4d6052ca9f2a1792f899a202a. Change-Id: Ic03e92260dc7725bc2fcf01b7c72a097d92c028e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9828 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-23 r/6876 fix(tvix/cli/derivation): fix populate_output_configurationFlorian Klink1-302/+81
The `if let` wasn't matching `outputHashAlgo` being unset, and didn't populate it in that case. Port the remaining commented-out testcases over to nix-lang based tests. Change-Id: I140b5643b9ed9d29f9522ec65d98d0b12262d728 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9825 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-23 r/6875 refactor(tvix/cli): rename `a` to `algo_str`Florian Klink1-2/+3
Change-Id: I65d307126647841b50db008c21c155441a6b6cb5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9818 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-10-23 r/6874 test(tvix/cli/derivation): add test cases for output path calcFlorian Klink4-0/+49
Get some test coverage with some calls to builtins.derivation. The expected output paths were calculated with Nix. Change-Id: I2ce205ea5244e8ef939d9cacb033283fc6f15d17 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9817 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-21T19·53+0000