about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2023-11-15 r/7017 refactor(tvix/store/pathinfosvc): inline GRPCPathInfoSvc::from_urlFlorian Klink2-28/+27
Change-Id: Ib53b5525ae13c276e61b7f564673b7c6144ffc0e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10025 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7016 feat(tvix/castore/src/channel): move from_url testsFlorian Klink2-73/+61
These gRPC PathInfoService tests were actually not too useful in here, what we're mostly testing is the channel construction, so move it to there. Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-15 r/7015 refactor(tvix/store/pathinfosvc): inline MemoryPathInfoSvc::from_urlFlorian Klink2-88/+47
Change-Id: If27eb518d372f4004b7b38fc765a42957f2a6b50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10023 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7014 refactor(tvix/store): remove from_url from PathInfoService traitFlorian Klink4-31/+13
We don't gain much from making this part of the trait, it's still up to `tvix_store::pathinfoservice::from_addr` to do most of the construction. Move it out of the trait and into the specific *Service impls directly. This allows further refactorings in followup CLs. Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-10 r/6988 feat(tvix/tools/narinfo2parquet): initedef6-0/+9703
Convert turbofetch output to queryable Parquet. Change-Id: I076f5a431f8aab8cfe7d973bdc9fe019cebde111 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9989 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6987 chore(nix-compat/store_path): use hex_literaledef1-8/+6
Change-Id: Id093a0131aa7e3ac532daffbf5a883ca213c83ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/9996 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 r/6986 feat(nix-compat/narinfo): turn flags into bitfieldsedef4-21/+30
Change-Id: I8b95723444013e97bc6ec8d282c7135b1aede114 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9987 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6985 feat(nix-compat/narinfo): permit out-of-order referencesedef1-10/+45
This appears in the cache.nixos.org dataset. Change-Id: I2eadafe8441e0132a448828026553da2dc7c12aa Reviewed-on: https://cl.tvl.fyi/c/depot/+/9994 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 r/6984 feat(nix-compat/narinfo): accept hex-encoded NarHashedef1-3/+40
This appears in the cache.nixos.org dataset. Change-Id: I35921f7ef148f6681081a4e371abb8c9cc98854d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9993 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6983 feat(nix-compat/nixbase32): use data_encoding::DecodeErroredef4-30/+28
Rather than having our own error type, just make decoding errors use the same common error type. Change-Id: Ie2c86972f3745c695253adc3214444ac0ab8db6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6982 feat(nix-compat/narinfo): default compression to bzip2edef1-2/+36
Change-Id: Ibdebc0a69672c00026c8748b93fcd2a83a0e69a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9992 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 r/6981 feat(nix-compat/narinfo): track presence of unknown fieldsedef1-1/+5
Change-Id: Ia3f8a86209a0045ff98322b56a21ae20220fbe99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9991 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6980 feat(nix-compat/narinfo): permit non-SHA256 CAHash::Naredef1-3/+28
This appears in the cache.nixos.org dataset. Change-Id: I055b60b9950a1a6a36c1b0576b957e11e1d4264b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9990 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6979 feat(tvix/tools/turbofetch): initedef9-0/+8624
Change-Id: I2efa6f94f57e812c52371256a4e62d1d54ff5057 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9925 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-07 r/6978 refactor(tvix/store/fs): mv sparse -> populated directoriesFlorian Klink3-317/+54
Do this upgrade whenever someone is actually interested in the children of a directory, but that directory doesn't contain a more detailed listing. This is much more predictable, and removes a bunch of confusing code from the inode tracker itself. Change-Id: Ib3a13694d6d5d22887d2d04ae429592137f39cb4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9982 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6977 refactor(tvix/store/fs): simplify name_in_root_to_ino_and_dataFlorian Klink1-82/+72
Have it return libc::ENOENT errors rather than an Option<…>. Also avoid having to traverse inode_data multiple times, by synthesizing the Arc<…> on our own in the insert case. In that case, the data is quite small, so cloning it is faster than traversing a second time. Change-Id: I7ab14bac8bb23859ed8d166a12070d4f4749b6d4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9981 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6976 refactor(tvix/store/fs): move code to get_directory_children helperFlorian Klink1-122/+76
As already established in the two previous CLs, these two pieces of code where doing the same. Move to a get_directory_children helper. Change-Id: Id6876f0c34f3f40a31a22d59a2cdbfef39e2d8de Reviewed-on: https://cl.tvl.fyi/c/depot/+/9980 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-07 r/6975 refactor(tvix/store/fs): reduce write lock, return childrenFlorian Klink1-47/+56
Very similar to the previous CL Change-Id: I0df07ddca742b7b9485d48771c8d295dc3aa7136 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9979 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6974 refactor(tvix/src/fs): reduce write lock, avoid inode_tracker lookupFlorian Klink1-39/+50
Code after this big match block only cares about parent_digest and children, so there's no need to do another inode_tracker.get in there. This also allows removing another if let block, right after, as we don't need to destructure parent_data anymore. Change-Id: I68fbbe3304194670caee5a453722369afa4e77ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/9978 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6973 refactor(tvix/store/fs): move inode for store_path lookup to helperFlorian Klink1-19/+13
This makes it much harder to keep the read lock around for too long, and the code a bit easier to understand. Change-Id: I7d99c85cadd433cad444b8edd34e2c43d7eaf5a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9977 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6972 fix(tvix/castore): B3Digest::{to_vec -> as_slice}edef7-19/+17
Not a single call site actually makes use of the Vec. Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-07 r/6971 feat(tvix/nix-compat): add drvfmtFlorian Klink2-0/+48
This small tool formats A-Term in a more readable format. It's a lossy conversion for non-valid UTF-8 environment values. Change-Id: I65a51054d7faf528321bc2d9fc4425180a7813f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9970 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-06 r/6963 docs(tvix/eval): document where EvalIO methods are usedVincent Ambo1-7/+34
Change-Id: I335f2ba4420973861c2a22125995b45a34d3608c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9969 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-11-06 r/6962 chore(tvix/nar-bridge): bump castore-goFlorian Klink4-23/+23
Change-Id: I81814ae326fb90ab50201cd06e3d9c0a3b15aec5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9958 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6961 chore(tvix/store-go): bump castore-goFlorian Klink3-90/+31
Change-Id: I740251785f4ffb6c9305f430e0f2c3418dfaa15d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9957 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6960 feat(tvix): add clippy target in CIFlorian Klink1-0/+24
This will invoke `cargo clippy` on the codebase, ensuring we spot things early on. Relates to b/321. Change-Id: Ifba13e5bef995d33b24346c6cee134e84d6566c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9106 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6959 chore(tvix): bump proptest dependencyVincent Ambo3-99/+78
This *might* contain a fix for a clippy lint thrown by that crate. Relates to b/321. Change-Id: Ia7ebd3e26e0feb8bcc7a6c811b1e583f9016fd9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9966 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 r/6958 chore(tvix): add missing clippy attributes & configVincent Ambo4-0/+9
For cases where clippy lints don't apply to us, or something is misfiring, add appropriate configuration. Relates to b/321. Change-Id: I0af453910b4a4112bf685b2a8e9a73de10ec87ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/9965 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-05 r/6957 refactor(tvix/eval): use IntoIterator trait for owned NixAttrs iterVincent Ambo1-15/+22
Uses the standard library IntoIterator trait for the construction of our iterators. Clippy complains about duplicating this. While doing this, I opted to rename the `IntoIter` type into something that is more useful to users, in case somebody ends up working with these manually. This fixes a clippy lint, and is related to b/321. Change-Id: I851fde0d7b8b38d182343a0fd6d9f8dd2a33ee11 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9963 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 r/6956 refactor(tvix/nix-compat): check presence with btree_map's entry APIVincent Ambo1-8/+9
Walking a btree_map twice is more expensive than copying a string, especially because the cloning only happens in the (non-hot) error path. This fixes a clippy lint, so it's related to b/321. Change-Id: I2ccfd0bc46792a45d277f47564e595b87107d8be Reviewed-on: https://cl.tvl.fyi/c/depot/+/9962 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-05 r/6955 chore(tvix): fix trivial clippy lintsVincent Ambo14-36/+35
Relates to b/321. Change-Id: I37284f89b186e469eb432e2bbedb37aa125a6ad4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9961 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 r/6954 feat(tvix/castore): extend blobstore protos for verified streamingFlorian Klink5-91/+264
This pdates the proto docstrings a bit, especially w.r.t. verified streaming. It also adds send_chunks, send_bao fields to StatBlobRequest (renamed from BlobMeta) Change-Id: I590cc8646d86b73bca9f38a9b6d9ea15e4df5cb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9951 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 r/6953 fix(tvix/*store-go): fix depot checksFlorian Klink2-2/+2
This wasn't doing anything, because $(regenerate) was a directory: /nix/store/cxfxvz76zh9vb7x26h3cx98gkv234jz2-pb-go-check: line 2: /nix/store/my1nd1qvg5iis38rfyn1pm2c7ib7myn5-regenerate: Is a directory Change-Id: I6ebed1d7b84dceb885c1f25527ce62d973146819 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9959 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6952 chore(tvix/store-go): regenerate proto bindingsFlorian Klink1-25/+223
As surfaced in cl/9959, we didn't actually spot having to run the regeneration in CI. Change-Id: Iabdb79339d2a220a6cb368f9c5c70166f1ce9eb3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9960 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6951 feat(tvix/store/nar): add read_nar methodFlorian Klink2-0/+338
This adds a function that can ingest a NAR file into a given blob_service and directory_service, returning the root node. Passing around the directory_putter is a bit annoying, but at least for this linear "reading through a NAR" case, we don't need it to be clone or sync. Change-Id: Ia26d596f5dffc6dbd69aa39a82b59bba41685a9c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9915 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-05 r/6950 refactor(tvix/nix-compat): remove unused importsFlorian Klink1-2/+1
Change-Id: I64523df2344233d9e424812d94b1c7ed2ecb1a74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9956 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 r/6949 refactor(tvix/nix-compat): use matches! macroFlorian Klink1-5/+1
Flagged by ``#[warn(clippy::match_like_matches_macro)]`.` Change-Id: If07109e5ec01b05df898119f9a577196dfe11b37 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9955 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-05 r/6948 refactor(tvix/nix-compat): drop useless try_into().expect()Florian Klink1-1/+1
This already has the right type. Change-Id: I8f5850a41f9e97f1bc5f2a45ca05cf7439665c9d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9954 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-05 r/6947 refactor(tvix/store/tests/nar_renderer): use castorepb consistentlyFlorian Klink1-10/+8
Change-Id: I40f33044c838a7cfb9b3665fdea8e70e6886343e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9953 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-05 r/6946 feat(tvix/castore): bump [Directory,File]Node size to u64Florian Klink16-57/+55
Having more than 4GiB files is quite possible (think about the NixOS graphical installer, and an uncompressed iso of it). No wire format changes. Change-Id: Ia78a07e4c554e91b93c5b9f8533266e4bd7f22b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9950 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-04 r/6940 fix(nix-compat/nixbase32): mark encode_len/decode_len constedef2-10/+3
Change-Id: Ib688bbb37cd54cfcd01e5cb3a8c376414ee8311e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9926 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-04 r/6939 refactor(tvix/glue): move builtins into separate directoryFlorian Klink7-166/+177
Change-Id: I25b7197458dbfbde8623545dc0a0286eb2744f10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9911 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-04 r/6938 refactor(tvix/glue): move nix_path handling to helper in tvix-glueFlorian Klink3-11/+14
Change-Id: I2327560c4cf0d3f90e253e3c2f47cb29c762461e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9910 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-04 r/6937 refactor(tvix/glue): use add_derivation_builtins helperFlorian Klink1-7/+2
Change-Id: I284c82612f1c8c81a8b7711fe63c19778ff2fdf0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9909 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-11-04 r/6936 refactor(tvix): move tvix glue code into glue crateFlorian Klink17-22/+154
There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 r/6932 refactor(tvix/eval): delay allocation when comparing attr valuesVincent Ambo1-4/+4
Delays allocation (through cloning) of the values to be compared until *after* the keys have been compared. Change-Id: I7d68c27d7a0fbcdcc387db7c092bce50ca4b94ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/9900 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-03 r/6931 chore(tvix/eval): add a marker for sorted borrowed attrs iterationVincent Ambo1-0/+6
Similar to `into_iter_sorted`, add a marker function for call sites that want *borrowed* sorted iteration. Change-Id: I7c6f14e1ac43fdb14b861b3da183eb5d12bba139 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9899 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-03 r/6930 refactor(tvix/eval): more efficiently intersect attributesVincent Ambo1-9/+70
builtins.intersectAttrs is used a _lot_ in nixpkgs eval, for whatever reason. We previously had a very inefficient implementation that would allocate for each comparison. It stuck out like a sore thumb in perf analysis. This moves to a custom algorithm with two iterators, one for the left and one for the right side, advancing them along the (borrowed) map keys until a match is found and allocation is required. I've not made any effort to reduce the verbosity of this code, I don't think it's worth it. On my machine this reduces the mean runtime of evaluating `nixpkgs.emacs.outPath` by ~8%. Change-Id: Ie506d82cb8d5f45909628f771a6b73e0eca16b27 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9898 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-03 r/6929 fix(tvix): fix `cargo test` for `nix_oracle`Florian Klink1-0/+1
As described in https://b.tvl.fyi/issues/313, the `nix_oracle` tests currently fail if run without Nix 2.3 in `$PATH`. This is not a problem for people running Nix 2.3 on their systems, and neither is it for the tests running inside Nix derivations itself (as we hardcode Nix 2.3 there), but for interactive `cargo test` runs, either using `mg shell //tvix:shell`, or `nix-shell` (from the tvix workspace) it currently is broken. Adding Nix 2.3 to the shell fixes that. Change-Id: Idbe01aa92d63eb1a3dab5797b9be5eba1e8d7f7a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9907 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-02 r/6926 fix(tvix/eval/benches): use black_box properlyFlorian Klink1-6/+3
The purpose of black_box is to actually prevent the compiler from being able to optimize computation of the benchmarked function away. To accomplish this, we need to actually *use* black_box to blackbox the input data away, rather than the return type. Change-Id: I5438982f57509fbf7b85034346a2739d76aef1fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/9902 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-21T20·13+0000