about summary refs log tree commit diff
path: root/tvix
AgeCommit message (Collapse)AuthorFilesLines
2023-11-19 r/7039 refactor(tvix/nix-compat): move narinfo into separate modFlorian Klink1-2/+4
Change-Id: Id85f979e46946da0345483cbbc6de3dd29c94c63 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10077 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 r/7038 feat(tvix/store/pathinfoservice): implement NixHTTPPathInfoServiceFlorian Klink6-10/+1043
NixHTTPPathInfoService acts as a bridge in between the Nix HTTP Binary cache protocol provided by Nix binary caches such as cache.nixos.org, and the Tvix Store Model. It implements the [PathInfoService] trait in an interesting way: Every [PathInfoService::get] fetches the .narinfo and referred NAR file, inserting components into a [BlobService] and [DirectoryService], then returning a [PathInfo] struct with the root. Due to this being quite a costly operation, clients are expected to layer this service with store composition, so they're only ingested once. The client is expected to be (indirectly) using the same [BlobService] and [DirectoryService], so able to fetch referred Directories and Blobs. [PathInfoService::put] and [PathInfoService::nar] are not implemented and return an error if called. This behaves very similar to the nar-bridge-pathinfo code in nar-bridge, except it's now in Rust. Change-Id: Ia03d4fed9d0657965d100299af97cd917a03f2f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10069 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 r/7037 refactor(tvix/castore/blobservice): rm AsyncBufRead from BlobReaderFlorian Klink1-4/+1
There's no need to already require this to be buffered here. Change-Id: Ib9a11b194e0754d87ab8d2ef0b8cb0f4edc01229 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10074 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-19 r/7035 feat(nix-compat/nar/reader): provide passthrough buffered I/Oedef2-18/+80
Allow taking advantage of the buffer of the underlying reader to avoid unnecessary copies of file data. We can't easily implement the methods of BufRead directly, since we have some extra I/O to perform in the final consume() invocation. That could be resolved at the cost of additional bookkeeping, but this will suffice for now. Change-Id: I8100cf0abd79e7469670b8596bd989be5db44a91 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10089 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-19 r/7034 fix(nix-compat/nar/reader): require BufReadedef2-4/+4
We rely on being able to make small reads cheaply, so this was already an implicit practical requirement. Requiring it explicitly removes a performance footgun, and makes further optimisations possible. Change-Id: I7f65880a41b1d6b5e6bf2e52dfe47d4c49b34bcd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10088 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-19 r/7033 fix(nix-compat/store_path): valid names ⊊ UTF-8edef1-1/+2
We don't need to validate UTF-8 separately, since valid names are a strict subset of ASCII, and therefore a strict subset of UTF-8. Change-Id: I3261bf0efe3480b5b315074efafcf5e47a6c5a65 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10087 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-19 r/7032 fix(tvix): patch futures::AsyncBufReadExt::fill_bufedef5-24/+84
This fixes EOF handling for buffered readers. Link: https://github.com/rust-lang/futures-rs/pull/2801 Change-Id: Ie98ca6a3e1de38500b0195e9b62511501acb1d2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/10086 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-19 r/7031 chore(tvix): upgrade futures to 0.3.29edef2-36/+36
Change-Id: I8fd63be3cbec8766fd6d72cd9271989a19774816 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10085 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-18 r/7030 refactor(tvix/store/fs): simpllify readFlorian Klink1-19/+3
We can just use take(size) to restrict reading to that as a max. Change-Id: I0fbda74e4fb98ffeababae86a325233416029acf Reviewed-on: https://cl.tvl.fyi/c/depot/+/10072 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-18 r/7029 feat(tvix/store): From<&nix_compat::...::NarInfo<'_>> for PathInfoFlorian Klink2-2/+176
This allows converting from the NarInfo falling out of the NarInfo parser (which is a bit annoying to handle due to lifetimes) to the PathInfo proto struct. The narinfo field, containing most of the data from the original NARInfo file, as well as the references (bytes) are populated. The node field is not populated, because it requires ingesting the NAR itself to describe the root node. Change-Id: I9c04dd6ad4cae556b455188a4255e34b4f6443c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10067 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-11-18 r/7028 refactor(tvix/nix-compat): no impl <StorePathRef<'_>> for StorePathFlorian Klink1-10/+8
This suggests it's cheap to convert around, but name actually does allocate. Move to a `to_owned(&self) -> StorePath`, to better signal that this does allocate. Change-Id: Ifaf7c21599e2a467d06e2b4ae1364228370275db Reviewed-on: https://cl.tvl.fyi/c/depot/+/10066 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-18 r/7027 feat(tvix/castore): fix tracing instrument in `MemoryBlobService`Ryan Lahfa1-2/+3
Change-Id: Iedba57e8b3e1a44f14f5baa1e981275d4b02eb56 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10070 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-18 r/7026 feat(tvix/castore): impl From<std::io::Error> for ErrorFlorian Klink1-0/+10
Make it less annoying to convert from io::Error to this. We already have one direction, doesn't hurt to have the other too. Change-Id: I9fe2c6da608c9d54910ee8c397572aadb1d90d99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10068 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-17 r/7025 refactor(tvix/castore/tonic): use match in channel_from_urlFlorian Klink1-45/+48
Having random if blocks and returning from them is error-prone. Also, turns out we only need the unprefixed scheme in the fallback case, so move it down to there. Change-Id: Ifcb09279c963f8a39e0dbabe145990263f3d7cf9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10041 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-16 r/7024 docs(tvix/glue): fix doc-comment referenceFlorian Klink1-1/+1
This has been renamed to descend_to in cl/9373. Change-Id: Ia6201fb81c7d4fa953d311451cfff95373549a50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10045 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-11-15 r/7023 refactor(tvix/castore/utils): drop unused DuplexStreamWrapperFlorian Klink1-13/+1
This wasn't used at all, let's remove it. Change-Id: I426e3d93c32ebe65247ae5cf8d05b5bf686be2d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10044 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-11-15 r/7022 refactor(tvix/castore/tonic): make async, support wait-connect=?Florian Klink11-182/+170
This moves the sync `channel::from_url` to a async `tonic::channel_from_url`. It now allows connecting non-lazily if `wait- connect=1` is set in the URL params. Also, make the pingpong tests for blobsvc and directorysvc use the wait- connect=1 codepath. Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7021 refactor(tvix/castore): remove DirectoryService::from_urlFlorian Klink5-187/+98
Make directoryservice::from_addr use the more specific constructors. Change-Id: I9fee2afed77692505988d631d9fe246d9843d25a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10029 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7020 refactor(tvix/castore/blobsvc): remove BlobService::from_urlFlorian Klink5-241/+98
Make blobservice::from_addr use the more specific constructors. Change-Id: Id9637e279d6910ce6d92ff0086a984be5c65a8c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10028 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7019 refactor(tvix/store/pathinfosvc/from_addr): use test_caseFlorian Klink1-93/+46
All we do is constructing some strings, and checking if from_addr succeeds or not. This can be written in a much more concise way using test_case. Use lazy_static to provide temporary directories. Also add some more grpc-related test cases. Change-Id: Ia310dd01f617f7628f1e7e21304ac70da2ab3534 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10027 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-15 r/7018 refactor(tvix/store/pathinfosvc): inline SledPathInfoSvc::from_urlFlorian Klink2-136/+81
Change-Id: I0d905228df086a422bb30322add7236ca41e807b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10026 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7017 refactor(tvix/store/pathinfosvc): inline GRPCPathInfoSvc::from_urlFlorian Klink2-28/+27
Change-Id: Ib53b5525ae13c276e61b7f564673b7c6144ffc0e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10025 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7016 feat(tvix/castore/src/channel): move from_url testsFlorian Klink2-73/+61
These gRPC PathInfoService tests were actually not too useful in here, what we're mostly testing is the channel construction, so move it to there. Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-15 r/7015 refactor(tvix/store/pathinfosvc): inline MemoryPathInfoSvc::from_urlFlorian Klink2-88/+47
Change-Id: If27eb518d372f4004b7b38fc765a42957f2a6b50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10023 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7014 refactor(tvix/store): remove from_url from PathInfoService traitFlorian Klink4-31/+13
We don't gain much from making this part of the trait, it's still up to `tvix_store::pathinfoservice::from_addr` to do most of the construction. Move it out of the trait and into the specific *Service impls directly. This allows further refactorings in followup CLs. Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-10 r/6988 feat(tvix/tools/narinfo2parquet): initedef6-0/+9703
Convert turbofetch output to queryable Parquet. Change-Id: I076f5a431f8aab8cfe7d973bdc9fe019cebde111 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9989 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6987 chore(nix-compat/store_path): use hex_literaledef1-8/+6
Change-Id: Id093a0131aa7e3ac532daffbf5a883ca213c83ed Reviewed-on: https://cl.tvl.fyi/c/depot/+/9996 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 r/6986 feat(nix-compat/narinfo): turn flags into bitfieldsedef4-21/+30
Change-Id: I8b95723444013e97bc6ec8d282c7135b1aede114 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9987 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6985 feat(nix-compat/narinfo): permit out-of-order referencesedef1-10/+45
This appears in the cache.nixos.org dataset. Change-Id: I2eadafe8441e0132a448828026553da2dc7c12aa Reviewed-on: https://cl.tvl.fyi/c/depot/+/9994 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 r/6984 feat(nix-compat/narinfo): accept hex-encoded NarHashedef1-3/+40
This appears in the cache.nixos.org dataset. Change-Id: I35921f7ef148f6681081a4e371abb8c9cc98854d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9993 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6983 feat(nix-compat/nixbase32): use data_encoding::DecodeErroredef4-30/+28
Rather than having our own error type, just make decoding errors use the same common error type. Change-Id: Ie2c86972f3745c695253adc3214444ac0ab8db6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9995 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6982 feat(nix-compat/narinfo): default compression to bzip2edef1-2/+36
Change-Id: Ibdebc0a69672c00026c8748b93fcd2a83a0e69a5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9992 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-10 r/6981 feat(nix-compat/narinfo): track presence of unknown fieldsedef1-1/+5
Change-Id: Ia3f8a86209a0045ff98322b56a21ae20220fbe99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9991 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6980 feat(nix-compat/narinfo): permit non-SHA256 CAHash::Naredef1-3/+28
This appears in the cache.nixos.org dataset. Change-Id: I055b60b9950a1a6a36c1b0576b957e11e1d4264b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9990 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-10 r/6979 feat(tvix/tools/turbofetch): initedef9-0/+8624
Change-Id: I2efa6f94f57e812c52371256a4e62d1d54ff5057 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9925 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-07 r/6978 refactor(tvix/store/fs): mv sparse -> populated directoriesFlorian Klink3-317/+54
Do this upgrade whenever someone is actually interested in the children of a directory, but that directory doesn't contain a more detailed listing. This is much more predictable, and removes a bunch of confusing code from the inode tracker itself. Change-Id: Ib3a13694d6d5d22887d2d04ae429592137f39cb4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9982 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6977 refactor(tvix/store/fs): simplify name_in_root_to_ino_and_dataFlorian Klink1-82/+72
Have it return libc::ENOENT errors rather than an Option<…>. Also avoid having to traverse inode_data multiple times, by synthesizing the Arc<…> on our own in the insert case. In that case, the data is quite small, so cloning it is faster than traversing a second time. Change-Id: I7ab14bac8bb23859ed8d166a12070d4f4749b6d4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9981 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6976 refactor(tvix/store/fs): move code to get_directory_children helperFlorian Klink1-122/+76
As already established in the two previous CLs, these two pieces of code where doing the same. Move to a get_directory_children helper. Change-Id: Id6876f0c34f3f40a31a22d59a2cdbfef39e2d8de Reviewed-on: https://cl.tvl.fyi/c/depot/+/9980 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-07 r/6975 refactor(tvix/store/fs): reduce write lock, return childrenFlorian Klink1-47/+56
Very similar to the previous CL Change-Id: I0df07ddca742b7b9485d48771c8d295dc3aa7136 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9979 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6974 refactor(tvix/src/fs): reduce write lock, avoid inode_tracker lookupFlorian Klink1-39/+50
Code after this big match block only cares about parent_digest and children, so there's no need to do another inode_tracker.get in there. This also allows removing another if let block, right after, as we don't need to destructure parent_data anymore. Change-Id: I68fbbe3304194670caee5a453722369afa4e77ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/9978 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6973 refactor(tvix/store/fs): move inode for store_path lookup to helperFlorian Klink1-19/+13
This makes it much harder to keep the read lock around for too long, and the code a bit easier to understand. Change-Id: I7d99c85cadd433cad444b8edd34e2c43d7eaf5a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9977 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6972 fix(tvix/castore): B3Digest::{to_vec -> as_slice}edef7-19/+17
Not a single call site actually makes use of the Vec. Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-07 r/6971 feat(tvix/nix-compat): add drvfmtFlorian Klink2-0/+48
This small tool formats A-Term in a more readable format. It's a lossy conversion for non-valid UTF-8 environment values. Change-Id: I65a51054d7faf528321bc2d9fc4425180a7813f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9970 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-06 r/6963 docs(tvix/eval): document where EvalIO methods are usedVincent Ambo1-7/+34
Change-Id: I335f2ba4420973861c2a22125995b45a34d3608c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9969 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-11-06 r/6962 chore(tvix/nar-bridge): bump castore-goFlorian Klink4-23/+23
Change-Id: I81814ae326fb90ab50201cd06e3d9c0a3b15aec5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9958 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6961 chore(tvix/store-go): bump castore-goFlorian Klink3-90/+31
Change-Id: I740251785f4ffb6c9305f430e0f2c3418dfaa15d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9957 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6960 feat(tvix): add clippy target in CIFlorian Klink1-0/+24
This will invoke `cargo clippy` on the codebase, ensuring we spot things early on. Relates to b/321. Change-Id: Ifba13e5bef995d33b24346c6cee134e84d6566c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9106 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 r/6959 chore(tvix): bump proptest dependencyVincent Ambo3-99/+78
This *might* contain a fix for a clippy lint thrown by that crate. Relates to b/321. Change-Id: Ia7ebd3e26e0feb8bcc7a6c811b1e583f9016fd9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9966 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 r/6958 chore(tvix): add missing clippy attributes & configVincent Ambo4-0/+9
For cases where clippy lints don't apply to us, or something is misfiring, add appropriate configuration. Relates to b/321. Change-Id: I0af453910b4a4112bf685b2a8e9a73de10ec87ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/9965 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-05 r/6957 refactor(tvix/eval): use IntoIterator trait for owned NixAttrs iterVincent Ambo1-15/+22
Uses the standard library IntoIterator trait for the construction of our iterators. Clippy complains about duplicating this. While doing this, I opted to rename the `IntoIter` type into something that is more useful to users, in case somebody ends up working with these manually. This fixes a clippy lint, and is related to b/321. Change-Id: I851fde0d7b8b38d182343a0fd6d9f8dd2a33ee11 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9963 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-11T22·24+0000