about summary refs log tree commit diff
path: root/tvix/store
AgeCommit message (Collapse)AuthorFilesLines
2023-11-15 r/7019 refactor(tvix/store/pathinfosvc/from_addr): use test_caseFlorian Klink1-93/+46
All we do is constructing some strings, and checking if from_addr succeeds or not. This can be written in a much more concise way using test_case. Use lazy_static to provide temporary directories. Also add some more grpc-related test cases. Change-Id: Ia310dd01f617f7628f1e7e21304ac70da2ab3534 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10027 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-15 r/7018 refactor(tvix/store/pathinfosvc): inline SledPathInfoSvc::from_urlFlorian Klink2-136/+81
Change-Id: I0d905228df086a422bb30322add7236ca41e807b Reviewed-on: https://cl.tvl.fyi/c/depot/+/10026 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7017 refactor(tvix/store/pathinfosvc): inline GRPCPathInfoSvc::from_urlFlorian Klink2-28/+27
Change-Id: Ib53b5525ae13c276e61b7f564673b7c6144ffc0e Reviewed-on: https://cl.tvl.fyi/c/depot/+/10025 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7016 feat(tvix/castore/src/channel): move from_url testsFlorian Klink1-73/+1
These gRPC PathInfoService tests were actually not too useful in here, what we're mostly testing is the channel construction, so move it to there. Change-Id: Ic8c07558a1b28b46f863d5c39bcaa3a79cea007a Reviewed-on: https://cl.tvl.fyi/c/depot/+/10024 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-15 r/7015 refactor(tvix/store/pathinfosvc): inline MemoryPathInfoSvc::from_urlFlorian Klink2-88/+47
Change-Id: If27eb518d372f4004b7b38fc765a42957f2a6b50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10023 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-15 r/7014 refactor(tvix/store): remove from_url from PathInfoService traitFlorian Klink4-31/+13
We don't gain much from making this part of the trait, it's still up to `tvix_store::pathinfoservice::from_addr` to do most of the construction. Move it out of the trait and into the specific *Service impls directly. This allows further refactorings in followup CLs. Change-Id: I99b93ef4acd83637a2f4888a1e586f1ca96390dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/10022 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6978 refactor(tvix/store/fs): mv sparse -> populated directoriesFlorian Klink3-317/+54
Do this upgrade whenever someone is actually interested in the children of a directory, but that directory doesn't contain a more detailed listing. This is much more predictable, and removes a bunch of confusing code from the inode tracker itself. Change-Id: Ib3a13694d6d5d22887d2d04ae429592137f39cb4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9982 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6977 refactor(tvix/store/fs): simplify name_in_root_to_ino_and_dataFlorian Klink1-82/+72
Have it return libc::ENOENT errors rather than an Option<…>. Also avoid having to traverse inode_data multiple times, by synthesizing the Arc<…> on our own in the insert case. In that case, the data is quite small, so cloning it is faster than traversing a second time. Change-Id: I7ab14bac8bb23859ed8d166a12070d4f4749b6d4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9981 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6976 refactor(tvix/store/fs): move code to get_directory_children helperFlorian Klink1-122/+76
As already established in the two previous CLs, these two pieces of code where doing the same. Move to a get_directory_children helper. Change-Id: Id6876f0c34f3f40a31a22d59a2cdbfef39e2d8de Reviewed-on: https://cl.tvl.fyi/c/depot/+/9980 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-11-07 r/6975 refactor(tvix/store/fs): reduce write lock, return childrenFlorian Klink1-47/+56
Very similar to the previous CL Change-Id: I0df07ddca742b7b9485d48771c8d295dc3aa7136 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9979 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6974 refactor(tvix/src/fs): reduce write lock, avoid inode_tracker lookupFlorian Klink1-39/+50
Code after this big match block only cares about parent_digest and children, so there's no need to do another inode_tracker.get in there. This also allows removing another if let block, right after, as we don't need to destructure parent_data anymore. Change-Id: I68fbbe3304194670caee5a453722369afa4e77ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/9978 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6973 refactor(tvix/store/fs): move inode for store_path lookup to helperFlorian Klink1-19/+13
This makes it much harder to keep the read lock around for too long, and the code a bit easier to understand. Change-Id: I7d99c85cadd433cad444b8edd34e2c43d7eaf5a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9977 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-07 r/6972 fix(tvix/castore): B3Digest::{to_vec -> as_slice}edef1-4/+4
Not a single call site actually makes use of the Vec. Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-11-05 r/6958 chore(tvix): add missing clippy attributes & configVincent Ambo1-0/+1
For cases where clippy lints don't apply to us, or something is misfiring, add appropriate configuration. Relates to b/321. Change-Id: I0af453910b4a4112bf685b2a8e9a73de10ec87ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/9965 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-05 r/6955 chore(tvix): fix trivial clippy lintsVincent Ambo1-1/+1
Relates to b/321. Change-Id: I37284f89b186e469eb432e2bbedb37aa125a6ad4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9961 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2023-11-05 r/6951 feat(tvix/store/nar): add read_nar methodFlorian Klink2-0/+338
This adds a function that can ingest a NAR file into a given blob_service and directory_service, returning the root node. Passing around the directory_putter is a bit annoying, but at least for this linear "reading through a NAR" case, we don't need it to be clone or sync. Change-Id: Ia26d596f5dffc6dbd69aa39a82b59bba41685a9c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9915 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-05 r/6947 refactor(tvix/store/tests/nar_renderer): use castorepb consistentlyFlorian Klink1-10/+8
Change-Id: I40f33044c838a7cfb9b3665fdea8e70e6886343e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9953 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-11-05 r/6946 feat(tvix/castore): bump [Directory,File]Node size to u64Florian Klink5-11/+11
Having more than 4GiB files is quite possible (think about the NixOS graphical installer, and an uncompressed iso of it). No wire format changes. Change-Id: Ia78a07e4c554e91b93c5b9f8533266e4bd7f22b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9950 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-11-02 r/6920 chore(tvix/castore): move data model docs to hereFlorian Klink3-120/+18
These describe the castore data model, so it should live in the castore crate. Also, some minor edits to //tvix/store/docs/api.md, to honor the move of the castore bits to tvix-castore. Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-10-27 r/6893 refactor(tvix): turn nullary enum variants into unit variantsedef2-7/+7
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6888 refactor(nix-compat/store_path): make digest and name privateedef4-7/+11
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-27 r/6883 refactor(tvix): condense long bytestringsedef1-4/+1
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-22 r/6870 feat(tvix/store/protos): add CA fieldFlorian Klink3-1/+63
This adds support to represent the `CA` field found in some .narinfo files. As `deriver`, it's also a subfield of the `narinfo` field. Extending nix-compat with a more accessible data structure that can take care of formatting, as well as writing validation functions in Rust + Golang, and integrating it into nar-bridge is something for a followup CL. Change-Id: I71e9c30957bcd03051a491aa54d7baac25b6dd2d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9795 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-22 r/6869 refactor(tvix/store): move deriver field into narinfo submessageFlorian Klink5-20/+24
This information is present in the .narinfo files, it should have gone there. Change-Id: Ib43d0cf30c2795bf1fe77c46646174353ade0458 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9794 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-17 r/6848 chore(tvix/store-go): rename go moduleFlorian Klink2-2/+2
`code.tvl.fyi/tvix/store/protos` now points to a directory that only contains the `.proto` files, while all golang tooling and .pb.go files live in tvix/store-go. As discussed in https://cl.tvl.fyi/c/depot/+/9787/comment/fc5d155c_1bd38e3a/, the amount of people currently using this is still small, so rename the go.mod now, while it doesn't yet hurt. Also, use code.tvl.fyi/tvix/castore-go instead of code.tvl.fyi/tvix/ castore/protos, to make use of cl/9791. Change-Id: I9ea89957d7c29dfae4c893b9aae8ac8a0bad2d8e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9792 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 r/6844 chore(tvix): move store golang bindings to tvix/store-goFlorian Klink14-1953/+34
Similar to the castore-go CL before, this also updates the store-go bindings to the new layout. Change-Id: Id73d7ad43f7d70171ab021728e303300c5db71f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9788 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-16 r/6826 refactor(tvix/store): clippyFlorian Klink1-1/+1
Change-Id: I31ba268bda21c1f07c63520a269dd62f7b309e99 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9748 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-16 r/6823 refactor(tvix/store/fs): clippyFlorian Klink2-7/+14
Change-Id: I11247e24475f0e49dc22f2d10bf3e1e276737899 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9709 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-14 r/6810 refactor(tvix/store): Upgrade tokio-listener to get tonic supportConnor Brewster4-136/+11
Tonic support was added to tokio-listener upstream which removes the need for use to have tonic compatibility wrapper types around it. See: https://github.com/vi/tokio-listener/pull/2 Fixes b/311 Change-Id: I04a2dbb3bc3c8bfe9339583c0b46070c7ec97811 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9721 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-14 r/6806 feat(tvix/store/protos): use Validate() function on root nodeFlorian Klink3-36/+21
This updates the code to make use of the new Validate() function defined on a Node. Change-Id: I9b6ed694661f41e700f19cc78d53d2224b61852d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9718 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-14 r/6805 feat(tvix/store/protos): add Deriver field to PathInfoFlorian Klink8-39/+144
This uses the newly introduced StorePath message type to add a Deriver field to the PathInfo message. Support for validation is added to both the golang and rust implementation. This includes extending unit tests. Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6804 feat(tvix/store/protos): add StorePath messageFlorian Klink3-48/+137
This encodes a store path a bit more concise, which is used in the Deriver field. Change-Id: Ibfb54d3b206917e51970d1d5fe94fcedb901704b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9646 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6803 refactor(tvix/store/fs): make fetch_directory_inode_data asyncFlorian Klink1-30/+32
To make this easier, move it outside of TvixStoreFs, and accept the DirectoryService as a function argument, so we don't need to worry about the lifetime of self. This also aligns with how we spawn async tasks inside the rest of TvixStoreFs. Change-Id: I3b95072209d32039f05aed122240f2d6db7ad172 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9713 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-14 r/6802 refactor(tvix/*store): helper for channel creation from urlFlorian Klink1-41/+4
This moves the repetitive code to parse a URL and create a channel connected to it into `tvix_castore::channel::from_url`. Part of b/308 Change-Id: Idd342cd71cad5e78a9b258b38c1b227993e75310 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9707 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 r/6801 refactor(tvix/*store): reorganize from_urlFlorian Klink1-14/+9
Move the channel creation depending on the string-based URL into its own block. Change-Id: I546b769acd2296b548eb966b62c495f910266df5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9706 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6800 refactor(tvix/*store): remove some grpc_client let bindingsFlorian Klink1-20/+17
We had to have these all while the traits where sync, and there was a lot of spawning and moving. Most of this can now be removed in favor of some inline `.clone()`. Change-Id: Id5466c32a403100bc3347866b3172e06a792e311 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9705 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-13 r/6795 refactor(tvix/store/protos): remove unused mustBlobDigest functionFlorian Klink1-10/+0
Change-Id: I79a73de0cb06abcf16a6579ae298876004ab0ba8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9719 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-12 r/6794 refactor(tvix/castore): factor out node checksFlorian Klink2-34/+35
Implement `validate()` on `node::Node`, and call it from PathInfo's validate() too. Node-related errors are moved to a ValidateNodeError error type. This additionally adds some more validations for symlink targets (they must not be empty, and not contain null bytes). Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-12 r/6793 refactor(tvix/store/fs): use while let statementFlorian Klink1-22/+16
Suggested by clippy. Change-Id: I686b1abcb40fab42017972a3f814fae29fa70c25 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9712 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-12 r/6792 refactor(tvix/store/proto): merge two match statements into oneFlorian Klink1-29/+23
Change-Id: I3daca008dff5527169f5916f4845234e8f3263cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9711 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-12 r/6791 refactor(tvix/store): clippy lintFlorian Klink4-4/+4
Change-Id: I03aa5c6cb0167ff6cb7312d81d1e43116fcb6d70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9710 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-12 r/6789 feat(tvix/store): enable tls[-roots] feature in tonicFlorian Klink1-1/+1
This teaches tonic to speak https, so `grpc+https` becomes a working connection string. Change-Id: I9c550ea665b829725b545a164cf6f78587bd5213 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9714 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-12 r/6788 fix(tvix/store/fs): fix typoedef1-3/+3
Change-Id: I132987dda1073dcc0b30e50e26682d6c8724d988 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9689 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-11 r/6786 feat(tvix/store): validate nar_sha256Florian Klink2-2/+26
Change-Id: I4c4dcdb75ea7748f2ab01a0bab218596b90b7b58 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9648 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-11 r/6785 feat(tvix/store/protos): validate NarSha256Florian Klink2-1/+17
Change-Id: I016d1b020b04850f0dca68c2ea96643230dada10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9649 Reviewed-by: Brian McGee <brian@bmcgee.ie> Tested-by: BuildkiteCI
2023-10-11 r/6782 docs(tvix/store/protos): explain a bit more what we're doingFlorian Klink1-1/+3
Explain a bit better what we do in the codeblock below, and add a newline to make it clear we don't just document the first branch. Change-Id: Ifb142d9984e6d2cbca648525c10298f2fcbdfedf Reviewed-on: https://cl.tvl.fyi/c/depot/+/9645 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de>
2023-10-10 r/6774 refactor(tvix/store/protos): have Export accept root nodeFlorian Klink2-43/+34
We don't need the full PathInfo message, only the root node. Change-Id: I667045ed766875dfbf8ac126a50b02baa2df67a4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9604 Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-10 r/6772 feat(tvix/store/protos): add ExportFlorian Klink8-1/+482
Export will traverse a given PathInfo structure, and write the contents in NAR format to the passed Writer. It uses directoryLookupFn and blobLookupFn to resolve references. This is being moved over from nar-bridge. We need to keep the code there around until we can bump go.mod to storev1 with this merged, but the tests can already be moved entirely. Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602 Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie> Autosubmit: flokli <flokli@flokli.de>
2023-10-10 r/6769 feat(tvix/store): use tvix_compat::nar::writer::asyncedef3-69/+101
Change-Id: Iad36872244df6f2225a2884f6b20cacd8f918b31 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9619 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu>
2023-10-09 r/6761 chore(tvix/*store): use google.golang.org/protobufFlorian Klink2-3/+3
The other one is deprecated, according to https://pkg.go.dev/github.com/golang/protobuf/proto. Change-Id: I502d5bfa7431df0dea3d564f1e0ecfea14c90e10 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9597 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Brian McGee <brian@bmcgee.ie>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-24T04·10+0000