about summary refs log tree commit diff
path: root/tvix/store
AgeCommit message (Collapse)AuthorFilesLines
2023-03-10 r/5909 refactor(tvix/store): remove anyhow::Result in directory it testFlorian Klink1-3/+1
There's no `?` being used. Change-Id: Ifbfd88f6056e0064d11458cb335d05b333ec7b6b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8127 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-10 r/5908 refactor(tvix/store): remove useless clone in testFlorian Klink1-1/+1
Change-Id: If62887cbc3338110cb7c815ec2ee2160fae04353 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8111 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5907 chore(tvix/store): move protos into separate mod.rsFlorian Klink6-3/+6
This allows adding more stuff into this namespace, from different files. Also move tests on proto-related code from src/tests to src/proto/tests. Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5906 feat(tvix/store): add blobreaderFlorian Klink2-0/+386
A BlobReader can be used to read a blob, which might consist out of multiple chunks. Chunks are fetched from a ChunkService. Change-Id: I1806225b0052adaa4a2320b79b744e554e524ee3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8088 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5894 feat(tvix/store): add blobserviceFlorian Klink4-0/+146
This adds a BlobService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5893 feat(tvix/store): add chunkserviceFlorian Klink4-0/+148
This adds the simpler ChunkService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: Icb0fdc41b37b44e9e9e4f548d0f4acae1d83b71e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8086 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-06 r/5892 feat(tvix/store): impl From<PoisonError> for ErrorFlorian Klink1-0/+7
Change-Id: Ib61e276b45c0102e383a7e7e641172b151369b03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8207 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5891 feat(tvix/store): add errorsFlorian Klink2-0/+15
Change-Id: I7be0a4eaa29840e1a3329bccb791975de22828ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8085 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-05 r/5890 docs(tvix/store): add document describing why we don't use git treesFlorian Klink1-0/+57
This came up recently again, it makes sense to document the reasoning behind the decision. Change-Id: Ic51d5bc7998c70e8b070b6f42877d8e88613935b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8223 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-03-05 r/5889 docs(tvix/store): move link hrefs to bottomFlorian Klink1-2/+3
Change-Id: Id8e32ee4de98fbe48266d417279194cd93e968cb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8222 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-02-15 r/5854 refactor(tvix/store): remove needless borrow in build.rsFlorian Klink1-1/+1
Change-Id: I2c2a70e5ffbb080702f57218aa9dcfe53d08870f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8110 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-02-15 r/5853 refactor(tvix/store/proto): use .cloned()Florian Klink1-10/+4
Instead of using an explicit closure to clone elements, use .cloned(). Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-15 r/5852 refactor(tvix/store): avoid casting to the same typeFlorian Klink1-1/+1
The size field already is u32, we don't need to convert here. Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-07 r/5840 docs(store/protos): update comment on blobstore Read and PutFlorian Klink3-8/+56
Further emphasize Read() can be used to ask for blobs OR chunks, and that clients usually want to stat and then request (smaller) chunks, rather than reading whole blobs. Also clarify that the chunking used to send BlobChunks over has nothing to do with the chunk sizes communicated in a Stat() request. Change-Id: Ia615d190aae570611de2655b11342a14d0b75976 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5794 feat(tvix/store): add write_nar functionFlorian Klink5-0/+351
This adds a function that consumes a [proto::node::Node] pointing to the root of a (store) path, and writes the contents in NAR serialization to the passed [std::io::Write]. We need this in various places: - tvix-store's calculate_nar() RPC method needs to render a NAR stream to get the nar hash, which is necessary to give things imported in the store a "NAR-based" store path. - communication with (remote) Nix (via daemon protocol) needs a NAR representation. - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a NAR representation. Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5790 feat(tvix/store): implement iteration over Directory nodesFlorian Klink3-1/+163
This provides a `Directory.nodes()` function, returning an iterator over all three node types in an ordered fashion. Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5789 feat(tvix/store/proto): implement get_name for node::Node, pub traitFlorian Klink1-3/+13
Make the trait public, so consumers can use it. Also, implement it for node::Node, so we can later use this to access the name from all three if we don't care about the enum type. Change-Id: Iae530a16b705493645e61947852c03273876cc55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5788 refactor(tvix): introduce nix-compat crateFlorian Klink7-374/+6
Move nixbase32 and store_path into this. This allows //tvix/cli to not pull in //tvix/store for now. Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5781 fix(tvix/store/nixbase32): fix encoder/decoderFlorian Klink2-69/+114
Replace our data_encoding usage with the implementation taken from https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32 Also uncomment some of the unit tests, and add a regression test for a NIXBASE32.encode with a 32 bytes sequence. The previous implementation of NIXBASE32.encode is wrong in that case: ``` ❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= 0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk ❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C 00000000 b3 a2 4d e9 7a 8f db c8 35 b9 83 31 69 50 10 30 |..M.z...5..1iP.0| 00000010 b8 97 70 31 bc b5 4b 3b 3a c1 37 40 f8 46 ab 30 |..p1..K;:.7@.F.0| 00000020 ``` Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5780 refactor(tvix/store/tests): move Directory tests into directory.rsFlorian Klink2-286/+286
There's gonna be more tests coming, let's move them into a separate file everything Directory-related into a separate module. Change-Id: I78a0f263925528907a22724b028f75cce644b329 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-30 r/5779 feat(tvix/store/proto): implement get_name for all nodesFlorian Klink1-0/+24
Also add a `NamedNode` trait. We'll later use this to access names from all three individually. Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5778 docs(tvix/store/protos): add docstring for Directory::digest()Florian Klink1-0/+2
Change-Id: I361dbca444a267fea28cd212d563ee2d03497c16 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7952 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5777 docs(tvix/store/protos): fix docstringFlorian Klink1-2/+2
Change-Id: Idf835e9648ab6c95c29960f3f8176153d4888b27 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7951 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5727 chore(tvix/store): add fastcdc crateFlorian Klink1-6/+3
This is used for content-defined chunking. Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5724 feat(tvix/store/directory): validate Directory and sizesFlorian Klink2-10/+98
This calls out to Directory::validate() for all received Directory messages, and also makes sure the sizes we refer a Directory message as matches the sizes that have been calculated. Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5723 feat(tvix/store/directory): deduplicate Directory messagesFlorian Klink2-9/+82
We can omit sending Directory messages to clients that have already been sent in the same stream. We can also omit storing a Directory message if we already have it - they're content-addressed anyways. Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5722 feat(tvix/store): add SledDirectoryServiceFlorian Klink6-37/+411
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5720 chore(tvix/store): add tonic-mockFlorian Klink1-0/+1
Upstream seems to be dead, so we're using https://github.com/tyrchen/ tonic-mock/pull/3 here. According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment- 1241164173, we might not need this crate at all, but for now, it gets the job done and is less code to write in the tests. Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5719 refactor(tvix/store): simplify test a bitFlorian Klink1-24/+17
Import more things, and use expect_err to unpack the response. Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 r/5698 docs(tvix/store): fix docstring for StorePath::to_absolute_pathFlorian Klink1-1/+1
Change-Id: I8b36c85fe11e4fb62e5d28f6900cd80d89a5cc41 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7854 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-18 r/5697 feat(tvix/store): rename and expose StorePath::validate_nameFlorian Klink1-2/+3
Change-Id: I8a16c214c7c644756d9d54187beba8c80ccfb39c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7853 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-14 r/5658 chore(tvix/store): derive Clone for StorePathVincent Ambo1-1/+1
Change-Id: Id325afa766b0bd2a3d04cf538f13e77692a22687 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7829 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-12 r/5649 feat(tvix/store): implement PathInfoService with sledFlorian Klink7-37/+169
This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-12 r/5648 chore(tvix/store): update ParseStorePathError messagesFlorian Klink1-3/+3
Change-Id: I49799393a1e35b4475566819111beb57a628c555 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7801 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-12 r/5647 chore(tvix/store): update ValidatePathInfoError messagesFlorian Klink1-1/+1
Change-Id: I8f16e25d998a74ac110ae99eec0edbbfd720c8dd Reviewed-on: https://cl.tvl.fyi/c/depot/+/7800 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 r/5615 refactor(tvix/store): move protobuf build config one level upVincent Ambo2-35/+13
This embeds the build config directly at the point where `Cargo.nix` is imported, making it transparent to library consumers. Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-06 r/5614 feat(tvix/store): document StorePath a bit more.Florian Klink1-0/+10
Change-Id: Ifab28d97ddc22a2073c5df5e6e2cefb51b4b9191 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7777 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 r/5613 refactor(tvix/store): rename NixPath to StorePathFlorian Klink5-50/+53
As discussed in #tvl, this is a more common term for it. Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-06 r/5611 feat(tvix/store): add NixPath::to_absolute_pathFlorian Klink1-0/+11
This provides a function returning a string starting with the store path prefix, the counterpart of `from_absolute_path`. Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-04 r/5586 feat(tvix/store): implement Nixpath::from_absolute_pathFlorian Klink1-1/+39
This allows constructing a NixPath from an absolute path. It pops off the STORE_DIR prefix and the trailing slash and returns an error if it couldn't be found. Change-Id: Ib540e353c63cc247ac15e20414b0db2caf695ef4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7751 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5572 feat(tvix/store/nixpath): DIGEST_SIZE public, use more constsFlorian Klink1-12/+18
Rename PATH_HASH_SIZE to DIGEST_SIZE. It's a digest, not hash (we don't necessarily have the internal hash state anymore), and the fact it's about (Nix)Paths is already visible from the module name. Also expose ENCODED_DIGEST_SIZE, so we don't need to do the calculation inside from_string() method, and it becomes more clear this is a constant. Change-Id: I0e7577dd7a6e503039037b986313b214e995d826 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7725 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: jrhahn <mail.jhahn@gmail.com> Tested-by: BuildkiteCI
2023-01-03 r/5571 feat(tvix/store/nixpath): expose digest and name fieldsFlorian Klink1-2/+2
These can be accessed directly. Change-Id: I71dc84f982820d53f319efefbed9b9055034954d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7724 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5570 feat(src/proto): add PathInfo.validate()Florian Klink3-0/+307
This provides validation of PathInfo messages, and ensures the output hashes are properly parsed from the root node names. NixPath already has a more extensive test suite for various wrong NixPaths, so it's omitted from here. Change-Id: I5d69118df5816daabb521ddb19d178bddd1caacf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7684 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-03 r/5569 chore(tvix/store): make importableFlorian Klink4-19/+27
This allows other crates to import tvix_store. Rename the bin crate to tvix-store-bin, to avoid having multiple crates with the same name (https://github.com/rust-lang/cargo/issues/6313) Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5568 chore(tvix/store/nixbase32): address clippyFlorian Klink1-1/+1
Change-Id: Ib47a55e39ed752492b9732439de5f327a7fa601e Reviewed-on: https://cl.tvl.fyi/c/depot/+/7723 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5567 chore(tvix/store/main): address clippyFlorian Klink1-1/+1
Change-Id: Ie400a7109d3aa6b66153c0ab25a38abf06984cbf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7722 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5558 feat(tvix/store): add logging with tracingFlorian Klink5-9/+44
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/ tracing. Annotate all method handlers with an instrument macro, and warn! a message for them being unimplemented. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5557 feat(tvix/store): implement reflectionFlorian Klink5-25/+68
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5556 refactor(tvix/store): make nixbase32 reversal more idiomaticedef1-7/+4
Change-Id: Ibe550f9573b0f45ee95453b50c7510e49b07c719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7685 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-30 r/5555 feat(tvix/store): refactor digest conversionJürgen Hahn1-5/+1
This refactors how the original digest type (Vec<u8>) is converted to [u8; PATH_HASH_SIZE]. Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-05T23·13+0000