about summary refs log tree commit diff
path: root/tvix/store/src
AgeCommit message (Collapse)AuthorFilesLines
2023-03-10 r/5907 chore(tvix/store): move protos into separate mod.rsFlorian Klink6-3/+6
This allows adding more stuff into this namespace, from different files. Also move tests on proto-related code from src/tests to src/proto/tests. Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5906 feat(tvix/store): add blobreaderFlorian Klink2-0/+386
A BlobReader can be used to read a blob, which might consist out of multiple chunks. Chunks are fetched from a ChunkService. Change-Id: I1806225b0052adaa4a2320b79b744e554e524ee3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8088 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5894 feat(tvix/store): add blobserviceFlorian Klink4-0/+146
This adds a BlobService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5893 feat(tvix/store): add chunkserviceFlorian Klink4-0/+148
This adds the simpler ChunkService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: Icb0fdc41b37b44e9e9e4f548d0f4acae1d83b71e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8086 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-06 r/5892 feat(tvix/store): impl From<PoisonError> for ErrorFlorian Klink1-0/+7
Change-Id: Ib61e276b45c0102e383a7e7e641172b151369b03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8207 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5891 feat(tvix/store): add errorsFlorian Klink2-0/+15
Change-Id: I7be0a4eaa29840e1a3329bccb791975de22828ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8085 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-15 r/5853 refactor(tvix/store/proto): use .cloned()Florian Klink1-10/+4
Instead of using an explicit closure to clone elements, use .cloned(). Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-15 r/5852 refactor(tvix/store): avoid casting to the same typeFlorian Klink1-1/+1
The size field already is u32, we don't need to convert here. Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-31 r/5794 feat(tvix/store): add write_nar functionFlorian Klink5-0/+351
This adds a function that consumes a [proto::node::Node] pointing to the root of a (store) path, and writes the contents in NAR serialization to the passed [std::io::Write]. We need this in various places: - tvix-store's calculate_nar() RPC method needs to render a NAR stream to get the nar hash, which is necessary to give things imported in the store a "NAR-based" store path. - communication with (remote) Nix (via daemon protocol) needs a NAR representation. - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a NAR representation. Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5790 feat(tvix/store): implement iteration over Directory nodesFlorian Klink3-1/+163
This provides a `Directory.nodes()` function, returning an iterator over all three node types in an ordered fashion. Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5789 feat(tvix/store/proto): implement get_name for node::Node, pub traitFlorian Klink1-3/+13
Make the trait public, so consumers can use it. Also, implement it for node::Node, so we can later use this to access the name from all three if we don't care about the enum type. Change-Id: Iae530a16b705493645e61947852c03273876cc55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5788 refactor(tvix): introduce nix-compat crateFlorian Klink6-373/+4
Move nixbase32 and store_path into this. This allows //tvix/cli to not pull in //tvix/store for now. Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5781 fix(tvix/store/nixbase32): fix encoder/decoderFlorian Klink2-69/+114
Replace our data_encoding usage with the implementation taken from https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32 Also uncomment some of the unit tests, and add a regression test for a NIXBASE32.encode with a 32 bytes sequence. The previous implementation of NIXBASE32.encode is wrong in that case: ``` ❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= 0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk ❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C 00000000 b3 a2 4d e9 7a 8f db c8 35 b9 83 31 69 50 10 30 |..M.z...5..1iP.0| 00000010 b8 97 70 31 bc b5 4b 3b 3a c1 37 40 f8 46 ab 30 |..p1..K;:.7@.F.0| 00000020 ``` Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5780 refactor(tvix/store/tests): move Directory tests into directory.rsFlorian Klink2-286/+286
There's gonna be more tests coming, let's move them into a separate file everything Directory-related into a separate module. Change-Id: I78a0f263925528907a22724b028f75cce644b329 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-30 r/5779 feat(tvix/store/proto): implement get_name for all nodesFlorian Klink1-0/+24
Also add a `NamedNode` trait. We'll later use this to access names from all three individually. Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5778 docs(tvix/store/protos): add docstring for Directory::digest()Florian Klink1-0/+2
Change-Id: I361dbca444a267fea28cd212d563ee2d03497c16 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7952 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5777 docs(tvix/store/protos): fix docstringFlorian Klink1-2/+2
Change-Id: Idf835e9648ab6c95c29960f3f8176153d4888b27 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7951 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5724 feat(tvix/store/directory): validate Directory and sizesFlorian Klink2-10/+98
This calls out to Directory::validate() for all received Directory messages, and also makes sure the sizes we refer a Directory message as matches the sizes that have been calculated. Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5723 feat(tvix/store/directory): deduplicate Directory messagesFlorian Klink2-9/+82
We can omit sending Directory messages to clients that have already been sent in the same stream. We can also omit storing a Directory message if we already have it - they're content-addressed anyways. Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5722 feat(tvix/store): add SledDirectoryServiceFlorian Klink6-37/+411
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5719 refactor(tvix/store): simplify test a bitFlorian Klink1-24/+17
Import more things, and use expect_err to unpack the response. Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 r/5698 docs(tvix/store): fix docstring for StorePath::to_absolute_pathFlorian Klink1-1/+1
Change-Id: I8b36c85fe11e4fb62e5d28f6900cd80d89a5cc41 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7854 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-18 r/5697 feat(tvix/store): rename and expose StorePath::validate_nameFlorian Klink1-2/+3
Change-Id: I8a16c214c7c644756d9d54187beba8c80ccfb39c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7853 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-14 r/5658 chore(tvix/store): derive Clone for StorePathVincent Ambo1-1/+1
Change-Id: Id325afa766b0bd2a3d04cf538f13e77692a22687 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7829 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-12 r/5649 feat(tvix/store): implement PathInfoService with sledFlorian Klink6-37/+167
This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-12 r/5648 chore(tvix/store): update ParseStorePathError messagesFlorian Klink1-3/+3
Change-Id: I49799393a1e35b4475566819111beb57a628c555 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7801 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-12 r/5647 chore(tvix/store): update ValidatePathInfoError messagesFlorian Klink1-1/+1
Change-Id: I8f16e25d998a74ac110ae99eec0edbbfd720c8dd Reviewed-on: https://cl.tvl.fyi/c/depot/+/7800 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 r/5614 feat(tvix/store): document StorePath a bit more.Florian Klink1-0/+10
Change-Id: Ifab28d97ddc22a2073c5df5e6e2cefb51b4b9191 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7777 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 r/5613 refactor(tvix/store): rename NixPath to StorePathFlorian Klink5-50/+53
As discussed in #tvl, this is a more common term for it. Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-06 r/5611 feat(tvix/store): add NixPath::to_absolute_pathFlorian Klink1-0/+11
This provides a function returning a string starting with the store path prefix, the counterpart of `from_absolute_path`. Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-04 r/5586 feat(tvix/store): implement Nixpath::from_absolute_pathFlorian Klink1-1/+39
This allows constructing a NixPath from an absolute path. It pops off the STORE_DIR prefix and the trailing slash and returns an error if it couldn't be found. Change-Id: Ib540e353c63cc247ac15e20414b0db2caf695ef4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7751 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5572 feat(tvix/store/nixpath): DIGEST_SIZE public, use more constsFlorian Klink1-12/+18
Rename PATH_HASH_SIZE to DIGEST_SIZE. It's a digest, not hash (we don't necessarily have the internal hash state anymore), and the fact it's about (Nix)Paths is already visible from the module name. Also expose ENCODED_DIGEST_SIZE, so we don't need to do the calculation inside from_string() method, and it becomes more clear this is a constant. Change-Id: I0e7577dd7a6e503039037b986313b214e995d826 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7725 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: jrhahn <mail.jhahn@gmail.com> Tested-by: BuildkiteCI
2023-01-03 r/5571 feat(tvix/store/nixpath): expose digest and name fieldsFlorian Klink1-2/+2
These can be accessed directly. Change-Id: I71dc84f982820d53f319efefbed9b9055034954d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7724 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5570 feat(src/proto): add PathInfo.validate()Florian Klink3-0/+307
This provides validation of PathInfo messages, and ensures the output hashes are properly parsed from the root node names. NixPath already has a more extensive test suite for various wrong NixPaths, so it's omitted from here. Change-Id: I5d69118df5816daabb521ddb19d178bddd1caacf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7684 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-03 r/5569 chore(tvix/store): make importableFlorian Klink2-17/+17
This allows other crates to import tvix_store. Rename the bin crate to tvix-store-bin, to avoid having multiple crates with the same name (https://github.com/rust-lang/cargo/issues/6313) Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5568 chore(tvix/store/nixbase32): address clippyFlorian Klink1-1/+1
Change-Id: Ib47a55e39ed752492b9732439de5f327a7fa601e Reviewed-on: https://cl.tvl.fyi/c/depot/+/7723 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5567 chore(tvix/store/main): address clippyFlorian Klink1-1/+1
Change-Id: Ie400a7109d3aa6b66153c0ab25a38abf06984cbf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7722 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5558 feat(tvix/store): add logging with tracingFlorian Klink4-9/+42
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/ tracing. Annotate all method handlers with an instrument macro, and warn! a message for them being unimplemented. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5557 feat(tvix/store): implement reflectionFlorian Klink2-6/+25
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5556 refactor(tvix/store): make nixbase32 reversal more idiomaticedef1-7/+4
Change-Id: Ibe550f9573b0f45ee95453b50c7510e49b07c719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7685 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-30 r/5555 feat(tvix/store): refactor digest conversionJürgen Hahn1-5/+1
This refactors how the original digest type (Vec<u8>) is converted to [u8; PATH_HASH_SIZE]. Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-30 r/5554 feat(tvix/store): initial dummy implementationFlorian Klink4-2/+128
This replaces the hello world example from tvix-store with an actual gRPC endpoint, implementing all of BlobService, DirectoryService and PathInfoService. All RPC methods currently respond with the unimplemented gRPC status. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5552 feat(tvix/store): parameterise validate_digest errorFlorian Klink1-4/+7
Similar to cl/7682, we also want to make that error configurable. Change-Id: I64f1a4570b3d75af4741abe10c2855959766e107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7708 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 r/5551 feat(tvix/store): add nixpathJürgen Hahn2-0/+143
This implements the NixPath structure. NixPath allow to parse a string to a nix path. If the parsing fails, a DecodeError will be raised. Change-Id: I28363cdcfb27f04bf21a11c0d130b461667e3720 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7706 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 r/5550 feat(tvix/store): add nixbase32 modFlorian Klink2-0/+120
This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 r/5549 feat(tvix/store): parameterise validate_node_name errorFlorian Klink1-9/+8
We'll be using validate_node_name in other places in a bit, where returning a ValidateDirectoryError is not appropriate. Accept a function mapping a string to error as a second argument, and pass ValidateDirectoryError::InvalidName at the current call sites. Change-Id: I45cbb0deb4763061ad912c6b18a112c727795a17 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7682 Tested-by: BuildkiteCI Reviewed-by: jrhahn <mail.jhahn@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2022-12-29 r/5548 fix(tvix/store): fix typoFlorian Klink1-1/+1
Change-Id: I351a2bd4e007443aef1b97f50aecffc095ac60b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7681 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: jrhahn <mail.jhahn@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 r/5533 chore(tvix/store): address clippy warningsFlorian Klink1-17/+14
The only warnings left are unused warnings, but that'll change once we have a real implementation, and not just tests. Change-Id: I28912281b5e66735be37e999cc8ef4b8b09028fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7669 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 r/5532 chore(tvix/store): trim Cargo.tomlFlorian Klink1-1/+0
lazy_static is only used in tests, and anyhow isn't used at all (yet). This can be dropped. Change-Id: Ic41ff3f9bb93cfa600c3485e85464f78a3976504 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7668 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 r/5531 chore(tvix/store): move tests into separate fileFlorian Klink3-289/+288
Move them from the bottom of src/proto.rs to its own src/tests/mod.rs. Also drop the test_ prefix, this is not golang. Change-Id: I2e0b6b9812264f3d9721c0766936f08157fadc66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7667 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-12-03T17·26+0000