about summary refs log tree commit diff
path: root/tvix/store/src
AgeCommit message (Collapse)AuthorFilesLines
2023-03-10 r/5931 refactor(tvix/store/tests/nar_renderer): only upload EMPTY_BLOBFlorian Klink1-16/+14
This also uploaded HELLOWORLD_BLOB_CONTENTS before, but it's not referred from anywhere in the fixture. Change-Id: I823133afe0f08d18a59e2ac4e4d4bb7d34ce8a2b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8158 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5930 refactor(tvix/store/tests): move nar contents to fixturesFlorian Klink2-90/+88
Change-Id: I89a700240e7250a4d93eb8c4235e79c29ecb7f64 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8157 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5929 refactor(tvix/store/tests): move fixtures into separate moduleFlorian Klink7-95/+94
Change-Id: I362dbf0899e4dc42114fd2e6a8fa7f537e9ea138 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8156 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5928 refactor(tvix/store/tests): move gen_*_service() into helperFlorian Klink7-71/+57
This allows hiding to tests what exact implementation we're using, when testing things that do something with a store, but don't care what's used for underlying storage. Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5927 refactor(tvix/store): move blob splitting into a BlobWriter structFlorian Klink4-76/+172
This will moves the chunking-as-we-receive logic that so far only lived in grpc_blobservice_wrapper.rs into a generic BlobWriter. Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5926 feat(tvix/store): do not buffer blob dataFlorian Klink2-73/+139
Use the FastCDC::cut function to ask fastcd for cutting points as we receive the data. Make sure to keep the last chunk in the temporary buffer, as we might not actually cut at the end. Also, use rayon to calculate the blake3 hash if the input data is > 128KiB. Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5925 feat(tvix/store): bump fastcdc, use v2020 versionFlorian Klink1-1/+1
This switches away from the less canonical "ronomon" version to the implementation as described in the [paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et al., in 2020. That version uses 64-bit hash values and tends to be faster than both the ronomon and v2016 versions, and produces the same chunking as the 2016 version. As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's the recommended choice. The crate also gained support for streaming version of chunkers: https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be useful. Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5924 feat(tvix/store): add --json argFlorian Klink1-2/+27
This configures logging as JSON. Change-Id: I22cdda84de215bfceda4e9d47bc8d487a5451a6e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8130 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5923 chore(tvix/store): remove old implementationsFlorian Klink8-784/+0
These were implementing against the (more complicated) gRPC interface, for which we now have a wrapper. Change-Id: I0a8284493718ab99618a1d21a76df4d173edb899 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8100 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5922 refactor(tvix/store): use GRPC Wrappers and new servicesFlorian Klink2-10/+30
This swaps out the implementation used in the main entrypoint. We now use the non-gRPC aware Sled*Service, and wrap it with GRPC*ServiceWrapper to spin up the gRPC server. Change-Id: I5a1236c0612eee85e49891062040127c8bd95058 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8106 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5921 feat(tvix/store): add tests for GRPCPathInfoServiceWrapperFlorian Klink2-0/+108
Change-Id: I22de33c28c132fc9710ddecd0e9158ee3b3ef94b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8108 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5920 feat(tvix/store): add GRPCPathInfoServiceWrapperFlorian Klink2-0/+87
This exposes a proto::pathinfo_service_server::PathInfoService for a directoryservice::PathInfoService and a way to calculate NARs. Change-Id: I30cd058562f83d063c78b84976ec97190de49400 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8097 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5919 feat(tvix/store): add pathinfoserviceFlorian Klink4-0/+165
This adds a PathInfoService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5918 feat(tvix/store): add NARCalculationService trait and implFlorian Klink2-0/+61
This adds a NARCalculationService trait, which will take a root node, and provide a proto::CalculateNarResponse in return. It also adds a NonCachingNARCalculationService implementation, that will simply always render the NAR in memory to calculate the size and sha256, without any caching. Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5917 feat(tvix/store): validate blob size in NARRendererFlorian Klink3-0/+115
Make sure the blob size in the current proto node matches what we get back from the blob backend. Change-Id: I939fa18f37c7bc86ada8a495c7be622e69ec47f8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8129 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5916 refactor(tvix/store): don't use anyhow::Result<()> in testsFlorian Klink1-11/+6
It's much easier to just unwrap the TempDir::new() result. Change-Id: I7b05bc18f3146401e30e1cc2bb412503c5171a66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8128 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5915 chore(tvix/store): move NAR rendering logic into Renderer structFlorian Klink5-181/+275
This moves the logic rendering NARs to a struct using the previously introduced, more granular BlobService, ChunkService and DirectoryService. Instead of passing them around to the helper functions, they're kept as members of a struct. Remove the async invocations in the nar_renderer tests, there's nothing async in here. Change-Id: Ic6d24aaad68a1fda46ce29f2cdb5f7b87f481d5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8095 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5914 feat(tvix/store): add tests for GRPCDirectoryServiceWrapperFlorian Klink2-0/+263
Change-Id: Iccf70664cd5dd8ea22e0b43002d2ff1a9680870a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8102 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5913 feat(tvix/store): add GRPCDirectoryServiceWrapperFlorian Klink2-0/+232
This exposes a proto::directory_service_server::DirectoryService for a directoryservice::DirectoryService. Change-Id: I191a6e44e5bce10d904b5f10f0073d6adaf1f572 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8094 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5912 feat(tvix/store): add directoryserviceFlorian Klink4-0/+182
This adds a DirectoryService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: Ida61524b2ca949e1b3a78089a5aa7d9f9800c8d7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8093 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5911 feat(tvix/store): add tests for GRPCBlobServiceWrapperFlorian Klink2-0/+211
Change-Id: I0c9a2d887b5b94c43afee97a8a1e701cc9f74f23 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8098 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5910 feat(tvix/store/blobservice): add GRPCBlobServiceWrapperFlorian Klink3-0/+244
This takes a BlobService and ChunkService in the constructor, and provides a [proto::blob_service_server::BlobService] trait for it. Implementing proto::blob_service_server::BlobService is a lot of surface to cover, and providing this wrapper will make individual implementations taking care of how to store chunks or chunking information much simpler. Change-Id: Ia7b46484fb3ac9104354d496ff2922dca96ff7b9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8092 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5909 refactor(tvix/store): remove anyhow::Result in directory it testFlorian Klink1-3/+1
There's no `?` being used. Change-Id: Ifbfd88f6056e0064d11458cb335d05b333ec7b6b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8127 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-10 r/5908 refactor(tvix/store): remove useless clone in testFlorian Klink1-1/+1
Change-Id: If62887cbc3338110cb7c815ec2ee2160fae04353 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8111 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5907 chore(tvix/store): move protos into separate mod.rsFlorian Klink6-3/+6
This allows adding more stuff into this namespace, from different files. Also move tests on proto-related code from src/tests to src/proto/tests. Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5906 feat(tvix/store): add blobreaderFlorian Klink2-0/+386
A BlobReader can be used to read a blob, which might consist out of multiple chunks. Chunks are fetched from a ChunkService. Change-Id: I1806225b0052adaa4a2320b79b744e554e524ee3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8088 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5894 feat(tvix/store): add blobserviceFlorian Klink4-0/+146
This adds a BlobService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5893 feat(tvix/store): add chunkserviceFlorian Klink4-0/+148
This adds the simpler ChunkService trait, and an implementation for it using sled, and one using a HashMap. Change-Id: Icb0fdc41b37b44e9e9e4f548d0f4acae1d83b71e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8086 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-06 r/5892 feat(tvix/store): impl From<PoisonError> for ErrorFlorian Klink1-0/+7
Change-Id: Ib61e276b45c0102e383a7e7e641172b151369b03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8207 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-06 r/5891 feat(tvix/store): add errorsFlorian Klink2-0/+15
Change-Id: I7be0a4eaa29840e1a3329bccb791975de22828ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/8085 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-15 r/5853 refactor(tvix/store/proto): use .cloned()Florian Klink1-10/+4
Instead of using an explicit closure to clone elements, use .cloned(). Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-15 r/5852 refactor(tvix/store): avoid casting to the same typeFlorian Klink1-1/+1
The size field already is u32, we don't need to convert here. Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-31 r/5794 feat(tvix/store): add write_nar functionFlorian Klink5-0/+351
This adds a function that consumes a [proto::node::Node] pointing to the root of a (store) path, and writes the contents in NAR serialization to the passed [std::io::Write]. We need this in various places: - tvix-store's calculate_nar() RPC method needs to render a NAR stream to get the nar hash, which is necessary to give things imported in the store a "NAR-based" store path. - communication with (remote) Nix (via daemon protocol) needs a NAR representation. - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a NAR representation. Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5790 feat(tvix/store): implement iteration over Directory nodesFlorian Klink3-1/+163
This provides a `Directory.nodes()` function, returning an iterator over all three node types in an ordered fashion. Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5789 feat(tvix/store/proto): implement get_name for node::Node, pub traitFlorian Klink1-3/+13
Make the trait public, so consumers can use it. Also, implement it for node::Node, so we can later use this to access the name from all three if we don't care about the enum type. Change-Id: Iae530a16b705493645e61947852c03273876cc55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-31 r/5788 refactor(tvix): introduce nix-compat crateFlorian Klink6-373/+4
Move nixbase32 and store_path into this. This allows //tvix/cli to not pull in //tvix/store for now. Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5781 fix(tvix/store/nixbase32): fix encoder/decoderFlorian Klink2-69/+114
Replace our data_encoding usage with the implementation taken from https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32 Also uncomment some of the unit tests, and add a regression test for a NIXBASE32.encode with a 32 bytes sequence. The previous implementation of NIXBASE32.encode is wrong in that case: ``` ❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= 0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk ❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C 00000000 b3 a2 4d e9 7a 8f db c8 35 b9 83 31 69 50 10 30 |..M.z...5..1iP.0| 00000010 b8 97 70 31 bc b5 4b 3b 3a c1 37 40 f8 46 ab 30 |..p1..K;:.7@.F.0| 00000020 ``` Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5780 refactor(tvix/store/tests): move Directory tests into directory.rsFlorian Klink2-286/+286
There's gonna be more tests coming, let's move them into a separate file everything Directory-related into a separate module. Change-Id: I78a0f263925528907a22724b028f75cce644b329 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-30 r/5779 feat(tvix/store/proto): implement get_name for all nodesFlorian Klink1-0/+24
Also add a `NamedNode` trait. We'll later use this to access names from all three individually. Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5778 docs(tvix/store/protos): add docstring for Directory::digest()Florian Klink1-0/+2
Change-Id: I361dbca444a267fea28cd212d563ee2d03497c16 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7952 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 r/5777 docs(tvix/store/protos): fix docstringFlorian Klink1-2/+2
Change-Id: Idf835e9648ab6c95c29960f3f8176153d4888b27 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7951 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5724 feat(tvix/store/directory): validate Directory and sizesFlorian Klink2-10/+98
This calls out to Directory::validate() for all received Directory messages, and also makes sure the sizes we refer a Directory message as matches the sizes that have been calculated. Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5723 feat(tvix/store/directory): deduplicate Directory messagesFlorian Klink2-9/+82
We can omit sending Directory messages to clients that have already been sent in the same stream. We can also omit storing a Directory message if we already have it - they're content-addressed anyways. Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5722 feat(tvix/store): add SledDirectoryServiceFlorian Klink6-37/+411
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5719 refactor(tvix/store): simplify test a bitFlorian Klink1-24/+17
Import more things, and use expect_err to unpack the response. Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 r/5698 docs(tvix/store): fix docstring for StorePath::to_absolute_pathFlorian Klink1-1/+1
Change-Id: I8b36c85fe11e4fb62e5d28f6900cd80d89a5cc41 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7854 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-18 r/5697 feat(tvix/store): rename and expose StorePath::validate_nameFlorian Klink1-2/+3
Change-Id: I8a16c214c7c644756d9d54187beba8c80ccfb39c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7853 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-14 r/5658 chore(tvix/store): derive Clone for StorePathVincent Ambo1-1/+1
Change-Id: Id325afa766b0bd2a3d04cf538f13e77692a22687 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7829 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-12 r/5649 feat(tvix/store): implement PathInfoService with sledFlorian Klink6-37/+167
This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-12 r/5648 chore(tvix/store): update ParseStorePathError messagesFlorian Klink1-3/+3
Change-Id: I49799393a1e35b4475566819111beb57a628c555 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7801 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-07T22·50+0000