about summary refs log tree commit diff
path: root/tvix/store
AgeCommit message (Collapse)AuthorFilesLines
2023-01-21 r/5727 chore(tvix/store): add fastcdc crateFlorian Klink1-6/+3
This is used for content-defined chunking. Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5724 feat(tvix/store/directory): validate Directory and sizesFlorian Klink2-10/+98
This calls out to Directory::validate() for all received Directory messages, and also makes sure the sizes we refer a Directory message as matches the sizes that have been calculated. Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 r/5723 feat(tvix/store/directory): deduplicate Directory messagesFlorian Klink2-9/+82
We can omit sending Directory messages to clients that have already been sent in the same stream. We can also omit storing a Directory message if we already have it - they're content-addressed anyways. Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5722 feat(tvix/store): add SledDirectoryServiceFlorian Klink6-37/+411
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5720 chore(tvix/store): add tonic-mockFlorian Klink1-0/+1
Upstream seems to be dead, so we're using https://github.com/tyrchen/ tonic-mock/pull/3 here. According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment- 1241164173, we might not need this crate at all, but for now, it gets the job done and is less code to write in the tests. Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-21 r/5719 refactor(tvix/store): simplify test a bitFlorian Klink1-24/+17
Import more things, and use expect_err to unpack the response. Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 r/5698 docs(tvix/store): fix docstring for StorePath::to_absolute_pathFlorian Klink1-1/+1
Change-Id: I8b36c85fe11e4fb62e5d28f6900cd80d89a5cc41 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7854 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-18 r/5697 feat(tvix/store): rename and expose StorePath::validate_nameFlorian Klink1-2/+3
Change-Id: I8a16c214c7c644756d9d54187beba8c80ccfb39c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7853 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-14 r/5658 chore(tvix/store): derive Clone for StorePathVincent Ambo1-1/+1
Change-Id: Id325afa766b0bd2a3d04cf538f13e77692a22687 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7829 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-12 r/5649 feat(tvix/store): implement PathInfoService with sledFlorian Klink7-37/+169
This uses [sled](https://github.com/spacejam/sled) to store PathInfo objects. Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-12 r/5648 chore(tvix/store): update ParseStorePathError messagesFlorian Klink1-3/+3
Change-Id: I49799393a1e35b4475566819111beb57a628c555 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7801 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-12 r/5647 chore(tvix/store): update ValidatePathInfoError messagesFlorian Klink1-1/+1
Change-Id: I8f16e25d998a74ac110ae99eec0edbbfd720c8dd Reviewed-on: https://cl.tvl.fyi/c/depot/+/7800 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 r/5615 refactor(tvix/store): move protobuf build config one level upVincent Ambo2-35/+13
This embeds the build config directly at the point where `Cargo.nix` is imported, making it transparent to library consumers. Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-06 r/5614 feat(tvix/store): document StorePath a bit more.Florian Klink1-0/+10
Change-Id: Ifab28d97ddc22a2073c5df5e6e2cefb51b4b9191 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7777 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 r/5613 refactor(tvix/store): rename NixPath to StorePathFlorian Klink5-50/+53
As discussed in #tvl, this is a more common term for it. Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-06 r/5611 feat(tvix/store): add NixPath::to_absolute_pathFlorian Klink1-0/+11
This provides a function returning a string starting with the store path prefix, the counterpart of `from_absolute_path`. Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-04 r/5586 feat(tvix/store): implement Nixpath::from_absolute_pathFlorian Klink1-1/+39
This allows constructing a NixPath from an absolute path. It pops off the STORE_DIR prefix and the trailing slash and returns an error if it couldn't be found. Change-Id: Ib540e353c63cc247ac15e20414b0db2caf695ef4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7751 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5572 feat(tvix/store/nixpath): DIGEST_SIZE public, use more constsFlorian Klink1-12/+18
Rename PATH_HASH_SIZE to DIGEST_SIZE. It's a digest, not hash (we don't necessarily have the internal hash state anymore), and the fact it's about (Nix)Paths is already visible from the module name. Also expose ENCODED_DIGEST_SIZE, so we don't need to do the calculation inside from_string() method, and it becomes more clear this is a constant. Change-Id: I0e7577dd7a6e503039037b986313b214e995d826 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7725 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: jrhahn <mail.jhahn@gmail.com> Tested-by: BuildkiteCI
2023-01-03 r/5571 feat(tvix/store/nixpath): expose digest and name fieldsFlorian Klink1-2/+2
These can be accessed directly. Change-Id: I71dc84f982820d53f319efefbed9b9055034954d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7724 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5570 feat(src/proto): add PathInfo.validate()Florian Klink3-0/+307
This provides validation of PathInfo messages, and ensures the output hashes are properly parsed from the root node names. NixPath already has a more extensive test suite for various wrong NixPaths, so it's omitted from here. Change-Id: I5d69118df5816daabb521ddb19d178bddd1caacf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7684 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-03 r/5569 chore(tvix/store): make importableFlorian Klink4-19/+27
This allows other crates to import tvix_store. Rename the bin crate to tvix-store-bin, to avoid having multiple crates with the same name (https://github.com/rust-lang/cargo/issues/6313) Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5568 chore(tvix/store/nixbase32): address clippyFlorian Klink1-1/+1
Change-Id: Ib47a55e39ed752492b9732439de5f327a7fa601e Reviewed-on: https://cl.tvl.fyi/c/depot/+/7723 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 r/5567 chore(tvix/store/main): address clippyFlorian Klink1-1/+1
Change-Id: Ie400a7109d3aa6b66153c0ab25a38abf06984cbf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7722 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5558 feat(tvix/store): add logging with tracingFlorian Klink5-9/+44
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/ tracing. Annotate all method handlers with an instrument macro, and warn! a message for them being unimplemented. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5557 feat(tvix/store): implement reflectionFlorian Klink5-25/+68
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools like evans automatically discover available services, without having to specify the path to the .proto files client-side. It's behind a reflection feature flag, which is enabled by default. Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5556 refactor(tvix/store): make nixbase32 reversal more idiomaticedef1-7/+4
Change-Id: Ibe550f9573b0f45ee95453b50c7510e49b07c719 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7685 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-30 r/5555 feat(tvix/store): refactor digest conversionJürgen Hahn1-5/+1
This refactors how the original digest type (Vec<u8>) is converted to [u8; PATH_HASH_SIZE]. Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-30 r/5554 feat(tvix/store): initial dummy implementationFlorian Klink5-2/+131
This replaces the hello world example from tvix-store with an actual gRPC endpoint, implementing all of BlobService, DirectoryService and PathInfoService. All RPC methods currently respond with the unimplemented gRPC status. Co-Authored-By: Márton Boros <martonboros@gmail.com> Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-30 r/5553 chore(tvix/store/protos): more idiomatic goFlorian Klink1-3/+1
Pointed out by edef in https://cl.tvl.fyi/c/depot/+/7648/comment/4551ba4b_e89ade36/#, thanks! Change-Id: I6b0d317bb0210521622483cdf4cb557bc637a100 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7709 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 r/5552 feat(tvix/store): parameterise validate_digest errorFlorian Klink1-4/+7
Similar to cl/7682, we also want to make that error configurable. Change-Id: I64f1a4570b3d75af4741abe10c2855959766e107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7708 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 r/5551 feat(tvix/store): add nixpathJürgen Hahn2-0/+143
This implements the NixPath structure. NixPath allow to parse a string to a nix path. If the parsing fails, a DecodeError will be raised. Change-Id: I28363cdcfb27f04bf21a11c0d130b461667e3720 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7706 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 r/5550 feat(tvix/store): add nixbase32 modFlorian Klink3-3/+125
This implements the nix-specific base32 encoding and decoding, exposing a subset of the API that the data-encoding crate provides. Nix uses a custom alphabet, no padding, and encodes bytes in reverse order. The latter one is the reason we can't just use the data-encoding crate directly. Three odd corner case tests ported over from go-nix failed. We opened b/235 to further investigate. Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-29 r/5549 feat(tvix/store): parameterise validate_node_name errorFlorian Klink1-9/+8
We'll be using validate_node_name in other places in a bit, where returning a ValidateDirectoryError is not appropriate. Accept a function mapping a string to error as a second argument, and pass ValidateDirectoryError::InvalidName at the current call sites. Change-Id: I45cbb0deb4763061ad912c6b18a112c727795a17 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7682 Tested-by: BuildkiteCI Reviewed-by: jrhahn <mail.jhahn@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2022-12-29 r/5548 fix(tvix/store): fix typoFlorian Klink1-1/+1
Change-Id: I351a2bd4e007443aef1b97f50aecffc095ac60b1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7681 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: jrhahn <mail.jhahn@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 r/5533 chore(tvix/store): address clippy warningsFlorian Klink2-18/+15
The only warnings left are unused warnings, but that'll change once we have a real implementation, and not just tests. Change-Id: I28912281b5e66735be37e999cc8ef4b8b09028fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7669 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 r/5532 chore(tvix/store): trim Cargo.tomlFlorian Klink2-2/+3
lazy_static is only used in tests, and anyhow isn't used at all (yet). This can be dropped. Change-Id: Ic41ff3f9bb93cfa600c3485e85464f78a3976504 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7668 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-29 r/5531 chore(tvix/store): move tests into separate fileFlorian Klink3-289/+288
Move them from the bottom of src/proto.rs to its own src/tests/mod.rs. Also drop the test_ prefix, this is not golang. Change-Id: I2e0b6b9812264f3d9721c0766936f08157fadc66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7667 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5528 feat(tvix/store/protos): rename Get to Read, add Stat methodFlorian Klink3-88/+434
Stat exposes metadata about a given blob, such as more granular chunking, baos. It implicitly allows checking for existence too, as asking this for a non-existing Blob will return a Status::not_found grpc error. The previous version returned a Status::not_found error on the Get request too, but there was no chance to prevent the server from starting to stream (except sending an immediate cancellation). Being able to check whether something exists in a BlobStore helps to prevent from uploading in first place. The granular chunking bits are an optional optimization - if the BlobStore implements no more granular chunking, the Stat response can simply contain a single chunk. Read returns a stream of BlobChunk, which is just a stream of bytes - not necessarily using the chunking that's returned in the reply of a Stat() call. It can be used to read blobs or chunks. Change-Id: I4b6030ef184ace5484c84ca273b49d710433731d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7652 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5527 feat(tvix/store): make blobstore stream chunksFlorian Klink3-172/+171
This changes the RPC methods to return/consume a stream of chunks, instead of a very big message containing the whole blob, to keep message sizes in manageable sizes (less than 4MiB). Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5525 chore(tvix/store/protos): add PathInfoService::CalculateNAR()Florian Klink3-39/+222
Expose the NAR calculation to a separate `CalculateNAR` method, which responds with the NAR size and sha256 hash. Contrary to what cl/7618 and cl/7620 initially did, don't add different other request types. In the CalculateNARResponse message, there's now some duplication in the (optional) `narinfo` field of a PathInfo, but I'm not entirely sure if we want to drop the fields from there yet. Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5524 chore(tvix/store/protos): drop non-sha256 NAR hashesFlorian Klink2-196/+41
Nix actually doesn't support anything else than sha256 NAR hashes, so there's little reason to communicate anything else in here. Co-Authored-By: edef <edef@unfathomable.blue> Change-Id: I760370bd6cabd02028e001a74c454ef9296eb600 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7619 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5523 chore(tvix/store/protos): move (Root)Node into separate messageFlorian Klink2-124/+191
This is gonna get used in another place in a second. Co-Authored-By: edef <edef@unfathomable.blue> Change-Id: I347c11c8d24379628b7ed09d2c90670c576e686a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7617 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5520 feat(tvix/store): implement Directory::validate()Florian Klink2-1/+329
Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-28 r/5519 chore(tvix/store): use DUMMY_DIGEST, tooFlorian Klink1-1/+1
Missed from cl/7649. Change-Id: Id2d382567f91cba9cdd792fe44bc0a83d040ec69 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7656 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 r/5518 chore(tvix/store): use Directory::default in some more placesFlorian Klink1-18/+6
Change-Id: Iebad5e0b38f9506933fa855e1bfb2611d27ff572 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7655 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-27 r/5510 chore(tvix/store): use lazy_static and dummy digestsFlorian Klink2-2/+11
While there's currently nothing in here checking the size of the digest, we should use something that passes the to-be-introduced validate() function. Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 r/5509 feat(tvix/store/protos): implement Directory.Validate()Florian Klink2-0/+297
Validate thecks the Directory message for invalid data, such as: - violations of name restrictions - invalid digest lengths - not properly sorted lists - duplicate names in the three lists Change-Id: I8d43a13797793c64097e526ef3bd482c9606c87b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7648 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-27 r/5508 feat(tvix/store/protos): implement Size() and Digest() for DirectoryFlorian Klink4-0/+251
This adds Size() and Digest() functions for the golang version. Change-Id: If71445a9bb26100bb4076ac4f5c96945b33919f9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7325 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 r/5496 feat(txix/store/protos): init go.mod, generate .pb.go filesFlorian Klink9-0/+2390
This allows importing the generated .pb.go files into other go projects. I initially looked at buildGo.protos, but it doesn't work for multi-.proto files, and actually having LSP support for the generated structs is nice, too. Change-Id: Idbd448008010790a10a0ea42e4059dbb609eaf1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7322 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 r/5495 feat(tvix/store/protos): add go_package optionFlorian Klink5-0/+10
Change-Id: I0898b8a0a78e704219da38e5acaabef1e640d4e4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7321 Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-06T08·11+0000