about summary refs log tree commit diff
path: root/tvix/store/src/proto/tests
AgeCommit message (Collapse)AuthorFilesLines
2024-08-17 r/8506 refactor(tvix/castore): add PathComponent type for checked componentsFlorian Klink1-2/+2
This encodes a verified component on the type level. Internally, it contains a bytes::Bytes. The castore Path/PathBuf component() and file_name() methods now return this type, the old ones returning bytes were renamed to component_bytes() and component_file_name() respectively. We can drop the directory_reject_invalid_name test - it's not possible anymore to pass an invalid name to Directories::add. Invalid names in the Directory proto are still being tested to be rejected in the validate_invalid_names tests. Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-08-17 r/8504 refactor(tvix/castore): remove `name` from NodesFlorian Klink1-16/+20
Nodes only have names if they're contained inside a Directory, or if they're a root node and have something else possibly giving them a name externally. This removes all `name` fields in the three different Nodes, and instead maintains it inside a BTreeMap inside the Directory. It also removes the NamedNode trait (they don't have a get_name()), as well as Node::rename(self, name), and all [Partial]Ord implementations for Node (as they don't have names to use for sorting). The `nodes()`, `directories()`, `files()` iterators inside a `Directory` now return a tuple of Name and Node, as does the RootNodesProvider. The different {Directory,File,Symlink}Node struct constructors got simpler, and the {Directory,File}Node ones became infallible - as there's no more possibility to represent invalid state. The proto structs stayed the same - there's now from_name_and_node and into_name_and_node to convert back and forth between the two `Node` structs. Some further cleanups: The error types for Node validation were renamed. Everything related to names is now in the DirectoryError (not yet happy about the naming) There's some leftover cleanups to do: - There should be a from_(sorted_)iter and into_iter in Directory, so we can construct and deconstruct in one go. That should also enable us to implement conversions from and to the proto representation that moves, rather than clones. - The BuildRequest and PathInfo structs are still proto-based, so we still do a bunch of conversions back and forth there (and have some ugly expect there). There's not much point for error handling here, this will be moved to stricter types in a followup CL. Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205 Tested-by: BuildkiteCI Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-13 r/8486 refactor(tvix/castore): move *Node and Directory to crate rootFlorian Klink1-2/+2
*Node and Directory are types of the tvix-castore model, not the tvix DirectoryService model. A DirectoryService only happens to send Directories. Move types into individual files in a nodes/ subdirectory, as it's gotten too cluttered in a single file, and (re-)export all types from the crate root. This has the effect that we now cannot poke at private fields directly from other files inside `crate::directoryservice` (as it's not all in the same file anymore), but that's a good thing, it now forces us to go through the proper accessors. For the same reasons, we currently also need to introduce the `rename` functions on each *Node directly. A followup is gonna move the names out of the individual enum kinds, so we can better represent "unnamed nodes". Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199 Reviewed-by: yuka <yuka@yuka.dev> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-13 r/8484 refactor(tvix/castore): use Directory struct separate from proto oneYureka1-16/+14
This uses our own data type to deal with Directories in the castore model. It makes some undesired states unrepresentable, removing the need for conversions and checking in various places: - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere. - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones. Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-19 r/7965 chore(tvix/store): migrate import.rs and tests/pathinfo.rs to rstestFlorian Klink1-69/+53
Also, rename the DUMMY_NAME constant in the fixtures to DUMMY_PATH, which aligns more with the ToString representation and from_bytes conversions we have on StorePath[Ref]. Change-Id: I39763c9dfa84c5d86f2fd0171b3a4d36fd72f267 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11464 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-13 r/7892 docs(tvix/nix-compat): rename as_narinfo() to to_narinfo()Florian Klink1-1/+1
This actually does a bit of allocation. There's two Vecs, one for references and one for signatures. We can get rid of the Vec at nar_hash. Change-Id: Ie025309b6678f83f5b961d49ff75dcfc7da145a1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11395 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-05 r/7856 feat(tvix/store/proto): avoid clone in PathInfo::validate()Florian Klink1-9/+8
Have this return a StorePathRef<'_>, rather than a StorePath, and leave it up to the caller to possibly convert it to a owned StorePath. This avoids some allocations, if we only want to validate. Change-Id: I5cf8e246fe02bd4e631f46a5cb86d3f77a728a0d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11361 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-28 r/7794 refactor(tvix/store): drop proto/tests/{utils,grpc_pathinfoservice}.rsFlorian Klink2-76/+0
As for grpc_pathinfoservice, we test this in the rstest-based tests, with more implementations. tests/utils.rs is unused now. Change-Id: I62b192170d181e4fb432bf7dfff2057cb048b52c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11279 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-03-28 r/7787 refactor(tvix/store/tests/fixtures): use [u8; 20]Florian Klink2-5/+6
This makes the fixture more use-able when interacting with the trait, the Bytes are only useful for the gRPC version. Change-Id: Iaaea1adc6df18491f236a28c4343f5b4ee5fcfd3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11271 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-01-21 r/7435 chore(tvix/store): Use BoxStream type aliasConnor Brewster1-2/+3
The BoxStream type alias is a more concise and easier to read than the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type. Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-12 r/7371 refactor(tvix/store/pathinfoservice): make more genericFlorian Klink1-2/+2
We don't need Arcs in most of the cases, we're fine with some container. Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-01 r/7307 refactor(tvix/castore/tests): let gen_*_service return BoxesFlorian Klink1-2/+2
Only convert to and reuse an Arc<…> where needed. Change-Id: I2c1bc69cca5a4a3ebd3bdb33d6e28e1f5fb86cb9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10514 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-22 r/7255 refactor(tvix/store): remove Arc<> from PathInfoService::from_addrFlorian Klink1-1/+1
This makes PathInfoService::from_addr return a Box<dyn PathInfoService>, rather than an Arc<dyn …>, and leaves it up to the consumers to rewrap it into an Arc where needed. This allows us to drop the Arc for the tvix-store daemon subcommand. Change-Id: Ic83aa2ade6c51912281bd17c7eef7252e152b2d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10409 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-11-22 r/7051 feat(tvix/store): add as_narinfo() for PathInfoFlorian Klink1-1/+43
This allows seeing a PathInfo as a nix_compat::narinfo::NarInfo<'_>. It doesn't allocate any new data, but the NarInfo<'_> view allows us to access things like signature verification, or rendering out (alternations of this) as strings. Change-Id: Id0d8d7feeb626ee02c3d8a4932f24ace77022619 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10108 Reviewed-by: edef <edef@edef.eu> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-18 r/7029 feat(tvix/store): From<&nix_compat::...::NarInfo<'_>> for PathInfoFlorian Klink1-1/+110
This allows converting from the NarInfo falling out of the NarInfo parser (which is a bit annoying to handle due to lifetimes) to the PathInfo proto struct. The narinfo field, containing most of the data from the original NARInfo file, as well as the references (bytes) are populated. The node field is not populated, because it requires ingesting the NAR itself to describe the root node. Change-Id: I9c04dd6ad4cae556b455188a4255e34b4f6443c5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10067 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-10-27 r/6893 refactor(tvix): turn nullary enum variants into unit variantsedef1-5/+5
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-22 r/6869 refactor(tvix/store): move deriver field into narinfo submessageFlorian Klink1-4/+6
This information is present in the .narinfo files, it should have gone there. Change-Id: Ib43d0cf30c2795bf1fe77c46646174353ade0458 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9794 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-14 r/6805 feat(tvix/store/protos): add Deriver field to PathInfoFlorian Klink1-0/+31
This uses the newly introduced StorePath message type to add a Deriver field to the PathInfo message. Support for validation is added to both the golang and rust implementation. This includes extending unit tests. Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2023-10-12 r/6794 refactor(tvix/castore): factor out node checksFlorian Klink1-4/+27
Implement `validate()` on `node::Node`, and call it from PathInfo's validate() too. Node-related errors are moved to a ValidateNodeError error type. This additionally adds some more validations for symlink targets (they must not be empty, and not contain null bytes). Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-10-11 r/6786 feat(tvix/store): validate nar_sha256Florian Klink1-0/+13
Change-Id: I4c4dcdb75ea7748f2ab01a0bab218596b90b7b58 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9648 Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-05 r/6707 test(tvix/store/pathinfo): add more tests for referencesFlorian Klink1-42/+72
This should cover all error cases produced. Change-Id: If31816d9b087551d86d7913df55df8f9f44bb554 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9546 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-05 r/6706 feat(tvix/store): add validation for referencesFlorian Klink1-2/+2
This validates the size of reference digests in the PathInfo message, as well as inside the narinfo submessage. If narinfo is set, they need to parse to StorePath, and have the same digest there as in the PathInfo message. `proto::tests::pathinfo::validate_references` needed to be updated, because we actually did not populate the proper references before. Change-Id: I9545b2487aab9fe0d229c26aceba5ddc5e6daafd Reviewed-on: https://cl.tvl.fyi/c/depot/+/9545 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-05 r/6705 refactor(tvix/store): mv *Err*::Invalid{Node,}DigestLenFlorian Klink1-2/+2
There's other digests in the PathInfo structure, that also might have wrong digest lengths. Rename this to give some room for them, and update the error message a bit as we go. Change-Id: I06562664721156e658f2ed14ba1de907377d284b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9543 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-09-22 r/6629 refactor(tvix): move castore into tvix-castore crateFlorian Klink7-759/+33
This splits the pure content-addressed layers from tvix-store into a `castore` crate, and only leaves PathInfo related things, as well as the CLI entrypoint in the tvix-store crate. Notable changes: - `fixtures` and `utils` had to be moved out of the `test` cfg, so they can be imported from tvix-store. - Some ad-hoc fixtures in the test were moved to proper fixtures in the same step. - The protos are now created by a (more static) recipe in the protos/ directory. The (now two) golang targets are commented out, as it's not possible to update them properly in the same CL. This will be done by a followup CL once this is merged (and whitby deployed) Bug: https://b.tvl.fyi/issues/301 Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-05 r/6555 feat(tvix/store/pathinfosvc): provide listingFlorian Klink1-1/+3
This provides an additional method in the PathInfoService trait, as well as an RPC method on the gRPC layer to list all PathInfo objects in a PathInfoService. Change-Id: I7378f6bbd334bd6ac4e9be92505bd099a1c2b19a Reviewed-on: https://cl.tvl.fyi/c/depot/+/9216 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-08-20 r/6500 refactor(tvix/store): cargo clippyFlorian Klink1-1/+1
Change-Id: I3a80560d036e7ed08036b5e9f0974080d1a30ded Reviewed-on: https://cl.tvl.fyi/c/depot/+/9096 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-07-22 r/6439 feat(tvix/store/proto): use Bytes instead of Vec<u8>Florian Klink5-49/+57
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-07-21 r/6437 feat(tvix/store/digests): use bytes::Bytes instead of Vec<u8>Florian Klink1-6/+4
This will save us some copies, because a clone will simply create an additional pointer to the same data. Change-Id: I017a5d6b4c85a861b5541ebad2858ad4fbf8e8fa Reviewed-on: https://cl.tvl.fyi/c/depot/+/8978 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-07-21 r/6436 refactor(tvix/store): use bytes for node names and symlink targetsFlorian Klink5-65/+63
Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-07-21 r/6433 feat(tvix/nix-compat): have StorePath accept bytesFlorian Klink1-3/+4
The primary constructor for this is now from_bytes, from_string is simply calling .as_bytes() on the string, passing it along. The InvalidName error now contains a Vec<u8>, to encode the invalid name (which might not be a string anymore). from_absolute_path now accepts a &[u8] (even though we might want to make this a OSString of some sort). StorePath::validate_name has been degraded to a pub(crate) function. It's still used in src/derivation, even though it probably shouldn't at all - that cleanup is left for cl/8412 though. Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 r/6432 feat(tvix/nix-compat): fold NameError into ErrorFlorian Klink1-3/+3
This being a nested error makes things more complicated than necessary. Also, this caused BuildStorePathError to only hold NameError, so refactor these utility functions to either return Error, or BuildStorePathError. Change-Id: I046fb403780cc5135df8b8833a291fc2a90fd913 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8972 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6279 refactor(tvix/store/pathinfosvc): use Arc<dyn …>Florian Klink1-2/+6
This removes the use of generics, like previously done with Blob and Directory services. Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6272 refactor(tvix/store): use Box<dyn DirectoryService>Florian Klink1-3/+1
Once we support configuring services at runtime, we don't know what DirectoryService we're using at compile time. This also means, we can't explicitly use the is_closed method from GRPCPutter, without making it part of the DirectoryPutter itself. Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink1-6/+6
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6270 refactor(tvix/src/nar): drop NARCalculationServiceFlorian Klink1-2/+2
There's only one way to calculate NAR files, by walking through them. Things like caching such replies should be done closer to where we use these, composing NARCalculationService doesn't actually give us much. Instead, expose two functions, `nar::calculate_size_and_sha256` and `nar::writer_nar`, the latter writing NAR to a writer, the former using write_nar to only keeping the NAR size and digest. Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6269 feat(tvix/store): eliminate generics in BlobStoreFlorian Klink1-3/+1
To construct various stores at runtime, we need to eliminate associated types from the BlobService trait, and return Box<dyn …> instead of specific types. This also means we can't consume self in the close() method, so everything we write to is put in an Option<>, and during the first close we take from there. Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-23 r/6177 refactor(tvix/store/directorysvc): move from Vec<u8> to B3DigestFlorian Klink2-4/+8
This introduces a new struct, B3Digest, which internally holds a Vec<u8>, but only allows construction with 32 bytes. It also implements display, which will print the base64 representation. This should reduce some boilerplate when parsing Vec<u8>. Change-Id: Ia91aa40cb691916773abc8f93e6ed79a5fd34863 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8592 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-18 r/6164 fix(tvix/store/proto/grpc_blobsvc): clippyFlorian Klink1-1/+1
Change-Id: I0473562793f5fa2211cb10329ba1bca14b1acab9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8594 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-11 r/6133 refactor(tvix/store): remove ChunkServiceFlorian Klink2-160/+26
Whether chunking is involved or not, is an implementation detail of each Blobstore. Consumers of a whole blob shouldn't need to worry about that. It currently is not visible in the gRPC interface either. It shouldn't bleed into everything. Let the BlobService trait provide `open_read` and `open_write` methods, which return handles providing io::Read or io::Write, and leave the details up to the implementation. This means, our custom BlobReader module can go away, and all the chunking bits in there, too. In the future, we might still want to add more chunking-aware syncing, but as a syncing strategy some stores can expose, not as a fundamental protocol component. This currently needs "SyncReadIntoAsyncRead", taken and vendored in from https://github.com/tokio-rs/tokio/pull/5669. It provides a AsyncRead for a sync Read, which is necessary to connect our (sync) BlobReader interface to a GRPC server implementation. As an alternative, we could also make the BlobReader itself async, and let consumers of the trait (EvalIO) deal with the async-ness, but this is less of a change for now. In terms of vendoring, I initially tried to move our tokio crate to these commits, but ended up in version incompatibilities, so let's vendor it in for now. Change-Id: I5969ebbc4c0e1ceece47981be3b9e7cfb3f59ad0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8551 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-09 r/6088 refactor(nix-compat): Properly encapsulate store path constructionJohn Ericson1-3/+3
Before there was code scattered about (e.g. text hashing module and derivation output computation) constructing store paths from low level building blocks --- there was some duplication and it was easy to make nonsense store paths. Now, we have roughly the same "safe-ish" ways of constructing them as C++ Nix, and only those are exposed: - Make text hashed content-addressed store paths - Make other content-addressed store paths - Make input-addressed fixed output hashes Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-30 r/6060 refactor(tvix/nix-compat): move build_store_path out of derivationFlorian Klink1-4/+4
This doesn't have anything to do with ATerms, we just happen to be using the aterm representation of a Derivation as contents. Moving this into store_path/utils.rs makes these things much cleaner - Have a build_store_path_from_references function, and a build_store_path_from_fingerprint helper function that makes use of it. build_store_path_from_references is invoked from the derivation module which can be used to calculate the derivation path. In the derivation module, we also invoke build_store_path_from_fingerprint during the output path calculation. Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-16 r/6014 refactor(tvix/store/directorysvc): use [u8; 32] instead of Vec<u8>Florian Klink2-9/+15
Also, simplify the trait interface, only allowing lookups of Directory objects by their digest. Change-Id: I6eec28a8cb0557bed9b69df8b8ff99a5e0f8fe35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8313 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-10 r/5934 feat(tvix/store): add new_temporary for all Sled servicesFlorian Klink3-32/+21
This provides a service using /dev/shm, that's deleted once the reference is dropped. Refactor all tests to use these, which allows getting rid of most TempDir usage in the tests. The only place where we still use TempDir is in the importer tests, which work on a filesystem path. Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5929 refactor(tvix/store/tests): move fixtures into separate moduleFlorian Klink3-47/+3
Change-Id: I362dbf0899e4dc42114fd2e6a8fa7f537e9ea138 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8156 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5928 refactor(tvix/store/tests): move gen_*_service() into helperFlorian Klink3-43/+24
This allows hiding to tests what exact implementation we're using, when testing things that do something with a store, but don't care what's used for underlying storage. Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-10 r/5926 feat(tvix/store): do not buffer blob dataFlorian Klink1-0/+33
Use the FastCDC::cut function to ask fastcd for cutting points as we receive the data. Make sure to keep the last chunk in the temporary buffer, as we might not actually cut at the end. Also, use rayon to calculate the blake3 hash if the input data is > 128KiB. Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5921 feat(tvix/store): add tests for GRPCPathInfoServiceWrapperFlorian Klink2-0/+108
Change-Id: I22de33c28c132fc9710ddecd0e9158ee3b3ef94b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8108 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5914 feat(tvix/store): add tests for GRPCDirectoryServiceWrapperFlorian Klink2-0/+263
Change-Id: Iccf70664cd5dd8ea22e0b43002d2ff1a9680870a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8102 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5911 feat(tvix/store): add tests for GRPCBlobServiceWrapperFlorian Klink2-0/+211
Change-Id: I0c9a2d887b5b94c43afee97a8a1e701cc9f74f23 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8098 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-10 r/5909 refactor(tvix/store): remove anyhow::Result in directory it testFlorian Klink1-3/+1
There's no `?` being used. Change-Id: Ifbfd88f6056e0064d11458cb335d05b333ec7b6b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8127 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2025-01-01T11·56+0000