| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
We have two places where we parse protos and want their names to be
empty:
- Receiving a root node in a nar-bridge NAR request
- Processing the CalculateNAR gRPC call
We don't have any place where we want to keep a name as bytes::Bytes
around, yet we used the `into_name_bytes_and_node` method.
It was also a bit wrongly named - it wasn't very clear the name was
not validated, and that the function may fail.
This moves the "splitting off the name as bytes::Bytes" part into a
private helper, only leaving the `try_into_name_and_node` and
`try_into_anonymous_node` methods around.
Change-Id: I2c7fd9871d49ec67450d7efa6a30d96197fb319c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12664
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This switches the PathInfoService trait from using the proto-derived
PathInfo struct to a more restrictive struct, and updates all
implementations to use it.
It removes a lot of the previous conversion and checks, as invalid
states became nonrepresentable, and validations are expressed on the
type level.
PathInfoService implementations consuming protobuf need to convert and
do the verification internally, and can only return the strongly typed
variant.
The nix_compat::narinfo::NarInfo conversions for the proto PathInfo
are removed, we only keep a version showing a NarInfo representation for
the strong struct.
Converting back to a PathInfo requires the root node now, but is
otherwise trivial, so left to the users.
Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I6fdfdb44063efebb44a8f0097b6b81a828717e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12588
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Don't use DirectoryError, but PathComponentError.
Also add checks for too long path components.
Change-Id: Ia9deb9dd0351138baadb2e9c9454c3e019d5a45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12229
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
|
|
Replace the hand-rolled code comparing names with a try_fold. Also, make
it slightly stricter here, detecting duplicates in the same fields
earlier.
Change-Id: I9c560838ece88c3b8b339249a8ecbf3b05969538
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12226
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
This encodes a verified component on the type level. Internally, it
contains a bytes::Bytes.
The castore Path/PathBuf component() and file_name() methods now
return this type, the old ones returning bytes were renamed to
component_bytes() and component_file_name() respectively.
We can drop the directory_reject_invalid_name test - it's not possible
anymore to pass an invalid name to Directories::add.
Invalid names in the Directory proto are still being tested to be
rejected in the validate_invalid_names tests.
Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.
This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.
It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).
The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.
The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.
The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.
Some further cleanups:
The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)
There's some leftover cleanups to do:
- There should be a from_(sorted_)iter and into_iter in Directory, so
we can construct and deconstruct in one go.
That should also enable us to implement conversions from and to the
proto representation that moves, rather than clones.
- The BuildRequest and PathInfo structs are still proto-based, so we
still do a bunch of conversions back and forth there (and have some
ugly expect there). There's not much point for error handling here,
this will be moved to stricter types in a followup CL.
Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
*Node and Directory are types of the tvix-castore model, not the tvix
DirectoryService model. A DirectoryService only happens to send
Directories.
Move types into individual files in a nodes/ subdirectory, as it's
gotten too cluttered in a single file, and (re-)export all types from
the crate root.
This has the effect that we now cannot poke at private fields directly
from other files inside `crate::directoryservice` (as it's not all in
the same file anymore), but that's a good thing, it now forces us to go
through the proper accessors.
For the same reasons, we currently also need to introduce the `rename`
functions on each *Node directly.
A followup is gonna move the names out of the individual enum kinds, so
we can better represent "unnamed nodes".
Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This uses our own data type to deal with Directories in the castore model.
It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:
- In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
- In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.
Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This adds `Directory::add` which is a convenience helper for adding
nodes into a `Directory` while preserving sorted order.
This implements `Ord` and `PartialOrd` for `FileNode`, `SymlinkNode`,
and `DirectoryNode` so `binary_search` can be used.
Change-Id: I94b86bdef5d0da55aa352e098988b9704cafca19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11481
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This drops pretty much all of castore/utils.rs.
There were only two things left in there, both a bit messy and only used
for tests:
Some `gen_*_service()` helper functions. These can be expressed by
`from_addr("memory://")`.
The other thing was some plumbing code to test the gRPC layer, by
exposing a in-memory implementation via gRPC, and then connecting to
that channel via a gRPC client again.
Previous CLs moved the connection setup code to
{directory,blob}service::tests::utils, close to where we exercise them,
the new rstest-based tests.
The tests interacting directly on the gRPC types are removed, all
scenarios that were in there show now be covered through the rstest ones
on the trait level.
Change-Id: I450ccccf983b4c62145a25d81c36a40846664814
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11223
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I5f39d0e613b651458b168cfd9df0693d7f01d704
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11246
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
- Adjust to ecl 23.9.9 release
- Regenerate go protos after protoc-gen-go update
- Drop dhall fork which hasn't kept up with 1.42.*
- Address new clippy warnings:
- Variant naming of Error::ValidationError
- Simplify .try_into().unwrap()
- Drop unnecessary identity function
- Test module must be last in file
- Drop unused `pub use`
- Update agenix to 0.15.0. Current master has a installCheckPhase that
doesn't work with C++ Nix 2.3.*:
https://github.com/ryantm/agenix/commit/a23aa271bec82d3e962bafb994595c1c4a62b133#commitcomment-137185861
Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
|
|
Not a single call site actually makes use of the Vec.
Change-Id: I6cf31073c9f443d1702a21937a0c3938c2c643b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9988
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Having more than 4GiB files is quite possible (think about the NixOS
graphical installer, and an uncompressed iso of it).
No wire format changes.
Change-Id: Ia78a07e4c554e91b93c5b9f8533266e4bd7f22b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9950
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Implement `validate()` on `node::Node`, and call it from PathInfo's
validate() too. Node-related errors are moved to a ValidateNodeError
error type.
This additionally adds some more validations for symlink targets (they
must not be empty, and not contain null bytes).
Change-Id: Ib9b89f1c9c795e868a1533281239bc8a36d97c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
|
|
We use checked arithmetic for computing the total size, and verify
that size is in-bounds in Directory::validate.
If an out-of-bounds size makes it to the "unchecked" size method,
we either panic (in debug mode), or silently saturate to u32::MAX.
No new panic sites are added, since overflows in debug mode already
panic at the language level.
Change-Id: I95b8c066a42614fa447f08b4f8fe74e16fbe8bf9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9616
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Iebfa12df51c8c9d39b4d9eb2fccc6ffa312639cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9620
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Similar to gen_directorysvc_grpc_client, introduce a
gen_blobsvc_grpc_client function that provides a gRPC client connected
to a blobservice.
The test is update to use that client to test against, rather than the
server trait, removing the last usage of tonic_mock, so it's removed
as well.
Fixes b/243.
Change-Id: If746e8600588da247eb53a63b70fe72f139e9e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9564
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
We can use DuplexStream to create to bidirectional pairs, which avoids
manually waiting for unix sockets to pop up and connect, and creating
temporary directoires to create the unix sockets in.
Turns out, we also don't actually need to spawn the server in a separate
runtime, it works just fine these days. This might be due to all the
sync barriers in between being gone.
Change-Id: I6b79823bc6209cbcb343b7a498c64a2ba6e0aee7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9562
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
The tests were using the gRPC Service/Server interface, not the gRPC
clients.
Use our new gen_directorysvc_grpc_client() function to spin up a server
in a separate thread, and test via that client, rather than *just* the
server implementation.
This is more correct, because we're now really exercising the gRPC
stack, and some of the stream implementations are also only implemented
on the client.
Change-Id: I99434a2399856a44c9e6988d0b720f7a429d0ad1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9560
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.
Notable changes:
- `fixtures` and `utils` had to be moved out of the `test` cfg, so they
can be imported from tvix-store.
- Some ad-hoc fixtures in the test were moved to proper fixtures in the
same step.
- The protos are now created by a (more static) recipe in the protos/
directory.
The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)
Bug: https://b.tvl.fyi/issues/301
Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
