Age | Commit message (Collapse) | Author | Files | Lines |
|
Change-Id: I7be0a4eaa29840e1a3329bccb791975de22828ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8085
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This came up recently again, it makes sense to document the reasoning
behind the decision.
Change-Id: Ic51d5bc7998c70e8b070b6f42877d8e88613935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8223
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: Id8e32ee4de98fbe48266d417279194cd93e968cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8222
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I2c2a70e5ffbb080702f57218aa9dcfe53d08870f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8110
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Instead of using an explicit closure to clone elements, use .cloned().
Change-Id: I31f0f0bad2b4935e1a8d91fa0d14163c94182e1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8109
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
The size field already is u32, we don't need to convert here.
Change-Id: Ie29819aa2d1d8022e9bd73fcf05b140e45c967a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8107
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Further emphasize Read() can be used to ask for blobs OR chunks, and
that clients usually want to stat and then request (smaller) chunks,
rather than reading whole blobs.
Also clarify that the chunking used to send BlobChunks over has nothing
to do with the chunk sizes communicated in a Stat() request.
Change-Id: Ia615d190aae570611de2655b11342a14d0b75976
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This adds a function that consumes a [proto::node::Node] pointing to the
root of a (store) path, and writes the contents in NAR serialization to
the passed [std::io::Write].
We need this in various places:
- tvix-store's calculate_nar() RPC method needs to render a NAR stream
to get the nar hash, which is necessary to give things imported in
the store a "NAR-based" store path.
- communication with (remote) Nix (via daemon protocol) needs a NAR
representation.
- Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a
NAR representation.
Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This provides a `Directory.nodes()` function, returning an iterator over
all three node types in an ordered fashion.
Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Make the trait public, so consumers can use it.
Also, implement it for node::Node, so we can later use this to access
the name from all three if we don't care about the enum type.
Change-Id: Iae530a16b705493645e61947852c03273876cc55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Move nixbase32 and store_path into this.
This allows //tvix/cli to not pull in //tvix/store for now.
Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Replace our data_encoding usage with the implementation taken from
https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32
Also uncomment some of the unit tests, and add a regression test for a
NIXBASE32.encode with a 32 bytes sequence.
The previous implementation of NIXBASE32.encode is wrong in that case:
```
❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA=
0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk
❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C
00000000 b3 a2 4d e9 7a 8f db c8 35 b9 83 31 69 50 10 30 |..M.z...5..1iP.0|
00000010 b8 97 70 31 bc b5 4b 3b 3a c1 37 40 f8 46 ab 30 |..p1..K;:.7@.F.0|
00000020
```
Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
There's gonna be more tests coming, let's move them into a separate file
everything Directory-related into a separate module.
Change-Id: I78a0f263925528907a22724b028f75cce644b329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Also add a `NamedNode` trait. We'll later use this to access names from
all three individually.
Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: I361dbca444a267fea28cd212d563ee2d03497c16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7952
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: Idf835e9648ab6c95c29960f3f8176153d4888b27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7951
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This is used for content-defined chunking.
Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This calls out to Directory::validate() for all received Directory
messages, and also makes sure the sizes we refer a Directory message as
matches the sizes that have been calculated.
Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
We can omit sending Directory messages to clients that have already been
sent in the same stream.
We can also omit storing a Directory message if we already have it -
they're content-addressed anyways.
Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Upstream seems to be dead, so we're using https://github.com/tyrchen/
tonic-mock/pull/3 here.
According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment-
1241164173, we might not need this crate at all, but for now, it gets
the job done and is less code to write in the tests.
Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Import more things, and use expect_err to unpack the response.
Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: I8b36c85fe11e4fb62e5d28f6900cd80d89a5cc41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7854
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: I8a16c214c7c644756d9d54187beba8c80ccfb39c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7853
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: Id325afa766b0bd2a3d04cf538f13e77692a22687
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7829
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This uses [sled](https://github.com/spacejam/sled) to store PathInfo
objects.
Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: I49799393a1e35b4475566819111beb57a628c555
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7801
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: I8f16e25d998a74ac110ae99eec0edbbfd720c8dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7800
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This embeds the build config directly at the point where `Cargo.nix`
is imported, making it transparent to library consumers.
Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ifab28d97ddc22a2073c5df5e6e2cefb51b4b9191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7777
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
As discussed in #tvl, this is a more common term for it.
Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This provides a function returning a string starting with the store
path prefix, the counterpart of `from_absolute_path`.
Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This allows constructing a NixPath from an absolute path. It pops off
the STORE_DIR prefix and the trailing slash and returns an error if it
couldn't be found.
Change-Id: Ib540e353c63cc247ac15e20414b0db2caf695ef4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7751
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Rename PATH_HASH_SIZE to DIGEST_SIZE.
It's a digest, not hash (we don't necessarily have the internal hash
state anymore), and the fact it's about (Nix)Paths is already visible
from the module name.
Also expose ENCODED_DIGEST_SIZE, so we don't need to do the calculation
inside from_string() method, and it becomes more clear this is a
constant.
Change-Id: I0e7577dd7a6e503039037b986313b214e995d826
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7725
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Tested-by: BuildkiteCI
|
|
These can be accessed directly.
Change-Id: I71dc84f982820d53f319efefbed9b9055034954d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7724
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This provides validation of PathInfo messages, and ensures the output
hashes are properly parsed from the root node names.
NixPath already has a more extensive test suite for various wrong
NixPaths, so it's omitted from here.
Change-Id: I5d69118df5816daabb521ddb19d178bddd1caacf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7684
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This allows other crates to import tvix_store.
Rename the bin crate to tvix-store-bin, to avoid having multiple crates
with the same name (https://github.com/rust-lang/cargo/issues/6313)
Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: Ib47a55e39ed752492b9732439de5f327a7fa601e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7723
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Change-Id: Ie400a7109d3aa6b66153c0ab25a38abf06984cbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7722
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/
tracing.
Annotate all method handlers with an instrument macro, and warn! a
message for them being unimplemented.
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools
like evans automatically discover available services, without having to
specify the path to the .proto files client-side.
It's behind a reflection feature flag, which is enabled by default.
Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Change-Id: Ibe550f9573b0f45ee95453b50c7510e49b07c719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7685
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
|
|
This refactors how the original digest type (Vec<u8>) is converted
to [u8; PATH_HASH_SIZE].
Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This replaces the hello world example from tvix-store with an actual
gRPC endpoint, implementing all of BlobService, DirectoryService and
PathInfoService.
All RPC methods currently respond with the unimplemented gRPC status.
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
Pointed out by edef in
https://cl.tvl.fyi/c/depot/+/7648/comment/4551ba4b_e89ade36/#, thanks!
Change-Id: I6b0d317bb0210521622483cdf4cb557bc637a100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7709
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
Similar to cl/7682, we also want to make that error configurable.
Change-Id: I64f1a4570b3d75af4741abe10c2855959766e107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7708
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This implements the NixPath structure. NixPath allow to parse a string to a nix path. If the parsing fails, a DecodeError will be raised.
Change-Id: I28363cdcfb27f04bf21a11c0d130b461667e3720
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This implements the nix-specific base32 encoding and decoding, exposing
a subset of the API that the data-encoding crate provides.
Nix uses a custom alphabet, no padding, and encodes bytes in reverse
order. The latter one is the reason we can't just use the data-encoding
crate directly.
Three odd corner case tests ported over from go-nix failed. We opened
b/235 to further investigate.
Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We'll be using validate_node_name in other places in a bit, where
returning a ValidateDirectoryError is not appropriate.
Accept a function mapping a string to error as a second argument, and
pass ValidateDirectoryError::InvalidName at the current call sites.
Change-Id: I45cbb0deb4763061ad912c6b18a112c727795a17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7682
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I351a2bd4e007443aef1b97f50aecffc095ac60b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7681
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
|