| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Parsing of the narinfo file sets the compression field to None instead
of Some("none"). The mapping selecting the decompression reader expected
the former in //tvix/store/src/pathinfoservice/nix_http.rs.
Change-Id: I254a825b88a4016aab087446bdc0c7b6286de40c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12007
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Align these with the way it's called in the ed25519 crates.
Change-Id: Ia52d3bb9bf831dc6b5f7d5356f5ac62135672883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12013
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
|
|
Change-Id: Ia4035aca43cf9d3f7de982dd154715120ba25496
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11927
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We were wrongly comparing the raw row_key with the store path
digest, but we hexlower-encode the digest in the row key (via
derive_pathinfo_key).
Update the logic to fix that.
Change-Id: I8916d8de9fb8b25a6986d4158faa91ec97c57347
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11926
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Introduces a helper function within tvix-tracing that returns a reqwest
tracing middleware that will ingest the traceparent if otlp is enabled.
It is feature flagged in tvix-tracing so not every consumer of that
library automatically has reqwest in its dependencies.
Tested using netcat to verify that the `traceparent` header is there if
otlp is enabled and missing if otlp feature is disabled.
Change-Id: I5abccae777b725f5ff7382e3686165383c477a39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11886
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
We don't need to require these things for these impl blocks yet.
Change-Id: I3cec958a637a4f900bdd38abd00e9133bf75ce46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11865
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
|
|
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.
The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.
This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.
Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This is currently still taking a noticeable amount of time, so make sure
we show it is happening.
Change-Id: I13d18785fbf41ae4479e1ea58d61ece1d7485719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11847
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This wraps ingest_nar, but also keeps track of the number of bytes read,
and calculates the sha256 digest of it.
Make use of it in the NixHTTPPathInfoService, where this code is coming
from.
Change-Id: I9c54e93d3ec8ed9ede87aed43e04d114fb06897b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Introduce a new common crate that contains tracing boilerplate which then
can be used in the cli, tvix-store and tvix-build crates.
It has otlp as an optional feature, which is currently only used by
tvix-store.
Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
|
|
Since cl/…, a PathInfoService doesn't need to implement `calculate_nar`
anymore, so most of them don't actually have a handle to a
{Blob,Directory}Service anymore.
This means, we can simplify the construction of them for test cases
a lot.
Change-Id: I100e9e1c9b00a049b4d6136c57aad4cdb04461c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11691
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This allows querying two PathInfoService implementations sequentially,
and inserts into the "near" one if it's not there yet.
There is no negative cache, and put / listing is not implemented (for
now).
Change-Id: I24c3d0e0c3c2f0524a6cc7b2f3cbc33eb20cf92b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11636
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Add instrumentation to the get() and put() implementations of all
PathInfoService.
Use the nixbase32 representation of the digest, not the base64 one.
Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We don't want to use the std::sync::RwLock here, as it blocks.
This also means we don't need to deal with the error cases anymore.
The list() implementation is updated to use try_stream, which means we
can now avoid collecting everything into a Vec before returning from it.
Change-Id: I9057dcc410dc553e6b1be3f20d5ee830569e8218
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11611
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This provides an implementation of PathInfoService storing PathInfo in
memory up to a certain capacity, then evicting these that have been used
the least recently.
Change-Id: I9d738687caf4f181a957f72245f26b92832313cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11622
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
These are not used anymore.
Change-Id: I9c348391c9600e9319f171faf3eda7175ebf7076
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11621
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
These are not used anymore.
Change-Id: I6c16b4d80ddaabcb75fec3ea3e32b923b7719485
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11620
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
This shouldn't be part of the PathInfoService trait.
Pretty much none of the PathInfoServices do implement it, and requiring
them to implement it means they also cannot make use of this calculation
already being done by other PathInfoServices.
Move it out into its own NarCalculationService trait, defined somewhere
at tvix_store::nar, and have everyone who wants to trigger nar
calculation use nar_calculation_service directly, which now is an
additional field in TvixStoreIO for example.
It being moved outside the PathInfoService trait doesn't prohibit
specific implementations to implement it (like the GRPC client for the
`PathInfoService` does.
This is currently wired together in a bit of a hacky fashion - as of
now, everything uses the naive implementation that traverses blob and
directoryservice, rather than composing it properly. I want to leave
that up to a later CL, dealing with other parts of store composition
too.
Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This does IO, which might take a longer amount of time than what we want
to be blocking the normal executor.
Use spawn_blocking instead. I didn't add it for the constructors, as we
only call these once.
Change-Id: I9a1063099bac9582ca9681043c58c1edc780c5ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11618
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
We already have the same code in make_grpc_path_info_service_client.
Change-Id: Ibcd60831af8a061a2c3bb2f960f52a43d06cf6fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11615
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Autosubmit was too fast and submitted this before my push went through.
Flagged in https://cl.tvl.fyi/c/depot/+/11580/comment/40a56824_7be73334/
Change-Id: I1f835ae60c2c6fd2db6654c1b1c71d90bee141af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11595
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Rename read_nar to ingest_nar, and have it use the async nar reader
version, and the ingest_entries machinery.
This means we can now drop all code dealing with manually assembling
castore nodes.
Update our consumer, NixHTTPPathInfoService to use the new API.
As we now accept an AsyncRead, we don't need to do any blocking here
anymore, and can use the same async-compression crate as in the fetching
logic (and support some more compression formats out of the box).
Change-Id: I8646d20bd8603f8da47b5c84bc9e4ac236eb7f1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11580
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Write this a bit more compact, by using map_err(|e| …) and ?.
Ideally we'd get rid of the error mapping entirely, by using proper
error types, but that's left for a followup.
Change-Id: I68dc72b162ac89c5ff82d8c2bc26e1c808a0affd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11584
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
The emulator and bigtable client are quite big. Remove them from the
default //tvix:shell.
Put the tests behind a `integration` feature flag, and add a variant
with that enabled to CI, and drop the bigtable tools from //tvix:shell.
Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Also, rename the DUMMY_NAME constant in the fixtures to DUMMY_PATH,
which aligns more with the ToString representation and from_bytes
conversions we have on StorePath[Ref].
Change-Id: I39763c9dfa84c5d86f2fd0171b3a4d36fd72f267
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11464
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Change-Id: Id673cd9c574b00a739de78aab2ee86f9d9b1ed40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11463
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This exposes `user.tvix.castore.{blob,directory}.digest` xattr keys for
files and directories:
```
❯ getfattr -d /tmp/tvix/06jrrv6wwp0nc1m7fr5bgdw012rfzfx2-nano-7.2-info
getfattr: Removing leading '/' from absolute path names
user.tvix.castore.directory.digest="b3:SuYDcUM9RpWcnA40tYB1BtYpR0xw72v3ymhKDQbBfe4="
❯ getfattr -d /tmp/tvix/156a89x10c3kaby9rgf3fi4k0p6r9wl1-etc-shells
getfattr: Removing leading '/' from absolute path names
user.tvix.castore.blob.digest="b3:pZkwZoHN+/VQ8wkaX0wYVXZ0tV/HhtKlSqiaWDK7uRs="
```
It's currently mostly used for debugging, though it might be useful for
tvix-castore-aware syncing programs using the filesystem too.
Change-Id: I26ac3cb9fe51ffbf7f880519f26741549cb5ab6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11422
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
|
|
This kept retrying indefinitely if the socket didn't appear.
Change-Id: I4d4ef61df73cef6abda698501432f370abc8a82c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11406
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Change-Id: Idb7d7144be1917fbaf83e9fd76c5b2ebb3df98d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11400
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
Put behind the "cloud" backend, like in the `tvix-castore` crate.
Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Remove usage of the gen_{blob,directory}_service() helper functions
from utils. We populate Memory{Blob,Directory}Services here directly, as
test_case and rstest doesn't compose well.
Change-Id: I0fb48aadb8c818f508b18ceb83c85eb91359442a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11278
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: Ib114a4d141ca829520aed34600678d613994f875
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11277
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This introduces rstest-based tests. We also add fixtures for creating
some BlobService / DirectoryService out of thin air.
To test a PathInfoService, we don't really care too much about its
internal storage - ensuring they work is up to the castore tests.
Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This makes the fixture more use-able when interacting with the trait,
the Bytes are only useful for the gRPC version.
Change-Id: Iaaea1adc6df18491f236a28c4343f5b4ee5fcfd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11271
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I7fc06ae97a50d04b8c36292b3457c112242a7a70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11270
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This will allow feature-flagging some of the backends.
Change-Id: Ie92914c3e2ad870eee87e73b3b5abe605fb56fe7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11202
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This will give us the sha256: prefix, and hashes we're more used to, in
that context.
Change-Id: I72e42fe685e365ba9baa7cd81001387d239fa7c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11194
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
If there's an unexpected test failure, print it out, rather than just
saying something is false even though it should be true.
Use .expect() for this, which displays the error if it failed.
We can't use expect_err(), as our stores are not display'able, so use an
assertion with a message there.
Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I5f4e93aceec373eb3377e3f8fbb55a5142cc1363
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10740
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
The BoxStream type alias is a more concise and easier to read than
the full `Pin<Box<dyn Stream<Item = ...> + Send + ...>>` type.
Change-Id: I5b7bccfd066ded5557e01f7895f4cf5c4a33bd44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10677
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
|
|
We don't need Arcs in most of the cases, we're fine with some container.
Change-Id: Ic4f8acb5b9d93e2b0923bb607463fb91e9d0e4fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10606
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Removes some more needs for Arcs.
Change-Id: I9a9f4b81641c271de260e9ffa98313a32944d760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10578
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: I557ed67a04112ba3e363303637d1988309250a93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Only convert to and reuse an Arc<…> where needed.
Change-Id: I2c1bc69cca5a4a3ebd3bdb33d6e28e1f5fb86cb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10514
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
cl/10378 did already move store/fs to castore/fs, but we kept the tests
in tvix-store, as they were populating a PathInfoService to make nodes
appear in the mount root.
Update these tests to now just insert root nodes into a BTreeMap, and
ensure we can use that as a RootNodes too.
Change-Id: Iad7d1ee4f9423eb6e3a1da33f433842c9ae0de1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10410
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
With the recent introduction of the RootNodes trait, there's nothing in
the fs module pulling in tvix-store dependencies, so it can live in
tvix-castore.
This allows other crates to make use of TvixStoreFS, without having to
pull in tvix-store.
For example, a tvix-build using a fuse mountpoint at /nix/store doesn't
need a PathInfoService to hold the root nodes that should be present,
but just a list.
tvix-store now has a pathinfoservice/fs module, which contains the
necessary glue logic to implement the RootNodes trait for a
PathInfoService.
To satisfy Rust orphan rules for trait implementations, we had to add a
small wrapper struct. It's mostly hidden away by the make_fs helper
function returning a TvixStoreFs.
It can't be entirely private, as its still leaking into the concrete
type of TvixStoreFS.
tvix-store still has `fuse` and `virtiofs` features, but they now simply
enable these features in the `tvix-castore` crate they depend on.
The tests for the fuse functionality stay in tvix-store for now, as
they populate the root nodes through a PathInfoService.
Once above mentioned "list of root nodes" implementation exists, we
might want to shuffle this around one more time.
Fixes b/341.
Change-Id: I989f664827a5a361b23b34368d242d10c157c756
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
|
|
This makes PathInfoService::from_addr return a Box<dyn PathInfoService>,
rather than an Arc<dyn …>, and leaves it up to the consumers to rewrap
it into an Arc where needed.
This allows us to drop the Arc for the tvix-store daemon subcommand.
Change-Id: Ic83aa2ade6c51912281bd17c7eef7252e152b2d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10409
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
|
