| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
tvix-castore already supports composition without any additional feature
flags, the only thing that can be explicitly enabled is referring to
other stores via an anonymous url. Rename that feature flag to
"xp-composition-url-refs".
tvix-store effectively only controls the CLI surface, so rename this to
"xp-composition-cli".
The "store" in the feature name was dropped, as it's already apparent
from being in the tvix-[ca]store crate.
Change-Id: I1175dc6280cbba4cbcdfb7fd4b35fce713b45fc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12747
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
|
|
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: Ia0370ca46cb1c6122a452b1d117160536b632c7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
|
|
This allows specifying an url in place of a named reference to another
composition entry, if the castore crate has been compiled with the
xp-store-composition feature.
Example: `--directory-service-addr cache://?near=memory://&far=memory://`
This would be equivalent to the instantiation via toml file:
```toml
[memory1]
type = "memory"
[memory2]
type = "memory"
[default]
type = "cache"
near = "memory1"
far = "memory2"
```
Note that each anonymous url causes a distinct instance to be created.
Change-Id: Iee5a07a94b063b5e767c704d9cad0114fa843164
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12146
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Over the past couple of months we've been using redb instead of sled as
the default filesystem-based database in PS and DS. I am confident that
we can get rid of sled completely now, and just keep redb.
Change-Id: I11fa1e4453e280253855f8eade990b37eb6965ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12567
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
|
|
This provides a DirectoryService implementation which uses
redb (https://github.com/cberner/redb) as the database. It provides both
in-memory and persistent on-filesystem implementations.
Change-Id: Id8f7c812e2cf401cccd1c382b19907b17a6887bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12038
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I3c94ecb5958294b5973c6fcdf5ee9c0d37fa54ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11976
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
|
|
We previously called ObjectStoreBlobService::parse_url, which passes an
empty list of options when constructing the ObjectStore.
This is most likely not what we want. The more reasonable thing to do is
pass along the query string (pairs) as options to
`object_store::parse_url_opts`, and remove them from the plain URL we
pass to object_store itself.
Change-Id: Ic2cb1dca2a2980a863165d81baa3323a355cf3fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11897
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This introduces optional helper function in tvix/tracing for trace
propagation and uses these helper in the `tvix-store`.
The GRPCBlobService, GRPCDirectoryService and GRPCPathInfoService now
accept a generic client, meaning the client can be generated with either
`::new` or `::with_interceptor`.
This was tested and validated by starting a `tvix-store daemon` and
`tvix-store import`.
Change-Id: I4b194483bf09266820104b4b56e4a135dca2b77a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I1636012be2e8ee3ae64f7bc62fd28bfe0cb2bca5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11668
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
The emulator and bigtable client are quite big. Remove them from the
default //tvix:shell.
Put the tests behind a `integration` feature flag, and add a variant
with that enabled to CI, and drop the bigtable tools from //tvix:shell.
Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This adds a Directory service using
https://cloud.google.com/bigtable/docs/ as a K/V store.
Directory (closures) are put in individual keys.
We don't do any bucketed upload of directory closures (yet), as castore/
fs does query individually, does not request recursively (and buffers).
This will be addressed by store composition at some point.
Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I52b14652822766421d66e95bc646ed7baecc705f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11369
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
|
|
This will allow feature-flagging some of the backends.
Change-Id: Iddbdb89d3cf9c966a2c25b06b03e6917b284cae5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11201
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
|
|
If there's an unexpected test failure, print it out, rather than just
saying something is false even though it should be true.
Use .expect() for this, which displays the error if it failed.
We can't use expect_err(), as our stores are not display'able, so use an
assertion with a message there.
Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
|
|
While we currently mostly use it in an Arc, as we need to clone it
inside PathInfoService, there might be other usecases not requiring it
to be Clone.
Change-Id: Ia05bb370340792a048e2036be30e285ef1e63870
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10483
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
We don't really require the Path to be a PathBuf, we don't even require
it to be a Path, we only need it to be AsRef<Path>>.
This removes some conversion in the from_addr cases, which can just
reuse `url.path()` (a `&str`).
Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
This moves the sync `channel::from_url` to a async
`tonic::channel_from_url`. It now allows connecting non-lazily if `wait-
connect=1` is set in the URL params.
Also, make the pingpong tests for blobsvc and directorysvc use the wait-
connect=1 codepath.
Change-Id: Ibeea33117c8121814627e7f6aba0e943ae2e92ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10030
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Make directoryservice::from_addr use the more specific constructors.
Change-Id: I9fee2afed77692505988d631d9fe246d9843d25a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10029
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.
Notable changes:
- `fixtures` and `utils` had to be moved out of the `test` cfg, so they
can be imported from tvix-store.
- Some ad-hoc fixtures in the test were moved to proper fixtures in the
same step.
- The protos are now created by a (more static) recipe in the protos/
directory.
The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)
Bug: https://b.tvl.fyi/issues/301
Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
