| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Derivation that depends on `${d.drvPath}` generates a `NixContextElement::Derivation(drvPath)` context string.
In turn, this makes the dependent derivation depend on *ALL* outputs of that derivation.
Note that a dependency on `${d.drvPath}` generates an input source
dependency too.
This is a complete implementation of the context string system in Nix on
the "input population" side.
The test coverage of this change is taken care in cl/11264.
Change-Id: I97fe5f7c772a6b1cc4366bee071aa691a11fcde6
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11261
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Iddcee013e6d187f9a71329f0cccc6a0205aeb497
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11288
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
This is currently unused, and even if it would be used, it'd probably
be with `default-features = false`, explicitly enabling what's needed.
Change-Id: I3175a19adf8d65f0b805e5234267e460cd5e68d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11287
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
|
|
This provides an emulator for Google Bigtable.
This has also been sent to nixpkgs in
https://github.com/NixOS/nixpkgs/pull/298044, and can be dropped (modulo
the patches) once we bump past it.
Change-Id: Id7809a02065ce479d6d29813d1de454cd230d2df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11211
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Nixpkgs make use of it and this is a 2.3 builtin that was missed.
Change-Id: If36d748eed4a5f777344556c0802392c76c8d53c
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11260
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Make this function async, and do the block_on on the (single) callsite.
Change-Id: Ib8b0b54ab5370fe02ef95f38a45d8866868a9d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11285
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Replace the (single) callsite with some code interacting with the tokio
runtime to block on the async version.
Change-Id: I3976496ae77b2bb8734603f303655834265e3f0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11284
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Let's get rid of these sync helpers, they make this less understandable.
Change-Id: I3c7294647849db2747762722247c65e4e2947757
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11283
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: If3a45d2f8008b75524ead704b05320364cc60d92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11282
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
We need to define behaviours and add tests for these.
Change-Id: Id5825fafbf47897d8de42503ea6006eb131b1082
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11281
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This drops pretty much all of castore/utils.rs.
There were only two things left in there, both a bit messy and only used
for tests:
Some `gen_*_service()` helper functions. These can be expressed by
`from_addr("memory://")`.
The other thing was some plumbing code to test the gRPC layer, by
exposing a in-memory implementation via gRPC, and then connecting to
that channel via a gRPC client again.
Previous CLs moved the connection setup code to
{directory,blob}service::tests::utils, close to where we exercise them,
the new rstest-based tests.
The tests interacting directly on the gRPC types are removed, all
scenarios that were in there show now be covered through the rstest ones
on the trait level.
Change-Id: I450ccccf983b4c62145a25d81c36a40846664814
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11223
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
As for grpc_pathinfoservice, we test this in the rstest-based tests,
with more implementations.
tests/utils.rs is unused now.
Change-Id: I62b192170d181e4fb432bf7dfff2057cb048b52c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11279
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Remove usage of the gen_{blob,directory}_service() helper functions
from utils. We populate Memory{Blob,Directory}Services here directly, as
test_case and rstest doesn't compose well.
Change-Id: I0fb48aadb8c818f508b18ceb83c85eb91359442a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11278
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: Ib114a4d141ca829520aed34600678d613994f875
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11277
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I975729cb97d69d080fb63fbb8f81b4ac46cbfb7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11276
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Previous CLs ensured BlobService and DirectoryService is implemented on
AsRef<dyn BlobService> and AsRef<dyn DirectoryService> respectively, so
there's no need to require AsRef in here.
Change-Id: I1e8b93188ca784e1b9a592f3d35fcf6b9df75c75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11275
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This stops using the gen_{blob,directory}_service() functions from
tvix_castore::utils in favor of the rstest-based fixtures.
Change-Id: I4c2fff2926f34af63d3bd5c29c8e939dbdf7e062
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11274
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This introduces rstest-based tests. We also add fixtures for creating
some BlobService / DirectoryService out of thin air.
To test a PathInfoService, we don't really care too much about its
internal storage - ensuring they work is up to the castore tests.
Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This makes the fixture more use-able when interacting with the trait,
the Bytes are only useful for the gRPC version.
Change-Id: Iaaea1adc6df18491f236a28c4343f5b4ee5fcfd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11271
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I7fc06ae97a50d04b8c36292b3457c112242a7a70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11270
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This allows StorePathRef as keys in a hashmap, by deriving Hash. The same is already done for StorePath.
Change-Id: I3fc54c45787948116dcb27dfb5dc806b9b505835
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11269
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I7776635b17c44534223603d28cf59c7eebd976e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11229
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
At least terraform wants all parameters passed via JSON to be strings.
It can't accept maps.
This means, allowing to pass a `argstr` JSON dict isn't really possible.
However, terraform is perfectly able to JSON-encode a map. So accept a
`argstr_json` argument instead, which `jq` will JSON-decode before
further processing it.
I dropped `argstr` support again so the jq expression still fits on my
screen, if anyone else (started) using this, I'm happy to review a CL
adding this. We should probably move the jq expression to some multiline
format then, though.
Change-Id: I5ab7a1169ab7305d3ab02db31c27732d9d1ab4e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11228
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
|
|
If this is set to true (and only then), also invoke `nix-build` on the
previously-instantiated .drv to cause builds/substitutions on the local
machine.
There's no terraform example for this in here, but this is useful if you
want to perform builds locally, for example to upload nix-built blobs
elsewhere through terraform.
Change-Id: Idcf7b8527aa9c27f6f9ca60ca607c29d82e1cce9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11215
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Implementing the full connection handshake.
The integration test is a bit naive, but there's not much to test yet.
Tested this against cpp nix. We reach the stage where cppnix sends the
opcode.
Change-Id: I98322832848ee5b048f22105731b0adeb44b2ce0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11227
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We already did all the instantiation, grabbing the calculated output
path too is cheap.
Change-Id: Id591865c65159409da739f706a9de29a9f50456a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11214
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: Ifae9c41e1e3fa5e96f9b7e188181a44650ff8a04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11250
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
This allows us to use containers around BlobServices as BlobServices too.
Change-Id: I3c7feb074f42b4e07c550fb8dfa63cf81d448ab5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11249
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This creates test scenarios (using the DirectoryService trait) that we
want all DirectoryService implementations to pass.
Some of these tests are ported from proto::tests::grpc_directoryservice,
which tested this on the gRPC interface (rather than the trait),
some others ensure certain behaviour for which we only recently
introduced general checking logic (through ClosureValidator).
We also borrow some code related to setting up a gRPC DirectoryService
client (connecting to a server exposing a in-memory DiretoryService)
from castore::utils, this will be deleted once it's all ported over.
Change-Id: I6810215a76101f908e2aaecafa803c70d85bc552
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11247
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This allows us to use containers around DirectoryServices as DirectoryServices too.
Change-Id: I56cca27b3212858db8b12b874df0e567dd868711
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11248
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This uses DirectoryClosureValidator for validation and the sled batch
API to insert multiple directories at once.
Change-Id: I2d6dc513ccbc02e638f8d22173da5463e73182ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11222
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This greatly simplifies the code in this function, replacing it with a
much better tested (and more capable!) version of the validation logic.
It also enables the gRPC server frontend to make use of the
DirectoryPutter interface. While this might not be too visible in terms
of latency thanks to gRPC streams bursting, it also enables further
optimizations later (such as bucketing of directory closures).
Change-Id: I21f805aa72377dd5266de3b525905d9f445337d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11221
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
This simplifies a bunch of code, and gets rid of some TODOs.
Also, move it out of castore/utils, and into its own file.
Change-Id: Ie63e05a6cdfb2a73e878cf7107f9172aed1cdf13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11224
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
|
|
This can be used to validate a Directory closure (connected DAG of
Directories), and their insertion order.
Directories need to be inserted (via `add`), in an order from the leaves
to the root. During insertion, we validate as much as we can at that
time:
- individual validation of Directory messages
- validation of insertion order (no upload of not-yet-known Directories)
- validation of size fields of referred Directories
Internally it keeps all received Directories (and their sizes) in a HashMap,
keyed by digest.
Once all Directories have been inserted, a drain() function can be
called to get a (deduplicated and) validated list of directories, in
from-leaves-to-root order (to be stored somewhere).
While assembling that list, a check for graph connectivity is performed
too, to ensure there's no separate components being sent (and only one
root).
It adds a test suite for these cases, which is much nicer to test than
where we previously had these checks (only in the gRPC server wrapper).
Followup CLs will move the existing putters to use this.
Change-Id: Ie88c832924c170a24626e9e3e91d868497b5d7a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11220
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: I0686c802b902ed69334221367a2f00be758daec1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11256
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
|
|
Change-Id: I2959c201809ef0d77934992f809756a16108b96a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11255
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: aspen <root@gws.fyi>
|
|
First stab at factoring out handlers into a generalized handler
function.
This is still kind of confusing, but can be simplified later.
Change-Id: I42da047de83f6d489337d57059f85f793313443a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11245
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I5f39d0e613b651458b168cfd9df0693d7f01d704
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11246
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
We need to ensure the Directories are successfully uploaded before doing
any testing with them.
Change-Id: Iafa8deb86b3d5eb302ebfba3ced34385f67a7229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11244
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
Move the http calls into their own module, so we can trace the request
and provide a simple copy-to-replay command.
We have to work around a bug in the otel library, which would limit
our attribute value length to 128 bytes because it uses the wrong
option value.
~~~
`ifExists` is finally made more useful for dealing with optional
attributes in e.g. lists.
Change-Id: Iafab523e9ec4b00136db43f31fdc12aeefb7f77c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11241
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
Change-Id: Id774963178ba358447699d0297a6a1fbef5ac8fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11240
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
|
|
With this, you can `mdbook build` inside the docs/ directory.
Change-Id: I97df12ed839e3c7a8065563c039436e1a830281e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11243
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
We tried to be more strict than Nix, actually detecting if multiple
hashes were specified, or other garbage at the end.
However, Nix seems to just chop off at the end, so happily accepts
anything afterwards.
Example: https://github.com/NixOS/nixpkgs/pull/298041
Example: https://github.com/NixOS/nixpkgs/pull/298052
Change-Id: I2c1a49f51c8f8589a84df2fbf148e67e7380b550
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11234
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ia2bd60b8449592ef1f79ac4877554958eb0b0407
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11239
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
|
|
Move the functionality into two coarse modules.
There’s still the question about whether functions that change the
database tables should be in their own storage module, but let’s see
if it gets too confusing.
Change-Id: Ied1d47b353dd4597ffea35f111f440aad22e981d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11238
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
|
|
This documents the input and output format, and also removes some
references to Terraform and evaluating NixOS system configurations.
It can be used to evaluate anything.
Change-Id: I8492cc3e386f89b299469c78e586644ee82a708f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11213
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I16c25b43a92ba650f9862ab46d11f29e940cb7ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11226
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I87a40f79ee7e91bc4fe3dc0ee7818a533c729373
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11225
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This allows these messages to be put in HashSets.
Change-Id: Ia58094cafe53eb624578821d3d8d969c5d21a1d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11219
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Log the entire span with "trace" level, not just its `ret` level.
The level of the error value event defaults to ERROR, so we don't loose
these.
B3Digest implements Debug and Display the same way, so we can omit the
`(Display)` part in `ret(Display)` for them.
Change-Id: Id00d123a5798e5bdc9820dd97ae2b4d4eb5455f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11218
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
