| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Right now `builtins.hashFile` always reads the entire file into memory
before hashing, which is not ideal for large files. This replaces
`read_to_string` with `open_file` which allows calculating the hash of
the file without buffering it entirely into memory. Other callers can
continue to buffer into memory if they choose, but they still use the
`open_file` VM request and then call `read_to_string` or `read_to_end`
on the `std::io::Reader`.
Fixes b/380
Change-Id: Ifa1c8324bcee8f751604b0b449feab875c632fda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11236
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This adds a Directory service using
https://cloud.google.com/bigtable/docs/ as a K/V store.
Directory (closures) are put in individual keys.
We don't do any bucketed upload of directory closures (yet), as castore/
fs does query individually, does not request recursively (and buffers).
This will be addressed by store composition at some point.
Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
In case structured attrs is enabled, rather than adding a
space-separated list to the "outputs" environment variable, a "proper"
list is added to the JSON itself, at the "outputs" key.
Fixes b/395.
Reported-By: Alyssa Ross <hi@alyssa.is>
Change-Id: I2834ede9cfcf49d5e81e1056bf8f9bb9b65ddad8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11386
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Make it a bit more clear that this is the total padding length, not the
padding length we still need to write.
Change-Id: I9ff4aa16f256fda367b4b9295abf82ed01b1f989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11383
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
This is perfectly fine to track the position inside a reader too, so
rename it to reflect that.
Also make the docstring a bit less write-specific.
Change-Id: I831b0a8fe44a2477d4af96fefc692b9aabc378f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11382
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
We actually did construct other payload sizes than the test function was
intended to, because I'm too stupid to count. Fix that, and also update
a leftover _debug in a test name.
Change-Id: I1bff8551f2078afe82f789646b851f2fcff3e82c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11381
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Moving out the handshake route out of tvix-daemon to nix-compat.
We're bumping the protocol to version 37. It does not change anything
for us. Version 37 introduced a change in the build result
serialization. We do not implement that for now.
Tested tvix-daemon against Nix via the integration test to make sure
we did not break anything in the process.
Taking advantage of this refactor to remove the unused anyhow
dependency in tvix-daemon.
Change-Id: I8a2aad0b6dbb0aaa997446a612a365fd7d70336c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11379
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This allows using read_u64, write_u64, which is a bit easier to juggle
with.
Also, update names to align with the nix codebase, which makes it easier
to spot both the constant name as well as the value.
Leave the ASCII interpretation as a comment afterwards.
Change-Id: I0b9ab187acd22807e2785b0722aa4300dab37c51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11378
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
|
|
`primitive.rs` implements reading and writing primitive (fixed-length)
types in the wire format, used in the the nix daemon protocol and NAR
format.
Move worker-protocol specific magic bytes to worker_protocol.rs (and
possibly further split there once needed)
Change-Id: If681c01e9460294619f1d000229b81f0ac745810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11377
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
This includes String, &str etc. An example testcase with &str is
provided.
Change-Id: I900186d6ceb52f52bd41ef4596524c1f5b52470b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11376
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Use the same EMPTY_BYTES trick from BytesWriter to write out the
padding, rather than allocating a Vec.
Change-Id: Ifb4ba1b45b7388adbc135fc8e46fd3d3cedd30aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11375
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Only describe the format once (in `read_bytes`, and simplify the other
docstrings a bit.
Change-Id: Iff898f3c4173d506a357bc14bdffbf69c4c6e0e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11374
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
|
|
This isn't used anywhere so far, and if someone really wants to, they
can simply make the range open-ended.
Change-Id: Iae9bcaa1f7ea032dd3ee76c8c142a38b6b72894d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11373
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
|
|
We were wrongly using `'de` as a lifetime for both `Deserializer` and
`StorePathRef`.
This prevented Deserializing into a struct containing `StorePathRef`.
See https://serde.rs/lifetimes.html#the-deserializede-lifetime, the last
part of the paragraph:
The 'de lifetime should not appear in the type to which the Deserialize
impl applies.
- // Do not do this. Sooner or later you will be sad.
- impl<'de> Deserialize<'de> for Q<'de> {
+ // Do this instead.
+ impl<'de: 'a, 'a> Deserialize<'de> for Q<'a> {
This fixes it, and adds a test, deserializing into a `Container` struct.
It also fixes the existing test cases, deserialize_ref was actually
deserialize_owned, and deserialize_owned didn't exist yet - but they
alone are not enough to provoke the lifetime issues.
Change-Id: Iaed2832998cae5f192eafe7fd5243e82ff6e051e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11372
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Add the primitives necessary to read the client settings from the Nix
daemon wire protocol.
Introducing the read_string primitive. This trivial primitive parses a
read_bytes call, check the bytes are valid utf-8 bytes and wraps the
result in a String.
Change-Id: Ie1253523a6bd4e31e7924e9898a0898109da2fa0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11358
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I5049a3682a58ce848d80f413b2964331025a90a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11370
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Change-Id: I52b14652822766421d66e95bc646ed7baecc705f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11369
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
|
|
Change-Id: I6c594d2e670a681484b858c3e04bc25b9e5a2077
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11368
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Change-Id: Ie088bf03c739bf64abf3432175362a8f92f501c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11367
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Change-Id: I99ae0e27b4db4799db8af7cd6b9cc8d7f09227de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11366
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
This (and more) should now be covered by the generic testsuite
(in crate::blobservice::tests).
Change-Id: Ib3afc4f19f7e37a561b7398d43663dc941971f5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11365
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
This allows conditionalizing test cases on feature flags.
Change-Id: Ic9ed9ef52f703c973fda2ca4aae5f425e33b67de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11364
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
|
|
This deals with writing byte packets of larger sizes to an underlying
AsyncWrite.
Its constructor receives the expected size. It also deals with writing
padding if flush/shutdown is called after writing all the payload.
Change-Id: I8acbf992467f3862ffb8c7d669e8c0c8eced14c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11355
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
|
|
Have this return a StorePathRef<'_>, rather than a StorePath, and leave
it up to the caller to possibly convert it to a owned StorePath.
This avoids some allocations, if we only want to validate.
Change-Id: I5cf8e246fe02bd4e631f46a5cb86d3f77a728a0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11361
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Let's make this usable for the entire crate.
Change-Id: I754408908a00296ee80dd52680f84b8a7cb22317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11354
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
`appendContext s ctx` will just append a user-crafted context attrs to `s`.
The most important part of this builtin is to perform all the relevant invariant validations
to avoid letting the user craft invalid contexts which can never be built, e.g. invalid store paths,
inexistent derivations, etc.
This version is incomplete and full of TODOs, but passes all the Nix's
context strings tests, so we turn them on.
Change-Id: I625dc5e7c4f5b784f078b390f04b0ee5a8d65a7c
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11263
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Fixes b/392.
Output paths were created, depending on a plain store path but no
context string was attached to track that plain dependency.
Context string propagation tests are strengthened to prevent any
regression on this.
Change-Id: Ifd6671aeba6949324b0bb9f0f766b87db728d484
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11351
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Actually these are all u64 LE encoded on the wire.
Change-Id: I5ca22c7639607ac47117cd946e036a444271885a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11348
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Using all the primitives recently implemented to nix-compat to reach
the point where the Nix client start to send us operation requests.
Using a small integration test script (or the VM test, but let's face
it, it's too slow to be useful), we manage to reach the point where
we're able to read a store operation:
2024-03-21T18:53:27.624876Z INFO tvix_daemon: Incoming connection addr=unix
2024-03-21T18:53:27.625312Z INFO worker:perform_init_handshake: tvix_daemon: Trust sent conn=Connection(unix) conn=Connection(unix)
2024-03-21T18:53:27.625406Z INFO worker: tvix_daemon: Client hanshake succeeded conn=Connection(unix)
2024-03-21T18:53:27.625488Z INFO worker: tvix_daemon: Operation received op=SetOptions conn=Connection(unix)
We had to take some shortcuts wrt. stderr/log management. The CPP Nix
codebase is a bit confusing in that area. I'll need to spend more time
reading this to fully understand what's happening there. For now,
sending the STDERR_LAST command to the client does the trick.
Change-Id: I9b0e20a52d885e64fe29188496aac5334de61edd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11233
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This is used by the nix client to determine whether or not the daemon
trust it. The trust conditions check are daemon-specific, hence not
part of nix-compat.
Change-Id: Icbcba2f7f1fd58f67e7da72d22a264f5a3f3619d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11231
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Write counterpart of read_bytes. Despite its name, we mostly use it to
write strings (as in ascii strings) to the wire.
We also extract the padding calculation in its own function.
Change-Id: I8d936e989961107261b3089e4275acbd2c093a7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11230
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Now, it supports almost everything except `recursive = false;`, i.e. `flat`-ingestion
because we have no knob exposed in the tvix store import side to do it.
This has been tested to work.
Change-Id: I2e9da10ceccdfbf45b43c532077ed45d6306aa98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10597
Tested-by: BuildkiteCI
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
|
|
Instead of enforcing NAR SHA256 all the time, we generalize the
`PathInfo` constructor to take a `CAHash` argument which can drive
whether we are having a flat, NAR or text scheme.
With this, it is now possible to implement flat schemes in our
evaluation builtins, e.g. `builtins.path`.
Change-Id: I15bfee0ef4f0f428bfbd2f30c57c012cdcf6a976
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11286
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Bump to the latest u-root release. This seems to be compatible with go
1.22, too, which previously segfaulted the compiler in cl/11289.
Change-Id: Ie8a0d3cb51a435d6a0b194b38a9c83ee260618df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11291
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
|
|
This builtin only transforms any `NixContextElement::Derivation` into the trivial `NixContextElement::Plain`.
This is a forgetful functor on derivation-deep context strings.
The test coverage of this change is done in cl/11264.
Change-Id: Icd00778c97766be6db8a6bdabaa59e9724353ec5
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11262
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Derivation that depends on `${d.drvPath}` generates a `NixContextElement::Derivation(drvPath)` context string.
In turn, this makes the dependent derivation depend on *ALL* outputs of that derivation.
Note that a dependency on `${d.drvPath}` generates an input source
dependency too.
This is a complete implementation of the context string system in Nix on
the "input population" side.
The test coverage of this change is taken care in cl/11264.
Change-Id: I97fe5f7c772a6b1cc4366bee071aa691a11fcde6
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11261
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Nixpkgs make use of it and this is a 2.3 builtin that was missed.
Change-Id: If36d748eed4a5f777344556c0802392c76c8d53c
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11260
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Make this function async, and do the block_on on the (single) callsite.
Change-Id: Ib8b0b54ab5370fe02ef95f38a45d8866868a9d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11285
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Replace the (single) callsite with some code interacting with the tokio
runtime to block on the async version.
Change-Id: I3976496ae77b2bb8734603f303655834265e3f0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11284
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Let's get rid of these sync helpers, they make this less understandable.
Change-Id: I3c7294647849db2747762722247c65e4e2947757
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11283
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: If3a45d2f8008b75524ead704b05320364cc60d92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11282
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
We need to define behaviours and add tests for these.
Change-Id: Id5825fafbf47897d8de42503ea6006eb131b1082
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11281
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This drops pretty much all of castore/utils.rs.
There were only two things left in there, both a bit messy and only used
for tests:
Some `gen_*_service()` helper functions. These can be expressed by
`from_addr("memory://")`.
The other thing was some plumbing code to test the gRPC layer, by
exposing a in-memory implementation via gRPC, and then connecting to
that channel via a gRPC client again.
Previous CLs moved the connection setup code to
{directory,blob}service::tests::utils, close to where we exercise them,
the new rstest-based tests.
The tests interacting directly on the gRPC types are removed, all
scenarios that were in there show now be covered through the rstest ones
on the trait level.
Change-Id: I450ccccf983b4c62145a25d81c36a40846664814
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11223
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
As for grpc_pathinfoservice, we test this in the rstest-based tests,
with more implementations.
tests/utils.rs is unused now.
Change-Id: I62b192170d181e4fb432bf7dfff2057cb048b52c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11279
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Remove usage of the gen_{blob,directory}_service() helper functions
from utils. We populate Memory{Blob,Directory}Services here directly, as
test_case and rstest doesn't compose well.
Change-Id: I0fb48aadb8c818f508b18ceb83c85eb91359442a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11278
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: Ib114a4d141ca829520aed34600678d613994f875
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11277
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I975729cb97d69d080fb63fbb8f81b4ac46cbfb7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11276
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Previous CLs ensured BlobService and DirectoryService is implemented on
AsRef<dyn BlobService> and AsRef<dyn DirectoryService> respectively, so
there's no need to require AsRef in here.
Change-Id: I1e8b93188ca784e1b9a592f3d35fcf6b9df75c75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11275
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
This stops using the gen_{blob,directory}_service() functions from
tvix_castore::utils in favor of the rstest-based fixtures.
Change-Id: I4c2fff2926f34af63d3bd5c29c8e939dbdf7e062
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11274
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This introduces rstest-based tests. We also add fixtures for creating
some BlobService / DirectoryService out of thin air.
To test a PathInfoService, we don't really care too much about its
internal storage - ensuring they work is up to the castore tests.
Change-Id: Ia62af076ef9c9fbfcf8b020a781454ad299d972e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11272
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
