| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
We separate ingesting the trailer block into a Future of its own,
parametrised on the specifics of the trailer pattern.
This is intended to be used for future work on an async NAR reader,
which needs to read a terminating parenthesis as well as the regular
padding.
Thanks to @griff for suggesting separating the ingestion into its own
Future.
Change-Id: I36c2503baa67937046a63e9bf0cfc38201394025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11522
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: Ic683eab435576acc8f7e03f5684767ffa468851a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11521
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Pin<&mut T> is DerefMut when T: Unpin, so we don't actually need to
explicitly call get_mut.
Change-Id: Iaa312ec49c87100010e09c94f319e57e31da0cd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11520
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
We don't use it in the reader anymore.
Change-Id: I98fe204a747711464e9e7ca17df06fa9854eb344
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11519
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
We now *never* return the final bytes until we've read the padding
in full, so read_exact is safe to use.
This is implemented by TrailerReader, which splits the phases of
reading (and validating) the final 8-byte block, and providing
the contained payload bytes to the caller.
Change-Id: I0d05946a8af9c260a18d71d2b763ba7a68b3c27f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11518
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
Change-Id: I9c94239c308cfbc2e6dae871ba77fb33507433c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11517
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
We already require R: Unpin in the constructor, so there's not much use
to pin projection.
Change-Id: Ia7bf734dc3aa86ffa6d1d5de778939baa9676bb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11516
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This is semantically a RangeInclusive, since we can only have
0..=u64::MAX at most, and monomorphising on the bounds doesn't
buy us anything.
Change-Id: Ib601d7fd77d703d6c8c5ec27ac9e67bb122ce1c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11515
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Adds debug assertions to ensure that the reader's variants are upheld.
If any of the following happens, then the currently in use reader must
be abandoned:
* A directory or file reader encounters an error
* A directory or file reader is dropped before being fully read from
Additionally, a directory reader must not be read from again after it
has returned None.
These checks are only used when debug_assertions are on, so vanish in
release mode.
Resolves two TODO items added by edef
Change-Id: I27bd9643a632798db5351957506c166b9bd5ca4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11508
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Autosubmit: Aria Shrimpton <me@aria.rip>
Tested-by: BuildkiteCI
|
|
This is a ReadBuf, not a BufRead.
Change-Id: Ie80e894f4b24b77cdd60409ddfaa66dae0ffeec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11511
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
|
|
Change-Id: Ideed83d191b55e131720e598b7591e8375a26cfd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11510
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
|
|
Also, replace the useless read_file helper with std::fs::read.
Change-Id: I704ab42d1ffb9089716533125ff390b7441a8591
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11473
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I7c29e7d9e2b9cd66483ed44a9f2b3d291061f81d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11472
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: I053f2c7fc9fdf59de5d8697eab32644b860e40be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11471
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
Change-Id: If05fa3d0f41d45e3574e0c5ddff406cb33dc52f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11470
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
|
|
Change-Id: Ic466a27d61b95ca4d297abd6eb976c083e8b40af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11469
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I9ed5e728ff0706a37d0a238210a8dd23eb306033
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11468
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
|
|
Change-Id: Ibd3cd1bc0d9deaf018a66748cea56af51451703c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11467
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
|
|
This provides a nice wrapper struct to deal with versions.
Change-Id: I6acc03bc9f8d84a0583196073b52776c45d3fe92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11454
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Change-Id: Icd2431f27043f120eee1d156cfcdc9bc85519ead
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11412
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
While the compiler might realize it's a good idea to inline these,
especially considering this is in the same crate [1], it doesn't hurt to
add that annotation too.
Suggested in cl/11385.
[1]: https://nnethercote.github.io/perf-book/inlining.html
Change-Id: If679fce0e1aab9ec681ba60f689d386d6fd92d61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11403
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
|
|
There's little reason to keep the nar writer using Async{Read,Write}
traits from futures, while everything else async in tvix (and
nix-compat) uses tokio.
Change-Id: I8cd1efcd0dd5bb76471de997603c7b701a5095de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11391
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
|
|
This doesn't have much to do with the plain "wire" format, it's merely
one user of it.
Also, use the more "public" `wire::` API to read/write bytes, strings,
bools and u64s.
Change-Id: I98dddcc3004dfde7a0c009958fe84a840f77b188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11390
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
|
|
Move everything bytes-related into its own module, and re-export
both bytes and primitive in a flat space from wire/mod.rs.
Expose this if a `wire` feature flag is set. We only have `async` stuff
in here.
Change-Id: Ia4ce4791f13a5759901cc9d6ce6bd6bbcca587c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11389
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
|
|
This adds AsyncRead counterpart for read_bytes.
Change-Id: I751da9944984c7a523abee305f8f8a050e705f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11385
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
|
|
This is another representation about information Nix has on a Store
Path. It's encountered in the exportReferencesGraph feature.
Change-Id: Ia0c08c369f4af9e50ebe3a5053bd98fadaa0003d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11396
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Change-Id: I30294079129b0e5b4faa3272e09df982d3ef2178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11394
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
We use the (slightly more tolerant) from_str to deserialize, and
serialize out as SRI.
Change-Id: If76b0ed2d4e243904f02df34f6c90b976c0bab8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11393
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
|
|
Move the code implementing it from StorePath to StorePathRef, and have
the StorePath impls use that too.
Drop the debug_assert in every comparison - we have tests for this to
ensure it keeps working, and built up some confidence by piping a lot of
other store paths through it in the meantime.
Change-Id: I288bad3dfa597f68d63c4bcda7791f722b7a8ced
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11392
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
|
|
We forgot to only write the part of the padding that has not been
written already.
Unfortunately it seems the tokio mocking thing does buffer small writes,
so the tests succeeds even with the bug :-/
Change-Id: I5a6cf04212d559728639427c57207094d507ec75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11384
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
|
|
Make it a bit more clear that this is the total padding length, not the
padding length we still need to write.
Change-Id: I9ff4aa16f256fda367b4b9295abf82ed01b1f989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11383
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
This is perfectly fine to track the position inside a reader too, so
rename it to reflect that.
Also make the docstring a bit less write-specific.
Change-Id: I831b0a8fe44a2477d4af96fefc692b9aabc378f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11382
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
|
|
We actually did construct other payload sizes than the test function was
intended to, because I'm too stupid to count. Fix that, and also update
a leftover _debug in a test name.
Change-Id: I1bff8551f2078afe82f789646b851f2fcff3e82c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11381
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Moving out the handshake route out of tvix-daemon to nix-compat.
We're bumping the protocol to version 37. It does not change anything
for us. Version 37 introduced a change in the build result
serialization. We do not implement that for now.
Tested tvix-daemon against Nix via the integration test to make sure
we did not break anything in the process.
Taking advantage of this refactor to remove the unused anyhow
dependency in tvix-daemon.
Change-Id: I8a2aad0b6dbb0aaa997446a612a365fd7d70336c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11379
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This allows using read_u64, write_u64, which is a bit easier to juggle
with.
Also, update names to align with the nix codebase, which makes it easier
to spot both the constant name as well as the value.
Leave the ASCII interpretation as a comment afterwards.
Change-Id: I0b9ab187acd22807e2785b0722aa4300dab37c51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11378
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
|
|
`primitive.rs` implements reading and writing primitive (fixed-length)
types in the wire format, used in the the nix daemon protocol and NAR
format.
Move worker-protocol specific magic bytes to worker_protocol.rs (and
possibly further split there once needed)
Change-Id: If681c01e9460294619f1d000229b81f0ac745810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11377
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
This includes String, &str etc. An example testcase with &str is
provided.
Change-Id: I900186d6ceb52f52bd41ef4596524c1f5b52470b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11376
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Use the same EMPTY_BYTES trick from BytesWriter to write out the
padding, rather than allocating a Vec.
Change-Id: Ifb4ba1b45b7388adbc135fc8e46fd3d3cedd30aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11375
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Only describe the format once (in `read_bytes`, and simplify the other
docstrings a bit.
Change-Id: Iff898f3c4173d506a357bc14bdffbf69c4c6e0e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11374
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
|
|
This isn't used anywhere so far, and if someone really wants to, they
can simply make the range open-ended.
Change-Id: Iae9bcaa1f7ea032dd3ee76c8c142a38b6b72894d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11373
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
|
|
We were wrongly using `'de` as a lifetime for both `Deserializer` and
`StorePathRef`.
This prevented Deserializing into a struct containing `StorePathRef`.
See https://serde.rs/lifetimes.html#the-deserializede-lifetime, the last
part of the paragraph:
The 'de lifetime should not appear in the type to which the Deserialize
impl applies.
- // Do not do this. Sooner or later you will be sad.
- impl<'de> Deserialize<'de> for Q<'de> {
+ // Do this instead.
+ impl<'de: 'a, 'a> Deserialize<'de> for Q<'a> {
This fixes it, and adds a test, deserializing into a `Container` struct.
It also fixes the existing test cases, deserialize_ref was actually
deserialize_owned, and deserialize_owned didn't exist yet - but they
alone are not enough to provoke the lifetime issues.
Change-Id: Iaed2832998cae5f192eafe7fd5243e82ff6e051e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11372
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Add the primitives necessary to read the client settings from the Nix
daemon wire protocol.
Introducing the read_string primitive. This trivial primitive parses a
read_bytes call, check the bytes are valid utf-8 bytes and wraps the
result in a String.
Change-Id: Ie1253523a6bd4e31e7924e9898a0898109da2fa0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11358
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This deals with writing byte packets of larger sizes to an underlying
AsyncWrite.
Its constructor receives the expected size. It also deals with writing
padding if flush/shutdown is called after writing all the payload.
Change-Id: I8acbf992467f3862ffb8c7d669e8c0c8eced14c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11355
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
|
|
Let's make this usable for the entire crate.
Change-Id: I754408908a00296ee80dd52680f84b8a7cb22317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11354
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
|
|
Actually these are all u64 LE encoded on the wire.
Change-Id: I5ca22c7639607ac47117cd946e036a444271885a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11348
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Using all the primitives recently implemented to nix-compat to reach
the point where the Nix client start to send us operation requests.
Using a small integration test script (or the VM test, but let's face
it, it's too slow to be useful), we manage to reach the point where
we're able to read a store operation:
2024-03-21T18:53:27.624876Z INFO tvix_daemon: Incoming connection addr=unix
2024-03-21T18:53:27.625312Z INFO worker:perform_init_handshake: tvix_daemon: Trust sent conn=Connection(unix) conn=Connection(unix)
2024-03-21T18:53:27.625406Z INFO worker: tvix_daemon: Client hanshake succeeded conn=Connection(unix)
2024-03-21T18:53:27.625488Z INFO worker: tvix_daemon: Operation received op=SetOptions conn=Connection(unix)
We had to take some shortcuts wrt. stderr/log management. The CPP Nix
codebase is a bit confusing in that area. I'll need to spend more time
reading this to fully understand what's happening there. For now,
sending the STDERR_LAST command to the client does the trick.
Change-Id: I9b0e20a52d885e64fe29188496aac5334de61edd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11233
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
|
|
This is used by the nix client to determine whether or not the daemon
trust it. The trust conditions check are daemon-specific, hence not
part of nix-compat.
Change-Id: Icbcba2f7f1fd58f67e7da72d22a264f5a3f3619d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11231
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Write counterpart of read_bytes. Despite its name, we mostly use it to
write strings (as in ascii strings) to the wire.
We also extract the padding calculation in its own function.
Change-Id: I8d936e989961107261b3089e4275acbd2c093a7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11230
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
This allows StorePathRef as keys in a hashmap, by deriving Hash. The same is already done for StorePath.
Change-Id: I3fc54c45787948116dcb27dfb5dc806b9b505835
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11269
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
|
Change-Id: I7776635b17c44534223603d28cf59c7eebd976e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11229
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
|
