about summary refs log tree commit diff
AgeCommit message (Collapse)AuthorFilesLines
2024-04-15 r/7916 refactor(tvix/castore/fs): use consistent span field name for handleFlorian Klink1-5/+4
Use rq.handle in `release` too, and remove interpolating it into the log message itself. Also update the comment, we don't get ownership, just simply drop, and change the level to warn!, as suggested in cl/11425. Change-Id: If4e6cff6d8b580671b1548ae3862851db4af6694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11427 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-15 r/7915 feat(tvix/castore/fs): implement opendir/releasedirFlorian Klink2-68/+144
Similar to how we already handle opening files, implement opendir/ releasedir, and keep a map of dir_handles. They point to the rx side of a channel. This greatly improves performance listing the root of the filesystem when used inside tvix-store, as we don't need to re-request the listing (and skip to the desired position) all the time. Change-Id: I0d3ec4cb70a8792c5a1343439cf47d78d9cbb1d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11425 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-15 r/7914 refactor(tvix/castore/fs): use std::sync::MutexFlorian Klink1-21/+24
This allows us acquiring the lock in sync code still. Also, simplify some of the error handling a bit. Change-Id: I29e83b715f92808e95ecb0ae9de787339d1a371d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11424 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-15 r/7913 refactor(tvix/castore/fs): simplify some separate spawn and blocksFlorian Klink1-13/+6
We can just pass an async move closure to `self.tokio_handle.block_on` and make this a bit shorter. Change-Id: Iba674f34f22ba7a7de7c5bae59d64584884cb17c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11423 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-15 r/7912 feat(tvix/castore/fs): support extended attributesFlorian Klink7-5/+227
This exposes `user.tvix.castore.{blob,directory}.digest` xattr keys for files and directories: ``` ❯ getfattr -d /tmp/tvix/06jrrv6wwp0nc1m7fr5bgdw012rfzfx2-nano-7.2-info getfattr: Removing leading '/' from absolute path names user.tvix.castore.directory.digest="b3:SuYDcUM9RpWcnA40tYB1BtYpR0xw72v3ymhKDQbBfe4=" ❯ getfattr -d /tmp/tvix/156a89x10c3kaby9rgf3fi4k0p6r9wl1-etc-shells getfattr: Removing leading '/' from absolute path names user.tvix.castore.blob.digest="b3:pZkwZoHN+/VQ8wkaX0wYVXZ0tV/HhtKlSqiaWDK7uRs=" ``` It's currently mostly used for debugging, though it might be useful for tvix-castore-aware syncing programs using the filesystem too. Change-Id: I26ac3cb9fe51ffbf7f880519f26741549cb5ab6a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11422 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 r/7911 feat(tvix/nix-compat/path_info): derive Hash for ExportedPathInfoIlan Joselevich1-1/+1
Change-Id: Icd2431f27043f120eee1d156cfcdc9bc85519ead Reviewed-on: https://cl.tvl.fyi/c/depot/+/11412 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-13 r/7910 feat(tvix/castore/import): remove copying in find_ancestorFlorian Klink1-8/+2
We don't need to copy if we explicitly say that the returned Option<Path> may hold onto bytes from the passed in &DirEntry. Change-Id: Ib46b6fd2f8f19a45f8bef79c4c1d2fa6b490cad7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11410 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-13 r/7909 fix(tvix/eval): fix the `unsafeGetAttrPos` stub return valueRyan Lahfa1-1/+1
It was returning `{ line, col, file }` instead of `{ line, column, file }`. Note this is not a regression in Nix, Nix always had `column` since 2013. Change-Id: I4871a201b32324dff0432526d0449806a535828f Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11409 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-13 r/7908 fix(tvix): apply cbtemulator patch in tvix viewFlorian Klink2-2/+12
This tells josh to include the patch from third_party/overlays/patches/cbtemulator-uds.patch at nixpkgs/cbtemulator-uds.patch. We then use the `additionalOverlays` feature introduced earlier, and pass it an overlay adding the patch to `cbtemulator`. This only gets applied if tvix/shell.nix is not called with a pkgs argument, which is the case in the tvix view. Change-Id: I7bc5543460ddd30325ab0998e891c15fcaf8bcfe Reviewed-on: https://cl.tvl.fyi/c/depot/+/11408 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-04-13 r/7907 feat(3p/nixpkgs): add additionalOverlays optional argFlorian Klink2-1/+5
This allows applying additional overlays to nixpkgs before instantiating it. Change-Id: Iee486086e13bb73e6bd20a817b1106c3cd99c935 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11407 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-13 r/7906 feat(tvix/*store/bigtable): limit retries connecting to cbtemulatorFlorian Klink2-2/+6
This kept retrying indefinitely if the socket didn't appear. Change-Id: I4d4ef61df73cef6abda698501432f370abc8a82c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11406 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-13 r/7905 feat(tvix/nix-compat/wire/bytes): add #[inline] for ensure_* funcsFlorian Klink2-0/+2
While the compiler might realize it's a good idea to inline these, especially considering this is in the same crate [1], it doesn't hurt to add that annotation too. Suggested in cl/11385. [1]: https://nnethercote.github.io/perf-book/inlining.html Change-Id: If679fce0e1aab9ec681ba60f689d386d6fd92d61 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11403 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2024-04-13 r/7904 chore(3p/nixpkgs): drop most of cbtemulator expressionFlorian Klink1-31/+4
We bumped nixpkgs past that bump, so we only need to carry our downstream UDS support patch. Change-Id: I56504d71a8327cba2e43aa36b7b657f93cf35e51 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11405 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-13 r/7903 feat(tvix/buildkite): apply a 10 minutes timeoutFlorian Klink1-0/+1
We should cap the time of a `cargo build && cargo test` to a reasonable amount of time. Change-Id: If618198459ac267f9f6d7eb60466534a95a95c2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11404 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 r/7902 refactor(tvix/castore/import): rename ingest_entries function argFlorian Klink1-2/+2
This is a stream of DirEntry, so let's call it direntry_stream. Change-Id: I5b3cb4efba899d746393f75f6ece7eaa79424717 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11401 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-13 r/7901 refactor(tvix/boot): use tvix-store copyFlorian Klink1-33/+6
Instead of using an additional Nix build producing a directory structure containing xz-compressed NAR files and NARInfos, and uploading these via `curl` to NAR-bridge, which then decomposes them and uploads them to tvix-store, have tvix-store interpret the exported reference graph directly, and deal with the uploading on its own. Change-Id: I44ee0b132944c53d11abb688d2d017f0cc9c3d97 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11398 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-13 r/7900 feat(tvix/store/bin): add "copy" commandFlorian Klink4-0/+99
This allows copying individual store paths from the local /nix/store into tvix-store. As we don't support getting this information from Nix yet, we currently expect metadata to be provided externally: Nix' `exportReferencesGraph` feature contains pretty much all data we need for this. Expect a list of this information at a well-known key (`closure`, similar to nixpkgs' `pkgs/build-support/binary-cache/ default.nix`). We currently simply upload all store paths sequentially, without any parallelism or awareness in how the reference graph looks like. As long as the connected stores don't enforce this, this is fine, at least for now. Change-Id: Ib83c998465adddfdb110db994843c44e26b3d3d8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11397 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-13 r/7899 feat(tvix/eval): contextful JSON operationsRyan Lahfa11-33/+102
`toJSON` transform a Nix structure into a JSON string. For each context in that Nix structure, the JSON string must possess it. Thus, it is necessary to take the union of all contexts and attach it to the final structure. Unfortunately, the return type of `into_json` is a serde's JSON object, not a string. Therefore, it is not possible to reuse `NixString` machinery. Context tests are reinforced as Nix does not test those behaviors. Fixes b/393. Change-Id: I5afdbc4e18dd70469192c1aa657d1049ba330149 Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz> Reviewed-on: https://cl.tvl.fyi/c/depot/+/11266 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-13 r/7898 refactor(tvix/nix-compat): move nar writer to tokioFlorian Klink6-73/+57
There's little reason to keep the nar writer using Async{Read,Write} traits from futures, while everything else async in tvix (and nix-compat) uses tokio. Change-Id: I8cd1efcd0dd5bb76471de997603c7b701a5095de Reviewed-on: https://cl.tvl.fyi/c/depot/+/11391 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 r/7897 refactor(tvix/nix-compat): move worker_protocol into nix_daemon modFlorian Klink5-34/+34
This doesn't have much to do with the plain "wire" format, it's merely one user of it. Also, use the more "public" `wire::` API to read/write bytes, strings, bools and u64s. Change-Id: I98dddcc3004dfde7a0c009958fe84a840f77b188 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11390 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 r/7896 refactor(tvix/nix-compat): reorganize wire and bytesFlorian Klink12-96/+69
Move everything bytes-related into its own module, and re-export both bytes and primitive in a flat space from wire/mod.rs. Expose this if a `wire` feature flag is set. We only have `async` stuff in here. Change-Id: Ia4ce4791f13a5759901cc9d6ce6bd6bbcca587c7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11389 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 r/7895 docs(tvix/eval): remove link to cargo manifest referenceFlorian Klink2-4/+0
We don't have it in all Cargo.toml files either. Change-Id: Ifcb333a39603a7e402ab0068e0f0f1b5af769e13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11388 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de>
2024-04-13 r/7894 feat(tvix/nix-compat): add BytesReaderFlorian Klink6-4/+481
This adds AsyncRead counterpart for read_bytes. Change-Id: I751da9944984c7a523abee305f8f8a050e705f04 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11385 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de>
2024-04-13 r/7893 feat(tvix/nix-compat): introduce ExportedPathInfo structFlorian Klink2-0/+122
This is another representation about information Nix has on a Store Path. It's encountered in the exportReferencesGraph feature. Change-Id: Ia0c08c369f4af9e50ebe3a5053bd98fadaa0003d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11396 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 r/7892 docs(tvix/nix-compat): rename as_narinfo() to to_narinfo()Florian Klink2-5/+13
This actually does a bit of allocation. There's two Vecs, one for references and one for signatures. We can get rid of the Vec at nar_hash. Change-Id: Ie025309b6678f83f5b961d49ff75dcfc7da145a1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11395 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-13 r/7891 feat(tvix/nix-compat): impl Deserialize, Serialize for SignatureFlorian Klink1-1/+47
Change-Id: I30294079129b0e5b4faa3272e09df982d3ef2178 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11394 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-13 r/7890 feat(tvix/nix-compat): implement Serialize, Deserialize for NixHashFlorian Klink3-9/+50
We use the (slightly more tolerant) from_str to deserialize, and serialize out as SRI. Change-Id: If76b0ed2d4e243904f02df34f6c90b976c0bab8c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11393 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 r/7889 feat(nix-compat/store_path): impl [Partial]Ord for StorePathRefFlorian Klink1-16/+15
Move the code implementing it from StorePath to StorePathRef, and have the StorePath impls use that too. Drop the debug_assert in every comparison - we have tests for this to ensure it keeps working, and built up some confidence by piping a lot of other store paths through it in the meantime. Change-Id: I288bad3dfa597f68d63c4bcda7791f722b7a8ced Reviewed-on: https://cl.tvl.fyi/c/depot/+/11392 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-13 r/7888 chore(3p/overlays/patches): vendor cbtemulator UDS patchFlorian Klink2-10/+141
I updated https://github.com/googleapis/google-cloud-go/pull/9665 in the meantime, and GH decided to GC the patches. Vendor the patch in for now (manually stripping the `bigtable/` prefix in the path). Hopefully the PR itself gets merged soon. Change-Id: I5b7ba78ccaf5c792c1445818b23b52d6f17155a2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11402 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-12 r/7887 feat(tvix/store/pathinfo/grpc): instrument functionsFlorian Klink1-0/+6
Change-Id: Idb7d7144be1917fbaf83e9fd76c5b2ebb3df98d2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11400 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-12 r/7886 feat(tvix/castore/directory/grpc): instrument functionsFlorian Klink1-1/+3
Change-Id: I9cc0a6a32184773597556ab5f9250257aa18ca4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11399 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-12 r/7885 chore(tvix): bump rstest to 0.19.0Florian Klink6-12/+12
Change-Id: Ib2f5e84fdb8be1210b3507da67d4fe84f061651e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11387 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-09 r/7884 feat(tvix/store): add bigtable pathinfoservice backendFlorian Klink8-7/+499
Put behind the "cloud" backend, like in the `tvix-castore` crate. Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-04-09 r/7883 fix(tvix/nix-compat): fix writing padding partiallyFlorian Klink1-1/+28
We forgot to only write the part of the padding that has not been written already. Unfortunately it seems the tokio mocking thing does buffer small writes, so the tests succeeds even with the bug :-/ Change-Id: I5a6cf04212d559728639427c57207094d507ec75 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11384 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de>
2024-04-09 r/7882 fix(tvix): Avoid buffering file into memory in builtins.hashFileConnor Brewster9-74/+80
Right now `builtins.hashFile` always reads the entire file into memory before hashing, which is not ideal for large files. This replaces `read_to_string` with `open_file` which allows calculating the hash of the file without buffering it entirely into memory. Other callers can continue to buffer into memory if they choose, but they still use the `open_file` VM request and then call `read_to_string` or `read_to_end` on the `std::io::Reader`. Fixes b/380 Change-Id: Ifa1c8324bcee8f751604b0b449feab875c632fda Reviewed-on: https://cl.tvl.fyi/c/depot/+/11236 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-09 r/7881 feat(tvix/castore/directory): add bigtable backendFlorian Klink12-22/+2767
This adds a Directory service using https://cloud.google.com/bigtable/docs/ as a K/V store. Directory (closures) are put in individual keys. We don't do any bucketed upload of directory closures (yet), as castore/ fs does query individually, does not request recursively (and buffers). This will be addressed by store composition at some point. Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de>
2024-04-08 r/7880 fix(tvix/glue/derivation): fix __structuredAttrs "outputs" handlingFlorian Klink2-3/+12
In case structured attrs is enabled, rather than adding a space-separated list to the "outputs" environment variable, a "proper" list is added to the JSON itself, at the "outputs" key. Fixes b/395. Reported-By: Alyssa Ross <hi@alyssa.is> Change-Id: I2834ede9cfcf49d5e81e1056bf8f9bb9b65ddad8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11386 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-08 r/7879 refactor(tvix/nix-compat/wire): rename padding_len to total_padding_lenFlorian Klink1-3/+3
Make it a bit more clear that this is the total padding length, not the padding length we still need to write. Change-Id: I9ff4aa16f256fda367b4b9295abf82ed01b1f989 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11383 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 r/7878 refactor(tvix/nix-compat/wire): mv Bytes{WriterState,PacketPosition}Florian Klink1-24/+25
This is perfectly fine to track the position inside a reader too, so rename it to reflect that. Also make the docstring a bit less write-specific. Change-Id: I831b0a8fe44a2477d4af96fefc692b9aabc378f1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11382 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2024-04-08 r/7877 fix(tvix/nix-compat/wire): fix BytesWriter test payloadsFlorian Klink1-4/+4
We actually did construct other payload sizes than the test function was intended to, because I'm too stupid to count. Fix that, and also update a leftover _debug in a test name. Change-Id: I1bff8551f2078afe82f789646b851f2fcff3e82c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11381 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-08 r/7876 refactor(tvix/nix-compat): move handshake function to nix-compatPicnoir5-164/+121
Moving out the handshake route out of tvix-daemon to nix-compat. We're bumping the protocol to version 37. It does not change anything for us. Version 37 introduced a change in the build result serialization. We do not implement that for now. Tested tvix-daemon against Nix via the integration test to make sure we did not break anything in the process. Taking advantage of this refactor to remove the unused anyhow dependency in tvix-daemon. Change-Id: I8a2aad0b6dbb0aaa997446a612a365fd7d70336c Reviewed-on: https://cl.tvl.fyi/c/depot/+/11379 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-08 r/7875 refactor(tvix/nix-compat/wire): express magics as u64Florian Klink2-21/+15
This allows using read_u64, write_u64, which is a bit easier to juggle with. Also, update names to align with the nix codebase, which makes it easier to spot both the constant name as well as the value. Leave the ASCII interpretation as a comment afterwards. Change-Id: I0b9ab187acd22807e2785b0722aa4300dab37c51 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11378 Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Autosubmit: flokli <flokli@flokli.de>
2024-04-08 r/7874 refactor(tvix/nix-compat/wire): move magic bytes to worker_protocolFlorian Klink3-16/+18
`primitive.rs` implements reading and writing primitive (fixed-length) types in the wire format, used in the the nix daemon protocol and NAR format. Move worker-protocol specific magic bytes to worker_protocol.rs (and possibly further split there once needed) Change-Id: If681c01e9460294619f1d000229b81f0ac745810 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11377 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 r/7873 feat(tvix/nix-compat/wire): have write_bytes accept AsRef<[u8]> payloadsFlorian Klink2-5/+21
This includes String, &str etc. An example testcase with &str is provided. Change-Id: I900186d6ceb52f52bd41ef4596524c1f5b52470b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11376 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 r/7872 refactor(tvix/nix-compat/wire): simplify write_bytes a bitFlorian Klink2-10/+14
Use the same EMPTY_BYTES trick from BytesWriter to write out the padding, rather than allocating a Vec. Change-Id: Ifb4ba1b45b7388adbc135fc8e46fd3d3cedd30aa Reviewed-on: https://cl.tvl.fyi/c/depot/+/11375 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-08 r/7871 docs(tvix/nix-compat/wire): update docstringsFlorian Klink1-19/+21
Only describe the format once (in `read_bytes`, and simplify the other docstrings a bit. Change-Id: Iff898f3c4173d506a357bc14bdffbf69c4c6e0e0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11374 Tested-by: BuildkiteCI Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Autosubmit: flokli <flokli@flokli.de>
2024-04-07 r/7870 refactor(tvix/nix-compat): drop read_bytes_uncheckedFlorian Klink1-15/+19
This isn't used anywhere so far, and if someone really wants to, they can simply make the range open-ended. Change-Id: Iae9bcaa1f7ea032dd3ee76c8c142a38b6b72894d Reviewed-on: https://cl.tvl.fyi/c/depot/+/11373 Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr> Tested-by: BuildkiteCI
2024-04-07 r/7869 fix(nix-compat/store_path): fix Deserialize with borrowFlorian Klink1-3/+37
We were wrongly using `'de` as a lifetime for both `Deserializer` and `StorePathRef`. This prevented Deserializing into a struct containing `StorePathRef`. See https://serde.rs/lifetimes.html#the-deserializede-lifetime, the last part of the paragraph: The 'de lifetime should not appear in the type to which the Deserialize impl applies. - // Do not do this. Sooner or later you will be sad. - impl<'de> Deserialize<'de> for Q<'de> { + // Do this instead. + impl<'de: 'a, 'a> Deserialize<'de> for Q<'a> { This fixes it, and adds a test, deserializing into a `Container` struct. It also fixes the existing test cases, deserialize_ref was actually deserialize_owned, and deserialize_owned didn't exist yet - but they alone are not enough to provoke the lifetime issues. Change-Id: Iaed2832998cae5f192eafe7fd5243e82ff6e051e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11372 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 r/7868 feat(users/picnoir/tvix-daemon): implement set_options operationPicnoir1-9/+52
The protocol is more stateful than I initially thought. We need to keep track to a bunch of things, including but not limited to: the client settings, the client version. I moved things around a bit to keep this state along with the client socket. Change-Id: Ibd34fbe7821c20a460934ea1af0719f5de46e491 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11359 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-07 r/7867 feat(tvix/nix-compat): read client setting from wirePicnoir2-3/+225
Add the primitives necessary to read the client settings from the Nix daemon wire protocol. Introducing the read_string primitive. This trivial primitive parses a read_bytes call, check the bytes are valid utf-8 bytes and wraps the result in a String. Change-Id: Ie1253523a6bd4e31e7924e9898a0898109da2fa0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11358 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI