about summary refs log tree commit diff
path: root/tvix/nix-compat/src/nar
AgeCommit message (Collapse)AuthorFilesLines
2024-11-12 r/8907 feat(nix-daemon): Implement client handler.Vova Kryachko1-0/+2
This change includes only the basic nix handshake protocol handling and sets up a client session. The only supported operation at this point is SetOptions. Additional operations will be implemented in subsequent cls. Change-Id: I3eccd9e0ceb270c3865929543c702f1491768852 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12743 Autosubmit: Vladimir Kryachko <v.kryachko@gmail.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Reviewed-by: Brian Olsen <me@griff.name>
2024-09-23 r/8710 feat(tvix/nix-compat/nar/writer/sync): add file_manual_writeYureka1-0/+68
This is useful for building other NAR writers which use custom (async or optimized) I/O to write the blob parts of the NAR. Change-Id: I447c09914fb0c99044e2fa910d4213660dc51c64 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12437 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-09-07 r/8663 feat(tvix/nix-compat/nar/writer/sync): trait objects -> genericsYureka1-13/+10
This will later be used to allow the user to access the original writer. Change-Id: I1734600c5d580d18b16727f892cd2f6335cc9459 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12442 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-09-05 r/8653 chore(3p/sources): bump channels & overlays (2024-09-01)Vincent Ambo1-0/+2
Included changes: * users/aspen: explicitly use open-source nvidia driver This now has to be specified explicitly, otherwise evaluation fails with an error. * users/aspen: nixfmt -> nixfmt-classic * users/aspen: fixes for renamed packages & options * users/tazjin: fixes for renamed packages & options * 3p/overlays: remove cbtemulator patch (merged upstream) * tvix/shell: remove unnecessary patches (merged upstream) * 3p/rust-crates: mark libgit2_sys as broken * users/Profpatsch: mark git-db as broken * 3p/overlays: pick `mypaint` from stable channel * tvix: fix comments that clippy doesn't like anymore * tvix/glue: disable a misfiring clippy lint (applying its suggestion breaks code below) Change-Id: I6d3fc027694bbe7425a2d25dc53d65467a44f3b0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12403 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi> Reviewed-by: Profpatsch <mail@profpatsch.de> Autosubmit: tazjin <tazjin@tvl.su>
2024-08-19 r/8541 feat(tvix/nix-compat): entry locator in listing structuresRyan Lahfa2-1/+113
This implements a simple DFS locator in listing structures. It is interoperable with the Rust standard library paths, we also build our own errors to restrict path values to reasonable secure defaults, e.g. relative paths with no `..` component. Tests are added for this new feature for a positive and a negative check. In addition, a path validation test was added. The Windows-style prefix is gated on the Windows platform as UNIX does not parse `C:\\` as a `Component::Prefix(_)` but as a `Component::Normal(_)`. Change-Id: Iae2a80bebd8138e41af94aa7d09f2842c3c5a786 Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz> Reviewed-on: https://cl.tvl.fyi/c/depot/+/12255 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-08-19 r/8539 feat(tvix/nix-compat): add a basic listing deserializerRyan Lahfa4-0/+77
.ls files are useful to seek in a NAR without parsing it entirely. The responsibility of validating the files is on the caller. Change-Id: I5d1da28b5479c38f20ca5babe60e362a2217c9ea Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz> Reviewed-on: https://cl.tvl.fyi/c/depot/+/12196 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-09 r/8468 test(tvix/nix-compat/nar/wire): cover TOK_PAD_PARedef1-0/+2
Change-Id: Ib4da612a4144d062f55a22b437c725b8566bee70 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12172 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-08-09 r/8462 docs(tvix/nix-compat/nar/wire): fix TOK_EXE grammaredef1-1/+1
Change-Id: I12b640fe1cbe7a4c69efe4660040486e510deadc Reviewed-on: https://cl.tvl.fyi/c/depot/+/12171 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-06-03 r/8210 fix(tvix/nix-compat): require async and async,wire for some testsIlan Joselevich1-1/+1
Makes the following build: `cargo test --no-default-features --features async` `cargo test --no-default-features --features wire` Change-Id: I47ba0c944f08895f67ed3b861706ef2e4ba384b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11739 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-05-26 r/8174 fix(tvix/nix-compat): async nar reader requires wire featureFlorian Klink1-1/+1
It uses it internally. Change-Id: I8cb8fc9f567260d57f3a203407333d83beddf537 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11719 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-08 r/8095 refactor(nix-compat/nar/reader): reuse prev_name allocationedef5-47/+71
We reuse the prev_name allocation for Entry, instead of allocating and returning a separate Vec. We encode the `prev_name: None` case as an empty vector, since we don't allow empty names anyway, and the sorting is equivalent. Change-Id: I975b37ff873805f5ff099bc82128706891052247 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607 Reviewed-by: Brian Olsen <me@griff.name> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-08 r/8086 feat(nix-compat/wire/bytes/reader): support buffered readingedef1-0/+10
If our underlying reader supports AsyncBufRead, then we can too. Change-Id: If4b948c983400ca591c1c475bbcf7dc00d562040 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11545 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-08 r/8085 fix(nix-compat/wire/bytes/reader): always read trailer tag dataedef1-1/+1
Even if we have an aligned (or zero) size (and thus have no padding), we may still have a non-padding trailer. Since an exact read of only the user data bytes must always read the trailer bytes as well, this has to happen for aligned sizes too. For zero-sized trailers, we solve this by reading immediately, since no read is required of the user at all. user_len thus becomes a NonZeroU64, and the definition of "body data" is adjusted to always exclude the final block. Change-Id: I2c6bb51270fb22ad72a65076a3468673ab56ef68 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11597 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-05-04 r/8075 feat(nix-compat/nar/reader): async supportedef5-0/+565
This is a first cut at the async NAR reader, with some rough edges. Poisoning is left unimplemented for now, pending future work. Change-Id: Ifaafe0581a5e0e165a13357b909fb441f7bd8bab Reviewed-on: https://cl.tvl.fyi/c/depot/+/11524 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-29 r/8028 refactor(nix-compat/nar/reader): always enable poisoning *API*edef1-64/+66
The poisoning API is now always available, whether debug_assertions is enabled or not. When debug assertions are not enabled, it is equivalent to a unit struct, and is always considered ready and unpoisoned. Change-Id: I950237f4816d480330d9acab32661ed4f1663049 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11525 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-04-26 r/8017 chore(nix-compat/nar/wire): space Tag::Buf docs properlyedef1-0/+1
Change-Id: If15b14b5328b9e7f85ea3fa7644d327dfb50e1a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11523 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-25 r/8007 test(tvix/nix-compat): add debug assertions for nar readertcmal2-84/+426
Adds debug assertions to ensure that the reader's variants are upheld. If any of the following happens, then the currently in use reader must be abandoned: * A directory or file reader encounters an error * A directory or file reader is dropped before being fully read from Additionally, a directory reader must not be read from again after it has returned None. These checks are only used when debug_assertions are on, so vanish in release mode. Resolves two TODO items added by edef Change-Id: I27bd9643a632798db5351957506c166b9bd5ca4e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11508 Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: edef <edef@edef.eu> Autosubmit: Aria Shrimpton <me@aria.rip> Tested-by: BuildkiteCI
2024-04-13 r/7898 refactor(tvix/nix-compat): move nar writer to tokioFlorian Klink2-56/+51
There's little reason to keep the nar writer using Async{Read,Write} traits from futures, while everything else async in tvix (and nix-compat) uses tokio. Change-Id: I8cd1efcd0dd5bb76471de997603c7b701a5095de Reviewed-on: https://cl.tvl.fyi/c/depot/+/11391 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Brian Olsen <me@griff.name>
2024-01-19 r/7422 chore(3p/sources): Bump channels & overlayssterni1-30/+30
- Adjust to ecl 23.9.9 release - Regenerate go protos after protoc-gen-go update - Drop dhall fork which hasn't kept up with 1.42.* - Address new clippy warnings: - Variant naming of Error::ValidationError - Simplify .try_into().unwrap() - Drop unnecessary identity function - Test module must be last in file - Drop unused `pub use` - Update agenix to 0.15.0. Current master has a installCheckPhase that doesn't work with C++ Nix 2.3.*: https://github.com/ryantm/agenix/commit/a23aa271bec82d3e962bafb994595c1c4a62b133#commitcomment-137185861 Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615 Reviewed-by: aspen <root@gws.fyi> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2023-12-21 r/7251 docs(nix-compat/nar/reader): fix referenceFlorian Klink1-1/+1
Change-Id: I7ccdfe3c042f49aa962d1c7ba6337bd656ef05c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10405 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: edef <edef@edef.eu>
2023-11-19 r/7035 feat(nix-compat/nar/reader): provide passthrough buffered I/Oedef1-13/+75
Allow taking advantage of the buffer of the underlying reader to avoid unnecessary copies of file data. We can't easily implement the methods of BufRead directly, since we have some extra I/O to perform in the final consume() invocation. That could be resolved at the cost of additional bookkeeping, but this will suffice for now. Change-Id: I8100cf0abd79e7469670b8596bd989be5db44a91 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10089 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-11-19 r/7034 fix(nix-compat/nar/reader): require BufReadedef1-2/+2
We rely on being able to make small reads cheaply, so this was already an implicit practical requirement. Requiring it explicitly removes a performance footgun, and makes further optimisations possible. Change-Id: I7f65880a41b1d6b5e6bf2e52dfe47d4c49b34bcd Reviewed-on: https://cl.tvl.fyi/c/depot/+/10088 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 r/6898 fix(nix-compat/nar/writer): `use ByteSlice` closer to call siteedef2-2/+2
Otherwise, it'll generate unused import warnings in release mode. Change-Id: I0399d2617a8a8b973a2b8c18f4a0a613d70c48f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9864 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-18 r/6857 test(tvix/nix-compat/nar/writer/sync): add testsFlorian Klink2-0/+136
Change-Id: I04c556e7b57f20dd33f71b0379e6104e1702a8ee Reviewed-on: https://cl.tvl.fyi/c/depot/+/9785 Reviewed-by: Connor Brewster <cbrewster@hey.com> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-18 r/6856 feat(tvix/nix-compat/nar/reader): add testsFlorian Klink5-0/+122
Change-Id: If0abaf74a3b7d201af8972e54e3e34b20b8c0bc4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9764 Tested-by: BuildkiteCI Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-18 r/6855 fix(tvix/nix-compat/nar/wire): correct Node::Some doc commentedef1-1/+2
It is *eventually* followed by a Node, but there is some stuff in between. Change-Id: Ie7c7b462828bd3e066f4a7e774895f30b82763ef Reviewed-on: https://cl.tvl.fyi/c/depot/+/9768 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6854 refactor(nix-compat/nar/writer): use clone_intoedef2-4/+2
Equivalent to the existing code, but a little less cryptic. Change-Id: Ib9b2f9aedddc84d0e79840bba4cce01f92d9bc56 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9766 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6853 feat(tvix/nix-compat): NAR readeredef8-4/+530
Change-Id: I50d51baf62c0419eaf17f0dc262f728aaff9794d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9688 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 r/6852 docs(tvix/nix-compat/nar): document the wire formatedef1-2/+66
Change-Id: I6c8e23bad27fa6ada1b8973482b4d99190cf050d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9767 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-10 r/6768 feat(tvix/nix-compat): add nar::writer::asyncedef2-0/+239
Change-Id: Ib962f889836659fd278beec461ee329b00e64ea5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9615 Autosubmit: edef <edef@edef.eu> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-10-10 r/6767 refactor(tvix/nix-compat): move nar::writer to nar::writer::syncedef2-223/+227
This is preparation for adding an async port. Change-Id: Id638ec1f6f46e2f3935448184eed51e2233263fe Reviewed-on: https://cl.tvl.fyi/c/depot/+/9618 Tested-by: BuildkiteCI Autosubmit: edef <edef@edef.eu> Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 r/6766 fix(tvix/nix-compat): clean up the debug assertions a bitedef1-12/+11
Consistent error messages, and slightly nicer code layout. We avoid printing the input data, since we primarily want to point out the specific violated invariant. In the one place where we do want to, we use BStr's Debug implementation, since byte slices don't print nicely. Change-Id: I3a9a0c37be270ea5f16cf124922c254608fb849e Reviewed-on: https://cl.tvl.fyi/c/depot/+/9617 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu>
2023-10-09 r/6754 docs(tvix/nix-compat): mention Directory::close on Node::directoryedef1-0/+3
Change-Id: I19e6a3b63eb1b1609b96544b2529d71234bcb13b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9613 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: edef <edef@edef.eu> Tested-by: BuildkiteCI
2023-10-09 r/6753 docs(tvix/nix-compat): document the name ordering requirementedef1-0/+4
Change-Id: Ia2c5f0b357a23d4bfac1fd52cf01845c78ebfa22 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9612 Autosubmit: edef <edef@edef.eu> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-10-09 r/6752 fix(tvix/nix-compat): don't transiently allocate namesedef1-1/+1
The point of clearing and reusing the same Vec is to avoid transiently allocating for every directory entry. This was lost in cl/8974 when we switched from String to Vec<u8>. Change-Id: I65647e5c4e54e88f1fe45e9a752cb5154d98fb33 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9607 Autosubmit: edef <edef@edef.eu> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-09-21 r/6623 refactor(tvix/store): Asyncify PathInfoService and DirectoryServiceConnor Brewster1-1/+1
We've decided to asyncify all of the services to reduce some of the pains going back and for between sync<->async. The end goal will be for all the tvix-store internals to be async and then expose a sync interface for things like tvix eval io. Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369 Autosubmit: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-07-21 r/6436 refactor(tvix/store): use bytes for node names and symlink targetsFlorian Klink1-13/+14
Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-05-10 r/6129 feat(tvix/nix-compat/nar/writer): check for more data in readerFlorian Klink1-1/+14
We already returned UnexpectedEof in case the reader stopped returning bytes too early, but similarly we should also fail if there's still bytes left to be read in the reader passed. We normally use the NAR writer to produce new NAR files, so the readers point to the blobs we actually want to render, and having some data left in there should be an error. If for some reason the reader points to more data than just the blob, the `.take` method can be used to limit it to the (known) size. Change-Id: I9e8fa0a6dd9c794492abb6dc9e55995e619cb3bb Reviewed-on: https://cl.tvl.fyi/c/depot/+/8553 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 r/5793 refactor(tvix/nix-compat): absorb nar writerFlorian Klink3-0/+253
Expose it at `nix_compat::nar::writer`. Change-Id: I98a8acfa6277074f24608d64fb6c0082b41d10f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7969 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-27T13·53+0000