about summary refs log tree commit diff
path: root/tvix/eval/src/io.rs
AgeCommit message (Collapse)AuthorFilesLines
2024-08-09 r/8466 feat(tvix/eval): Implement builtins.readFileTypeIlan Joselevich1-2/+2
builtins.readFileType was added to Nix back in version 2.14. The tests were also moved out of notyetpassing in addition to the readDir fixtures they depend on. I caught a bug where we previously used std::fs::metadata (via the .metadata() method on File) which follows symlinks so it would always return false for is_symlink(). Instead we now use std::fs::symlink_metadata directly which does not follow symlinks, so tests now pass. This wasn't an issue for builtins.readDir as it uses walkdir and walkdir doesn't follow symlinks either. Change-Id: I58eb97bdb5ec95df4f6882f495f8c572fe7c6793 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12130 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-08-09 r/8463 feat(tvix/eval): Implement Display for io::FileTypeIlan Joselevich1-0/+13
In newer versions of Nix there's a builtins.readFileType builtin, we should try to avoid duplicating the enum -> string conversion by implementating Display before we implement builtins.readFileType. Change-Id: I579e95949a76eb33d2e7bda0000ed84859df765e Reviewed-on: https://cl.tvl.fyi/c/depot/+/12129 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com> Tested-by: BuildkiteCI
2024-07-27 r/8416 fix(tvix/eval): don't bubble up io errors from path_existsMatthew Tromp1-1/+2
path_exists was returning an error when certain common IO errors were encountered. e.g. in the path "/dev/null/.", path_exists would throw an error because the underlying call to Path::try_exists threw an error because null isn't a directory. But if null isn't a directory, then the path is invalid, so this should really be returning false. That's what nix's behavior is and that's what makes sense. The trait function isn't being changed because some other implementers (e.g. tvix_store_io) have actual errors they can throw. Fixes: b/411 Change-Id: I9e810e7a198bffe61365697c6d3d7e71f264c20b Reviewed-on: https://cl.tvl.fyi/c/depot/+/12042 Tested-by: BuildkiteCI Autosubmit: chickadee <matthewktromp@gmail.com> Reviewed-by: aspen <root@gws.fyi>
2024-06-26 r/8304 feat(tvix/eval): add file_type methodFlorian Klink1-0/+25
This allows peeking of the type at a given path. It's necessary, as an open() might not fail until you try to read() from it, and generally, stat'ing can be faster in some cases. Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871 Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-26 r/8175 fix(tvix/eval/io): OsStringExt and std::fs::File import are impure-onlyFlorian Klink1-2/+4
These are only needed when building with the impure feature enabled. This removes some warnings when building with --no-default-features. Change-Id: I3139d9133d4846aeb1b1b5f3830c0d078d047292 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11720 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2024-04-09 r/7882 fix(tvix): Avoid buffering file into memory in builtins.hashFileConnor Brewster1-10/+6
Right now `builtins.hashFile` always reads the entire file into memory before hashing, which is not ideal for large files. This replaces `read_to_string` with `open_file` which allows calculating the hash of the file without buffering it entirely into memory. Other callers can continue to buffer into memory if they choose, but they still use the `open_file` VM request and then call `read_to_string` or `read_to_end` on the `std::io::Reader`. Fixes b/380 Change-Id: Ifa1c8324bcee8f751604b0b449feab875c632fda Reviewed-on: https://cl.tvl.fyi/c/depot/+/11236 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-03-15 r/7700 docs(tvix): fix some docstringsFlorian Klink1-1/+1
Change-Id: Ife599387d0472cd746b992bd6755a2fb6a0e0dc4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11158 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-02-21 r/7587 fix(tvix/eval): allow reading non-UTF8 filesFlorian Klink1-5/+5
With our values using bstr now, we're not restricted to only reading files that contain valid UTF-8. Update our `read_to_string` function to `read_to_end` (named like `std::io::Read::read_to_end`), and have it return a Vec<u8>. Change-Id: I87f0291dc855a132689576559c891d66c30ddf2b Reviewed-on: https://cl.tvl.fyi/c/depot/+/11003 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Pádraic Ó Mhuiris <patrick.morris.310@gmail.com> Reviewed-by: flokli <flokli@flokli.de>
2023-12-12 r/7170 refactor(tvix): use io::Result for EvalIOFlorian Klink1-12/+12
This is just a alias for Result<_, io::Error>, but shorter. Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-06 r/6963 docs(tvix/eval): document where EvalIO methods are usedVincent Ambo1-7/+34
Change-Id: I335f2ba4420973861c2a22125995b45a34d3608c Reviewed-on: https://cl.tvl.fyi/c/depot/+/9969 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-08-20 r/6504 refactor(tvix/eval/io): cargo clippy &PathFlorian Klink1-1/+1
Change-Id: Ifec2a4fc93c482e41ff5993183643b5126a7728b Reviewed-on: https://cl.tvl.fyi/c/depot/+/9100 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-07-22 r/6439 feat(tvix/store/proto): use Bytes instead of Vec<u8>Florian Klink1-4/+4
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-07-21 r/6436 refactor(tvix/store): use bytes for node names and symlink targetsFlorian Klink1-5/+8
Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-05-25 r/6202 refactor(tvix/eval): stop borrowing &mut selfFlorian Klink1-12/+12
This does undo cl/8571. Change-Id: Ib14b4e7404f906e346304b6113860ae811afc94a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8631 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-22 r/6173 refactor(tvix/eval): use &Path instead of PathBufFlorian Klink1-9/+9
This allows getting rid of some clones in eval/src/vm/generators.rs. Change-Id: I330390307d3bcfeef19c98954c753ee55b1ccee3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8604 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-22 r/6171 refactor(tvix/eval/io): use io::Error instead of tvix_eval errorsFlorian Klink1-43/+29
We didn't return anything useful other than ErrorKind::IO anyways. We can use io::ErrorKind::Unsupported for DummyIO. Fixes b/271. Change-Id: Icb231e9b38168e8b6fa473bfa405d160357b317f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8602 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-14 r/6140 feat(tvix/eval/io): allow &mut self in EvalIOFlorian Klink1-12/+12
It's okay if these calls mutate some internal state inside an implementation. Change-Id: I12bb11bde0310778c3da1275696bf7de058863a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8571 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-03 r/5833 chore(tvix/eval): only use Rc with impure featureFlorian Klink1-2/+3
We only use Rc in `impl EvalIO for StdIO`, which is only included when building with the "impure" feature. Change-Id: Id29d647c899cbfcdda11abfb9fabd5aa7e24299f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8025 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-21 r/5467 feat(tvix/eval): use `EvalIO::import_path` when coercing pathsVincent Ambo1-1/+22
This "ties the knot" of importing files into a store when referring to them through path literals, e.g. inside of strings. I'm not yet sure if this interface is sufficient for builtins.path (which we haven't implemented at all yet), but it's enough to wire up eval & store initially. In the default implementations nothing interesting happens in this function at all. Change-Id: Ie01ff4161617d1e743a68dbd1a5e54c1b40c0990 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7582 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5465 feat(tvix/eval): builtins.storeDirAdam Joseph1-0/+6
Returns the store directory through EvalIO::store_dir. Note that this is _optional_ in Tvix, as an evaluation can occur in a context where there simply is no store directory. In those contexts, `builtins.storeDir` returns `null` in Tvix. This would only happen in contexts like Tvixbolt (or completely unrelated use-cases) in practice. Co-Authored-By: Vincent Ambo <tazjin@tvl.su> Change-Id: I5a752c7e89b2f75bd7efb082dbfa5b25e3b1ff3b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7452 Autosubmit: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5464 refactor(tvix/eval): use `EvalIO::read_dir` for equivalent builtinVincent Ambo1-2/+57
Change-Id: I6d782c07166f51587d2f1d06607823268debb5d5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7574 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5463 refactor(tvix/eval): use `EvalIO::path_exists` for the builtinVincent Ambo1-1/+16
Change-Id: I49822ce30137777865e7370ee86666636e277b35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7573 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5462 chore(tvix/eval): gate tvix_eval::StdIO behind the `impure` featureVincent Ambo1-0/+2
This shouldn't be available if we've built a "pure" crate. Change-Id: I7c85827ee212890252ff7e0b6242e2c52618cba5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7572 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5459 feat(tvix/eval): add EvalIO to public crate APIVincent Ambo1-2/+2
This lets users set the `io_handle` field on an `Evaluation`, which is then propagated to the VM. Change-Id: I616d7140724fb2b4db47c2ebf95451d5303a487a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7566 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5458 feat(tvix/eval): introduce initial EvalIO traitVincent Ambo1-0/+52
This trait is going to be used to abstract filesystem interactions in Tvix. For now, it only contains a `read_to_string` method that closely mirrors `std::fs::read_to_string`. As a first step, to see how this works in practice, we will thread through only this function to the various relevant parts. Two implementations are provided in tvix-eval itself: A dummy implementation (which just returns ErrorKind::NotImplemented for all operations), and a std implementation which delegates to `std` functions. Change-Id: Ied3e3bf4bd0e874dd84e166190e3873a0f923ddb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7565 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-22T18·24+0000