about summary refs log tree commit diff
path: root/tvix/cli
AgeCommit message (Collapse)AuthorFilesLines
2023-09-18 r/6606 refactor(tvix/store/blobsvc): make BlobStore asyncFlorian Klink3-77/+108
We previously kept the trait of a BlobService sync. This however had some annoying consequences: - It became more and more complicated to track when we're in a context with an async runtime in the context or not, producing bugs like https://b.tvl.fyi/issues/304 - The sync trait shielded away async clients from async worloads, requiring manual block_on code inside the gRPC client code, and spawn_blocking calls in consumers of the trait, even if they were async (like the gRPC server) - We had to write our own custom glue code (SyncReadIntoAsyncRead) to convert a sync io::Read into a tokio::io::AsyncRead, which already existed in tokio internally, but upstream ia hesitant to expose. This now makes the BlobService trait async (via the async_trait macro, like we already do in various gRPC parts), and replaces the sync readers and writers with their async counterparts. Tests interacting with a BlobService now need to have an async runtime available, the easiest way for this is to mark the test functions with the tokio::test macro, allowing us to directly .await in the test function. In places where we don't have an async runtime available from context (like tvix-cli), we can pass one down explicitly. Now that we don't provide a sync interface anymore, the (sync) FUSE library now holds a pointer to a tokio runtime handle, and needs to at least have 2 threads available when talking to a blob service (which is why some of the tests now use the multi_thread flavor). The FUSE tests got a bit more verbose, as we couldn't use the setup_and_mount function accepting a callback anymore. We can hopefully move some of the test fixture setup to rstest in the future to make this less repetitive. Co-Authored-By: Connor Brewster <cbrewster@hey.com> Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329 Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-04 r/6550 refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crateFlorian Klink4-4/+330
This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-08-20 r/6498 refactor(tvix/cli): cargo clippyFlorian Klink2-2/+1
Change-Id: I723bc9b29733f5ee4e3094b92696fbdeb4b15a33 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9094 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-08-20 r/6496 refactor(tvix/cli): stop parsing NixHash twiceFlorian Klink1-6/+2
We already have the parsed output_hash from above, no need to construct it again. Change-Id: Ie6d924ab446137c25c29fbeaf671aa7e5418262d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9110 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-08-14 r/6483 chore(tvix/cli): delete unused NixCompatIO codesterni2-122/+0
Change-Id: Icb91b102208fea512e04383ce9d65b0681af18ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/9079 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-07-31 r/6453 refactor(tvix/cli/refscan): use wu-manber crate with &[u8] supportFlorian Klink2-24/+10
PR'ed at https://github.com/tvlfyi/wu-manber/pull/1, and now merged. Change-Id: I8c71e359196396a1d42a3ea2ab7ac15b137b2db0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8992 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 r/6449 refactor(tvix/nix-compat): support non-unicode DerivationsFlorian Klink2-4/+18
Derivations can have non-unicode strings in their env values, so the ATerm representations are not necessarily String anymore, but Vec<u8>. Change-Id: Ic23839471eb7f68d9c3c30667c878830946b6607 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8990 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de>
2023-07-31 r/6447 docs(tvix/cli/refscan): fix commentFlorian Klink1-2/+2
The comment still mentions Aho-Corasick, which isn't correct. Change-Id: I846a2ca9ea7075c2456ca6ef04a132ff6161227a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8991 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-22 r/6439 feat(tvix/store/proto): use Bytes instead of Vec<u8>Florian Klink3-2/+3
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our bytes field cheaper to clone. It's a bit annoying to configure due to https://github.com/hyperium/tonic/issues/908, but the workaround does get the job done. Change-Id: I25714600b041bb5432d3adf5859b151e72b12778 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-07-21 r/6436 refactor(tvix/store): use bytes for node names and symlink targetsFlorian Klink2-4/+2
Some paths might use names that are not valid UTF-8. We should be able to represent them. We don't actually need to touch the PathInfo structures, as they need to represent StorePaths, which come with their own harder restrictions, which can't encode non-UTF8 data. While this doesn't change any of the wire format of the gRPC messages, it does however change the interface of tvix_eval::EvalIO - its read_dir() method does now return a list of Vec<u8>, rather than SmolStr. Maybe this should be OsString instead? Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-06-22 r/6344 feat(tvix/eval): allow extending builtins outside of tvix_evalEvgeny Zemtsov1-1/+1
The change allows applications that use tvix_serde for parsing nix-based configuration to extend the language with domain-specific set of features. Change-Id: Ia86612308a167c456ecf03e93fe0fbae55b876a6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8848 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-20 r/6336 fix(tvix/eval): only finalise formal arguments if defaultingsterni1-3/+2
When dealing with a formal argument in a function argument pattern that has a default expression, there are two different things that can happen at runtime: Either we select its value from the passed attribute successfully or we need to use the default expression. Both of these may be thunks and both of these may need finalisers. However, in the former case this is taken care of elsewhere, the value will always be finalised already if necessary. In the latter case we may need to finalise the thunk resulting from the default expression. However, the thunk corresponding to the expression may never end up in the local's stack slot. Since finalisation goes by stack slot (and not constants), we need to prevent a case where we don't fall back to the default expression, but finalise anyways. Previously, we worked around this by making `OpFinalise` ignore non-thunks. Since finalisation of already evaluated thunks still crashed, the faulty compilation of function pattern arguments could still cause a crash. As a new approach, we reinstate the old behavior of `OpFinalise` to crash whenever encountering something that is either not a thunk or doesn't need finalisation. This can also help catching (similar) miscompilations in the future. To then prevent the crash, we need to track whether we have fallen back or not at runtime. This is done using an additional phantom on the stack that holds a new `FinaliseRequest` value. When it comes to finalisation we check this value and conditionally execute `OpFinalise` based on its value. Resolves b/261 and b/265 (partially). Change-Id: Ic04fb80ec671a2ba11fa645090769c335fb7f58b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8705 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-12 r/6279 refactor(tvix/store/pathinfosvc): use Arc<dyn …>Florian Klink1-2/+4
This removes the use of generics, like previously done with Blob and Directory services. Change-Id: I7cc8bd1439b026c88e80c11e38aafc63c74e5e84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8751 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6273 refactor(tvix/store): use Arc instead of BoxFlorian Klink1-11/+6
This allows us to blob services without closing them before putting them in a box. We currently need to use Arc<_>, not Rc<_>, because the GRPC wrappers require Sync. Change-Id: I679c5f06b62304f5b0456cfefe25a0a881de7c84 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8738 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6272 refactor(tvix/store): use Box<dyn DirectoryService>Florian Klink1-3/+9
Once we support configuring services at runtime, we don't know what DirectoryService we're using at compile time. This also means, we can't explicitly use the is_closed method from GRPCPutter, without making it part of the DirectoryPutter itself. Change-Id: Icd2a1ec4fc5649a6cd15c9cc7db4c2b473630431 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8727 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-12 r/6271 feat(tvix/store/pathinfosvc): add calculate_nar methodFlorian Klink1-1/+2
Putting this in the PathInfoService trait makes much more sense, we can have direct control over where/how to cache the results in the implementation. This now requires each PathInfoService to hold pointers to BlobService and DirectoryService. Change-Id: I4faae780d43eae4beeb57bd5e190e6d1a5d3314e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8724 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-12 r/6270 refactor(tvix/src/nar): drop NARCalculationServiceFlorian Klink1-11/+2
There's only one way to calculate NAR files, by walking through them. Things like caching such replies should be done closer to where we use these, composing NARCalculationService doesn't actually give us much. Instead, expose two functions, `nar::calculate_size_and_sha256` and `nar::writer_nar`, the latter writing NAR to a writer, the former using write_nar to only keeping the NAR size and digest. Change-Id: Ie5d2cfea35470fdbb5cbf9da1136b0cdf0250266 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8723 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-06-12 r/6269 feat(tvix/store): eliminate generics in BlobStoreFlorian Klink1-2/+2
To construct various stores at runtime, we need to eliminate associated types from the BlobService trait, and return Box<dyn …> instead of specific types. This also means we can't consume self in the close() method, so everything we write to is put in an Option<>, and during the first close we take from there. Change-Id: Ia523b6ab2f2a5276f51cb5d17e81a5925bce69b6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8647 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-11 r/6266 fix(tvix/eval): emit only warnings on shadowed outputsLinus Heckemann2-5/+3
Unfortunately, nixpkgs has at least one case[1] where the out environment variable is shadowed -- though it doesn't cause a problem, since it's shadowed with the correct value, odd as this may be! [1]: https://github.com/NixOS/nixpkgs/blob/c7c298471676ac1c7789ab3c424fbcebecaa6791/pkgs/development/python-modules/pybind11/default.nix#L19 Change-Id: Ibf6790d2484dc9cce8e424feeb5886664d498dc3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8696 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-10 r/6261 chore(tvix/store): add fuse featureFlorian Klink1-1/+1
This brings in fuse (via the `fuser` crate), and adds pkg-config and libfuse to the dev shell, so `cargo build` can link against it. Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-06-02 r/6227 fix(tvix/cli): fix refscan when no paths are referencedLinus Heckemann1-4/+21
Before, the construction of a TwoByteWM would panic when no patterns were provided, as in `tvix --expr 'builtins.toFile "snens" "soos"'`. Change-Id: I25ed498c475523aec5baf8683b23059fadabb21c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8697 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-29 r/6214 fix(tvix/eval): thunk unary operator applicationssterni1-0/+3
Unary operator applications are thunked which can easily be observed by nix-instantiate --eval -E '[ (!true) (-1) ]' Unfortunately, there are few simple expressions where this makes a difference in the end result. Thus it only cropped up when using nixpkgs for cross compilation: Here we would compile the expression !(stdenv.cc.isGNU or false) to assemble python3Minimal's passthru attribute set (at least this seems to be the most likely explanation from the backtraces I've studied). This means that an unthunked <stdenv.cc.isGNU or false> OpForce OpInvert would be performed in order to assemble this attribute set, causing stdenv.cc to be evaluated too early, causing an infinite recursion. Resolves b/273. It seems that having a test suite that doesn't use --strict and relies on thunks rendered as <CODE> would be beneficial for catching such issues. I've not been able to find a test case with --strict that demonstrates the problem fixed in this CL. Change-Id: I640a5827b963f5b9d0f86fa2142e75e3a6bbee78 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8654 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-27 r/6213 feat(tvix/cli): add --no-warnings argumentFlorian Klink1-2/+8
This will prevent tvix from printing any warnings. As a followup, we can also thread this parameter through into the evaluator itself, to prevent warnings from being constructed in first place. Change-Id: I15381396f86573484bdd1a73d09034a665638e35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8646 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-25 r/6202 refactor(tvix/eval): stop borrowing &mut selfFlorian Klink2-8/+8
This does undo cl/8571. Change-Id: Ib14b4e7404f906e346304b6113860ae811afc94a Reviewed-on: https://cl.tvl.fyi/c/depot/+/8631 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-25 r/6201 refactor(tvix/cli): wrap NixCompatIO import_cache in RwLockFlorian Klink1-4/+13
This allows dropping all &mut self from the EvalIO trait signature. Change-Id: Ie127b0a459d2996636385d159fcc5f7147e74e2e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8630 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6193 feat(tvix/cli): use TvixStoreIO instead of NixCompatIOFlorian Klink2-1/+19
This switches tvix-cli over from using `NixCompatIO` to `TvixStoreIO`. For now, we simply instantiate in-memory services instead of getting fancy with constructors inside tvix-store, but long-term, we might want to support some URI syntax, to make this configurable at runtime. nixpkgs eval tests might be fine (and fast!) with a purely in-memory backend, but other usages might involve talking to a local tvix-store over gRPC (using the gRPC client, either unix domain socket or even further away remote), or running tvix-store in "embedded" mode (using another client than the gRPC client). Change-Id: I509afd3dc5ce3f2d52b0fb7067748fab820e26ab Reviewed-on: https://cl.tvl.fyi/c/depot/+/8572 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-24 r/6190 feat(tvix/cli): check nixpkgs eval for pkgs.hello.outPathFlorian Klink1-0/+1
We already evaluate this the same as Nix does, so worth adding it into CI. Change-Id: I529faccac6e5e16e0bc985ab4c3e0cd07bb23293 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8624 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-05-24 r/6189 feat(tvix/cli): introduce mkNixpkgsEvalCheck, add outPath checkFlorian Klink1-16/+23
This introduces a function that can be instantiated with an attribute path to instantiate, as well as the expected path (normally interpolated with the nix evaluator). Check both pkgs.stdenv.drvPath and pkgs.stdenv.outPath to match. Change-Id: Id667ed35fa159ff83fedb3017ef8d3271aa42695 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8606 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-05-22 r/6173 refactor(tvix/eval): use &Path instead of PathBufFlorian Klink2-6/+6
This allows getting rid of some clones in eval/src/vm/generators.rs. Change-Id: I330390307d3bcfeef19c98954c753ee55b1ccee3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8604 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-22 r/6171 refactor(tvix/eval/io): use io::Error instead of tvix_eval errorsFlorian Klink2-14/+12
We didn't return anything useful other than ErrorKind::IO anyways. We can use io::ErrorKind::Unsupported for DummyIO. Fixes b/271. Change-Id: Icb231e9b38168e8b6fa473bfa405d160357b317f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8602 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-18 r/6160 chore(tvix/cli): remove unnecessary RefCellVincent Ambo1-7/+5
This remained around from some previous time where this type was being cloned around directly, but it's not anymore. Change-Id: I2c61cf9cecb8e5d7d08644ed20642ee61357bc21 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8580 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-05-18 r/6159 refactor(tvix/cli): split CLI-specific IO logic into TvixIO typeVincent Ambo3-11/+84
This adds a wrapper type TvixIO<T: EvalIO>, which can wrap around an arbitrary EvalIO implementation and perform actions needed for the Tvix CLI (marking imported paths as known, and handling __corepkgs__). Change-Id: I5fc1ca199b9f94b21a89103b84575e0f8f58dff9 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8579 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-05-14 r/6140 feat(tvix/eval/io): allow &mut self in EvalIOFlorian Klink1-4/+4
It's okay if these calls mutate some internal state inside an implementation. Change-Id: I12bb11bde0310778c3da1275696bf7de058863a3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8571 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-11 r/6134 chore(tvix/*): bump to smol_str 0.2.0Florian Klink1-1/+1
Change-Id: Ic9ac1b6fecb564eafb41b265bf317cd385fdc170 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8560 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-04-09 r/6088 refactor(nix-compat): Properly encapsulate store path constructionJohn Ericson1-10/+8
Before there was code scattered about (e.g. text hashing module and derivation output computation) constructing store paths from low level building blocks --- there was some duplication and it was easy to make nonsense store paths. Now, we have roughly the same "safe-ish" ways of constructing them as C++ Nix, and only those are exposed: - Make text hashed content-addressed store paths - Make other content-addressed store paths - Make input-addressed fixed output hashes Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-30 r/6060 refactor(tvix/nix-compat): move build_store_path out of derivationFlorian Klink1-6/+13
This doesn't have anything to do with ATerms, we just happen to be using the aterm representation of a Derivation as contents. Moving this into store_path/utils.rs makes these things much cleaner - Have a build_store_path_from_references function, and a build_store_path_from_fingerprint helper function that makes use of it. build_store_path_from_references is invoked from the derivation module which can be used to calculate the derivation path. In the derivation module, we also invoke build_store_path_from_fingerprint during the output path calculation. Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-22 r/6038 feat(tvix/cli): add `--strict` flag for evaluationVincent Ambo1-0/+9
This toggles whether tvix will evaluate the top-level value and deep-force it, or return it potentially still containing thunks. Change-Id: Ie910941e3b6a0f16c5c0cb896d73947626335f4b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8326 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-03-15 r/6010 fix(tvix/cli): populate output configuration after walking argsVincent Ambo1-15/+14
Walking the arguments might encounter an `outputs` output, which might explicitly (for whatever reason) specify the `out` output. To prevent dropping FOD settings in this case, we have to populate that part of the configuration after walking the other attributes. Change-Id: Iee6a7f0a71e9c9699e79d35e6cb19e1ddb49395d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8312 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-03-15 r/6009 refactor(tvix/nix-compat): -derivation::Hash, +NixHashFlorian Klink1-11/+10
This stops using our own custom Hash structure, which was mostly only used because we had to parse the JSON representation somehow. Since cl/8217, there's a `NixHash` struct, which is better suited to hold this data. Converting the format requires a bit of serde labor though, but that only really matters when interacting with JSON representations (which we mostly don't). Change-Id: Idc5ee511e36e6726c71f66face8300a441b0bf4c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8304 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 r/5996 chore(tvix/cli): don't set COREPKGS anymoreFlorian Klink1-4/+0
Since cl/8213, tvix bundles corepkgs. Change-Id: I17fa4452a6a1d554ae67b4aed9d9b1e77cb495f5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8305 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-14 r/5995 refactor(tvix/nix-compat): replace calculate_drv_replacement_strFlorian Klink2-20/+30
Call this function derivation_or_fod_hash, and return a NixHash. This is more in line with how cppnix calls this, and allows using to_nix_hash_string() in some places. Change-Id: Iebf5355f08ed5c9a044844739350f829f874f0ce Reviewed-on: https://cl.tvl.fyi/c/depot/+/8293 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-03-13 r/5986 chore(tvix/cli): silence useless warningVincent Ambo1-0/+1
Change-Id: I18f567d5dcbbeb934428bf9111eb5fae5fef0342 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8261 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-03-13 r/5981 feat(tvix/cli): bundle corepkgs/fetchurl.nix with tvix-cliVincent Ambo3-1/+76
This file which ships with C++ Nix is required for evaluating nixpkgs. Like C++ Nix, we now inject a pseudo path in EvalIO from which this will resolve as <nix/fetchurl.nix> Change-Id: Ic948c476a2cfc6381d5655d308bc2d5fa25b7123 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8213 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-13 r/5964 refactor(tvix/eval): flatten call stack of VM using generatorsVincent Ambo2-291/+301
Warning: This is probably the biggest refactor in tvix-eval history, so far. This replaces all instances of trampolines and recursion during evaluation of the VM loop with generators. A generator is an asynchronous function that can be suspended to yield a message (in our case, vm::generators::GeneratorRequest) and receive a response (vm::generators::GeneratorResponsee). The `genawaiter` crate provides an interpreter for generators that can drive their execution and lets us move control flow between the VM and suspended generators. To do this, massive changes have occured basically everywhere in the code. On a high-level: 1. The VM is now organised around a frame stack. A frame is either a call frame (execution of Tvix bytecode) or a generator frame (a running or suspended generator). The VM has an outer loop that pops a frame off the frame stack, and then enters an inner loop either driving the execution of the bytecode or the execution of a generator. Both types of frames have several branches that can result in the frame re-enqueuing itself, and enqueuing some other work (in the form of a different frame) on top of itself. The VM will eventually resume the frame when everything "above" it has been suspended. In this way, the VM's new frame stack takes over much of the work that was previously achieved by recursion. 2. All methods previously taking a VM have been refactored into async functions that instead emit/receive generator messages for communication with the VM. Notably, this includes *all* builtins. This has had some other effects: - Some test have been removed or commented out, either because they tested code that was mostly already dead (nix_eq) or because they now require generator scaffolding which we do not have in place for tests (yet). - Because generator functions are technically async (though no async IO is involved), we lose the ability to use much of the Rust standard library e.g. in builtins. This has led to many algorithms being unrolled into iterative versions instead of iterator combinations, and things like sorting had to be implemented from scratch. - Many call sites that previously saw a `Result<..., ErrorKind>` bubble up now only see the result value, as the error handling is encapsulated within the generator loop. This reduces number of places inside of builtin implementations where error context can be attached to calls that can fail. Currently what we gain in this tradeoff is significantly more detailed span information (which we still need to bubble up, this commit does not change the error display). We'll need to do some analysis later of how useful the errors turn out to be and potentially introduce some methods for attaching context to a generator frame again. This change is very difficult to do in stages, as it is very much an "all or nothing" change that affects huge parts of the codebase. I've tried to isolate changes that can be isolated into the parent CLs of this one, but this change is still quite difficult to wrap one's mind and I'm available to discuss it and explain things to any reviewer. Fixes: b/238, b/237, b/251 and potentially others. Change-Id: I39244163ff5bbecd169fe7b274df19262b515699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2023-02-16 r/5858 chore(tvix/cli): iter instead of into_iter for referencesAaqa Ishtyaq1-1/+1
This CL removes calling into_iter on a reference, as it will not move out it's content into resulting iterator. Change-Id: Ifcc10b7cf33b98453570cbcec3eb82ffaba2ffcb Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/8126 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-04 r/5836 fix(tvix/cli): use tvlfyi/wu-manber fork for refscannerVincent Ambo1-1/+1
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1) but it's not clear whether upstream will accept patches, so for now lets point this directly at our fork. Change-Id: Iccdcedae3e9a8b783241431787c952561d032694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-03 r/5829 test(tvix/cli): eval nixpkgs.stdenv in CI and assert that it matchesVincent Ambo1-2/+26
Change-Id: If80194b5fdbf69512217bd4780e416e678045323 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8023 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-02 r/5827 fix(tvix/cli): keep tracking full paths in known_pathsVincent Ambo3-42/+84
We need to distinguish explicitly between the paths used for the scanner, and the paths that populate the derivation inputs. The full paths must be accessible from the result of the refscanner to populate drv fields correctly. This was previously hidden by debug changes that masked actual IO operations with no-ops. Change-Id: I037af6e6bbe2b573034d695f8779bee1b56bc125 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8022 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-02-02 r/5826 feat(tvix/cli): cache imported paths in NixCompatIOVincent Ambo1-2/+20
Creates a cache of imported literal files (e.g. `./default-builder.sh`) which avoids shelling out to Nix for each instance of the same file. Note that a better way to tackle this is to create memoizable thunks for these expressions in the compiler, but we are lacking a little bit of infrastructure for that at the moment. Change-Id: Ibc062b20d81e97dd3986e734d225a744e1779fe7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8015 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 r/5825 refactor(tvix/cli): use Wu-Manber string scanning for drv referencesVincent Ambo4-59/+37
Switch out the string-scanning algorithm used in the reference scanner. The construction of aho-corasick automata made up the vast majority of runtime when evaluating nixpkgs previously. While the actual scanning with a constructed automaton is relatively fast, we almost never scan for the same set of strings twice and the cost is not worth it. An algorithm that better matches our needs is the Wu-Manber multiple string match algorithm, which works efficiently on *long* and *random* strings of the *same length*, which describes store paths (up to their hash component). This switches the refscanner crate to a Rust implementation[0][1] of this algorithm. This has several implications: 1. This crate does not provide a way to scan streams. I'm not sure if this is an inherent problem with the algorithm (probably not, but it would need buffering). Either way, related functions and tests (which were actually unused) have been removed. 2. All strings need to be of the same length. For this reason, we truncate the known paths after their hash part (they are still unique, of course). 3. Passing an empty set of matches, or a match that is shorter than the length of a store path, causes the crate to panic. We safeguard against this by completely skipping the refscanning if there are no known paths (i.e. when evaluating the first derivation of an eval), and by bailing out of scanning a string that is shorter than a store path. On the upside, this reduces overall runtime to less 1/5 of what it was before when evaluating `pkgs.stdenv.drvPath`. [0]: Frankly, it's a random, research-grade MIT-licensed crate that I found on Github: https://github.com/jneem/wu-manber [1]: We probably want to rewrite or at least fork the above crate, and add things like a three-byte wide scanner. Evaluating large portions of nixpkgs can easily lead to more than 65k derivations being scanned for. Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-01T23·00+0000