about summary refs log tree commit diff
path: root/tvix/eval/src/vm.rs
AgeCommit message (Collapse)AuthorFilesLines
2023-03-13 r/5964 refactor(tvix/eval): flatten call stack of VM using generatorsVincent Ambo1-1218/+0
Warning: This is probably the biggest refactor in tvix-eval history, so far. This replaces all instances of trampolines and recursion during evaluation of the VM loop with generators. A generator is an asynchronous function that can be suspended to yield a message (in our case, vm::generators::GeneratorRequest) and receive a response (vm::generators::GeneratorResponsee). The `genawaiter` crate provides an interpreter for generators that can drive their execution and lets us move control flow between the VM and suspended generators. To do this, massive changes have occured basically everywhere in the code. On a high-level: 1. The VM is now organised around a frame stack. A frame is either a call frame (execution of Tvix bytecode) or a generator frame (a running or suspended generator). The VM has an outer loop that pops a frame off the frame stack, and then enters an inner loop either driving the execution of the bytecode or the execution of a generator. Both types of frames have several branches that can result in the frame re-enqueuing itself, and enqueuing some other work (in the form of a different frame) on top of itself. The VM will eventually resume the frame when everything "above" it has been suspended. In this way, the VM's new frame stack takes over much of the work that was previously achieved by recursion. 2. All methods previously taking a VM have been refactored into async functions that instead emit/receive generator messages for communication with the VM. Notably, this includes *all* builtins. This has had some other effects: - Some test have been removed or commented out, either because they tested code that was mostly already dead (nix_eq) or because they now require generator scaffolding which we do not have in place for tests (yet). - Because generator functions are technically async (though no async IO is involved), we lose the ability to use much of the Rust standard library e.g. in builtins. This has led to many algorithms being unrolled into iterative versions instead of iterator combinations, and things like sorting had to be implemented from scratch. - Many call sites that previously saw a `Result<..., ErrorKind>` bubble up now only see the result value, as the error handling is encapsulated within the generator loop. This reduces number of places inside of builtin implementations where error context can be attached to calls that can fail. Currently what we gain in this tradeoff is significantly more detailed span information (which we still need to bubble up, this commit does not change the error display). We'll need to do some analysis later of how useful the errors turn out to be and potentially introduce some methods for attaching context to a generator frame again. This change is very difficult to do in stages, as it is very much an "all or nothing" change that affects huge parts of the codebase. I've tried to isolate changes that can be isolated into the parent CLs of this one, but this change is still quite difficult to wrap one's mind and I'm available to discuss it and explain things to any reviewer. Fixes: b/238, b/237, b/251 and potentially others. Change-Id: I39244163ff5bbecd169fe7b274df19262b515699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2023-03-13 r/5962 feat(tvix/eval): add generator-related functions to RuntimeObserverVincent Ambo1-2/+2
These functions will be used by the changes in the VM to observe the runtime execution of generator frames, and provide a more linear view of the execution of the Tvix VM. Change-Id: I10b1b1933dedc065e7c61d5d6062f0aaeee0097e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8240 Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-03-07 r/5899 feat(tvix/eval): introduce generators moduleVincent Ambo1-0/+2
This module contains the request/response types for generators requesting actions from the VM. For most of these, an async helper function is added that will be used inside of generator functions to make use of these requests/responses instead of constructing them directly. Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-07 r/5898 refactor(tvix/eval): merge OpCall & OpTailCallVincent Ambo1-5/+0
As applies are thunked, there was no situation where OpCall could be emitted. In practice, all calls were already tail calls. Change-Id: Id0d441dcdd86f804d7cddd0cc14f589bbfc75e5b Reviewed-on: https://cl.tvl.fyi/c/depot/+/8147 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-02-03 r/5828 fix(tvix/eval): ensure all evaluated thunks are correctly memoizedVincent Ambo1-1/+2
This fixes a very complicated bug (b/246). Evaluation progresses *much* further after this, leading to several less complicated bugs likely being uncovered by this What was the problem? ===================== Previously, when evaluating a thunk, we had a code path that looked like this: match *thunk { ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => { let inner_repr = inner_thunk.0.borrow().clone(); drop(thunk); self.0.replace(inner_repr); } /* ... */ } This code path created a copy of the inner `ThunkRepr` of a nested thunk, and moved that copy into the `ThunkRepr` of the parent. The effect of this was that the original `ThunkRepr` (unforced!) lived on in the original thunk, without the memoization of the subsequent forcing applying to it. This had the result that Tvix would repeatedly evaluate these thunks without ever memoizing them, if they occured repeatedly as shared inner thunks. Most notably, this would *always* occur when builtins.import was used. What's the solution? ==================== I have completely rewritten `Thunk::force_trampoline_self` to make all flows that can occur in it explicit. I have also removed the outer loop inside of that function, and resorted to more use of trampolining instead. The function is now well-commented and it should be possible to read it from top-to-bottom and get a general sense of what is going on, though the trampolining itself (which is implemented in the VM) needs to be at least partially understood for this. What's the new problem(s)? ========================== One new (known) problem is that we have to construct `Error` instances in all error types here, but we do not have spans available in some thunk-related situations. Due to b/238 we cannot ask the VM for an arbitrary span from the callsite leading to the force. This means that there are now code paths where, under certain conditions, causing an evaluation error during thunk forcing will panic. To fix this we will need to investigate and fix b/238, and/or add a span tracking mechanism to thunks themselves. What other impacts does this have? ================================== With this commit, eval of nixpkgs mostly succeeds (things like stdenv evaluate to the same hashes for us and C++ Nix, meaning we now construct identical derivations without eval breaking). Due to this we progress much further into nixpkgs, which lets us uncover more additional bugs. For example, after this commit we can quickly see that cl/7949 introduces some kind of behavioural issue and should not be merged as-is (this was not apparent before). Additionally, tvix-eval is now seemingly very fast. When doing performance analysis of a nixpkgs eval, we now mostly see the code path for shelling out to C++ Nix to add things to the store in there. We still need those code paths, so we can not (yet) do a performance analysis beyond that. Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 r/5822 refactor(tvix/eval): import_cache can be a HashMapVincent Ambo1-2/+2
... instead of a BTreeMap, as we do not need ordering guarantees here. HashMaps are noticeably faster here (especially as we've been sorting essentially random data!). Change-Id: Ie92d74286df9f763c04c9b226ef1066ee8484c13 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8014 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-01-23 r/5746 fix(tvix/eval): force functors before applying themVincent Ambo1-0/+4
call_value in the VM expects the callable to be forced when calling it, which was not the case for functors. Change-Id: Id55a2fe32a9573be42aef8669e268df519a989cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/7909 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-21 r/5721 refactor(tvix/eval): administer antidote for poisonAdam Joseph1-4/+0
The codebase contains a lot of complexity and odd roundabout handling for shadowing globals. I'm pretty sure none of this is necessary, and all of it disappears if you simply make the globals part of the ordinary identifier resolution chain, with their own scope up above the root scope. Then the ordinary shadowing routines do the right thing, and no special cases or new terminology are required. This commit does that. Note by tazjin: This commit was originally abandoned when Adam decided not to take away reviewer bandwidth for this at the time (eval was still in a much earlier stage). As we've recently done some significant refactoring of globals initialisation this came up again, and it seems we can easily cover the use-cases of the poison tracking in other ways now, so I've rebased, updated and resurrected the CL. Co-Authored-By: Vincent Ambo <tazjin@tvl.su> Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: Ib3309a47a7b31fa5bf10466bade0d876b76ae462 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7089 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 r/5715 refactor(tvix/eval): keep globals alive through VM structVincent Ambo1-1/+15
This forces users to pass the fully constructed set of globals to the VM, making it harder to accidentally "lose" the set while weak references to it still exist. This doesn't modify any functionality, but is laying the foundation for simplifying some of the builtins behaviour that has grown more complex again. Change-Id: I5120f97861c65dc46d90b8a4e2c92ad32cc53e03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7877 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 r/5714 docs(tvix/eval): add doc comments to VM fieldsVincent Ambo1-0/+8
Change-Id: Ia4857c217de15aec8b61e1abd39e22c50e2d816a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7876 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-01-20 r/5706 feat(tvix/eval): add error contexts to annotate error kindsVincent Ambo1-14/+3
This makes it possible for users to add additional context to an error, which will then be rendered as an additional secondary span in the formatted error output. We should strive to do this basically anywhere errors are raised that can occur multiple times, *especially* during type casts. This was triggered by me debugging a type cast error attached to a fairly large-ish span (a builtin invocation). Change-Id: I51be41fabee00cf04de973935daf34fe6424e76f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7849 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 r/5583 refactor(tvix/eval): avoid unnecessary pop/push in OpForceVincent Ambo1-6/+1
Change-Id: Ic7559eaa43aa0dcc97babb7669770c0f7f959f1b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7754 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-29 r/5542 refactor(tvix/eval): remove extra Rc<..> around Value::AttrsVincent Ambo1-7/+6
The `im::OrdMap` is already small and cheap to copy while sharing memory, so this is not required anymore. Only the `KV` variant may have slightly larger content, but in practice this doesn't seem to make a difference when comparing the two variants and this one is less complicated. Change-Id: I64a563b209a2444125653777551373cb2989ca7d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7677 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-12-29 r/5534 refactor(tvix/eval): use im::Vector for NixList representationVincent Ambo1-4/+3
This is a persistent, structurally sharing data structure which is more efficient in some of our use-cases. I have verified the efficiency improvement using `hyperfine` repeatedly over expressions on nixpkgs. Lists are not the most performance-critical structure in Nix (that would be attribute sets), but we can already see a small (~5-10%) improvement. Note that there are a handful of cases where we still go via `Vec` that need to be fixed, most notable for `builtins.sort` which can not currently be implemented directly using `im::Vector` because of a restrictive type bound. Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-25 r/5486 fix(tvix/eval): fix current clippy warningsVincent Ambo1-8/+6
It's been a while since the last time, so quite a lot of stuff has accumulated here. Change-Id: I0762827c197b30a917ff470fd8ae8f220f6ba247 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7597 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-25 r/5485 refactor(tvix/eval): non-recursive thunk forcingAdam Joseph1-38/+128
Introduces continuation-passing-based trampolining of thunk forcing to avoid recursing when forcing deeply nested expressions. This is required for evaluating large expressions. This change was extracted out of cl/7362. Co-authored-by: Vincent Ambo <tazjin@tvl.su> Co-authored-by: Griffin Smith <grfn@gws.fyi> Change-Id: Ifc1747e712663684b2fff53095de62b8459a47f3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7551 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-21 r/5466 refactor(tvix/eval): use light spans in builtins.importVincent Ambo1-6/+6
Change-Id: I05732073155b430575babb6f076bf465aef98857 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7581 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5460 refactor(tvix/eval): use EvalIO::read_to_string in impure builtinsVincent Ambo1-0/+5
With this change, the behaviour of reading a string from a file path is controlled by the provided `EvalIO` structure. This is a huge step towards abstracting away I/O behaviour correctly. Change-Id: Ifde8e46cd863b16e0301dca45a434ad27560399f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7567 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 r/5459 feat(tvix/eval): add EvalIO to public crate APIVincent Ambo1-2/+11
This lets users set the `io_handle` field on an `Evaluation`, which is then propagated to the VM. Change-Id: I616d7140724fb2b4db47c2ebf95451d5303a487a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7566 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5457 refactor(tvix/eval): add a LightSpan type for lighter span trackingVincent Ambo1-1/+8
This type carries the information required for calculating a span (i.e. the chunk and offset), instead of the span itself. The span is then only calculated in cases where it is required (when throwing errors). This reduces the eval time for `builtins.length (builtins.attrNames (import <nixpkgs> {}))` by *one third*! The data structure in chunks that carries span information reduces in-memory size by trading off the speed of retrieving span information. This is because the span information is only actually required when throwing errors (or emitting warnings). However, somewhere along the way we grew a dependency on carrying span information in thunks (for correctly reporting error chains). Hitting the code paths for span retrieval was expensive, and carrying the spans in a different way would still be less cache-efficient. This change is the best tradeoff I could come up with. Refs: b/229. Change-Id: I27d4c4b5c5f9be90ac47f2db61941e123a78a77b Reviewed-on: https://cl.tvl.fyi/c/depot/+/7558 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5453 feat(tvix/eval): remove `derive(Copy)` from UpvaluesAdam Joseph1-4/+4
Change-Id: I0fa069fbeff6718a765ece948c2c1bce285496f7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7449 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5452 feat(tvix/eval): wrap Closure in Rc<> to match cppnix semanticsAdam Joseph1-3/+3
Change-Id: I595087eff943d38a9fc78a83d37e207bb2ab79bc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7443 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-02 r/5367 feat(tvix/eval): inline(always) tail_call_valueAdam Joseph1-0/+1
Rust doesn't do tail-call elimination (still!) so the best we can hope for here is to inline non-recursive invocations. Change-Id: I78949967e48b006fcbf31786d8f6281cd122f36f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7360 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: Adam Joseph <adam@westernsemico.com>
2022-12-02 r/5366 feat(tvix/eval): crude caching builtins.importAdam Joseph1-1/+4
Before this, tvix was spending most of its time furiously re-parsing and re-compiling nixpkgs, each time hoping to get a different result... Change-Id: I1c0cfbf9af622c276275b1f2fb8d4e976f1b5533 Signed-off-by: Adam Joseph <adam@westernsemico.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/7361 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-28 r/5350 feat(tvix/eval): implement equality on derivationsAdam Joseph1-0/+32
Change-Id: I344b66c39cbc4b426accc482aa8f6f2eb18db68a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7417 Autosubmit: Adam Joseph <adam@westernsemico.com> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-27 r/5345 feat(tvix/eval): non-recursive implementation of nix_eq()Adam Joseph1-7/+122
This passes all the function/thunk-pointer-equality tests in cl/7369. Change-Id: Ib47535ba2fc77a4f1c2cc2fd23d3a879e21d8b4c Signed-off-by: Adam Joseph <adam@westernsemico.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/7358 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-26 r/5340 feat(tvix/eval): use backtrace-on-stack-overflow crateAdam Joseph1-0/+9
The backtrace-on-stack-overflow create provides best-effort stack traces when a stack overflow happens. Since it's running on the (usually tiny) signal alternate stack this isn't easy. This is guarded by a new `backtrace_overflow` feature flag and never enabled (even if that feature is selected) for release builds. This is strictly for debugging; there's crazy unsafe voodoo in there. https://lib.rs/crates/backtrace-on-stack-overflow Example output: ``` Stack Overflow: 0: backtrace_on_stack_overflow::handle_sigsegv at /home/amjoseph/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-on-stack-overflow-0.2.0/src/lib.rs:93:40 1: <unknown> 2: __rust_probestack 3: tvix_eval::vm::VM::run_op at src/vm.rs:399 4: tvix_eval::vm::VM::run at src/vm.rs:388:23 5: tvix_eval::vm::VM::enter_frame at src/vm.rs:360:22 6: tvix_eval::value::thunk::Thunk::force at src/value/thunk.rs:116:25 7: tvix_eval::vm::VM::run_op at src/vm.rs:801:37 8: tvix_eval::vm::VM::run at src/vm.rs:388:23 9: tvix_eval::vm::VM::enter_frame at src/vm.rs:360:22 10: tvix_eval::value::thunk::Thunk::force at src/value/thunk.rs:116:25 ... ``` Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I1d8a2017f836be7bf91a2223e7adacb86fa1dbb2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7354 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-26 r/5339 fix(tvix/eval): OpAdd must weakly stringify if either arg is stringAdam Joseph1-1/+12
Tests included. Change-Id: I7a4905d6103813373e383e2e8629c5fd243d6bca Reviewed-on: https://cl.tvl.fyi/c/depot/+/7377 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: Adam Joseph <adam@westernsemico.com>
2022-11-26 r/5324 feat(tvix/eval): wrap Closure::upvalues in RcAdam Joseph1-1/+2
See cl/7372; Nix equality semantics require the ability to track pointer equality of upvalue-sets. Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I82ba517499cf370189a80355e4e46a5caaab7153 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7373 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-23 r/5301 feat(tvix/eval): make NixList::clone() cheapAdam Joseph1-2/+3
When we start unrecursivifying (sp?) things, Rust's borrow checker is going to be a headache; its magic only works when you use the CPU stack as your call stack. Fixing the borrow checker issues usually involves adding lots of `clone()`s. Right now `NixList` is the only variant of `Value` that isn't cheap to clone() -- all the others are either a wrapper around Rc or else are of bounded size. Note that this requires dropping the `DerefMut for NixList` instance and using `Vec<Value>` instead in those situations. Change-Id: I5a47df66855342aa2064f8f3cb7934ff422d26bd Signed-off-by: Adam Joseph <adam@westernsemico.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/7359 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-21 r/5297 fix(tvix/eval): ensure callable is forced when using call_withVincent Ambo1-1/+5
When passing multiple arguments, every intermediate callable needs to be forced as this is expected by the VM's call_value function. Also adds a debug assertion for this which makes it easier to spot exactly what went wrong. Change-Id: I3aa519cb6cdaab713bd18282bef901c4cd77c535 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7312 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-11-10 r/5276 feat(tvix/eval): detect division by zerojhahn1-1/+14
This detects if the second argument of a division is a zero (either as integer or as float). If so, an error message is displayed. This fixes b/219. Change-Id: I50203d14a71482bc757832a2c8dee08eb7d35c49 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7258 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-11-06 r/5254 refactor(tvix/eval): move `unwrap_or_clone_rc` to lib moduleVincent Ambo1-6/+1
This is more generally useful than just inside the VM, until it is stabilised in Rust itself. Change-Id: Id9aa3d5b533ff38e3d2c6b85ad484394fdd05dcf Reviewed-on: https://cl.tvl.fyi/c/depot/+/7186 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-05 r/5250 refactor(tvix/eval): rename Opcode::DataLocalIdx to DataStackIdxAdam Joseph1-4/+4
It is very confusing that this opcode is called DataLocalIdx, but it carries a StackIdx rather than a LocalIdx. It seems like this really ought to be called DataStackIdx, but maybe I've misunderstood; if so please explain it to me. Change-Id: I91f6ffa759412beef0b91d3c19ec0d873fe51b99 Signed-off-by: Adam Joseph <adam@westernsemico.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/7088 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-11-04 r/5240 fix(tvix_eval): {stack,local}_idx confusionAdam Joseph1-4/+4
The variable name `local_idx` is used here for a StackIdx, which invites confusion. Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I2e22db90acdc0d29586ee5b72ea18d42d93badcb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7086 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-10-29 r/5221 feat(tvix/eval): Implement comparison for listsGriffin Smith1-1/+1
Lists are compared lexicographically in C++ nix as of [0], and our updated nix test suites depend on this. This implements comparison of list values in `Value::nix_cmp` using a very similar algorithm to what C++ does - similarly to there, this requires passing in the VM so we can force thunks in the list elements as we go. [0]: https://github.com/NixOS/nix/commit/09471d2680292af48b2788108de56a8da755d661# Change-Id: I5d8bb07f90647a1fec83f775243e21af856afbb1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7070 Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-10-24 r/5193 refactor(tvix/eval): Implement value comparison with a methodGriffin Smith1-23/+18
Rather than implementing all of the interesting semantics of value comparison with a macro bound to the VM, implement the bulk of the logic with a method on Value itself that returns an Ordering, and then use the macro to implement the comparison against that Ordering. This has no functional change, but paves the way to implementing lexicographic comparison of list values, which is supported in the latest version of upstream nix. Change-Id: I8af1a020b41577021af5939f5edc160c407d4a9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/7069 Autosubmit: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-23 r/5187 fix(tvix/eval): Use natural arg order for call_withGriffin Smith1-1/+2
Since we push arguments onto a stack when calling multi-argument functions, we actually were ending up calling `call_with` with the arguments in the *reverse order* - we patched around this by passing the arguments in the reverse order for `foldl'`, but it makes more sense to have them just be the order that the function would be called with in user surface code instead. Change-Id: Ifddb98f46970ac89872383709c3ce758dc965c65 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7067 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-10-23 r/5179 chore(tvix/eval): return detailed TvixBug if an upvalue is missingVincent Ambo1-1/+17
When capturing an upvalue, return a detailed TvixBug error that contains metadata about what exactly was missing. This particular thing helps with debugging some scope issues we still seem to have. Change-Id: I1089a1df4b3bbc63411a4907c3641a5dc3fad984 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7058 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-10-22 r/5175 feat(tvix/eval): Implement builtins.deepSeqGriffin Smith1-56/+9
This is done via a new `deepForce` function on Value. Since values can be cyclical (for example, see the test-case), we need to do some extra work to avoid RefCell borrow errors if we ever hit a graph cycle: While deep-forcing values, we keep a set of thunks that we have already seen and avoid doing any work on the same thunk twice. The set is encapsulated in a separate type to stop potentially invalid pointers from leaking out. Finally, since deep_force is conceptually similar to `VM::force_for_output` (but more suited to usage in eval since it doesn't clone the values) this removes the latter, replacing it with the former. Co-Authored-By: Vincent Ambo <tazjin@tvl.su> Change-Id: Iefddefcf09fae3b6a4d161a5873febcff54b9157 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7000 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-22 r/5174 fix(tvix/eval): use top-level span for `force_with_output`Vincent Ambo1-6/+27
When forcing thunks in `force_with_output`, the call stack of the VM is actually empty (as the calls are synthetic and no longer part of the evaluation of the top-level expression). This means that Tvix crashed when constructing error spans for the `fallible` macro, as the assumption of there being an enclosing span was violated. To work around this, we instead pass the span for the whole top-level expression to force_for_output and set this as the span for the enclosing error chain. Existing output logic will already avoid printing the entire expression as an error span. This fixes b/213. Change-Id: I93978e0deaf5bcb0f47a6fa95b3f5bebef5bad4c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7052 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-10-21 r/5172 fix(tvix): distinguish search- and relative path resolution errorssterni1-1/+1
Failures to resolve a nix search path lookup in angle brackets can be caught using tryEval (if it reaches the runtime). Resolving relative paths (either to the current directory or the current user's home) can never be caught, even if they happen inside a thunk at runtime (which is currently the case for home-relative paths). Change-Id: I7f73221df66d82a381dd4063358906257826995a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7025 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-10-19 r/5159 feat(tvix/eval): deduplicate overlap between Closure and ThunkAdam Joseph1-27/+32
This commit deduplicates the Thunk-like functionality from Closure and unifies it with Thunk. Specifically, we now have one and only one way of breaking reference cycles in the Value-graph: Thunk. No other variant contains a RefCell. This should make it easier to reason about the behavior of the VM. InnerClosure and UpvaluesCarrier are no longer necessary. This refactoring allowed an improvement in code generation: `Rc<RefCell<>>`s are now created only for closures which do not have self-references or deferred upvalues, instead of for all closures. OpClosure has been split into two separate opcodes: - OpClosure creates non-recursive closures with no deferred upvalues. The VM will not create an `Rc<RefCell<>>` when executing this instruction. - OpThunkClosure is used for closures with self-references or deferred upvalues. The VM will create a Thunk when executing this opcode, but the Thunk will start out already in the `ThunkRepr::Evaluated` state, rather than in the `ThunkRepr::Suspeneded` state. To avoid confusion, OpThunk has been renamed OpThunkSuspended. Thanks to @sterni for suggesting that all this could be done without adding an additional variant to ThunkRepr. This does however mean that there will be mutating accesses to `ThunkRepr::Evaluated`, which was not previously the case. The field `is_finalised:bool` has been added to `Closure` to ensure that these mutating accesses are performed only on finalised Closures. Both the check and the field are present only if `#[cfg(debug_assertions)]`. Change-Id: I04131501029772f30e28da8281d864427685097f Signed-off-by: Adam Joseph <adam@westernsemico.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/7019 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-17 r/5154 feat(tvix/eval): Validate closed formalsGriffin Smith1-0/+15
Validate "closed formals" (formal parameters without an ellipsis) via a new ValidateClosedFormals op, which checks the arguments (in an attr set at the top of the stack) against the formal parameters on the Lambda in the current frame, and returns a new UnexpectedArgument error (including the span of the formals themselves!!) if any arguments aren't allowed Change-Id: Idcc47a59167a83be1832a6229f137d84e426c56c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7002 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-10-16 r/5147 fix(tvix/eval): resolve home relative paths at runtimesterni1-0/+20
Home relative paths depend on the environment to be resolved. We have elected to do everything that depends on the environment, e.g. resolving SPATH expressions using NIX_PATH, at runtime, so tvix evaluation would continue to behave correctly even if we separated the compilation and execution phases more, e.g. via serializing bytecode. Then the value of HOME, NIX_PATH etc. could reasonably change in the time until execution, yielding wrong results if the resolution results were cached in the bytecode. We also take the opportunity to fix the broken path concatenation previously found in the compiler, fixing b/205. Another thing we could consider is emitting a warning for home relative path literals, as they are by nature relatively fragile. One sideeffect of this change is that home path resolution errors become catchable which is not the case in C++ Nix. This will need to be fixed up in a subsequent change. Change-Id: I30bd69b575667c49170a9fdea23a020565d0f9ec Reviewed-on: https://cl.tvl.fyi/c/depot/+/7024 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2022-10-16 r/5146 refactor(tvix/eval): make OpFindFile use internal UnresolvedPathsterni1-9/+15
To assert that OpFindFile is only emitted for specially compiled SPATH expressions, as well as make sure it doesn't accidentally operate on “ordinary values”, introduce an UnresolvedPath internal value. If OpFindFile sees a non-UnresolvedPath value, it'll crash. Note that this change is not done purely for OpFindFile: We may want to compile SPATH expressions as function calls to __findFile (like C++ Nix does) in the future, so the UnresolvedPath value would definitely need to be an ordinary string again then. Rather, this change is done in preparation for resolving home dir relative paths at runtime (since they depend on the environment) for which we'll need a similar mechanism to OpFindFile. Change-Id: I6acf287f35197cd9e13377079f972b9d36e5b22e Reviewed-on: https://cl.tvl.fyi/c/depot/+/7023 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 r/5140 refactor(tvix/eval) remove Value::DynamicUpvalueMissingAdam Joseph1-4/+1
I believe this variant is left over from a previous implementation. If not, please let me know. Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I02a3bf2f63794d09e96a5a92a034c0ad3d1ff221 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7027 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-14 r/5131 docs(tvix/eval) vm: explain VM::{frames,stack,with_stack}Adam Joseph1-2/+12
Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I94ba31ae25c1ff744f929a722c76a0c33cc361ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/7016 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-10-14 r/5129 feat(tvix/eval): eliminate the only `unsafe` in the codebaseAdam Joseph1-4/+2
Maybe I misunderstood this part of the code, but the use of `unsafe` appears unnecessary here? In any event it is the one and only `unsafe` in the codebase. Hopefully getting to "no `unsafe` anywhere" is worth the extra never-taken branch caused by unwrap() instead of unwrap_unchecked()? Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I33fbd5aad9d8307ea82c24b6220412783e1973c6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7011 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-14 r/5128 refactor(tvix/eval): remove OpResolveWithOrUpvalueAdam Joseph1-19/+0
The case branch in vm.rs for OpResolveWithOrUpvalue is unreachable/deadcode. I believe this opcode is unnecessary, since it should always be statically detectable (at parse-time) whether a reference is to an upvalue (i.e. enclosing binding); otherwise, and only then, is with-resolution applicable. Perhaps I've misunderstood how with-resolution works. If so, please explain it to me and -1/-2 this CL. Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I4a90b9eb6cb3396df92a6a943d42ecc301871ba0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7009 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
generated by cgit-pink 1.4.1 (git 2.44.2) at 2024-11-04T00·27+0000