about summary refs log tree commit diff
path: root/tvix/eval/src/tests
AgeCommit message (Collapse)AuthorFilesLines
2023-06-15 r/6308 fix(tvix/eval): make tvix display values like nix-instantiate(1)sterni6-0/+55
In order for the test suite we have currently to be comparable to C++ Nix, we need to display values in the same way. This was largely the case except in some weird cases. * <CODE> for thunks and <CYCLE> for repeated thunks (?) are already in use. <CODE> formatting is tested by the oracle test suite already. * Instead of lambda, we need to use <LAMBDA> * <<primop>> and <<primop-app>> (a formatting C++ Nix uses nowhere) now are <PRIMOP> and <PRIMOP-APP>. We'll probably want to have a fancier display of values (in a separate trait) down the line. This could be used for interactive usage, e.g. the REPL or a potential debugger. There is a peculiarity with C++ Nix 2.3 formatting primops: import is considered a <<PRIMOP-APP>>, since it is internally implemented by means of scopedImport. This implementation detail no longer leaks in C++ Nix 2.13 nor in Tvix. <CYCLE> display is untested at the moment, since we exhibit a discrepancy to C++ Nix 2.3. Our current detection is more similar to C++ Nix 2.13—luckily it is also the more consistent of the two. See also b/245. Change-Id: I1d534434b02e470bf5475b3758920ea81e3420dc Reviewed-on: https://cl.tvl.fyi/c/depot/+/8760 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2023-06-14 r/6300 test(tvix/eval): move division by zero tests into tvix_testssterni2-0/+0
These were added by us in r/5276, so they should go into our test suite. Change-Id: I6dc74fc242f33c22a17e0b4aee546ccae886ac85 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8774 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-14 r/6299 test(tvix/eval): add test case for builtins set pointer equalitysterni2-0/+21
Unsupported by Tvix at the moment. Documents b/280. Change-Id: I48844feeefa9da8ed7e5d85300d52bb5650f82d2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8772 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-14 r/6298 test(tvix/eval): re-enable blackhole teststerni1-0/+0
Change-Id: Id933f3bd708aa3342b9fd6a5584e65ee11751ff8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8773 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org>
2023-06-12 r/6268 test(tvix/eval): builtins.substring's behavior with negative argssterni3-0/+9
Change-Id: Ie8b97d174e9d58e33bf08c9b9e0afeeddd089ba8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8700 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-06-07 r/6243 fix(tvix/eval): type check function argument with set patternsterni1-0/+2
C++ Nix forces and typechecks the passed argument even if it is not necessary in order to compute the return value of the function. I discovered this when I thought our formals miscompilation might be that we are too strict, but doesn't look like it in this case. Change-Id: Ifb3c92592293052c489d1e3ae8c7c54e4b6b4dc6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8701 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-26 r/6207 fix(tvix): don't call function eagerly in genList, map & mapAttrssterni2-0/+32
mapAttrs, map and genList call Nix functions provided by the caller and store the result of applying them in a Nix data structure that does not force all of its contents when forced itself. This means that when such a builtin application is forced, the Nix function calls performed by the builtin should not be forced: They may be forced later, but it is also possible that they will never be forced, e.g. in builtins.length (builtins.map (builtins.add 2) [ 1 2 3 ]) it is not necessary to compute a single application of builtins.add. Since request_call_with immediately performs the function call requested, Tvix would compute function applications unnecessarily before this change. Because this was not followed by a request_force, the impact of this was relatively low in Nix code (most functions return a new thunk after being applied), but it was enough to cause a lot of bogus builtins.trace applications when evaluating anything from `lib.modules`. The newly added test includes many cases where Tvix previously incorrectly applied a builtin, breaking a working expression. To fix this we add a new helper to construct a Thunk performing a function application at runtime from a function and argument given as `Value`s. This mimics the compiler's compile_apply(), but does itself not require a compiler, since the necessary Lambda can be constructed independently. I also looked into other builtins that call a Nix function to verify that they don't exhibit such a problem: - Many builtins immediately use the resulting value in a way that makes it necessary to compute all the function calls they do as soon as the outer builtin application is forced: * all * any * filter * groupBy * partition - concatMap needs to (shallowly) force the returned list for concatenation. - foldl' is strict in the application of `op` (I added a comment that makes this explicit). - genericClosure needs to (shallowly) force the resulting list and some keys of the attribute sets inside. Resolves b/272. Change-Id: I1fa53f744bcffc035da84c1f97ed25d146830446 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8651 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-22 r/6037 feat(tvix/eval): add Evaluation::strict to toggle top-level deepseqVincent Ambo1-0/+2
This makes it possible for callers to control whether they can receive partially evaluated values from an evaluation or not. We're actually flipping the default behaviour to non-strict top-level evaluation, which means that callers have to set `strict = true` on the Evaluation to get the previous behaviour. Change-Id: Ic048e9ba09c88866d4c3177d5fa07db11c4eb20e Reviewed-on: https://cl.tvl.fyi/c/depot/+/8325 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-03-17 r/6022 fix(tvix/eval): use coerce_to_string in builtins.substringVincent Ambo2-0/+6
This actually uses coercion under the hood in C++ Nix. See the test for an example. Change-Id: Id56b364acf269225b6829d0b600e0222f8b3608d Reviewed-on: https://cl.tvl.fyi/c/depot/+/8322 Reviewed-by: andi <andi@notmuch.email> Tested-by: BuildkiteCI
2023-03-13 r/5979 fix(tvix/eval): implement cppnix JSON-serialisation semanticsVincent Ambo2-0/+0
This drops the usage of serde::Serialize, as the trait can not be used to implement the correct semantics (function colouring!). Instead, a manual JSON serialisation function is written which correctly handles toString, outPath and other similar weirdnesses. Unexpectedly, the eval-okay-tojson test from the C++ Nix test suite now passes, too. This fixes an issue where serialising data structures containing derivations to JSON would fail. Change-Id: I5c39e3d8356ee93a07eda481410f88610f6dd9f8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8209 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-13 r/5977 fix(tvix/eval): handle toJSON on attribute sets with `outPath`Vincent Ambo4-0/+15
These are serialised as the serialisation of the value of that field. Change-Id: Ida51708b1f43ce09b0ec835f4e265918aa31dd09 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8205 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-13 r/5976 fix(tvix/eval): handle `__toString` when JSON-serialising attrsetsVincent Ambo4-0/+20
These must be serialised to a JSON string of the *result* of coercing the function application to a string. Change-Id: Ib7f49ccd950503ddbdbf99643cd59565e26b50da Reviewed-on: https://cl.tvl.fyi/c/depot/+/8204 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2023-03-13 r/5970 fix(tvix/eval): correctly thunk deferred formals accessVincent Ambo2-0/+7
Formals can be initialised with deferred default values (see the test cases), in which case they need an extra thunk to have something that can be finalised appropriately when the setup is done. Fixes: b/255 Change-Id: I380e3770be68eaa83ace96d450c7cead32dacc9f Reviewed-on: https://cl.tvl.fyi/c/depot/+/8196 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-13 r/5965 test(tvix/eval): add test for infinite recursion detectionVincent Ambo1-0/+1
Change-Id: Ief20544a44c3542fe40a5c09f81d0f064a346f44 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8149 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-13 r/5964 refactor(tvix/eval): flatten call stack of VM using generatorsVincent Ambo1-3/+3
Warning: This is probably the biggest refactor in tvix-eval history, so far. This replaces all instances of trampolines and recursion during evaluation of the VM loop with generators. A generator is an asynchronous function that can be suspended to yield a message (in our case, vm::generators::GeneratorRequest) and receive a response (vm::generators::GeneratorResponsee). The `genawaiter` crate provides an interpreter for generators that can drive their execution and lets us move control flow between the VM and suspended generators. To do this, massive changes have occured basically everywhere in the code. On a high-level: 1. The VM is now organised around a frame stack. A frame is either a call frame (execution of Tvix bytecode) or a generator frame (a running or suspended generator). The VM has an outer loop that pops a frame off the frame stack, and then enters an inner loop either driving the execution of the bytecode or the execution of a generator. Both types of frames have several branches that can result in the frame re-enqueuing itself, and enqueuing some other work (in the form of a different frame) on top of itself. The VM will eventually resume the frame when everything "above" it has been suspended. In this way, the VM's new frame stack takes over much of the work that was previously achieved by recursion. 2. All methods previously taking a VM have been refactored into async functions that instead emit/receive generator messages for communication with the VM. Notably, this includes *all* builtins. This has had some other effects: - Some test have been removed or commented out, either because they tested code that was mostly already dead (nix_eq) or because they now require generator scaffolding which we do not have in place for tests (yet). - Because generator functions are technically async (though no async IO is involved), we lose the ability to use much of the Rust standard library e.g. in builtins. This has led to many algorithms being unrolled into iterative versions instead of iterator combinations, and things like sorting had to be implemented from scratch. - Many call sites that previously saw a `Result<..., ErrorKind>` bubble up now only see the result value, as the error handling is encapsulated within the generator loop. This reduces number of places inside of builtin implementations where error context can be attached to calls that can fail. Currently what we gain in this tradeoff is significantly more detailed span information (which we still need to bubble up, this commit does not change the error display). We'll need to do some analysis later of how useful the errors turn out to be and potentially introduce some methods for attaching context to a generator frame again. This change is very difficult to do in stages, as it is very much an "all or nothing" change that affects huge parts of the codebase. I've tried to isolate changes that can be isolated into the parent CLs of this one, but this change is still quite difficult to wrap one's mind and I'm available to discuss it and explain things to any reviewer. Fixes: b/238, b/237, b/251 and potentially others. Change-Id: I39244163ff5bbecd169fe7b274df19262b515699 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8104 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2023-02-13 r/5848 fix(tvix/eval): skip runtime completely on compiler errorsVincent Ambo1-0/+12
This branch was missing, and an assumption elsewhere just executed the returned (broken) bytecode. This fixes b/253. Change-Id: I015023ba921bc08ea03882167f1f560feca25e50 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8090 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su>
2023-02-03 r/5828 fix(tvix/eval): ensure all evaluated thunks are correctly memoizedVincent Ambo1-1/+6
This fixes a very complicated bug (b/246). Evaluation progresses *much* further after this, leading to several less complicated bugs likely being uncovered by this What was the problem? ===================== Previously, when evaluating a thunk, we had a code path that looked like this: match *thunk { ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => { let inner_repr = inner_thunk.0.borrow().clone(); drop(thunk); self.0.replace(inner_repr); } /* ... */ } This code path created a copy of the inner `ThunkRepr` of a nested thunk, and moved that copy into the `ThunkRepr` of the parent. The effect of this was that the original `ThunkRepr` (unforced!) lived on in the original thunk, without the memoization of the subsequent forcing applying to it. This had the result that Tvix would repeatedly evaluate these thunks without ever memoizing them, if they occured repeatedly as shared inner thunks. Most notably, this would *always* occur when builtins.import was used. What's the solution? ==================== I have completely rewritten `Thunk::force_trampoline_self` to make all flows that can occur in it explicit. I have also removed the outer loop inside of that function, and resorted to more use of trampolining instead. The function is now well-commented and it should be possible to read it from top-to-bottom and get a general sense of what is going on, though the trampolining itself (which is implemented in the VM) needs to be at least partially understood for this. What's the new problem(s)? ========================== One new (known) problem is that we have to construct `Error` instances in all error types here, but we do not have spans available in some thunk-related situations. Due to b/238 we cannot ask the VM for an arbitrary span from the callsite leading to the force. This means that there are now code paths where, under certain conditions, causing an evaluation error during thunk forcing will panic. To fix this we will need to investigate and fix b/238, and/or add a span tracking mechanism to thunks themselves. What other impacts does this have? ================================== With this commit, eval of nixpkgs mostly succeeds (things like stdenv evaluate to the same hashes for us and C++ Nix, meaning we now construct identical derivations without eval breaking). Due to this we progress much further into nixpkgs, which lets us uncover more additional bugs. For example, after this commit we can quickly see that cl/7949 introduces some kind of behavioural issue and should not be merged as-is (this was not apparent before). Additionally, tvix-eval is now seemingly very fast. When doing performance analysis of a nixpkgs eval, we now mostly see the code path for shelling out to C++ Nix to add things to the store in there. We still need those code paths, so we can not (yet) do a performance analysis beyond that. Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 r/5784 test(tvix/eval): add tests for internal formals dependenciesVincent Ambo4-0/+8
Formals can depend on each other when using another formal as a default value. This test ensures that the compiler's declaration and initialisation order of formals is consistent with what actually happens in the VM. Change-Id: Ibdabe262554e8066d67fac1ebc3b5a48ef626e18 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7948 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-25 r/5754 feat(tvix/eval): implement builtins.fromTOMLFlorian Klink2-0/+0
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML testcase from nix_tests. It uses the `toml` crate, and the serde integration it brings with it. Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-25 r/5753 feat(tvix/eval): use lexical-core to format floatFlorian Klink2-2/+2
Apparently our naive implementation of float formatting, which simply used {:.5}, and trimmed trailing "0" strings not sufficient. It wrongly trimmed numbers with zeroes but no decimal point, like `10000` got trimmed to `1`. Nix uses `std::to_string` on the double, which according to https://en.cppreference.com/w/cpp/string/basic_string/to_string is equivalent to `std::sprintf(buf, "%f", value)`. https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated like this: > Precision specifies the exact number of digits to appear after > the decimal point character. The default precision is 6. In the > alternative implementation decimal point character is written even if > no digits follow it. For infinity and not-a-number conversion style > see notes. This doesn't seem to be the case though, and Nix uses scientific notation in some cases. There's a whole bunch of strategies to determine which is a more compact notation, and which notation should be used for a given number. https://github.com/rust-lang/rust/issues/24556 provides some pointers into various rabbit holes for those interested. This gist seems to be that currently a different formatting is not exposed in rust directly, at least not for public consumption. There is the [lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate though, which provides a way to format floats with various strategies and formats. Change our implementation of `TotalDisplay` for the `Value::Float` case to use that. We still need to do some post-processing, because Nix always adds the sign in scientific notation (and there's no way to configure lexical-core to do that), and lexical-core in some cases keeps the trailing zeros. Even with all that in place, there as a difference in `eval-okay- fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated the fixture to a less problematic number. With this, the testsuite passes again, and does for the upcoming CL introducing builtins.fromTOML, and enabling the nix testsuite bits for it, too. Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-23 r/5746 fix(tvix/eval): force functors before applying themVincent Ambo2-0/+8
call_value in the VM expects the callable to be forced when calling it, which was not the case for functors. Change-Id: Id55a2fe32a9573be42aef8669e268df519a989cd Reviewed-on: https://cl.tvl.fyi/c/depot/+/7909 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-22 r/5735 feat(tvix/eval): support builtins implemented in Nix itselfVincent Ambo2-0/+23
This makes it possible to inject builtins into the builtin set that are written in Nix code, and which at runtime are represented by a thunk that will compile them the first time they are used. Change-Id: Ia632367328f66fb2f26cb64ae464f8f3dc9c6d30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7891 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 r/5707 refactor(tvix/eval): directly return builtin tuples from macroVincent Ambo1-5/+1
All invocations of the builtin macro had to previously filter through the `builtin_tuple` function, but it's more sensible to directly return these from the macro. Change-Id: I45600ba84d56c9528d3e92570461c319eea595ce Reviewed-on: https://cl.tvl.fyi/c/depot/+/7825 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 r/5674 feat(tvix/eval): add builtins to builtinsVincent Ambo2-0/+2
This is a somewhat terrifying hack that enables us to support `builtins.builtins`, by running a "fake compilation" inside of a suspended native thunk that can resolve the weak pointer to the globals. With this implementation, the thunk at `builtins.builtins` actually resolves to the "real" `builtins` (verified with a new test). This is kind of ugly, and it's something users shouldn't use, but bubbling a warning out of this is difficult at the moment due to a little bit of trickery with how the spans in suspended native thunks work (they don't) (see b/237, b/238) Change-Id: I67d0e93246dd5b279c960aeda00402031aa12af3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7748 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-16 r/5664 feat(tvix/eval): implement builtins.toXMLVincent Ambo4-0/+0
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-12 r/5652 feat(tvix/eval): implement builtins.toJSONVincent Ambo4-0/+22
Implements `Serialize` for `tvix_eval::Value`. Special care is taken with serialisation of attribute sets, and forcing of thunks. The tests should cover both cases well. Change-Id: I9bb135bacf6f87bc6bd0bd88cef0a42308e6c335 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7803 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su>
2023-01-10 r/5640 feat(tvix/eval): implement serde::Deserialize for ValueRyan Lahfa2-1/+2
Co-Authored-By: Vincent Ambo <tazjin@tvl.su> Change-Id: Ib6f7d1f4f4faac36b44f5f75cccc57bf912cf606 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7626 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-07 r/5626 fix(tvix/eval): fix typo'd function name in testsVincent Ambo1-1/+1
Caught by sterni on cl/7783. Change-Id: I15d57b893ef22538fdd7e809f3b92861dd2bc1af Reviewed-on: https://cl.tvl.fyi/c/depot/+/7789 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-06 r/5619 refactor(tvix/eval): move mocked builtins.derivation to testsVincent Ambo1-2/+37
This placeholder should not live in the main crate anymore as we will be injecting the real one from outside of eval, but there are still language tests that depend on a (simple, mockable) version of it. Change-Id: I68ea169db15cbdbeed320930d3069e21e376c90d Reviewed-on: https://cl.tvl.fyi/c/depot/+/7783 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-01-06 r/5602 feat(tvix/eval): implement initial compiler::optimiser moduleVincent Ambo2-0/+22
This optimiser can rewrite some expressions into more efficient forms, and warn users about those cases. As a proof-of-concept, only some simple boolean comparisons are supported for now. Change-Id: I7df561118cfbad281fc99523e859bc66e7a1adcb Reviewed-on: https://cl.tvl.fyi/c/depot/+/7766 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 r/5594 test(tvix/eval): add test for builtins paritysterni3-0/+41
This will eventually force us to have a base builtins set in common with C++ Nix, i.e. all 2.3 builtins except the controversial builtins.valueSize. Change-Id: I2c767f07d6a14711911658e87da9f18ede57a143 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7747 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 r/5582 fix(tvix/eval): ' is allowed in nonfirst position in Nix identifierssterni2-0/+31
With this is_valid_nix_identifier should line up with the upstream lexer definition: ID [a-zA-Z\_][a-zA-Z0-9\_\'\-]* While we're working on this, add a simple test checking the various formatting rules. Interestingly, it would not be suitable as an identity test, since you have to write { "assert" = null; } in order to avoid an evaluation error, but C++ Nix is happy to print this as { assert = null; } – maybe should be considered to be a bug. Change-Id: I0a4e1ccb5033a80f3767fb8d1c4bba08d303c5d8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7744 Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 r/5581 refactor(tvix/eval): streamline construction of globals/builtinsVincent Ambo1-2/+1
Previously the construction of globals (a compiler-only concept) and builtins (a (now) user-facing API) was intermingled between multiple different modules, and kind of difficult to understand. The complexity of this had grown in large part due to the implementation of `builtins.import`, which required the notorious "knot-tying" trick using Rc::new_cyclic (see cl/7097) for constructing the set of globals. As part of the new `Evaluation` API users should have the ability to bring their own builtins, and control explicitly whether or not impure builtins are available (regardless of whether they're compiled in or not). To streamline the construction and allow the new API features to work, this commit restructures things by making these changes: 1. The `tvix_eval::builtins` module is now only responsible for exporting sets of builtins. It no longer has any knowledge of whether or not certain sets (e.g. only pure, or pure+impure) are enabled, and it has no control over which builtins are globally available (this is now handled in the compiler). 2. The compiler module is now responsible for both constructing the final attribute set of builtins from the set of builtins supplied by a user, as well as for populating its globals (that is identifiers which are available at the top-level scope). 3. The `Evaluation` API now carries a `builtins` field which is populated with the pure builtins by default, and can be extended by users. 4. The `import` feature has been moved into the compiler, as a special case. In general, builtins no longer have the ability to reference the "fix point" of the globals set. This should not change any functionality, and in fact preserves minor differences between Tvix/Nix that we already had (such as `builtins.builtins` not existing). Change-Id: Icdf5dd50eb81eb9260d89269d6e08b1e67811a2c Reviewed-on: https://cl.tvl.fyi/c/depot/+/7738 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-12-29 r/5541 refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrsVincent Ambo1-5/+5
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory between different iterations of a map. This slightly speeds up eval, but not significantly. Future work might include benchmarking whether using a `HashMap` and only ordering in cases where order is actually required would help. This switches to a fork of `im` that fixes some bugs with its OrdMap implementation. Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-12-25 r/5486 fix(tvix/eval): fix current clippy warningsVincent Ambo2-8/+6
It's been a while since the last time, so quite a lot of stuff has accumulated here. Change-Id: I0762827c197b30a917ff470fd8ae8f220f6ba247 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7597 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-21 r/5459 feat(tvix/eval): add EvalIO to public crate APIVincent Ambo1-2/+8
This lets users set the `io_handle` field on an `Evaluation`, which is then propagated to the VM. Change-Id: I616d7140724fb2b4db47c2ebf95451d5303a487a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7566 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-21 r/5439 refactor(tvix/eval): use new public API in test codeVincent Ambo1-39/+44
This removes internal uses of the previous crate::eval module, which is being removed. Change-Id: I5fb3c53460a9c5381853d0258f9ed074ab23c630 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7543 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi>
2022-12-15 r/5420 refactor(tvix): build Rust projects using crate2nixVincent Ambo5-0/+0
Introduces granular dependency builds using crate2nix, bootstrapped off the generated configuration from the newly introduced workspace (see cl/7533). This commit checks in the generated Cargo.nix file which can be regenerated with a parameterless invocation of `crate2nix generate` in `//tvix`. I tried generating this in IFD, but it turned out to be harder than what seemed worthwhile for now. In this setup, the various build targets for Rust projects end up being attributes of the imported `Cargo.nix` file at the `tvix.crates` attribute. These still lack configuration, however, which has been fixed in the various `default.nix` files of individual projects. Note that we (temporarily) lose the ability to build tvix-eval's benchmarks in CI. I haven't figured out what magic incantation summons them from the void again ... The `eval-okay-readDir` tests from both test suites have been disabled because they fail for unknown reasons when run in this new derivation. Somebody will have to debug it! Change-Id: I2014614ccb9c8951aedbd71df7966ca191a13695 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7538 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2022-12-02 r/5374 test(tvix/eval): check inner forcing despite declaring pointersterni2-0/+12
Maybe counter-intuitively the inner elements of a list or the attribute values of an attribute set will be forced despite pointer equality (but only one layer deep). Change-Id: I485d96452fb56f5fb342d39039c9137725b33d3f Reviewed-on: https://cl.tvl.fyi/c/depot/+/7371 Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2022-12-02 r/5373 test(tvix/eval): add a test for repeated keys in listToAttrsVincent Ambo2-0/+14
This came up in the Nix Language channel today and I thought it warranted a test case. We did actually implement this correctly. Change-Id: I4b37c92d06eb6e3a7f59ea3d10af38f2b0a93d53 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7493 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-02 r/5369 test(tvix/eval): verify pointer equality checkssterni2-0/+47
Change-Id: I9baf2810fbd5b6ee8bfe10fce5b64801a35c1d67 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7369 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: Adam Joseph <adam@westernsemico.com> Tested-by: BuildkiteCI
2022-12-02 r/5368 test(tvix/eval): verify pointer equality in list comparisonssterni2-0/+7
Change-Id: I617d402c8ecc7aaf607c4bdcd58a06ebddb71fac Reviewed-on: https://cl.tvl.fyi/c/depot/+/7370 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-11-26 r/5339 fix(tvix/eval): OpAdd must weakly stringify if either arg is stringAdam Joseph2-0/+8
Tests included. Change-Id: I7a4905d6103813373e383e2e8629c5fd243d6bca Reviewed-on: https://cl.tvl.fyi/c/depot/+/7377 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: Adam Joseph <adam@westernsemico.com>
2022-11-25 r/5310 test(tvix/eval): add eval-okay-closure-pointer-compareAdam Joseph2-0/+15
This test case checks two things: * A sanity check that "pointer equality for functions" means not just the lambda, but also the upvalues. * To be pointer-equal, it is not enough for the upvalues to be normal-form equal (i.e. `nix_eq()`-equal); the upvalues must be *pointer*-equal. The second part of the test case checks for this. Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I4e59327a6f199b8212e97197b212e3c3934bb3f0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7372 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-11-24 r/5308 test(tvix/eval): test limits of builtins.seq's forcingsterni2-2/+2
Change-Id: I6dfc9108220762ef3372cd2739e91d79c01a55e1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7366 Tested-by: BuildkiteCI Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su>
2022-11-21 r/5296 test(tvix/eval): enable eval-okay-closure test from nix_testsVincent Ambo2-1/+2
This function covers builtins.genericClosure, seemingly including weird behaviour around the order in which the work set is processed. For some reason, in C++ Nix the test expectation is written in XML which we do not yet support, so I have created a new expectation file using `nix-instantiate --eval --strict` on the file (yes, using C++ Nix). Change-Id: Id90e7117d120dc66d963a51083c4d8e8f2d9f181 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7311 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-11-10 r/5276 feat(tvix/eval): detect division by zerojhahn2-0/+2
This detects if the second argument of a division is a zero (either as integer or as float). If so, an error message is displayed. This fixes b/219. Change-Id: I50203d14a71482bc757832a2c8dee08eb7d35c49 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7258 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2022-11-06 r/5255 feat(tvix/eval): placeholder builtin implementationsVincent Ambo2-0/+0
Adds initial placeholders for builtins.{derivation, unsafeDiscardStringContext}. Change-Id: I67a126c9b9f9f4f11e2256e69b9a32ebd9eb1b0e Reviewed-on: https://cl.tvl.fyi/c/depot/+/7187 Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-11-04 r/5244 feat(tvix/eval): implement builtins.splitAdam Joseph2-0/+0
This implements builtins.split, and passes eval-okay-regex-split.nix (which is moved out of notyetpassing). Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: Ieb0975da2058966c697ee0e2f5b3f26ccabfae57 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7143 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-11-04 r/5242 test(tvix/eval): add a test case for groupBy with thunksAdam Joseph2-0/+7
We have to be careful implementing `builtins.groupBy`, since the list may contain thunks, and tvix's to_xxx() functions do not work on thunks. Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I182b6fc2d4296f864ed16744ef70b153e8e6978a Reviewed-on: https://cl.tvl.fyi/c/depot/+/7039 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI