about summary refs log tree commit diff
path: root/tvix/eval/src/tests/mod.rs (follow)
AgeCommit message (Collapse)AuthorFilesLines
2022-10-26 r/5199 feat(tvix/eval): include filename of failing test when failingAdam Joseph1-5/+5
Unfortunately we have to mangle test case filenames into rust-valid symbols, since test-generator doesn't use `r#"..."` (deliberately?). This means that when a test fails, there's nothing on the console you can copy-and-paste in order to view/edit the code of the failing test case. This commit (partially) fixes it by including the unmangled name in the panic!() string. However failures due to panic!()s inside the vm (including deliberate panics due to panic!()-debugging) still won't display an unmangled filename. Maybe we should reconsider the use of test-generator? Change-Id: I2208a859ffab1264f17f48fd303ff5e19675967e Signed-off-by: Adam Joseph <adam@westernsemico.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/7092 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-20 r/5168 fix(tvix/eval): restore .exp.xml files and skip in test suitesterni1-0/+7
Change-Id: Iebda5e0d99925a0a8c1d6ae1d7a35397d127bf31 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7047 Autosubmit: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-10-19 r/5161 feat(tvix/eval): expect not-yet-passing tests to failAdam Joseph1-14/+48
It is helpful to be able to use the test suite as a regression test: make a change to the compiler/vm, re-run the tests, and if there are any failures you know it's your fault. Right now we can't do that, because the expected-to-fail tests are mixed in with the expected-to-pass tests. So we can't use them as a regression test. Change-Id: Ied606882b9835a7effd7e75bfcf3e5f827e0a2c8 Signed-off-by: Adam Joseph <adam@westernsemico.com> Reviewed-on: https://cl.tvl.fyi/c/depot/+/7036 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-10-13 r/5123 feat(tvix/eval): enable the XFAIL testsAdam Joseph1-14/+45
This commit adds support for running the "expected failure" tests in both the nix and tvix test suites. I have disabled the eval-fail-blackhole.nix test because it gets stuck running forever. Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: Iba75ce6c8f2becab3c834fcfdd9f4fdc5a4bdb9f Reviewed-on: https://cl.tvl.fyi/c/depot/+/6990 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi>
2022-10-13 r/5117 feat(tvix/eval): allow to disable warningsAdam Joseph1-2/+3
The nix_tests test suite produces lots of warnings. We can't fix these, since they are kept in sync with upstream, so there's little point in cluttering up the console with them every time the tests are run. Let's add a clap flag "warnings" and TVIX_WARNINGS environment variable. The default is "true". The test runner overrides this default and mutes the warnings. Signed-off-by: Adam Joseph <adam@westernsemico.com> Change-Id: I4b065f96fe15838afcca6970491a54e248ae4df7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6985 Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-10-08 r/5068 feat(tvix/tests): Import default.nix inside directoryGriffin Smith1-1/+1
This requires actually passing the source directory into `interpret` in the eval tests, but otherwise this is fairly straightforward - if we're trying to import a directory, just push `default.nix` onto it and import that instead. Change-Id: I0b7d4234f81977e78d14dfa651bf0cf9721017e5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6893 Autosubmit: grfn <grfn@gws.fyi> Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-09-18 r/4911 refactor(tvix/eval): use Clap for arg+env parsingGriffin Smith1-2/+4
Refactor the environment variable and argument parsing for the tvix repl to use Clap instead of doing things ad-hoc, and thread through options obtained from environment variables via explicit arguments rather than obtaining them from the environment as they're needed. This makes adding more flags more sustainable, and also makes the binary fully self-documenting, including supported env vars, via `--help`. Change-Id: Ib1f6a0cd20056e8c9196760ff755fa5729667760 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6653 Autosubmit: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-01 r/4563 refactor(tvix/eval): use pretty_assertions for testsVincent Ambo1-4/+5
This makes for much more readable output especially when long strings are involved. Change-Id: I43dd73a0480535d7181a760788c42883a9b083f8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6229 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-08-28 r/4519 feat(tvix/eval): resolve relative path literalsVincent Ambo1-2/+2
Resolves relative paths (e.g. `./foo`) either relative to the location of the Nix file, or relative to the working directory if none is supplied. Change-Id: I70ec574657b221b458015117a004b6e4a9c25a30 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6185 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-08-25 r/4470 test(tvix/eval): add some eval-okay-* tests for trivial typesVincent Ambo1-16/+24
Change-Id: I85ccc07e08c67abf4fcd3752c58e1702943239ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/6135 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2022-08-24 r/4467 test(tvix/eval): add identity tests for literal evaluationVincent Ambo1-2/+20
Change-Id: Id3f37fbe341e15e9428ef1d579d61a514232c0e8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/6132 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-08-24 r/4464 chore(tvix/tests): gate Nix test suite behind `nix_tests` featureVincent Ambo1-0/+1
Once we have full coverage they should be enabled by default. Change-Id: Iace9e1ae9a9f901a0979ad336434004b8028fe8a Reviewed-on: https://cl.tvl.fyi/c/depot/+/6129 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi>
2022-08-24 r/4461 feat(tvix/tests): check in Nix' language test suiteVincent Ambo1-0/+28
This adds scaffolding code for running the Nix language test suite. The majority of eval-okay-* tests should eventually be runnable as-is by Tvix, however the eval-fail-* tests might not as we intend to have more useful error messages than upstream Nix. Change-Id: I4f3227f0889c55e4274b804a3072850fb78dd1bd Reviewed-on: https://cl.tvl.fyi/c/depot/+/6126 Tested-by: BuildkiteCI Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi>