about summary refs log tree commit diff
diff options
context:
space:
mode:
authorFlorian Klink <flokli@flokli.de>2024-05-10T05·46+0300
committerclbot <clbot@tvl.fyi>2024-05-11T14·59+0000
commitfd49dfa6ca9db0f3a026a581ae2a14f760ac251d (patch)
tree5f166d2f196c95147392943c1d36fbe7fca1e4b6
parentbdf4df1ac70088fe9ae784510223b6ead512b92d (diff)
docs(tvix/todo): update test suite section r/8109
nix_oracle is quite misleading, and fooled me into thinking this
validates all .exp files in the repo to match Nix output (which it
doesn't, that's done by verify-lang-tests).

Also, the whole test suite structure thing can use a bit more
explanation.

Change-Id: I2fadcc871843143270ad2ed9ac98de8287280e6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11609
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
-rw-r--r--tvix/docs/src/TODO.md17
1 files changed, 13 insertions, 4 deletions
diff --git a/tvix/docs/src/TODO.md b/tvix/docs/src/TODO.md
index 6644bb6bac03..50b41e3e50bd 100644
--- a/tvix/docs/src/TODO.md
+++ b/tvix/docs/src/TODO.md
@@ -10,11 +10,20 @@ Feel free to add new ideas. Before picking something, ask in `#tvix-dev` to make
 sure noone is working on this, or has some specific design in mind already.
 
 ## Cleanups
+### Nix language test suite
+ - Think about how to merge, but "categorize" `tvix_tests` in `glue` and `eval`.
+   We currently only have this split as they need a different feature set /
+   builtins.
  - move some of the rstest cases in `tvix-glue` to the `.nix`/`.exp` mechanism.
-   - Parts requiring test fixtures need some special convention.
-     Some of these also cannot be checked into the repo, like the import tests
-     adding special files to test filtering.
- - add `nix_oracle` mechanism from `tvix-eval` to `tvix-glue`.
+   Some of them need test fixtures, which cannot be represented in git (special
+   file types in the import tests for example). Needs some support from the test
+   suite to create these fixtures on demand.
+ - extend `verify-lang-tests/default.nix` mechanism to validate `tvix-eval` and
+   `tvix-glue` test cases (or the common structure above).
+ - absorb `eval/tests/nix_oracle.rs` into `tvix_tests`, or figure out why it's
+   not possible (and document) it. It looks like it's only as nix is invoked
+   with a different level of `--strict`, but the toplevel doc-comment suggests
+   its generic?
 
 ## Fixes towards correctness
  - `builtins.toXML` is missing string context. See b/398.