depot/nix/runTestsuite, branch refs/r/7267

style: format entire depot with nixpkgs-fmt

2022-01-31T16:11:53+00:00

This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni Reviewed-by: lukegb Reviewed-by: wpcarro Reviewed-by: Profpatsch Reviewed-by: kanepyork Reviewed-by: tazjin Reviewed-by: cynthia Reviewed-by: edef Reviewed-by: eta Reviewed-by: grfn

fix(nix/runTestsuite): don't crash on big test suites

2021-11-25T11:15:35+00:00

Having `prettyRes` in the execline script causes it to fail because of the argv limit if your test suite is long enough. For the succeeding one we can work around this by hashing it (since we only care that something changes if the test suite changes), in the case of the failing one where we want to print the results, we use runExecline's stdin mechanism. Change-Id: I2489f76acfbe809351f51caefe2a477328a70ee3

refactor(nix/runTestsuite): clean up runTestsuite

2021-09-17T11:47:41+00:00

* goodAss wasn't used before. Simplify it to just return a boolean, so we can use it for partitionTests later. * goodIt also returns unnecessary extra meta information which is not used. Cleaning that up makes the condition extremely small, so we can inline it into (what was) goodIts. * goodIts is just called in one place, so we can inline it into res. Change-Id: I70cf4fa3f61ce1467a2ee5319f841cdd42db6a66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3548 Tested-by: BuildkiteCI Reviewed-by: Profpatsch

feat(nix/runTestsuite): rebuild if test case list changes

2021-09-14T14:10:46+00:00

If the result of the assertions changes for a successful test suite (this happens if tests are reworded, added or removed), this makes sure the no-op derivation is rebuilt. This makes sure that test suites show up in buildkite on ocassions other than channel bumps, since they are only added to the job list if their `outPath` is missing nowadays (see cl/3427). Change-Id: Ia1050cca5eeed8b7da84c40f6154b40760a3047f Reviewed-on: https://cl.tvl.fyi/c/depot/+/3536 Tested-by: BuildkiteCI Reviewed-by: Profpatsch

fix(nix/runTestsuite): use s6-portable-utils

2021-03-26T11:01:20+00:00

Same as 221698c603dcb318c609b4d21cb2a9fada44a14c We had a bunch of instances of https://github.com/NixOS/nix/issues/2176, where nix would exit with a “killed by signal 9” error. According to Eelco in that issue, this is perfectly normal behaviour of course, and appears if the last command in a loop closes `stdout` or `stdin`, then the builder will SIGKILL it immediately. This is of course also a perfectly fine error message for that case. It turns out that mainly GNU coreutils exhibit this behaviour … Let’s see if using a more sane tool suite fixes that. Change-Id: Iaf9e542952ca36c02208a3f067f575ba978272b4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2663 Reviewed-by: Profpatsch Tested-by: BuildkiteCI

feat(nix/runTestsuite): add assertDoesNotThrow

2021-02-09T21:57:56+00:00

assertDoesNotThrow is like assertThrows, but fails if the expression throws. In that case the new unexpected-throw branch of AssertErrorContext is returned. Change-Id: I7195eb5df8965456e9ab9b69e35ec96b33f00a35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2476 Tested-by: BuildkiteCI Reviewed-by: Profpatsch

refactor(nix/runTestsuite): implement in terms of generic assertBool

2021-02-09T21:57:56+00:00

Refactor assertEq and assertThrows to be implemented in terms of a more generic assertBool to reduce code duplication and ease adding new assert types. To preserve meaningful error messages AssertResult is changed slightly: nope-eq and nope-throw have been replaced by a single nope branch which contains an AssertErrorContext which contains error information. To implement an assert assertBoolContext (which is not exposed) can be used: It takes an AssertErrorContext which is returned in case of an error and a boolean determining whether the assert was successful. The currently possible AssertErrorContext are: * should-throw: error result of assertThrows, formerly nope-throw * not-equal: error result of assertEq, formerly nope-eq Change-Id: Ifd6b3aa4187c90c3add2df63fa7c906c8f03fd2d Reviewed-on: https://cl.tvl.fyi/c/depot/+/2473 Tested-by: BuildkiteCI Reviewed-by: Profpatsch

feat(nix/runTestsuite): add assertThrows

2021-01-30T11:44:25+00:00

Uses `builtins.tryEval` to check that the expression throws when `deepSeq`-ed. Change-Id: I0d57cc37f473bb733f57a1b1c0d889084152fd2f Reviewed-on: https://cl.tvl.fyi/c/depot/+/2463 Tested-by: BuildkiteCI Reviewed-by: sterni

fix(nix/runTestsuite): wrap runTestsuite into derivation

2021-01-30T11:44:25+00:00

Previously we would throw or return `{}`, which doesn’t integrate nicely into our CI; thus, let’s wrap it into a derivation which either fails the build or doesn’t. Change-Id: I65880d86b8393094661e57a0b32aafe748bf1dd5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2462 Reviewed-by: sterni Tested-by: BuildkiteCI

feat(nix/runTestsuite): add runTestsuite

2020-06-28T03:52:40+00:00

This is a very simple test suite for nix expressions. It should help us set up a good suite of unit tests for our nix-based stuff. Since we allow import from derivation, these tests can also depend on derivations and e.g. use `builtins.readFile` to check outputs. This is a first PoC to get us going, we can always replace it by something different in the future if we don’t like it. Change-Id: I206c7b624db2b1dabd9c73ffce4f87e658919958 Reviewed-on: https://cl.tvl.fyi/c/depot/+/662 Reviewed-by: tazjin Tested-by: tazjin