about summary refs log tree commit diff
path: root/nix/runTestsuite (follow)
AgeCommit message (Collapse)AuthorFilesLines
2022-01-31 r/3723 style: format entire depot with nixpkgs-fmtVincent Ambo1-28/+48
This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2021-11-25 r/3093 fix(nix/runTestsuite): don't crash on big test suitessterni1-8/+9
Having `prettyRes` in the execline script causes it to fail because of the argv limit if your test suite is long enough. For the succeeding one we can work around this by hashing it (since we only care that something changes if the test suite changes), in the case of the failing one where we want to print the results, we use runExecline's stdin mechanism. Change-Id: I2489f76acfbe809351f51caefe2a477328a70ee3
2021-09-17 r/2881 refactor(nix/runTestsuite): clean up runTestsuitesterni1-16/+6
* goodAss wasn't used before. Simplify it to just return a boolean, so we can use it for partitionTests later. * goodIt also returns unnecessary extra meta information which is not used. Cleaning that up makes the condition extremely small, so we can inline it into (what was) goodIts. * goodIts is just called in one place, so we can inline it into res. Change-Id: I70cf4fa3f61ce1467a2ee5319f841cdd42db6a66 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3548 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-09-14 r/2857 feat(nix/runTestsuite): rebuild if test case list changessterni1-2/+5
If the result of the assertions changes for a successful test suite (this happens if tests are reworded, added or removed), this makes sure the no-op derivation is rebuilt. This makes sure that test suites show up in buildkite on ocassions other than channel bumps, since they are only added to the job list if their `outPath` is missing nowadays (see cl/3427). Change-Id: Ia1050cca5eeed8b7da84c40f6154b40760a3047f Reviewed-on: https://cl.tvl.fyi/c/depot/+/3536 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-03-26 r/2336 fix(nix/runTestsuite): use s6-portable-utilsProfpatsch1-2/+3
Same as 221698c603dcb318c609b4d21cb2a9fada44a14c We had a bunch of instances of https://github.com/NixOS/nix/issues/2176, where nix would exit with a “killed by signal 9” error. According to Eelco in that issue, this is perfectly normal behaviour of course, and appears if the last command in a loop closes `stdout` or `stdin`, then the builder will SIGKILL it immediately. This is of course also a perfectly fine error message for that case. It turns out that mainly GNU coreutils exhibit this behaviour … Let’s see if using a more sane tool suite fixes that. Change-Id: Iaf9e542952ca36c02208a3f067f575ba978272b4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2663 Reviewed-by: Profpatsch <mail@profpatsch.de> Tested-by: BuildkiteCI
2021-02-09 r/2198 feat(nix/runTestsuite): add assertDoesNotThrowsterni1-0/+7
assertDoesNotThrow is like assertThrows, but fails if the expression throws. In that case the new unexpected-throw branch of AssertErrorContext is returned. Change-Id: I7195eb5df8965456e9ab9b69e35ec96b33f00a35 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2476 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-02-09 r/2197 refactor(nix/runTestsuite): implement in terms of generic assertBoolsterni1-26/+46
Refactor assertEq and assertThrows to be implemented in terms of a more generic assertBool to reduce code duplication and ease adding new assert types. To preserve meaningful error messages AssertResult is changed slightly: nope-eq and nope-throw have been replaced by a single nope branch which contains an AssertErrorContext which contains error information. To implement an assert assertBoolContext (which is not exposed) can be used: It takes an AssertErrorContext which is returned in case of an error and a boolean determining whether the assert was successful. The currently possible AssertErrorContext are: * should-throw: error result of assertThrows, formerly nope-throw * not-equal: error result of assertEq, formerly nope-eq Change-Id: Ifd6b3aa4187c90c3add2df63fa7c906c8f03fd2d Reviewed-on: https://cl.tvl.fyi/c/depot/+/2473 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-01-30 r/2162 feat(nix/runTestsuite): add assertThrowsProfpatsch1-4/+22
Uses `builtins.tryEval` to check that the expression throws when `deepSeq`-ed. Change-Id: I0d57cc37f473bb733f57a1b1c0d889084152fd2f Reviewed-on: https://cl.tvl.fyi/c/depot/+/2463 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org>
2021-01-30 r/2161 fix(nix/runTestsuite): wrap runTestsuite into derivationProfpatsch1-8/+26
Previously we would throw or return `{}`, which doesn’t integrate nicely into our CI; thus, let’s wrap it into a derivation which either fails the build or doesn’t. Change-Id: I65880d86b8393094661e57a0b32aafe748bf1dd5 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2462 Reviewed-by: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2020-06-28 r/1104 feat(nix/runTestsuite): add runTestsuiteProfpatsch1-0/+121
This is a very simple test suite for nix expressions. It should help us set up a good suite of unit tests for our nix-based stuff. Since we allow import from derivation, these tests can also depend on derivations and e.g. use `builtins.readFile` to check outputs. This is a first PoC to get us going, we can always replace it by something different in the future if we don’t like it. Change-Id: I206c7b624db2b1dabd9c73ffce4f87e658919958 Reviewed-on: https://cl.tvl.fyi/c/depot/+/662 Reviewed-by: tazjin <mail@tazj.in> Tested-by: tazjin <mail@tazj.in>