Alcotezt UX improvements
The Alcotezt project has incurred some usability regressions on unit test execution.
Problems
Notably, it is no longer possible, or hard to:
- (A) Select all tests in a specific folder / package / file
- (B) Selecting sub-tests: we now group tests at a coarser-grain, which
makes it harder to selected individual cases.
- In alcotest, a "test suite" is a sequence of "tests" that are composed of a sequence of "test cases".
- In tezt, there is only "tests" that are loosely grouped by "files" and horizontally by "tags".
- With alcotezt, we group each alcotest "test" into a tezt "test". consequently, we can no longer execute individual test cases.
- (C) Invocation headers
- Test files have traditionally contained a "test header", as described here: https://tezos.gitlab.io/developer/testing.html#conventions, including an invocation header.
- During the project, we've broken most of the invocation headers. Here's an example of a broken header: https://gitlab.com/tezos/tezos/blob/master/src/proto_016_PtMumbai/lib_client/test/test_client_proto_context.ml#L29
- (D) Understanding where a failing test in the CI is located
- (E) Having both
runtestandrunteztis confusing
Constraints on solution
- opam tests must continue to work
- runtest/runtezt should work
- tezt entrypoint should work
-
tezt --listshould give legible output
Proposed solutions
- Problem A / D.
-
Modify Tezt's --matchto match on file names (tags?)? (MR: nomadic-labs/tezt!17 (merged)) -
Modify Tezt's --fileso that it can match basenames or full paths (MR: nomadic-labs/tezt!13 (merged)) -
Adding ~__FILE__toAlcotest(_lwt).run(MR: !7965 (merged)) -
Adding tags to protocol tests and cleaning up their titles ( !8109 (closed), !8390 (merged)) - The above fixes will also solve problem D: when tests will be registered by full path that path will appear in error messages.
-
- Problem B.
-
Experiment with splitting tests (!8227 (merged) and !8228 (merged))
-
- Problem C.
-
Fixing broken invocation headers: (in progress: #5172 (closed)). This makes the headers work, but more imprecise. -
Shorten test titles, and updating invocations headers to make them more precise (!8408 (merged), !8400 (merged))
-
- Problem E.
-
Merging runtezt into runtest (!7968 (closed) and !8502 (merged))
-
Edited by lykimquyen