Unit tezts / Alcotezt

Unit tezts / Alcotezt project

The test suite of octez is highly fractured. We currently use the following test frameworks:

system tests: pytest, tezt and flextesa
unit tests: alcotest, qcheck (mostly launched through alcotest), ppx_expect (or ppx_inline test) and ad-hoc ocaml unit tests,

The goal of this project is to make the test suite more coherent by replacing alcotest by Tezt. As pointed out in issue #1265 (closed), Tezt now has all the features of Alcotest, in addition to many features that improve developer experience (e.g. test selection) and that facilitates CI integration (e.g. load balancing). In addition, reducing the number of tools decreases productivity losses induced by developer context switches.

Overall plan :

Add a compatibility layer ( named alcotezt and migrate a small number of tests as proof of concept )
Migrate all other unit tests to alcotezt
Publish and update the docs on how to write tests directly in tezt without alcotezt
Deprecate alcotezt to write new tests
Phase out the compatibility layer

However, some technical issues (also discussed in the issue #1265 (closed)) should be resolved to migrate Alcotest to Tezt:

How can we keep package-level tests while integrating them all in a global test suite?
How will tezt unit tests be launched when installing opam packages --with-test
Unit tests are typically launched through dune runtest that runs tests in parallel. in my experience, this shortens wall-time test time by 1/2 - 1/3 on my machine as opposed to running tests sequentially. Should we implement parallel test execution in Tezt?

The project should resolve the issues mentioned above, and produce a proof of concept by translating the unit tests of at least one package in the source tree (e.g. src/lib_base).

Notes by Romain

https://codimd.nomadic-labs.com/rmAqM73dS2uiVCj2wRR5Yg

Estimate time to finish

Wrapping alcotest suites with Alcotezt (issue #4741 (closed))

We can divide the alcotest suites into two parts:

The first one is all the libraries outside of the protocol test: around 48 of them
And the protocol tests suites inside the protocol_alpha and other protocols.

Other libraries

Protocol tests

The time of translation is ETC: ~17 hours