diff --git a/.gitlab/ci/unittest.yml b/.gitlab/ci/unittest.yml index 10f6cccc346dee6ad1875bc35812236a636b0e83..cb024e67ce567ee28032efbb31e4b860cd785b2f 100644 --- a/.gitlab/ci/unittest.yml +++ b/.gitlab/ci/unittest.yml @@ -36,7 +36,10 @@ test-static-libs-patch: name: "$CI_JOB_NAME-$CI_COMMIT_SHA" paths: - test_results + reports: + junit: test_results/*.xml expire_in: 1 day + when: always # this section is updated using the script scripts/update_unit_test.sh ##BEGIN_UNITTEST## diff --git a/docs/developer/testing.rst b/docs/developer/testing.rst index 457b91979ccaae66eca974a4b8b270ce36426da5..fcd8411b8bb5d9d8e11620b0212d14897fb7fbf2 100644 --- a/docs/developer/testing.rst +++ b/docs/developer/testing.rst @@ -307,11 +307,8 @@ References: .. _gitlab_test_ci: -Executing tests ---------------- - Executing tests locally -~~~~~~~~~~~~~~~~~~~~~~~ +----------------------- Whereas executing the tests through the CI, as described below, is the standard and most convenient way of running the full test suite, they @@ -405,7 +402,7 @@ or use the ``--remove`` option of the instrumentation script: Known issues -~~~~~~~~~~~~ +"""""""""""" 1. Report generation may fail spuriously. @@ -437,7 +434,7 @@ Known issues opam pin add bisect_ppx.2.5.0 --dev-repo --yes Executing tests through the GitLab CI -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +------------------------------------- All tests are executed on all branches for each commit. For instances, to see the latest runs of the CI on the master branch, @@ -446,10 +443,37 @@ visit `this page annotated with a green checkmark icon if the CI passed, and a red cross icon if not. You can click the icon for more details. -By default, the CI runs the tests as a set of independent jobs in the -``test`` stage. This is to better exploit GitLab runner parallelism: one job -per ``pytest`` test file and one job for each OCaml package containing tests. -This produces a report that is well-integrated with the CI user interface. +The results of the test suite on terminated pipelines is presented on +the details of the merge request page corresponding to the +pipeline's branch (if any). For more information, see the `GitLab +documentation on Unit test reports +`__. + +By default, the ``test`` of the CI runs the tests as a set of independent jobs +that cluster the tests with a varying grain. This strikes a balance between exploiting GitLab +runner parallelism while limiting the number of jobs per +pipeline. The grain used varies slightly for different types of +tests: + +Python integration and regression tests + We run one job per ``pytest`` test file whose tests are marked + ``slow``. We run one job regrouping the set of + ``pytest``\ s per protocol that are not marked ``slow``. + +Tezt integration and regression tests + Tezt tests are grouped in 3 batch jobs. New tests increases the + size of the last batch. + +The OCaml package tests (Alcotest, Crowbar & QCheck) + The OCaml package tests are regrouped analogously to the ``pytest``\ s: + one job per protocol package, in addition to one job regrouping + tests for remaining packages. + +Flextesa + Flextesa tests run in one job per test. + +Adding tests to the CI +~~~~~~~~~~~~~~~~~~~~~~ When adding a new test that should be run in the CI (which should be the case for most automatic tests), you need to make sure that it is @@ -460,18 +484,25 @@ Python integration and regression tests Run ``./scripts/update_integration_test.sh`` in Tezos home. This will include your new test in :src:`.gitlab-ci.yml`. -Tests executed through Dune (Alcotest, Flextesa) +Tezt integration and regression tests + New Tezt tests will be included automatically in the CI. To + rebalance the Tezt batches, run ``./scripts/run-tezt-tests-ci.sh``. + +The OCaml package tests (Alcotest, Crowbar & QCheck) Run ``./scripts/update_unit_test.sh`` in Tezos home. This will include your new test in :src:`.gitlab-ci.yml`. -Other +Other (including Flextesa) For other types of tests, you need to manually modify the :src:`.gitlab-ci.yml`. Please refer to the `GitLab CI Pipeline Reference `_. A helpful tool for this task is the `CI Lint tool `_, and ``gitlab-runner``, introduced in the :ref:`next section `. -A second way to run the tests is to trigger manually the job +Launching tests manually and measuring coverage in the CI +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Another way to run the tests is to trigger manually the job ``test_coverage`` in stage ``test_coverage``, from the Gitlab CI web interface. This job simply runs ``dune build @runtest`` in the project directory, followed by ``make all`` in the directory ``tests_python``. This is slower @@ -482,7 +513,7 @@ The role of having this extra testing stage is twofold. - It can be launched locally in a container environment (see next section), - it can be used to generate a code coverage report, from the CI. -The report artefact can be downloaded or browsed from the CI page upon completion +The coverage report artefact can be downloaded or browsed from the CI page upon completion of ``test_coverage``. It can also be published on a publicly available webpage linked to the gitlab repository. This is done by triggering manually the ``pages`` job in the ``publish_coverage`` stage, from the Gitlab CI @@ -495,12 +526,6 @@ the pipeline, as well as on the pipeline number. Examples: ``https://nomadic-labs.gitlab.io/tezos/105822404/``, ``https://tezos.gitlab.io/tezos/1234822404/``. -The results of the tests suite on terminated pipelines is presented on -the details of the merge request page that corresponds to the -pipeline's branch (if any). For more information, see the `GitLab -documentation on Unit test reports -`__. - .. _executing_gitlab_ci_locally: Executing the GitLab CI locally diff --git a/scripts/test_wrapper.sh b/scripts/test_wrapper.sh index af03ee2cca55bf88a4a6323cd3abc099f94be9a0..0472aa092a2e79e4a28277d0ccea1af42328b4ed 100755 --- a/scripts/test_wrapper.sh +++ b/scripts/test_wrapper.sh @@ -26,10 +26,42 @@ LC_NUMERIC=C printf "Total runtime: %02d:%02d:%02.4f\n" "$dh" "$dm" "$ds" if [ $EXITCODE -eq 0 ]; then echo "Ok"; + nb_fail=0 else echo "Error"; echo "Exited with exitcode $EXITCODE" cat "test_results/$name.log" + nb_fail=1 fi +timestamp=$(date -Is) +hostname=$(hostname) + +cat > "test_results/$name.xml" < + + + +EOF + +if [ ! $EXITCODE -eq 0 ]; then + msg="Exited with exitcode $EXITCODE." + + # Add link to log artifact when running in CI + if [ -n "$CI_SERVER_URL" ]; then + url=$CI_SERVER_URL/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/-/jobs/$CI_JOB_ID/artifacts/file/test_results/$name.log + msg="$msg See logs at $url" + fi + + cat >> "test_results/$name.xml" < +EOF +fi + +cat >> "test_results/$name.xml" < + + +EOF + exit $EXITCODE