diff --git a/doc/README.md b/doc/README.md index b73300accabc9e1bc0b6cad1f11fa6e79d8fcfb4..e6356640c7db39349bebdfc96c8905a493c7a4b2 100644 --- a/doc/README.md +++ b/doc/README.md @@ -244,7 +244,7 @@ The following documentation relates to the DevOps **Verify** stage: |:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------| | [Code Quality reports](user/project/merge_requests/code_quality.md) | Analyze source code quality. | | [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Integration with GitLab. | -| [JUnit test reports](ci/junit_test_reports.md) | Display JUnit test reports on merge requests. | +| [Unit test reports](ci/unit_test_reports.md) | Display Unit test reports on merge requests. | | [Multi-project pipelines](ci/multi_project_pipelines.md) **(PREMIUM)** | Visualize entire pipelines that span multiple projects, including all cross-project inter-dependencies. | | [Pipeline Graphs](ci/pipelines/index.md#visualize-pipelines) | Visualize builds. | | [Review Apps](ci/review_apps/index.md) | Preview changes to your application right from a merge request. | diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index f30dba331b836cd1c4c63f8d06fd63c398507044..bacdeaad4cf56252925accbee3f35a455def9b69 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -436,7 +436,7 @@ Reports that go over the 20 MB limit won't be loaded. Affected reports: - [Merge Request security reports](../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports-ultimate) - [CI/CD parameter `artifacts:expose_as`](../ci/yaml/README.md#artifactsexpose_as) -- [JUnit test reports](../ci/junit_test_reports.md) +- [Unit test reports](../ci/unit_test_reports.md) ## Advanced Global Search limits diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index dc81ef0e25e7fc12ed16312259e62c8902f5f0d5..95a7787e0295aecc75bd42e73154a5ff9ed50288 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -155,8 +155,8 @@ Example of response > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202525) in GitLab 13.0. -CAUTION: **Caution:** -This API route is part of the [JUnit test report](../ci/junit_test_reports.md) feature. It is protected by a [feature flag](../development/feature_flags/index.md) that is **disabled** due to performance issues with very large data sets. +NOTE: **Note:** +This API route is part of the [Unit test report](../ci/unit_test_reports.md) feature. ```plaintext GET /projects/:id/pipelines/:pipeline_id/test_report diff --git a/doc/ci/README.md b/doc/ci/README.md index 6f15fa5255073ea0958dee29c27f9b8926dff19e..41cb7c1d39fa1a68034f841b52a2d634e8109930 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -132,7 +132,7 @@ Its feature set is listed on the table below according to DevOps stages. | [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | | [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **(PREMIUM)** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | | [Interactive Web Terminals](interactive_web_terminal/index.md) **(CORE ONLY)** | Open an interactive web terminal to debug the running jobs. | -| [JUnit tests](junit_test_reports.md) | Identify script failures directly on merge requests. | +| [Unit test reports](unit_test_reports.md) | Identify script failures directly on merge requests. | | [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | |---+---| | **Release** || diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index c97f4e51d3060e591152773d0667600944ad80e1..b4569721f580203c5170cedbe1629f7770457163 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -191,7 +191,7 @@ according to each stage (Verify, Package, Release). - Analyze your source code quality with [GitLab Code Quality](../../user/project/merge_requests/code_quality.md). - Determine the browser performance impact of code changes with [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md). **(PREMIUM)** - Determine the server performance impact of code changes with [Load Performance Testing](../../user/project/merge_requests/load_performance_testing.md). **(PREMIUM)** - - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [JUnit tests](../junit_test_reports.md). + - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [Unit tests](../unit_test_reports.md). - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. 1. **Package**: - Store Docker images with [Container Registry](../../user/packages/container_registry/index.md). diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index 8bc55a6e4f3e0099212ff6ef1b20073170f291ea..449f9bf5fcd6b20c5436514d05475d9801a98d47 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -1,281 +1,5 @@ --- -stage: Verify -group: Testing -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: reference +redirect_to: 'unit_test_reports.md' --- -# JUnit test reports - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. - -## Overview - -It is very common that a [CI/CD pipeline](pipelines/index.md) contains a -test job that will verify your code. -If the tests fail, the pipeline fails and users get notified. The person that -works on the merge request will have to check the job logs and see where the -tests failed so that they can fix them. - -You can configure your job to use JUnit test reports, and GitLab will display a -report on the merge request so that it's easier and faster to identify the -failure without having to check the entire log. - -If you don't use Merge Requests but still want to see the JUnit output without searching through job logs, the full [JUnit test reports](#viewing-junit-test-reports-on-gitlab) are available in the pipeline detail view. - -## Use cases - -Consider the following workflow: - -1. Your `master` branch is rock solid, your project is using GitLab CI/CD and - your pipelines indicate that there isn't anything broken. -1. Someone from your team submits a merge request, a test fails and the pipeline - gets the known red icon. To investigate more, you have to go through the job - logs to figure out the cause of the failed test, which usually contain - thousands of lines. -1. You configure the JUnit test reports and immediately GitLab collects and - exposes them in the merge request. No more searching in the job logs. -1. Your development and debugging workflow becomes easier, faster and efficient. - -## How it works - -First, GitLab Runner uploads all JUnit XML files as artifacts to GitLab. Then, -when you visit a merge request, GitLab starts comparing the head and base branch's -JUnit test reports, where: - -- The base branch is the target branch (usually `master`). -- The head branch is the source branch (the latest pipeline in each merge request). - -The reports panel has a summary showing how many tests failed, how many had errors -and how many were fixed. If no comparison can be done because data for the base branch -is not available, the panel will just show the list of failed tests for head. - -There are four types of results: - -1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch -1. **Newly encountered errors:** Test cases which passed on base branch and failed due to a - test error on head branch -1. **Existing failures:** Test cases which failed on base branch and failed on head branch -1. **Resolved failures:** Test cases which failed on base branch and passed on head branch - -Each entry in the panel will show the test name and its type from the list -above. Clicking on the test name will open a modal window with details of its -execution time and the error output. - -![Test Reports Widget](img/junit_test_report.png) - -## How to set it up - -To enable the JUnit reports in merge requests, you need to add -[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) -in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. -The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). - -In the following examples, the job in the `test` stage runs and GitLab -collects the JUnit test report from each job. After each job is executed, the -XML reports are stored in GitLab as artifacts and their results are shown in the -merge request widget. - -To make the JUnit output files browsable, include them with the -[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). - -NOTE: **Note:** -You cannot have multiple tests with the same name and class in your JUnit report. - -### Ruby example - -Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the JUnit output file. - -```yaml -## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec -ruby: - stage: test - script: - - bundle install - - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml - artifacts: - paths: - - rspec.xml - reports: - junit: rspec.xml -``` - -### Go example - -Use the following job in `.gitlab-ci.yml`, and ensure you use `-set-exit-code`, -otherwise the pipeline will be marked successful, even if the tests fail: - -```yaml -## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go -golang: - stage: test - script: - - go get -u github.com/jstemmer/go-junit-report - - go test -v 2>&1 | go-junit-report -set-exit-code > report.xml - artifacts: - reports: - junit: report.xml -``` - -### Java examples - -There are a few tools that can produce JUnit reports in Java. - -#### Gradle - -In the following example, `gradle` is used to generate the test reports. -If there are multiple test tasks defined, `gradle` will generate multiple -directories under `build/test-results/`. In that case, you can leverage glob -matching by defining the following path: `build/test-results/test/**/TEST-*.xml`: - -```yaml -java: - stage: test - script: - - gradle test - artifacts: - reports: - junit: build/test-results/test/**/TEST-*.xml -``` - -NOTE: **Note:** -Support for `**` was added in [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620). - -#### Maven - -For parsing [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) -and [Failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/) test -reports, use the following job in `.gitlab-ci.yml`: - -```yaml -java: - stage: test - script: - - mvn verify - artifacts: - reports: - junit: - - target/surefire-reports/TEST-*.xml - - target/failsafe-reports/TEST-*.xml -``` - -### Python example - -This example uses pytest with the `--junitxml=report.xml` flag to format the output -for JUnit: - -```yaml -pytest: - stage: test - script: - - pytest --junitxml=report.xml - artifacts: - reports: - junit: report.xml -``` - -### C/C++ example - -There are a few tools that can produce JUnit reports in C/C++. - -#### GoogleTest - -In the following example, `gtest` is used to generate the test reports. -If there are multiple gtest executables created for different architectures (`x86`, `x64` or `arm`), -you will be required to run each test providing a unique filename. The results -will then be aggregated together. - -```yaml -cpp: - stage: test - script: - - gtest.exe --gtest_output="xml:report.xml" - artifacts: - reports: - junit: report.xml -``` - -#### CUnit - -[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit XML reports](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: - -```yaml -cunit: - stage: test - script: - - ./my-cunit-test - artifacts: - reports: - junit: ./my-cunit-test.xml -``` - -### .NET example - -The [JunitXML.TestLogger](https://www.nuget.org/packages/JunitXml.TestLogger/) NuGet -package can generate test reports for .Net Framework and .Net Core applications. The following -example expects a solution in the root folder of the repository, with one or more -project files in sub-folders. One result file is produced per test project, and each file -is placed in a new artifacts folder. This example includes optional formatting arguments, which -improve the readability of test data in the test widget. A full .Net Core -[example is available](https://gitlab.com/Siphonophora/dot-net-cicd-test-logging-demo). - -```yaml -## Source code and documentation are here: https://github.com/spekt/junit.testlogger/ - -Test: - stage: test - script: - - 'dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=..\artifacts\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"' - artifacts: - when: always - paths: - - ./**/*test-result.xml - reports: - junit: - - ./**/*test-result.xml -``` - -## Viewing JUnit test reports on GitLab - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. -> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. - -If JUnit XML files are generated and uploaded as part of a pipeline, these reports -can be viewed inside the pipelines details page. The **Tests** tab on this page will -display a list of test suites and cases reported from the XML file. - -![Test Reports Widget](img/pipelines_junit_test_report_ui_v12_5.png) - -You can view all the known test suites and click on each of these to see further -details, including the cases that make up the suite. - -You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). - -## Viewing JUnit screenshots on GitLab - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. -> - It's deployed behind a feature flag, disabled by default. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-screenshots-feature-core-only). **(CORE ONLY)** - -If JUnit XML files contain an `attachment` tag, GitLab parses the attachment. - -Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. - -```xml - - [[ATTACHMENT|/absolute/path/to/some/file]] - -``` - -When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complete, the attached file will be visible on the pipeline details page. - -### Enabling the JUnit screenshots feature **(CORE ONLY)** - -This feature comes with the `:junit_pipeline_screenshots_view` feature flag disabled by default. - -To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the -following command: - -```ruby -Feature.enable(:junit_pipeline_screenshots_view) -``` +This document was moved to [unit_test_reports](unit_test_reports.md). diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 14669edf7ebf0e599344ca656c63b29ed5a08538..ea1994235fd0fb80914965aaebc9d2d7eef07611 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -11,7 +11,7 @@ type: reference ## Overview -GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](junit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. +GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index be6886fe6b2c62f8be8aca8f3979f909d6be25d9..de881ff160e076efcabfa1916f6c434db1a21717 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -75,13 +75,13 @@ If you also want the ability to browse the report output files, include the > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. > - Requires GitLab Runner 11.2 and above. -The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) +The `junit` report collects [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) as artifacts. Although JUnit was originally developed in Java, there are many -[third party ports](https://en.wikipedia.org/wiki/JUnit#Ports) for other +third party ports for other languages like JavaScript, Python, Ruby, and so on. -See [JUnit test reports](../junit_test_reports.md) for more details and examples. -Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool: +See [Unit test reports](../unit_test_reports.md) for more details and examples. +Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: ```yaml rspec: @@ -94,7 +94,7 @@ rspec: junit: rspec.xml ``` -The collected JUnit reports upload to GitLab as an artifact and display in merge requests. +The collected Unit test reports upload to GitLab as an artifact and display in merge requests. NOTE: **Note:** If the JUnit tool you use exports to multiple XML files, specify diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md new file mode 100644 index 0000000000000000000000000000000000000000..0a62eab38deaa5759e097e546d572e1e45917659 --- /dev/null +++ b/doc/ci/unit_test_reports.md @@ -0,0 +1,286 @@ +--- +stage: Verify +group: Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + +# Unit test reports + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39737) from JUnit test reports to Unit test reports in GitLab 13.4. + +## Overview + +It is very common that a [CI/CD pipeline](pipelines/index.md) contains a +test job that will verify your code. +If the tests fail, the pipeline fails and users get notified. The person that +works on the merge request will have to check the job logs and see where the +tests failed so that they can fix them. + +You can configure your job to use Unit test reports, and GitLab will display a +report on the merge request so that it's easier and faster to identify the +failure without having to check the entire log. Unit test reports currently +only support test reports in the JUnit report format. + +If you don't use Merge Requests but still want to see the unit test report +output without searching through job logs, the full +[Unit test reports](#viewing-unit-test-reports-on-gitlab) are available +in the pipeline detail view. + +## Use cases + +Consider the following workflow: + +1. Your `master` branch is rock solid, your project is using GitLab CI/CD and + your pipelines indicate that there isn't anything broken. +1. Someone from your team submits a merge request, a test fails and the pipeline + gets the known red icon. To investigate more, you have to go through the job + logs to figure out the cause of the failed test, which usually contain + thousands of lines. +1. You configure the Unit test reports and immediately GitLab collects and + exposes them in the merge request. No more searching in the job logs. +1. Your development and debugging workflow becomes easier, faster and efficient. + +## How it works + +First, GitLab Runner uploads all JUnit report format XML files as artifacts to GitLab. Then, +when you visit a merge request, GitLab starts comparing the head and base branch's +JUnit report format XML files, where: + +- The base branch is the target branch (usually `master`). +- The head branch is the source branch (the latest pipeline in each merge request). + +The reports panel has a summary showing how many tests failed, how many had errors +and how many were fixed. If no comparison can be done because data for the base branch +is not available, the panel will just show the list of failed tests for head. + +There are four types of results: + +1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch +1. **Newly encountered errors:** Test cases which passed on base branch and failed due to a + test error on head branch +1. **Existing failures:** Test cases which failed on base branch and failed on head branch +1. **Resolved failures:** Test cases which failed on base branch and passed on head branch + +Each entry in the panel will show the test name and its type from the list +above. Clicking on the test name will open a modal window with details of its +execution time and the error output. + +![Test Reports Widget](img/junit_test_report.png) + +## How to set it up + +To enable the Unit test reports in merge requests, you need to add +[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) +in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. +The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). + +In the following examples, the job in the `test` stage runs and GitLab +collects the Unit test report from each job. After each job is executed, the +XML reports are stored in GitLab as artifacts and their results are shown in the +merge request widget. + +To make the Unit test report output files browsable, include them with the +[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). + +NOTE: **Note:** +You cannot have multiple tests with the same name and class in your JUnit report format XML file. + +### Ruby example + +Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the Unit test report output file. + +```yaml +## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report format XML file with rspec +ruby: + stage: test + script: + - bundle install + - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml + artifacts: + paths: + - rspec.xml + reports: + junit: rspec.xml +``` + +### Go example + +Use the following job in `.gitlab-ci.yml`, and ensure you use `-set-exit-code`, +otherwise the pipeline will be marked successful, even if the tests fail: + +```yaml +## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report format XML file with go +golang: + stage: test + script: + - go get -u github.com/jstemmer/go-junit-report + - go test -v 2>&1 | go-junit-report -set-exit-code > report.xml + artifacts: + reports: + junit: report.xml +``` + +### Java examples + +There are a few tools that can produce JUnit report format XML file in Java. + +#### Gradle + +In the following example, `gradle` is used to generate the test reports. +If there are multiple test tasks defined, `gradle` will generate multiple +directories under `build/test-results/`. In that case, you can leverage glob +matching by defining the following path: `build/test-results/test/**/TEST-*.xml`: + +```yaml +java: + stage: test + script: + - gradle test + artifacts: + reports: + junit: build/test-results/test/**/TEST-*.xml +``` + +NOTE: **Note:** +Support for `**` was added in [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620). + +#### Maven + +For parsing [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) +and [Failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/) test +reports, use the following job in `.gitlab-ci.yml`: + +```yaml +java: + stage: test + script: + - mvn verify + artifacts: + reports: + junit: + - target/surefire-reports/TEST-*.xml + - target/failsafe-reports/TEST-*.xml +``` + +### Python example + +This example uses pytest with the `--junitxml=report.xml` flag to format the output +into the JUnit report XML format: + +```yaml +pytest: + stage: test + script: + - pytest --junitxml=report.xml + artifacts: + reports: + junit: report.xml +``` + +### C/C++ example + +There are a few tools that can produce JUnit report format XML files in C/C++. + +#### GoogleTest + +In the following example, `gtest` is used to generate the test reports. +If there are multiple gtest executables created for different architectures (`x86`, `x64` or `arm`), +you will be required to run each test providing a unique filename. The results +will then be aggregated together. + +```yaml +cpp: + stage: test + script: + - gtest.exe --gtest_output="xml:report.xml" + artifacts: + reports: + junit: report.xml +``` + +#### CUnit + +[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit report format XML files](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: + +```yaml +cunit: + stage: test + script: + - ./my-cunit-test + artifacts: + reports: + junit: ./my-cunit-test.xml +``` + +### .NET example + +The [JunitXML.TestLogger](https://www.nuget.org/packages/JunitXml.TestLogger/) NuGet +package can generate test reports for .Net Framework and .Net Core applications. The following +example expects a solution in the root folder of the repository, with one or more +project files in sub-folders. One result file is produced per test project, and each file +is placed in a new artifacts folder. This example includes optional formatting arguments, which +improve the readability of test data in the test widget. A full .Net Core +[example is available](https://gitlab.com/Siphonophora/dot-net-cicd-test-logging-demo). + +```yaml +## Source code and documentation are here: https://github.com/spekt/junit.testlogger/ + +Test: + stage: test + script: + - 'dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=..\artifacts\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"' + artifacts: + when: always + paths: + - ./**/*test-result.xml + reports: + junit: + - ./**/*test-result.xml +``` + +## Viewing Unit test reports on GitLab + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. +> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. + +If JUnit report format XML files are generated and uploaded as part of a pipeline, these reports +can be viewed inside the pipelines details page. The **Tests** tab on this page will +display a list of test suites and cases reported from the XML file. + +![Test Reports Widget](img/pipelines_junit_test_report_ui_v12_5.png) + +You can view all the known test suites and click on each of these to see further +details, including the cases that make up the suite. + +You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). + +## Viewing JUnit screenshots on GitLab + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. +> - It's deployed behind a feature flag, disabled by default. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-screenshots-feature-core-only). **(CORE ONLY)** + +If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. + +Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. + +```xml + + [[ATTACHMENT|/absolute/path/to/some/file]] + +``` + +When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complete, the attached file will be visible on the pipeline details page. + +### Enabling the JUnit screenshots feature **(CORE ONLY)** + +This feature comes with the `:junit_pipeline_screenshots_view` feature flag disabled by default. + +To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the +following command: + +```ruby +Feature.enable(:junit_pipeline_screenshots_view) +``` diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 5b598a19a6eea7ce3fef6eb5ca5ce3adc7892c49..c8c8c8b1ab75fd8fa2a0b684cd8c6475da78ac74 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -74,7 +74,7 @@ that need to be stored. Also, a job may depend on artifacts from previous jobs i case the Runner will download them using a dedicated API endpoint. Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts -is reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request. +are reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request. Job status transitions are not all automated. A user may run [manual jobs](../../ci/yaml/README.md#whenmanual), cancel a pipeline, retry specific failed jobs or the entire pipeline. Anything that diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index a9ee9d8e5072aca7e542b081c9b18f557dadacb0..69276f0677b1a5dd57a29e5ca5e0e9c6d9ab6d0e 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -19,7 +19,7 @@ A. Consider you're a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request 1. You gather feedback from your team 1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md) -1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD +1. You verify your changes with [Unit test reports](../../../ci/unit_test_reports.md) in GitLab CI/CD 1. You avoid using dependencies whose license is not compatible with your project with [License Compliance reports](../../compliance/license_compliance/index.md) **(ULTIMATE)** 1. You request the [approval](merge_request_approvals.md) from your manager **(STARTER)** 1. Your manager: diff --git a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md index e5ebc46d58f764e31b3cf579da62a030c23635ba..b298c62a5e6185c29b919ea104cd53cc679a878a 100644 --- a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md +++ b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md @@ -19,7 +19,7 @@ or link to useful information directly from merge requests: | [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. | | [Display arbitrary job artifacts](../../../ci/yaml/README.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../../../ci/pipelines/job_artifacts.md) in merge requests. | | [GitLab CI/CD](../../../ci/README.md) | Build, test, and deploy your code in a per-branch basis with built-in CI/CD. | -| [JUnit test reports](../../../ci/junit_test_reports.md) | Configure your CI jobs to use JUnit test reports, and let GitLab display a report on the merge request so that it’s easier and faster to identify the failure without having to check the entire job log. | +| [Unit test reports](../../../ci/unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it’s easier and faster to identify the failure without having to check the entire job log. | | [License Compliance](../../compliance/license_compliance/index.md) **(ULTIMATE)** | Manage the licenses of your dependencies. | | [Metrics Reports](../../../ci/metrics_reports.md) **(PREMIUM)** | Display the Metrics Report on the merge request so that it's fast and easy to identify changes to important metrics. | | [Multi-Project pipelines](../../../ci/multi_project_pipelines.md) **(PREMIUM)** | When you set up GitLab CI/CD across multiple projects, you can visualize the entire pipeline, including all cross-project interdependencies. |