From ea02540fae8d0e26e2609403a0edbe08ba4f4dd5 Mon Sep 17 00:00:00 2001 From: rickywiens Date: Tue, 18 Aug 2020 11:43:05 -0500 Subject: [PATCH 1/2] Rename JUnit reports to Unit Test Reports - Having the name of the feature be "JUnit test reports" can cause confusion over what type of support this feature has. While it requires the JUnit format, many different languages support that format. For a non-GitLab user this could lead away from investigating a feature that might work well for them, just because their primary use case does not involve Java. --- doc/README.md | 2 +- doc/administration/instance_limits.md | 2 +- doc/api/pipelines.md | 4 +- doc/ci/README.md | 2 +- doc/ci/introduction/index.md | 2 +- doc/ci/junit_test_reports.md | 280 +---------------- doc/ci/metrics_reports.md | 2 +- doc/ci/pipelines/job_artifacts.md | 6 +- doc/ci/unit_test_reports.md | 285 ++++++++++++++++++ doc/development/cicd/index.md | 2 +- doc/user/project/merge_requests/index.md | 2 +- .../testing_and_reports_in_merge_requests.md | 2 +- 12 files changed, 300 insertions(+), 291 deletions(-) create mode 100644 doc/ci/unit_test_reports.md diff --git a/doc/README.md b/doc/README.md index b73300accabc9e..e6356640c7db39 100644 --- a/doc/README.md +++ b/doc/README.md @@ -244,7 +244,7 @@ The following documentation relates to the DevOps **Verify** stage: |:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------| | [Code Quality reports](user/project/merge_requests/code_quality.md) | Analyze source code quality. | | [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Integration with GitLab. | -| [JUnit test reports](ci/junit_test_reports.md) | Display JUnit test reports on merge requests. | +| [Unit test reports](ci/unit_test_reports.md) | Display Unit test reports on merge requests. | | [Multi-project pipelines](ci/multi_project_pipelines.md) **(PREMIUM)** | Visualize entire pipelines that span multiple projects, including all cross-project inter-dependencies. | | [Pipeline Graphs](ci/pipelines/index.md#visualize-pipelines) | Visualize builds. | | [Review Apps](ci/review_apps/index.md) | Preview changes to your application right from a merge request. | diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index f30dba331b836c..bacdeaad4cf562 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -436,7 +436,7 @@ Reports that go over the 20 MB limit won't be loaded. Affected reports: - [Merge Request security reports](../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports-ultimate) - [CI/CD parameter `artifacts:expose_as`](../ci/yaml/README.md#artifactsexpose_as) -- [JUnit test reports](../ci/junit_test_reports.md) +- [Unit test reports](../ci/unit_test_reports.md) ## Advanced Global Search limits diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index dc81ef0e25e7fc..95a7787e0295ae 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -155,8 +155,8 @@ Example of response > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202525) in GitLab 13.0. -CAUTION: **Caution:** -This API route is part of the [JUnit test report](../ci/junit_test_reports.md) feature. It is protected by a [feature flag](../development/feature_flags/index.md) that is **disabled** due to performance issues with very large data sets. +NOTE: **Note:** +This API route is part of the [Unit test report](../ci/unit_test_reports.md) feature. ```plaintext GET /projects/:id/pipelines/:pipeline_id/test_report diff --git a/doc/ci/README.md b/doc/ci/README.md index 6f15fa5255073e..41cb7c1d39fa1a 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -132,7 +132,7 @@ Its feature set is listed on the table below according to DevOps stages. | [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | | [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **(PREMIUM)** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | | [Interactive Web Terminals](interactive_web_terminal/index.md) **(CORE ONLY)** | Open an interactive web terminal to debug the running jobs. | -| [JUnit tests](junit_test_reports.md) | Identify script failures directly on merge requests. | +| [Unit test reports](unit_test_reports.md) | Identify script failures directly on merge requests. | | [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | |---+---| | **Release** || diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md index c97f4e51d3060e..b4569721f58020 100644 --- a/doc/ci/introduction/index.md +++ b/doc/ci/introduction/index.md @@ -191,7 +191,7 @@ according to each stage (Verify, Package, Release). - Analyze your source code quality with [GitLab Code Quality](../../user/project/merge_requests/code_quality.md). - Determine the browser performance impact of code changes with [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md). **(PREMIUM)** - Determine the server performance impact of code changes with [Load Performance Testing](../../user/project/merge_requests/load_performance_testing.md). **(PREMIUM)** - - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [JUnit tests](../junit_test_reports.md). + - Perform a series of tests, such as [Container Scanning](../../user/application_security/container_scanning/index.md) **(ULTIMATE)**, [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)**, and [Unit tests](../unit_test_reports.md). - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. 1. **Package**: - Store Docker images with [Container Registry](../../user/packages/container_registry/index.md). diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index 8bc55a6e4f3e00..449f9bf5fcd6b2 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -1,281 +1,5 @@ --- -stage: Verify -group: Testing -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers -type: reference +redirect_to: 'unit_test_reports.md' --- -# JUnit test reports - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. - -## Overview - -It is very common that a [CI/CD pipeline](pipelines/index.md) contains a -test job that will verify your code. -If the tests fail, the pipeline fails and users get notified. The person that -works on the merge request will have to check the job logs and see where the -tests failed so that they can fix them. - -You can configure your job to use JUnit test reports, and GitLab will display a -report on the merge request so that it's easier and faster to identify the -failure without having to check the entire log. - -If you don't use Merge Requests but still want to see the JUnit output without searching through job logs, the full [JUnit test reports](#viewing-junit-test-reports-on-gitlab) are available in the pipeline detail view. - -## Use cases - -Consider the following workflow: - -1. Your `master` branch is rock solid, your project is using GitLab CI/CD and - your pipelines indicate that there isn't anything broken. -1. Someone from your team submits a merge request, a test fails and the pipeline - gets the known red icon. To investigate more, you have to go through the job - logs to figure out the cause of the failed test, which usually contain - thousands of lines. -1. You configure the JUnit test reports and immediately GitLab collects and - exposes them in the merge request. No more searching in the job logs. -1. Your development and debugging workflow becomes easier, faster and efficient. - -## How it works - -First, GitLab Runner uploads all JUnit XML files as artifacts to GitLab. Then, -when you visit a merge request, GitLab starts comparing the head and base branch's -JUnit test reports, where: - -- The base branch is the target branch (usually `master`). -- The head branch is the source branch (the latest pipeline in each merge request). - -The reports panel has a summary showing how many tests failed, how many had errors -and how many were fixed. If no comparison can be done because data for the base branch -is not available, the panel will just show the list of failed tests for head. - -There are four types of results: - -1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch -1. **Newly encountered errors:** Test cases which passed on base branch and failed due to a - test error on head branch -1. **Existing failures:** Test cases which failed on base branch and failed on head branch -1. **Resolved failures:** Test cases which failed on base branch and passed on head branch - -Each entry in the panel will show the test name and its type from the list -above. Clicking on the test name will open a modal window with details of its -execution time and the error output. - -![Test Reports Widget](img/junit_test_report.png) - -## How to set it up - -To enable the JUnit reports in merge requests, you need to add -[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) -in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. -The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). - -In the following examples, the job in the `test` stage runs and GitLab -collects the JUnit test report from each job. After each job is executed, the -XML reports are stored in GitLab as artifacts and their results are shown in the -merge request widget. - -To make the JUnit output files browsable, include them with the -[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). - -NOTE: **Note:** -You cannot have multiple tests with the same name and class in your JUnit report. - -### Ruby example - -Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the JUnit output file. - -```yaml -## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec -ruby: - stage: test - script: - - bundle install - - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml - artifacts: - paths: - - rspec.xml - reports: - junit: rspec.xml -``` - -### Go example - -Use the following job in `.gitlab-ci.yml`, and ensure you use `-set-exit-code`, -otherwise the pipeline will be marked successful, even if the tests fail: - -```yaml -## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go -golang: - stage: test - script: - - go get -u github.com/jstemmer/go-junit-report - - go test -v 2>&1 | go-junit-report -set-exit-code > report.xml - artifacts: - reports: - junit: report.xml -``` - -### Java examples - -There are a few tools that can produce JUnit reports in Java. - -#### Gradle - -In the following example, `gradle` is used to generate the test reports. -If there are multiple test tasks defined, `gradle` will generate multiple -directories under `build/test-results/`. In that case, you can leverage glob -matching by defining the following path: `build/test-results/test/**/TEST-*.xml`: - -```yaml -java: - stage: test - script: - - gradle test - artifacts: - reports: - junit: build/test-results/test/**/TEST-*.xml -``` - -NOTE: **Note:** -Support for `**` was added in [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620). - -#### Maven - -For parsing [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) -and [Failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/) test -reports, use the following job in `.gitlab-ci.yml`: - -```yaml -java: - stage: test - script: - - mvn verify - artifacts: - reports: - junit: - - target/surefire-reports/TEST-*.xml - - target/failsafe-reports/TEST-*.xml -``` - -### Python example - -This example uses pytest with the `--junitxml=report.xml` flag to format the output -for JUnit: - -```yaml -pytest: - stage: test - script: - - pytest --junitxml=report.xml - artifacts: - reports: - junit: report.xml -``` - -### C/C++ example - -There are a few tools that can produce JUnit reports in C/C++. - -#### GoogleTest - -In the following example, `gtest` is used to generate the test reports. -If there are multiple gtest executables created for different architectures (`x86`, `x64` or `arm`), -you will be required to run each test providing a unique filename. The results -will then be aggregated together. - -```yaml -cpp: - stage: test - script: - - gtest.exe --gtest_output="xml:report.xml" - artifacts: - reports: - junit: report.xml -``` - -#### CUnit - -[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit XML reports](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: - -```yaml -cunit: - stage: test - script: - - ./my-cunit-test - artifacts: - reports: - junit: ./my-cunit-test.xml -``` - -### .NET example - -The [JunitXML.TestLogger](https://www.nuget.org/packages/JunitXml.TestLogger/) NuGet -package can generate test reports for .Net Framework and .Net Core applications. The following -example expects a solution in the root folder of the repository, with one or more -project files in sub-folders. One result file is produced per test project, and each file -is placed in a new artifacts folder. This example includes optional formatting arguments, which -improve the readability of test data in the test widget. A full .Net Core -[example is available](https://gitlab.com/Siphonophora/dot-net-cicd-test-logging-demo). - -```yaml -## Source code and documentation are here: https://github.com/spekt/junit.testlogger/ - -Test: - stage: test - script: - - 'dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=..\artifacts\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"' - artifacts: - when: always - paths: - - ./**/*test-result.xml - reports: - junit: - - ./**/*test-result.xml -``` - -## Viewing JUnit test reports on GitLab - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. -> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. - -If JUnit XML files are generated and uploaded as part of a pipeline, these reports -can be viewed inside the pipelines details page. The **Tests** tab on this page will -display a list of test suites and cases reported from the XML file. - -![Test Reports Widget](img/pipelines_junit_test_report_ui_v12_5.png) - -You can view all the known test suites and click on each of these to see further -details, including the cases that make up the suite. - -You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). - -## Viewing JUnit screenshots on GitLab - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. -> - It's deployed behind a feature flag, disabled by default. -> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-screenshots-feature-core-only). **(CORE ONLY)** - -If JUnit XML files contain an `attachment` tag, GitLab parses the attachment. - -Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. - -```xml - - [[ATTACHMENT|/absolute/path/to/some/file]] - -``` - -When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complete, the attached file will be visible on the pipeline details page. - -### Enabling the JUnit screenshots feature **(CORE ONLY)** - -This feature comes with the `:junit_pipeline_screenshots_view` feature flag disabled by default. - -To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the -following command: - -```ruby -Feature.enable(:junit_pipeline_screenshots_view) -``` +This document was moved to [unit_test_reports](unit_test_reports.md). diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 14669edf7ebf0e..ea1994235fd0fb 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -11,7 +11,7 @@ type: reference ## Overview -GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](junit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. +GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index be6886fe6b2c62..f4b00887ef6e84 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -77,10 +77,10 @@ If you also want the ability to browse the report output files, include the The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) as artifacts. Although JUnit was originally developed in Java, there are many -[third party ports](https://en.wikipedia.org/wiki/JUnit#Ports) for other +third party ports for other languages like JavaScript, Python, Ruby, and so on. -See [JUnit test reports](../junit_test_reports.md) for more details and examples. +See [Unit test reports](../unit_test_reports.md) for more details and examples. Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool: ```yaml @@ -94,7 +94,7 @@ rspec: junit: rspec.xml ``` -The collected JUnit reports upload to GitLab as an artifact and display in merge requests. +The collected Unit test reports upload to GitLab as an artifact and display in merge requests. NOTE: **Note:** If the JUnit tool you use exports to multiple XML files, specify diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md new file mode 100644 index 00000000000000..96500092fafdb7 --- /dev/null +++ b/doc/ci/unit_test_reports.md @@ -0,0 +1,285 @@ +--- +stage: Verify +group: Testing +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + +# Unit test reports + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. + +## Overview + +It is very common that a [CI/CD pipeline](pipelines/index.md) contains a +test job that will verify your code. +If the tests fail, the pipeline fails and users get notified. The person that +works on the merge request will have to check the job logs and see where the +tests failed so that they can fix them. + +You can configure your job to use Unit test reports, and GitLab will display a +report on the merge request so that it's easier and faster to identify the +failure without having to check the entire log. Unit test reports currently +only support test reports in the JUnit format. + +If you don't use Merge Requests but still want to see the unit test report +output without searching through job logs, the full +[Unit test reports](#viewing-unit-test-reports-on-gitlab) are available +in the pipeline detail view. + +## Use cases + +Consider the following workflow: + +1. Your `master` branch is rock solid, your project is using GitLab CI/CD and + your pipelines indicate that there isn't anything broken. +1. Someone from your team submits a merge request, a test fails and the pipeline + gets the known red icon. To investigate more, you have to go through the job + logs to figure out the cause of the failed test, which usually contain + thousands of lines. +1. You configure the Unit test reports and immediately GitLab collects and + exposes them in the merge request. No more searching in the job logs. +1. Your development and debugging workflow becomes easier, faster and efficient. + +## How it works + +First, GitLab Runner uploads all JUnit report XML files as artifacts to GitLab. Then, +when you visit a merge request, GitLab starts comparing the head and base branch's +JUnit report XML files, where: + +- The base branch is the target branch (usually `master`). +- The head branch is the source branch (the latest pipeline in each merge request). + +The reports panel has a summary showing how many tests failed, how many had errors +and how many were fixed. If no comparison can be done because data for the base branch +is not available, the panel will just show the list of failed tests for head. + +There are four types of results: + +1. **Newly failed tests:** Test cases which passed on base branch and failed on head branch +1. **Newly encountered errors:** Test cases which passed on base branch and failed due to a + test error on head branch +1. **Existing failures:** Test cases which failed on base branch and failed on head branch +1. **Resolved failures:** Test cases which failed on base branch and passed on head branch + +Each entry in the panel will show the test name and its type from the list +above. Clicking on the test name will open a modal window with details of its +execution time and the error output. + +![Test Reports Widget](img/junit_test_report.png) + +## How to set it up + +To enable the Unit test reports in merge requests, you need to add +[`artifacts:reports:junit`](pipelines/job_artifacts.md#artifactsreportsjunit) +in `.gitlab-ci.yml`, and specify the path(s) of the generated test reports. +The reports must be `.xml` files, otherwise [GitLab returns an Error 500](https://gitlab.com/gitlab-org/gitlab/-/issues/216575). + +In the following examples, the job in the `test` stage runs and GitLab +collects the Unit test report from each job. After each job is executed, the +XML reports are stored in GitLab as artifacts and their results are shown in the +merge request widget. + +To make the Unit test report output files browsable, include them with the +[`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). + +NOTE: **Note:** +You cannot have multiple tests with the same name and class in your JUnit report XML file. + +### Ruby example + +Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the Unit test report output file. + +```yaml +## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec +ruby: + stage: test + script: + - bundle install + - bundle exec rspec --format progress --format RspecJunitFormatter --out rspec.xml + artifacts: + paths: + - rspec.xml + reports: + junit: rspec.xml +``` + +### Go example + +Use the following job in `.gitlab-ci.yml`, and ensure you use `-set-exit-code`, +otherwise the pipeline will be marked successful, even if the tests fail: + +```yaml +## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go +golang: + stage: test + script: + - go get -u github.com/jstemmer/go-junit-report + - go test -v 2>&1 | go-junit-report -set-exit-code > report.xml + artifacts: + reports: + junit: report.xml +``` + +### Java examples + +There are a few tools that can produce JUnit reports in Java. + +#### Gradle + +In the following example, `gradle` is used to generate the test reports. +If there are multiple test tasks defined, `gradle` will generate multiple +directories under `build/test-results/`. In that case, you can leverage glob +matching by defining the following path: `build/test-results/test/**/TEST-*.xml`: + +```yaml +java: + stage: test + script: + - gradle test + artifacts: + reports: + junit: build/test-results/test/**/TEST-*.xml +``` + +NOTE: **Note:** +Support for `**` was added in [GitLab Runner 13.0](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620). + +#### Maven + +For parsing [Surefire](https://maven.apache.org/surefire/maven-surefire-plugin/) +and [Failsafe](https://maven.apache.org/surefire/maven-failsafe-plugin/) test +reports, use the following job in `.gitlab-ci.yml`: + +```yaml +java: + stage: test + script: + - mvn verify + artifacts: + reports: + junit: + - target/surefire-reports/TEST-*.xml + - target/failsafe-reports/TEST-*.xml +``` + +### Python example + +This example uses pytest with the `--junitxml=report.xml` flag to format the output +for JUnit: + +```yaml +pytest: + stage: test + script: + - pytest --junitxml=report.xml + artifacts: + reports: + junit: report.xml +``` + +### C/C++ example + +There are a few tools that can produce JUnit reports in C/C++. + +#### GoogleTest + +In the following example, `gtest` is used to generate the test reports. +If there are multiple gtest executables created for different architectures (`x86`, `x64` or `arm`), +you will be required to run each test providing a unique filename. The results +will then be aggregated together. + +```yaml +cpp: + stage: test + script: + - gtest.exe --gtest_output="xml:report.xml" + artifacts: + reports: + junit: report.xml +``` + +#### CUnit + +[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit XML reports](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: + +```yaml +cunit: + stage: test + script: + - ./my-cunit-test + artifacts: + reports: + junit: ./my-cunit-test.xml +``` + +### .NET example + +The [JunitXML.TestLogger](https://www.nuget.org/packages/JunitXml.TestLogger/) NuGet +package can generate test reports for .Net Framework and .Net Core applications. The following +example expects a solution in the root folder of the repository, with one or more +project files in sub-folders. One result file is produced per test project, and each file +is placed in a new artifacts folder. This example includes optional formatting arguments, which +improve the readability of test data in the test widget. A full .Net Core +[example is available](https://gitlab.com/Siphonophora/dot-net-cicd-test-logging-demo). + +```yaml +## Source code and documentation are here: https://github.com/spekt/junit.testlogger/ + +Test: + stage: test + script: + - 'dotnet test --test-adapter-path:. --logger:"junit;LogFilePath=..\artifacts\{assembly}-test-result.xml;MethodFormat=Class;FailureBodyFormat=Verbose"' + artifacts: + when: always + paths: + - ./**/*test-result.xml + reports: + junit: + - ./**/*test-result.xml +``` + +## Viewing Unit test reports on GitLab + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. +> - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. + +If JUnit XML report files are generated and uploaded as part of a pipeline, these reports +can be viewed inside the pipelines details page. The **Tests** tab on this page will +display a list of test suites and cases reported from the XML file. + +![Test Reports Widget](img/pipelines_junit_test_report_ui_v12_5.png) + +You can view all the known test suites and click on each of these to see further +details, including the cases that make up the suite. + +You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a-pipelines-test-report). + +## Viewing JUnit screenshots on GitLab + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0. +> - It's deployed behind a feature flag, disabled by default. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-screenshots-feature-core-only). **(CORE ONLY)** + +If JUnit XML report files contain an `attachment` tag, GitLab parses the attachment. + +Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. + +```xml + + [[ATTACHMENT|/absolute/path/to/some/file]] + +``` + +When [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/6061) is complete, the attached file will be visible on the pipeline details page. + +### Enabling the JUnit screenshots feature **(CORE ONLY)** + +This feature comes with the `:junit_pipeline_screenshots_view` feature flag disabled by default. + +To enable this feature, ask a GitLab administrator with [Rails console access](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the +following command: + +```ruby +Feature.enable(:junit_pipeline_screenshots_view) +``` diff --git a/doc/development/cicd/index.md b/doc/development/cicd/index.md index 5b598a19a6eea7..c8c8c8b1ab75fd 100644 --- a/doc/development/cicd/index.md +++ b/doc/development/cicd/index.md @@ -74,7 +74,7 @@ that need to be stored. Also, a job may depend on artifacts from previous jobs i case the Runner will download them using a dedicated API endpoint. Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts -is reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request. +are reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request. Job status transitions are not all automated. A user may run [manual jobs](../../ci/yaml/README.md#whenmanual), cancel a pipeline, retry specific failed jobs or the entire pipeline. Anything that diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index a9ee9d8e5072ac..69276f0677b1a5 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -19,7 +19,7 @@ A. Consider you're a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request 1. You gather feedback from your team 1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md) -1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD +1. You verify your changes with [Unit test reports](../../../ci/unit_test_reports.md) in GitLab CI/CD 1. You avoid using dependencies whose license is not compatible with your project with [License Compliance reports](../../compliance/license_compliance/index.md) **(ULTIMATE)** 1. You request the [approval](merge_request_approvals.md) from your manager **(STARTER)** 1. Your manager: diff --git a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md index e5ebc46d58f764..b298c62a5e6185 100644 --- a/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md +++ b/doc/user/project/merge_requests/testing_and_reports_in_merge_requests.md @@ -19,7 +19,7 @@ or link to useful information directly from merge requests: | [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. | | [Display arbitrary job artifacts](../../../ci/yaml/README.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../../../ci/pipelines/job_artifacts.md) in merge requests. | | [GitLab CI/CD](../../../ci/README.md) | Build, test, and deploy your code in a per-branch basis with built-in CI/CD. | -| [JUnit test reports](../../../ci/junit_test_reports.md) | Configure your CI jobs to use JUnit test reports, and let GitLab display a report on the merge request so that it’s easier and faster to identify the failure without having to check the entire job log. | +| [Unit test reports](../../../ci/unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it’s easier and faster to identify the failure without having to check the entire job log. | | [License Compliance](../../compliance/license_compliance/index.md) **(ULTIMATE)** | Manage the licenses of your dependencies. | | [Metrics Reports](../../../ci/metrics_reports.md) **(PREMIUM)** | Display the Metrics Report on the merge request so that it's fast and easy to identify changes to important metrics. | | [Multi-Project pipelines](../../../ci/multi_project_pipelines.md) **(PREMIUM)** | When you set up GitLab CI/CD across multiple projects, you can visualize the entire pipeline, including all cross-project interdependencies. | -- GitLab From c5553c7ecb0a0f06a5f3bffc4329cc247e2faac8 Mon Sep 17 00:00:00 2001 From: Ricky Wiens Date: Thu, 20 Aug 2020 14:20:19 +0000 Subject: [PATCH 2/2] Apply 2 suggestion(s) to 2 file(s) --- doc/ci/pipelines/job_artifacts.md | 4 ++-- doc/ci/unit_test_reports.md | 27 ++++++++++++++------------- 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index f4b00887ef6e84..de881ff160e076 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -75,13 +75,13 @@ If you also want the ability to browse the report output files, include the > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. > - Requires GitLab Runner 11.2 and above. -The `junit` report collects [JUnit XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) +The `junit` report collects [JUnit report format XML files](https://www.ibm.com/support/knowledgecenter/en/SSQ2R2_14.1.0/com.ibm.rsar.analysis.codereview.cobol.doc/topics/cac_useresults_junit.html) as artifacts. Although JUnit was originally developed in Java, there are many third party ports for other languages like JavaScript, Python, Ruby, and so on. See [Unit test reports](../unit_test_reports.md) for more details and examples. -Below is an example of collecting a JUnit XML file from Ruby's RSpec test tool: +Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: ```yaml rspec: diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index 96500092fafdb7..0a62eab38deaa5 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -7,7 +7,8 @@ type: reference # Unit test reports -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/45318) in GitLab 11.2. Requires GitLab Runner 11.2 and above. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/39737) from JUnit test reports to Unit test reports in GitLab 13.4. ## Overview @@ -20,7 +21,7 @@ tests failed so that they can fix them. You can configure your job to use Unit test reports, and GitLab will display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire log. Unit test reports currently -only support test reports in the JUnit format. +only support test reports in the JUnit report format. If you don't use Merge Requests but still want to see the unit test report output without searching through job logs, the full @@ -43,9 +44,9 @@ Consider the following workflow: ## How it works -First, GitLab Runner uploads all JUnit report XML files as artifacts to GitLab. Then, +First, GitLab Runner uploads all JUnit report format XML files as artifacts to GitLab. Then, when you visit a merge request, GitLab starts comparing the head and base branch's -JUnit report XML files, where: +JUnit report format XML files, where: - The base branch is the target branch (usually `master`). - The head branch is the source branch (the latest pipeline in each merge request). @@ -84,14 +85,14 @@ To make the Unit test report output files browsable, include them with the [`artifacts:paths`](yaml/README.md#artifactspaths) keyword as well, as shown in the [Ruby example](#ruby-example). NOTE: **Note:** -You cannot have multiple tests with the same name and class in your JUnit report XML file. +You cannot have multiple tests with the same name and class in your JUnit report format XML file. ### Ruby example Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the Unit test report output file. ```yaml -## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec +## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report format XML file with rspec ruby: stage: test script: @@ -110,7 +111,7 @@ Use the following job in `.gitlab-ci.yml`, and ensure you use `-set-exit-code`, otherwise the pipeline will be marked successful, even if the tests fail: ```yaml -## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report with go +## Use https://github.com/jstemmer/go-junit-report to generate a JUnit report format XML file with go golang: stage: test script: @@ -123,7 +124,7 @@ golang: ### Java examples -There are a few tools that can produce JUnit reports in Java. +There are a few tools that can produce JUnit report format XML file in Java. #### Gradle @@ -166,7 +167,7 @@ java: ### Python example This example uses pytest with the `--junitxml=report.xml` flag to format the output -for JUnit: +into the JUnit report XML format: ```yaml pytest: @@ -180,7 +181,7 @@ pytest: ### C/C++ example -There are a few tools that can produce JUnit reports in C/C++. +There are a few tools that can produce JUnit report format XML files in C/C++. #### GoogleTest @@ -201,7 +202,7 @@ cpp: #### CUnit -[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit XML reports](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: +[CUnit](https://cunity.gitlab.io/cunit/) can be made to produce [JUnit report format XML files](https://cunity.gitlab.io/cunit/group__CI.html) automatically when run using its `CUnitCI.h` macros: ```yaml cunit: @@ -244,7 +245,7 @@ Test: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24792) in GitLab 12.5 behind a feature flag (`junit_pipeline_view`), disabled by default. > - The feature flag was removed and the feature was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/216478) in GitLab 13.3. -If JUnit XML report files are generated and uploaded as part of a pipeline, these reports +If JUnit report format XML files are generated and uploaded as part of a pipeline, these reports can be viewed inside the pipelines details page. The **Tests** tab on this page will display a list of test suites and cases reported from the XML file. @@ -261,7 +262,7 @@ You can also retrieve the reports via the [GitLab API](../api/pipelines.md#get-a > - It's deployed behind a feature flag, disabled by default. > - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enabling-the-junit-screenshots-feature-core-only). **(CORE ONLY)** -If JUnit XML report files contain an `attachment` tag, GitLab parses the attachment. +If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. Upload your screenshots as [artifacts](pipelines/job_artifacts.md#artifactsreportsjunit) to GitLab. The `attachment` tag **must** contain the absolute path to the screenshots you uploaded. -- GitLab