Provide image integration test job and helpers to Secure analyzer projects

Problem to solve

As a developer working on a GitLab Secure analyzer project, I'd like to write specs for the analyzer image, but I can't reuse the RSpec shared examples used to test the gemnasium image (Dependency Scanning). I need an easy way to set up image integration tests similar to the ones of the Gemnasium projects without additional maintenance cost.

User experience goal

As a developer of a GitLab Secure analyzer, I write RSpec examples to test the analyzer image. These examples use shared examples for image testing. I then set up a CI job that simply calls rspec in the context of an image that provides the shared examples as well as the system dependencies needed to run these shared examples.

Proposal

• Make the shared examples generic enough to handle Dependency Scanning, Secret Detection, and SAST.
• Create a Ruby gem that combines the shared examples, the Ruby dependencies, and the Ruby classes developers commonly use to write image specs.
• Create a new project that builds an image where Ruby, rspec, and the aforementioned Ruby gem are installed. The image also provides the system libraries needed to run the specs (such as ruby-json), the Docker CLI, and the JSON schemas needed for schema validation.
• Document this project, and explain how to use it to write image integration tests.
• Use this in the gemnasium project, ~"Category:Dependency Scanning".
• Invite project maintainers of Category:SAST and Category:Secret Detection to use it to.

The project name is to be defined. See discussion.

Optional: Add the image test job to the shared CI config, and set the rules:exists parameter so that the job is skipped when there are no image integration tests.

Implementation plan

• Update the gemnasium project to make the shared examples generic
  • Include all security report schemas, so that it can validate any report gitlab-org/security-products/analyzers/gemnasium!212 (merged)
• Import shared examples and dependencies into a new integration-test project
• Make this new project build a Docker image that contains all we need for integration tests
• Use this Docker image in the gemnasium project
• Update README of integration-test project and cover image integration tests using rspec gitlab-org/security-products/analyzers/integration-test!5 (merged)
• Share with other Secure teams, and encourage them to use the same testing tool => brown bag session on July 29th

Follow-up issues:

• Move definition of image test job to shared CI config (optional, if doesn't impact other Secure analyzers)
• Convert job integration tests to image integration tests when relevant
• Publish gitlab-secure_integration-test Rubygem, so that it can be used out of the integration-test image 

Further details

The gemnasium project has an image test job that runs the specs defined in spec/image_spec.rb. These specs directly use generic code:

• shared examples for the generated report
• shared examples for the scan itself (exit code)
• a helper class that runs the analyzer image and collects the generated report as well as the exit code

This generic code could be used to test the Docker images of other analyzer projects.

Documentation

The new project should document:

• how to use the gem in rspec examples
  • shared examples for reports
  • shared examples for scans
  • runner class
• how to check reports using the QA scripts
• how to use the Docker image
  • job template for image integration tests
  • job template for job integration tests

Availability & Testing

• The Comparable module is tested using rspec. It's used in shared examples for images.
• Shared examples for image are tested when testing QA scripts.
• The QA scripts are tested by comparing expected report to similar and different reports.
• TBD: Should we test the shared examples for reports?
• TBD: Should we test the runner class?

What does success look like, and how can we measure that?

Developers working on Secure analyzer projects can easily set up image integration tests.

Is this a cross-stage feature?

Potentially. This could be used by devopsprotect.

Links / references