Unstable vulnerability ordering on security reports
Summary
Our test projects within security-products/tests occasionally need their fixtures (qa/expect/gl-sast-report.json) updated to reflect changes in the reports, usually due to new advisories in external DBs. There are however some false-negatives when report diffing fails due to ordering issues. While the sast analyzer should consistently be sorting reports there is an issue to be investigated.
Previous occurrences of report changes:
- gitlab-org/security-products/tests/sast@1994c45b
- https://gitlab.com/gitlab-org/security-products/tests/sast/issues/3
Previous discussion on ordering:
Example Project
Broken pipeline due to report changes: https://gitlab.com/gitlab-org/security-products/tests/js-yarn/pipelines/43743205
What is the current bug behavior?
Test projects occasionally generate reports with a different vulnerability order than fixtures, but no changes in set of identified vulnerabilities.
What is the expected correct behavior?
- Reports should have a consistent order of vulnerabilities.
- Test projects should only fail on differences between vulnerabilities within reports, not differences in report order.
Possible fixes
Review common lib sorting/deduping logic to ensure order is consistent across reports.
If it's not possible to guarantee a consistent order, our pipeline diffing should sort reports itself, but this is not ideal.