Security dashboard - Exclude No Longer Detected by adding undetected_since field to vulnerability PG and ES

What does this MR do and why?

This MR:

• Updates the application logic to create a new detection transition record
• Add ES migration to add undetected_since field
• Update ee/lib/search/elastic/references/vulnerability.rb with new schema version and field setup
• Add the field in ee/lib/search/elastic/types/vulnerability.rb
• ES sync is triggered when undetected_since is updated in which is already done as part of the Vulnerabilities::DetectionTransitions::InsertService class

Previous MRs:

• Add detection_transitions table to track no lon... (!214443 - merged) • Subashis Chakraborty • 18.7
• Add service to insert detection transitions (!214890 - merged) • Subashis Chakraborty • 18.7

Related to #578567 #578566 #578568

Query plan: https://console.postgres.ai/gitlab/gitlab-production-sec/sessions/46473/commands/141742

References

• Closes #578567
• Closes #578568
• Blocks #582985
• Related to #578566

Validation steps

1. Make sure you have Elasticsearch running locally. You can follow this guide: https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/elasticsearch.md#setup

2. Start Rails console by running rails c

3. Enable the feature flag and mark migration as complete:

Feature.enable(:new_security_dashboard_exclude_no_longer_detected)

migration = Elastic::DataMigrationService.find_by_name!(:add_undetected_since_field_to_vulnerability)
migration.save!(completed: true)
Elastic::DataMigrationService.drop_migration_has_finished_cache!(migration)

4. Verify undetected_since is populated:

project = Project.find xx  # Use the ID of a project on your local dev
finding = project.vulnerability_findings.first
vulnerability = finding.vulnerability 
vulnerability.id # Keep note of this ID, we will need it for the `curl` command in step 6.


    Vulnerabilities::DetectionTransition.create!(
      vulnerability_occurrence_id: vulnerability.finding.id,
      project_id: vulnerability.project.id,
      detected: false,
      created_at: Time.now
    )

5. Queue and process the indexing:

::Elastic::ProcessBookkeepingService.track!(
      Search::Elastic::References::Vulnerability.new(vulnerability.id, "group_#{vulnerability.project.namespace.root_ancestor.id}")
    )
    ::Elastic::ProcessBookkeepingService.new.execute

6. Verify in ES. Replace VULN_ID with the vulnerability ID you printed in step 4.

    curl -s "http://localhost:9200/gitlab-development-vulnerabilities/_search?pretty" \
      -H "Content-Type: application/json" \
      -d '{ "query": { "term": { "vulnerability_id": { "value": <VULN_ID> } } }, "_source": ["undetected_since"] }'

undetected_since should be present and populated.

7. Verify undetected_since is removed when it's detected

    Vulnerabilities::DetectionTransition.create!(
      vulnerability_occurrence_id: vulnerability.finding.id,
      project_id: vulnerability.project.id,
      detected: true,
      created_at: Time.now
    )

8. Queue and process again:

    ::Elastic::ProcessBookkeepingService.track!(
      Search::Elastic::References::Vulnerability.new(vulnerability.id, "group_#{vulnerability.project.namespace.root_ancestor.id}")
    )
    ::Elastic::ProcessBookkeepingService.new.execute

9. Now verify in ES by running the same curl command. Replace VULN_ID with the vulnerability ID you printed in step 4.

    curl -s "http://localhost:9200/gitlab-development-vulnerabilities/_search?pretty" \
      -H "Content-Type: application/json" \
      -d '{ "query": { "term": { "vulnerability_id": { "value": 2181 } } }, "_source": ["undetected_since"] }'

undetected_since should be null

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.