Telemetry - track validity checks usage

Overview

As we build validity checks in GitLab, it will be helpful to better understand how often this feature is being used.

• Feature Performance (Stability) - Dashboard
  • How long on average does the UpdateTokenStatusService take to run?
  • What is the frequency of nil token returns from the Secret Analyzer? (transferred from Validity checks Metrics (#532964 - closed) on August 1st, initially in [Experiment] Display GitLab token Status in Vul... (&16589))
    • See this discussion for more context
• Product Insights (Usability)
  • How many tokens are being processed per job? (Job contains 5 active secrets, 1 of which was a token that we checked for validity.)
    • Example: A job found the following secrets glpat token token A token B token C token D
      • Validity check: 1/5 (right now we only check for gitlab tokens)
      • Then separate this by project
    • Then separate this by namespace
  • What is the count per rule/pattern type separated by status (Active, Inactive, Possibly active)
    • Example: glpat token
      • Active: 5
      • Inactive: 10
      • Possibly active: 30
    • Then separate this by project
    • Then separate this by namespace
• Adoption (Engagement)
  • Validity Checks is enabled per project
  • Number of times refresh token button is used
    • Separate by UI vs. API
    • Then separate this by project
    • Then separate this by namespace
  • Are customers disabling Validity checks after it was enabled? Track when validity checks changes from on to off.

Resources

• Secret Detection Metrics handbook page

Implementation Plan

How long on average does the UpdateTokenStatusService take to run?

• Likely want to use prometheus to track this, specifically sidekiq_jobs_completion_seconds comes to mind
  • Draft: Track number of tokens sent to UpdateTok... (!204639 - closed)

What is the frequency of nil token returns from the Secret Analyzer?

• Can use database metrics to count nil or ruGuKmzNt0EMRvSYQo0PTbx4RUiPDbmIFAReJ5ojFZY

How many tokens are being processed per job?

• Internal Tracking Event for ee/app/services/security/secret_detection/update_token_status_service.rb grab length of relation
  • Draft: Track number of tokens sent to UpdateTok... (!204639 - closed)

What is the count per rule/pattern type separated by status (Active, Inactive, Possibly active)

• Database metric from project security setting, join with the corresponding project id to grab project and namespace data
  • Draft: Track number of tokens sent to UpdateTok... (!204639 - closed)

Validity Checks is enabled per project

• database metric for validity_checks_enabled == true example
  • Track number of project that enabled Validity C... (!204150 - merged)

Number of times refresh token button UI is used

• internal tracking event ee/app/assets/javascripts/vulnerabilities/components/validity_check.vue inside refreshValidityCheck

Number of times refresh token API is used

• internal tracking event for the refresh token controller

Are customers disabling Validity checks after it was enabled?

• internal tracking event app/assets/javascripts/security_configuration/components/pipeline_secret_detection_feature_card.vue inside onValidityChecksToggle when checked == false which would occur when the user is actively turning off the toggle