Gitaly Features should use Feature Toggles controlled by Cog

Use admin screen sliders instead of environment variables to toggle feature flags

---

Gitaly relies heavily on feature flags to toggle migration features on and off. At present, toggling these feature flags is an intensive operation:

• Create an MR to toggle the environment variable on in https://dev.gitlab.org/cookbooks/chef-repo
• Create an MR to toggle the environment variable off in https://dev.gitlab.org/cookbooks/chef-repo
• Merge the first MR and use knife to provision the changes
• Ensure that all unicorn workers have been recycled
• Test the feature
• Merge the second MR and use knife to provision the changes
• Ensure that all unicorn workers have been recycled

This is an intensive, slow and error-prone process.

The Gitaly Team would like to make toggling features easier.

Suggested Solution

For each feature flag in Gitaly, we would like to be able to have a UI control in the admin area to toggle the feature.

Because these flags are global to the application and cannot be controlled on a per-host basis, we will implement them as probabilities, represented in the admin UI as a percentage, that the feature flag is toggled for a call.

The simplest way to represent this in the UI for now would be a pulldown:

In the example above:

• Default would fall back to the current status of the feature: opt-in, opt-out. For more details on these statii consult the migration process document.

  • When default is set and the feature is opt-in, the legacy codepath will always be used.
  • When default is set and the feature is opt-out, the migrated codepath will always be used.

• When a percentage value is selected, that percent of the total requests to the migration site will use the migrated codepath. For example, when set to 5%, 1 in 20 requests will go through Gitaly while the remainder will go through Rugged.

• When set to on, all requests will be routed through the migrated codepath

Implementation

A feature flag probability is represented as a nullable integer with a value between 0 and 100 inclusive.

• Null represents: default
• 0 represents off
• 100 represents on
• 1-99 represent a percentage probability that a feature is enabled

This is kept in current_application_settings and in the application_settings

module Gitlab
  module GitalyClient
  
    module MigrationStatus
      DISABLED = 1
      OPT_IN = 2
      OPT_OUT = 3
    end
    
    # ....

    def self.feature_enabled?(feature, status)
      if status == MigrationStatus::DISABLED 
        return false
      end 

      # For feature x look at `gitaly_flag_x` application setting...
      feature_status = current_application_settings['gitaly_flag_' + feature.to_s]

      if feature_status.nil? 
        # If the feature is the default (nil) turn it on if it's opt-out
        # otherwise leave if off
        return status == MigrationStatus::OPT_OUT 
      end

      case feature_status
      # No need for random number generation if the feature is on or off...
      when 0
        false
      when 1
        true
      else
        # Probabilistically enable this feature
        Random.rand() * 100 < feature_status
      end
    end

    # Add a status argument to existing method
    def self.migrate(feature, status)
      is_enabled  = feature_enabled?(feature, status)
      # ... existing implementation
    end

At migration-sites, we then add the status argument:

    Gitlab::GitalyClient.migrate(:is_ancestor, Gitlab::GitalyClient::MigrationStatus::OPT_IN) do |is_enabled|
      if is_enabled
        # ...
      else
        # ...
      end
    end

Additional

• We need to ensure that during testing, the current_application_settings are ignored so that the test always behaves in a deterministic manner.