[go: up one dir, main page]

[Feature flag] Rollout of `locked_paths_mergeability_check`

Summary

This issue is to roll out the feature on production, that is currently behind the locked_paths_mergeability_check feature flag.

Owners

  • Most appropriate Slack channel to reach out to: #g_create_source-code-be
  • Best individual to reach out to: @jwoodwardgl

Expectations

What are we expecting to happen?

What can go wrong and how would we detect it?

Rollout Steps

Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command.

Rollout on non-production environments

  • Verify the MR with the feature flag is merged to master and have been deployed to non-production environments with /chatops run auto_deploy status 78c2978a
  • Deploy the feature flag at a percentage (recommended percentage: 50%) with /chatops run feature set locked_paths_mergeability_check <rollout-percentage> --actors --dev --pre --staging --staging-ref
  • Monitor that the error rates did not increase (repeat with a different percentage as necessary).
  • Enable the feature globally on non-production environments with /chatops run feature set locked_paths_mergeability_check true --dev --pre --staging --staging-ref
  • Verify that the feature works as expected. The best environment to validate the feature in is staging-canary as this is the first environment deployed to. Make sure you are configured to use canary.
  • If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking deployments.
    • See #qa-staging Slack channel and look for the following messages:
      • test kicked off: Feature flag locked_paths_mergeability_check has been set to true on **gstg**
      • test result: This pipeline was triggered due to toggling of locked_paths_mergeability_check feature flag

For assistance with end-to-end test failures, please reach out via the #test-platform Slack channel. Note that end-to-end test failures on staging-ref don't block deployments.

Specific rollout on production

For visibility, all /chatops commands that target production should be executed in the #production Slack channel and cross-posted (with the command results) to the responsible team's Slack channel.

  • Ensure that the feature MRs have been deployed to both production and canary with /chatops run auto_deploy status <merge-commit-of-your-feature>
  • Depending on the type of actor you are using, pick one of these options:
    • For project-actor: /chatops run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss,gitlab-com/www-gitlab-com locked_paths_mergeability_check true
    • For group-actor: /chatops run feature set --group=gitlab-org,gitlab-com locked_paths_mergeability_check true
    • For user-actor: /chatops run feature set --user=jwoodwardgl locked_paths_mergeability_check true
  • Verify that the feature works for the specific actors.

Preparation before global rollout

  • Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable.
  • Check if the feature flag change needs to be accompanied with a change management issue. Cross link the issue here if it does.
  • Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the @sre-oncall Slack alias.
  • Ensure that documentation exists for the feature, and the version history text has been updated.
  • Leave a comment on the feature issue announcing estimated time when this feature flag will be enabled on GitLab.com.
  • Ensure that any breaking changes have been announced following the release post process to ensure GitLab customers are aware.
  • Notify the #support_gitlab-com Slack channel and your team channel (more guidance when this is necessary in the dev docs).
  • Ensure that the feature flag rollout plan is reviewed by another developer familiar with the domain.

Global rollout on production

For visibility, all /chatops commands that target production should be executed in the #production Slack channel and cross-posted (with the command results) to the responsible team's Slack channel (#g_create_code-review).

  • (Optional) Incrementally roll out the feature on production environment.
    • Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net.
    • Perform actor-based rollout: /chatops run feature set locked_paths_mergeability_check <rollout-percentage> --actors
  • Enable the feature globally on production environment: /chatops run feature set locked_paths_mergeability_check true
  • Observe appropriate graphs on https://dashboards.gitlab.net and verify that services are not affected.
  • Leave a comment on [the feature issue][main-issue] announcing that the feature has been globally enabled.
  • Wait for at least one day for the verification term.

Rollback Steps

  • This feature can be disabled on production by running the following Chatops command:
/chatops run feature set locked_paths_mergeability_check false
  • Disable the feature flag on non-production environments:
/chatops run feature set locked_paths_mergeability_check false --dev --pre --staging --staging-ref
  • Delete feature flag from all environments:
/chatops run feature delete locked_paths_mergeability_check --dev --pre --staging --staging-ref --production
Edited by Joe Woodward