Geo: Feature flag changes do not propagate immediately to secondaries

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem

I am seeing weird behavior locally. When I toggle a feature flag :geo_package_file_replication on the primary, the cache invalidation event is sent and processed, but the feature flag does not return the updated value in Rails console or in rails-web behavior (when I check the expected change in the UI). But when I exit and start Rails console it updates in console as well as in the UI (restarting rails-web also worked). Any ideas?

I believe this happens because the Feature class loads the stored feature flags n the following order:

• Memoized: using Gitlab::SafeRequestStore or @flipper
• L1: using Process cache
• L2: using Redis cache
• DB: using a single SQL query

The feature flag does not return the updated value in the Rails console or in the UI because any of these caches are still valid since we only clear the Redis cache while processing the event on a Geo secondary node.

Proposal

CacheInvalidationEventStore is only fired from https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/ee/feature.rb#L39, so it's only used for feature flags, so we can do whatever we want, specific to feature flags, in https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/geo/log_cursor/events/cache_invalidation_event.rb#L18.