Bug: StrategyError during concurrent project creation from templates with Object storage backend

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Close this issue

Summary

Gitlab::ImportExport::AfterExportStrategies::BaseAfterExportStrategy::StrategyError exception during concurrent project creation from instance templates when using Object storage as the backend. The issue appears when multiple project creation requests are processed simultaneously.

This bug also happens on Gitlab.com. This script reproduces it:

create_gitlab_projects-com.sh

Steps to reproduce

Configure GitLab to use Object storage as the storage backend
Create an instance template that can be used for project creation

Send 10 concurrent API requests to create projects from this template with approximately 1 second delay between each request:

for i in {1..10}; do
  curl -X POST "https://gitlab.example.com/api/v4/projects" \
    --header "Content-Type: application/json" \
    --header "PRIVATE-TOKEN: <your_token>" \
    --data "{\"name\": \"$PROJECT_NAME\", \"path\": \"$PROJECT_PATH\", \"namespace_id\": $NAMESPACE_ID, \"use_custom_template\": true, \"template_project_id\": $TEMPLATE_ID
 }" &
  sleep 1
done

Example Project

What is the current bug behavior?

When multiple project creation requests using the same template are processed concurrently, some requests fail with Gitlab::ImportExport::AfterExportStrategies::BaseAfterExportStrategy::StrategyError exception. This appears to be a race condition related to how the export strategies handle concurrent access to the Object storage backend.

What is the expected correct behavior?

All concurrent project creation requests should complete successfully without throwing StrategyError exceptions, regardless of the number of simultaneous requests or the storage backend being used.

Relevant logs and/or screenshots

This happened 60 times on Gitlab.com in the past 7 days: https://log.gprd.gitlab.net/app/r/s/Cmz1o

{"severity":"ERROR","time":"2025-04-30T20:32:59.565Z","correlation_id":"01JT46E36DJZ8B7ZSTKG4872J3","meta.caller_id":"ProjectTemplateExportWorker","meta.remote_ip":"217.103.27.244","meta.feature_category":"importers","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","meta.organization_id":1,"meta.root_caller_id":"POST /api/:version/projects","exception.class":"Gitlab::ImportExport::AfterExportStrategies::BaseAfterExportStrategy::StrategyError","exception.message":"Gitlab::ImportExport::AfterExportStrategies::BaseAfterExportStrategy::StrategyError","exception.backtrace":["lib/gitlab/import_export/after_export_strategies/base_after_export_strategy.rb:64:in `ensure_export_ready!'","lib/gitlab/import_export/after_export_strategies/base_after_export_strategy.rb:31:in `execute'","app/services/projects/import_export/export_service.rb:51:in `execute_after_export_action'","app/services/projects/import_export/export_service.rb:21:in `execute'","app/workers/project_export_worker.rb:30:in `perform'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:220:in `execute_job'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:185:in `block (4 levels) in process'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:180:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","ee/lib/gitlab/sidekiq_middleware/set_session/server.rb:21:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/identity/restore.rb:12:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/skip_jobs.rb:51:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/resource_usage_limit/middleware.rb:16:in `perform'","lib/gitlab/sidekiq_middleware/resource_usage_limit/server.rb:8:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:35:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/concurrency_limit/middleware.rb:37:in `perform'","lib/gitlab/sidekiq_middleware/concurrency_limit/server.rb:8:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/click_house/migration_support/sidekiq_middleware.rb:7:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/pause_control/strategies/base.rb:31:in `perform'","lib/gitlab/sidekiq_middleware/pause_control/strategy_handler.rb:22:in `perform'","lib/gitlab/sidekiq_middleware/pause_control/server.rb:8:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call'","lib/gitlab/application_context.rb:173:in `block in use'","gitlab-labkit (0.37.0) lib/labkit/context.rb:35:in `with_context'","lib/gitlab/application_context.rb:173:in `use'","lib/gitlab/application_context.rb:96:in `with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'","lib/gitlab/database/query_analyzer.rb:83:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/set_ip_address.rb:7:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/server_metrics.rb:111:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:139:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:139:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:110:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `block in call'","gems/gitlab-safe_request_store/lib/gitlab/safe_request_store.rb:66:in `enabling_request_store'","gems/gitlab-safe_request_store/lib/gitlab/safe_request_store.rb:59:in `ensure_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:7:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","gitlab-labkit (0.37.0) lib/labkit/middleware/sidekiq/server.rb:21:in `block in call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:180:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","gitlab-labkit (0.37.0) lib/labkit/middleware/sidekiq/context/server.rb:16:in `block in call'","gitlab-labkit (0.37.0) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.37.0) lib/labkit/middleware/sidekiq/context/server.rb:15:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:173:in `invoke'","gitlab-labkit (0.37.0) lib/labkit/middleware/sidekiq/server.rb:20:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/shard_awareness_validator.rb:10:in `block in call'","lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'","lib/gitlab/sidekiq_middleware/shard_awareness_validator.rb:9:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","marginalia (1.11.1) lib/marginalia/sidekiq_instrumentation.rb:9:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","sentry-sidekiq (5.22.1) lib/sentry/sidekiq/sentry_context_middleware.rb:54:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","vendor/gems/sidekiq/lib/sidekiq/job/interrupt_handler.rb:9:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:183:in `block in traverse'","vendor/gems/sidekiq/lib/sidekiq/metrics/tracking.rb:26:in `track'","vendor/gems/sidekiq/lib/sidekiq/metrics/tracking.rb:134:in `call'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:182:in `traverse'","vendor/gems/sidekiq/lib/sidekiq/middleware/chain.rb:173:in `invoke'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:184:in `block (3 levels) in process'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:145:in `block (6 levels) in dispatch'","vendor/gems/sidekiq/lib/sidekiq/job_retry.rb:118:in `local'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:144:in `block (5 levels) in dispatch'","vendor/gems/sidekiq/lib/sidekiq/rails.rb:27:in `block in call'","activesupport (7.0.8.7) lib/active_support/execution_wrapper.rb:92:in `wrap'","activesupport (7.0.8.7) lib/active_support/reloader.rb:72:in `block in wrap'","activesupport (7.0.8.7) lib/active_support/execution_wrapper.rb:92:in `wrap'","activesupport (7.0.8.7) lib/active_support/reloader.rb:71:in `wrap'","vendor/gems/sidekiq/lib/sidekiq/rails.rb:26:in `call'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:139:in `block (4 levels) in dispatch'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:281:in `stats'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:134:in `block (3 levels) in dispatch'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:133:in `block (2 levels) in dispatch'","vendor/gems/sidekiq/lib/sidekiq/job_retry.rb:85:in `global'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:132:in `block in dispatch'","vendor/gems/sidekiq/lib/sidekiq/job_logger.rb:40:in `prepare'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:131:in `dispatch'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:183:in `block (2 levels) in process'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:182:in `handle_interrupt'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:182:in `block in process'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:181:in `handle_interrupt'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:181:in `process'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:86:in `process_one'","vendor/gems/sidekiq/lib/sidekiq/processor.rb:76:in `run'","vendor/gems/sidekiq/lib/sidekiq/component.rb:10:in `watchdog'","vendor/gems/sidekiq/lib/sidekiq/component.rb:19:in `block in safe_thread'"],"user.username":"root","tags.queue":"default","tags.jid":"accfc96f5a6bb11b0d5b99c1","tags.program":"sidekiq","tags.locale":"en","tags.feature_category":"importers","tags.correlation_id":"01JT46E36DJZ8B7ZSTKG4872J3","extra.importer":"Import/Export","extra.project_id":1,"extra.project_name":"Project Template","extra.project_path":"instancetemplates/project-template","extra.import_jid":null}

Output of checks

This bug happens on GitLab.com

Possible fixes

Edited Jun 26, 2025 by 🤖 GitLab Bot 🤖