"Reference not found" after creating branch, no pipeline created
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
This issue has been created after a customer reported this behaviour, and we confirmed it was happening on GitLab.com (based on Elasticsearch Sidekiq logs).
When a branch gets created, this results in a POST /api/v4/internal/post_receive request to GitLab with the changes. GitLab processes these changes, which includes creating pipelines. The Ci::CreatePipelineService checks if the ref exists via Gitlab::Ci::Pipeline::Chain::Validate::Repository. If the branch doesn't exist, it returns Reference not found:
https://gitlab.com/gitlab-org/gitlab/-/blob/94f870b8e4b965a41dd2ad576d50f7eeb271f117/lib/gitlab/ci/pipeline/chain/validate/repository.rb#L13
It seems like there might be two different ways to check if a branch exists: 1) Check the cached data, and 2) Call Gitaly using Gitaly::RefExistsRequest
As mentioned previously, we can see logs for GitLab.com where POST /api/v4/internal/post_receive requests resulted in no pipeline created because Reference not found. That is despite the pipeline_params.before having the value 0000000000000000000000000000000000000000 for that ref.
Could outdated cached branch data be the cause of this problem?
Similar issue to #500751 (closed), which has been fixed.
Steps to reproduce
Unable to reproduce reliably, but customer see this intermittently, and we can see this occurs on GitLab.com
Example Project
What is the current bug behavior?
After branch creation, pipelines can't run for that ref because Reference not found. This problem resolves itself over a period of time.
What is the expected correct behavior?
After branch creation, pipelines can run for that ref, even those related to the post_receive request for that ref creation
Relevant logs and/or screenshots
Elasticsearch query (Sidekiq index pattern)
json.message:"Error creating pipeline" AND json.pipeline_params.before:"0000000000000000000000000000000000000000" AND json.errors:"Reference not found"https://log.gprd.gitlab.net/app/r/s/uGAZV
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)(we will only investigate if the tests are passing)