Manual pipeline creation fails with `Gitlab::RequestContext::RequestDeadlineExceeded: Request takes longer than 57 seconds`
We are seeing internal timeouts when manually creating pipelines from the primary branch of the project; the project is configured to run between 100 and 400 stages within each pipeline using a variety of rules to determine what stages will run. When manually running the pipeline it will occasionally fail with the error "Pipeline cannot be run. Something went wrong on our end. Please try again."
When viewing logs for successful pipelines for the same project I see 20-30 second processing times for successful pipelines, which could easily to exceed internal timeouts when Com is under moderate load.
https://sentry.gitlab.net/gitlab/gitlabcom/issues/1586686/?referrer=gitlab_plugin
Gitlab::RequestContext::RequestDeadlineExceeded: Request takes longer than 57 seconds
gitlab/request_context.rb:31:in `ensure_deadline_not_exceeded!'
raise RequestDeadlineExceeded,
gitlab/gitaly_client.rb:172:in `execute'
Gitlab::RequestContext.instance.ensure_deadline_not_exceeded!
gitlab/gitaly_client/call.rb:18:in `block in call'
GitalyClient.execute(@storage, @service, @rpc, @request, remote_storage: @remote_storage, timeout: @timeout, &block)
gitlab/gitaly_client/call.rb:55:in `recording_request'
yield
gitlab/gitaly_client/call.rb:17:in `call'
response = recording_request do
...
(207 additional frame(s) were not displayed)Proposal
Move pipeline creation asynchronously but provide consistent UX. E.g. page waits until the pipeline is created, then redirect to the pipeline page. In Sidekiq we don't have such strict deadline.