[go: up one dir, main page]

grpc: Return RESOURCE_EXHAUSTED when rate-limited by Rails

Closes #6737 (closed)

Gitaly may invoke Rails endpoints such as POST /internal/allowed during the course of RPC execution. Hook-based RPCs for example will query the endpoint to perform an authentication check.

When clients inadvertently send too many concurrent requests to a GitLab API that invokes one of these Gitaly RPCs, Rails may return an HTTP 429 error when Gitaly tries to perform the authentication check. When this happens, the error is treated as an internal failure, and Gitaly returns an INTERNAL status code in the gRPC response.

INTERNAL errors directly contribute to the error rate calculations for the Gitaly service and have a significant impact on its apdex score. More importantly, the INTERNAL status code is not representative of the actual error and doesn't give the caller the opportunity to "back off".

Let's check explicitly for 429 errors and propagate this information up the call chain. Modifying the NotAllowedError struct to include a new RateLimited field seems the least obtrusive way to do this. The field is set based on the error type received from the client call. We extract it within the existing statushandler middleware to return the more appropriate RESOURCE_EXHAUSTED gRPC status.

A small change was required in the UserDeleteBranch handler to prevent it from prematurely returning an INTERNAL error.

Edited by James Liu

Merge request reports

Loading