From 4636ed9bdeab3f3a664c52e8876b17391c0bbc19 Mon Sep 17 00:00:00 2001 From: Will Chandler Date: Thu, 28 Oct 2021 17:52:44 +0000 Subject: [PATCH] Don't limit number of Gitaly client keepalives Long-running RPCs, such as `ForkRepository`, may take several hours to complete. While Sidekiq waits for the RPC to complete it should send keepalive pings to Gitaly/Praefect to prevent load balancers from killing the connection. However, the default value for `GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA` is only 2, with pings sent at 5 minute intervals. As a result, Sidekiq will only send keepalives for the first 5 minutes, then leave the connection idle for up to 6 hours and putting long-running RPCs at risk of failure. This commit sets `GRPC_ARG_HTTP2_MAX_PINGS_WITHOUT_DATA` to 0, so Sidekiq can send an unlimited number of keepalives and keep long-running RPCs active. Note that pings are still sent at 5 minute intervals with this change. Changelog: fixed --- lib/gitlab/gitaly_client.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/gitlab/gitaly_client.rb b/lib/gitlab/gitaly_client.rb index a824f97e197504..cc3f20ab774350 100644 --- a/lib/gitlab/gitaly_client.rb +++ b/lib/gitlab/gitaly_client.rb @@ -56,7 +56,8 @@ def self.channel_args # https://gitlab.com/gitlab-org/gitaly/-/blob/bf9f52bc/client/dial.go#L78 { 'grpc.keepalive_time_ms': 20000, - 'grpc.keepalive_permit_without_calls': 1 + 'grpc.keepalive_permit_without_calls': 1, + 'grpc.http2.max_pings_without_data': 0 } end private_class_method :channel_args -- GitLab