From 8e57886ef55a1602fe2896a49fd955263a69bf3b Mon Sep 17 00:00:00 2001 From: Nailia Iskhakova Date: Wed, 20 Sep 2023 16:51:26 +0200 Subject: [PATCH 1/2] Add 2k Sidekiq configuration Signed-off-by: Nailia Iskhakova --- .../reference_architectures/2k_users.md | 132 +++++++++++++++++- 1 file changed, 131 insertions(+), 1 deletion(-) diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 8f22343e770515..20de390f35ce04 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -26,6 +26,7 @@ For a full list of reference architectures, see | PostgreSQL1 | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | Redis2 | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | | Gitaly5 | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Sidekiq6 | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | GitLab Rails6 | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Object storage4 | - | - | - | - | - | @@ -591,6 +592,136 @@ To configure Gitaly with TLS: +## Configure Sidekiq + +Sidekiq requires connection to the [Redis](#configure-redis), +[PostgreSQL](#configure-postgresql) and [Gitaly](#configure-gitaly) instances. +It also requires a connection to [Object Storage](#configure-the-object-storage) as recommended. + +To configure the Sidekiq server, on the server node you want to use for Sidekiq: + +1. SSH in to the Sidekiq server. +1. [Download and install](https://about.gitlab.com/install/) the Linux + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration: + + + + ```ruby + roles ["sidekiq_role"] + + # External URL + external_url 'https://gitlab.example.com' + + ## Redis connection details + gitlab_rails['redis_port'] = '6379' + gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server + gitlab_rails['redis_password'] = 'Redis Password' + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the Gitaly setup + gitlab_rails['gitaly_token'] = 'gitalysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + + ## PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + + ## Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false + + # Sidekiq + sidekiq['enable'] = true + sidekiq['listen_address'] = "0.0.0.0" + + ## Set number of Sidekiq queue processes to the same number as available CPUs + sidekiq['queue_groups'] = ['*'] * 4 + + ## Set number of Sidekiq threads per queue process to the recommend number of 20 + sidekiq['max_concurrency'] = 20 + + ## Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + + # Object Storage + ## This is an example for configuring Object Storage on GCP + ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true + gitlab_rails['object_store']['connection'] = { + 'provider' => 'Google', + 'google_project' => '', + 'google_json_key_location' => '' + } + gitlab_rails['object_store']['objects']['artifacts']['bucket'] = "" + gitlab_rails['object_store']['objects']['external_diffs']['bucket'] = "" + gitlab_rails['object_store']['objects']['lfs']['bucket'] = "" + gitlab_rails['object_store']['objects']['uploads']['bucket'] = "" + gitlab_rails['object_store']['objects']['packages']['bucket'] = "" + gitlab_rails['object_store']['objects']['dependency_proxy']['bucket'] = "" + gitlab_rails['object_store']['objects']['terraform_state']['bucket'] = "" + + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_project' => '', + 'google_json_key_location' => '' + } + gitlab_rails['backup_upload_remote_directory'] = "" + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Linux package node you configured and add or replace + the file of the same name on this server. If this is the first Linux package node you are configuring then you can skip this step. + +1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: + + ```shell + sudo touch /etc/gitlab/skip-auto-reconfigure + ``` + + Only a single designated node should handle migrations as detailed in the + [GitLab Rails post-configuration](#gitlab-rails-post-configuration) section. + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#reconfigure-a-linux-package-installation). + +1. Verify the GitLab services are running: + + ```shell + sudo gitlab-ctl status + ``` + + The output should be similar to the following: + + ```plaintext + run: logrotate: (pid 192292) 2990s; run: log: (pid 26374) 93048s + run: node-exporter: (pid 26864) 92997s; run: log: (pid 26446) 93036s + run: sidekiq: (pid 26870) 92996s; run: log: (pid 26391) 93042s + ``` + +NOTE: +If you find that the environment's Sidekiq job processing is slow with long queues, +more nodes can be added as required. You can also tune your Sidekiq nodes to +run [multiple Sidekiq processes](../sidekiq/extra_sidekiq_processes.md). + + + ## Configure GitLab Rails This section describes how to configure the GitLab application (Rails) component. @@ -648,7 +779,6 @@ On each node perform the following: node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' puma['listen'] = '0.0.0.0' - sidekiq['listen_address'] = "0.0.0.0" # Configure Sidekiq with 2 workers and 20 max concurrency sidekiq['max_concurrency'] = 20 -- GitLab From 267f232f6ead9cea17821284eaf49d83ce5d55da Mon Sep 17 00:00:00 2001 From: Nailia Iskhakova Date: Wed, 4 Oct 2023 18:32:02 +0200 Subject: [PATCH 2/2] Update 3k and 5k with new Sidekiq specs Signed-off-by: Nailia Iskhakova --- doc/administration/reference_architectures/3k_users.md | 10 +++------- doc/administration/reference_architectures/5k_users.md | 8 ++------ 2 files changed, 5 insertions(+), 13 deletions(-) diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 19f6fbd222eb91..7994cc77a29ba1 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -40,7 +40,7 @@ For a full list of reference architectures, see | Gitaly5 6 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Praefect5 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq7 | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Sidekiq7 | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | GitLab Rails7 | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage4 | - | - | - | - | @@ -71,7 +71,7 @@ card "**Internal Load Balancer**" as ilb #9370DB together { collections "**GitLab Rails** x3" as gitlab #32CD32 - collections "**Sidekiq** x4" as sidekiq #ff8dd1 + collections "**Sidekiq** x2" as sidekiq #ff8dd1 } together { @@ -202,8 +202,6 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.141`: Praefect PostgreSQL 1 (non HA) - `10.6.0.71`: Sidekiq 1 - `10.6.0.72`: Sidekiq 2 -- `10.6.0.73`: Sidekiq 3 -- `10.6.0.74`: Sidekiq 4 - `10.6.0.41`: GitLab application 1 - `10.6.0.42`: GitLab application 2 - `10.6.0.43`: GitLab application 3 @@ -1701,8 +1699,6 @@ The following IPs will be used as an example: - `10.6.0.71`: Sidekiq 1 - `10.6.0.72`: Sidekiq 2 -- `10.6.0.73`: Sidekiq 3 -- `10.6.0.74`: Sidekiq 4 To configure the Sidekiq nodes, one each one: @@ -1763,7 +1759,7 @@ Updates to example must be made at: sidekiq['listen_address'] = "0.0.0.0" ## Set number of Sidekiq queue processes to the same number as available CPUs - sidekiq['queue_groups'] = ['*'] * 2 + sidekiq['queue_groups'] = ['*'] * 4 ## Set number of Sidekiq threads per queue process to the recommend number of 20 sidekiq['max_concurrency'] = 20 diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index f011f0832f0dac..025cc57dc520ce 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -37,7 +37,7 @@ costly-to-operate environment by using the | Gitaly5 6 | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | | Praefect5 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq7 | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Sidekiq7 | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | GitLab Rails7 | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage4 | - | - | - | - | @@ -68,7 +68,7 @@ card "**Internal Load Balancer**" as ilb #9370DB together { collections "**GitLab Rails** x3" as gitlab #32CD32 - collections "**Sidekiq** x4" as sidekiq #ff8dd1 + collections "**Sidekiq** x2" as sidekiq #ff8dd1 } together { @@ -199,8 +199,6 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.141`: Praefect PostgreSQL 1 (non HA) - `10.6.0.71`: Sidekiq 1 - `10.6.0.72`: Sidekiq 2 -- `10.6.0.73`: Sidekiq 3 -- `10.6.0.74`: Sidekiq 4 - `10.6.0.41`: GitLab application 1 - `10.6.0.42`: GitLab application 2 - `10.6.0.43`: GitLab application 3 @@ -1689,8 +1687,6 @@ examples include the Object storage configuration. - `10.6.0.71`: Sidekiq 1 - `10.6.0.72`: Sidekiq 2 -- `10.6.0.73`: Sidekiq 3 -- `10.6.0.74`: Sidekiq 4 To configure the Sidekiq nodes, one each one: -- GitLab