From c7b31cd2e85e97bd0ec3a22c383d6e1633eefc75 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 17 Sep 2021 12:58:49 +0100 Subject: [PATCH 01/12] Update Redis and NFS notes on 10k --- .../reference_architectures/10k_users.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index f9397e6dbcaf5a..7d03e4d0309be5 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -33,7 +33,7 @@ full list of reference architectures, see | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage4 | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | @@ -137,8 +137,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +168,8 @@ To set up GitLab and its components to accommodate up to 10,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. -- GitLab From 46c71d0d561a807b2e11d81e33d19a766f5a465e Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 17 Sep 2021 13:29:27 +0100 Subject: [PATCH 02/12] Colocate Redis Sentinel with Redis in RA docs --- .../reference_architectures/10k_users.md | 436 ++++-------------- 1 file changed, 102 insertions(+), 334 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 88af96b2625ca8..3eba96dcbd5ab2 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -23,9 +23,7 @@ full list of reference architectures, see | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node3 | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Redis - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | @@ -82,11 +80,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -190,15 +183,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.61`: Redis - Queues Primary - `10.6.0.62`: Redis - Queues Replica 1 - `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -789,15 +776,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.61`: Redis - Queues Primary - `10.6.0.62`: Redis - Queues Replica 1 - `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 ### Providing your own Redis instance @@ -809,7 +790,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -827,8 +808,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -840,8 +825,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -875,10 +871,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -886,11 +878,15 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`, e.g. ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -902,15 +898,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - # The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -946,10 +946,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a `gitlab-ctl reconfigure`, they will get their configuration restored by @@ -964,133 +960,15 @@ are supported and can be added if needed. -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - -
- - Back to setup components - -
- -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster -This is the section where we install and set up the new Redis Queues instances. +This is the section where we install and set up the new Redis Persistent instances. Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1100,8 +978,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1113,8 +995,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1138,13 +1031,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -#### Configure the replica Redis Queues nodes +#### Configure the replica Redis Persistent nodes -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1152,8 +1041,12 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1165,16 +1058,20 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1199,10 +1096,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a `gitlab-ctl reconfigure`, they will get their configuration restored by @@ -1217,124 +1110,6 @@ are supported and can be added if needed. -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. - -
- - Back to setup components - -
- ## Configure Gitaly Cluster [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. @@ -1854,30 +1629,30 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly Cluster @@ -2029,30 +1804,30 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actionable gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2413,9 +2188,7 @@ services where applicable): | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node3 | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Redis - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | @@ -2471,11 +2244,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white -- GitLab From 3b1c62eaf5195511ff71cc0cea68e3cdf58b9418 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 17 Sep 2021 14:07:27 +0100 Subject: [PATCH 03/12] Removed unneeded sections --- doc/administration/reference_architectures/10k_users.md | 5 ----- doc/administration/reference_architectures/25k_users.md | 5 ----- doc/administration/reference_architectures/50k_users.md | 5 ----- 3 files changed, 15 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 3eba96dcbd5ab2..0b49106a037c1f 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -946,11 +946,6 @@ a node and change its status from primary to replica (and vice versa). 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index f500434d75b642..8f1f0a8108cf65 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -956,11 +956,6 @@ You can specify multiple roles, like sentinel and Redis, as: `roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index b071b48cbd95e1..05e6224a8ba433 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -964,11 +964,6 @@ You can specify multiple roles, like sentinel and Redis, as: `roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. -- GitLab From d12faa9ab63a48e958bb372b10d1cc29b64534a5 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 17 Sep 2021 14:18:54 +0100 Subject: [PATCH 04/12] Fix broken links --- doc/administration/reference_architectures/10k_users.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 0b49106a037c1f..97c88bce1f80b3 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1894,7 +1894,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2038,7 +2038,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better +It's recommended over [NFS](#configure-nfs) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. -- GitLab From 3dce9d646e97cb7ef64234fb24d4a111cad6223c Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 17 Sep 2021 14:31:05 +0100 Subject: [PATCH 05/12] Fix furthern links and mistaken removals --- doc/administration/reference_architectures/10k_users.md | 7 +------ doc/administration/reference_architectures/25k_users.md | 5 +++++ doc/administration/reference_architectures/50k_users.md | 5 +++++ 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 97c88bce1f80b3..b151e4ac0bbf1e 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1091,11 +1091,6 @@ a node and change its status from primary to replica (and vice versa). 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -2108,7 +2103,7 @@ cluster alongside your instance, read how to -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 8f1f0a8108cf65..f500434d75b642 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -956,6 +956,11 @@ You can specify multiple roles, like sentinel and Redis, as: `roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 05e6224a8ba433..b071b48cbd95e1 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -964,6 +964,11 @@ You can specify multiple roles, like sentinel and Redis, as: `roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about [roles](https://docs.gitlab.com/omnibus/roles/). +These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after +a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a +`gitlab-ctl reconfigure`, they will get their configuration restored by +the same Sentinels. + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. -- GitLab From 6b924efe29cbbbe9c1cda374cc87b854ce98f7d2 Mon Sep 17 00:00:00 2001 From: admiralboom Date: Fri, 17 Sep 2021 14:17:22 -0400 Subject: [PATCH 06/12] Updating the 25k ref arch to drop the separate sentinel nodes from redis combining sentinel with redis. --- .../reference_architectures/10k_users.md | 2 +- .../reference_architectures/25k_users.md | 502 +++++------------- 2 files changed, 124 insertions(+), 380 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index b151e4ac0bbf1e..c6210e089104bb 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -31,7 +31,7 @@ full list of reference architectures, see | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage4 | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index f500434d75b642..b379afb475b5be 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -22,10 +22,8 @@ full list of reference architectures, see | PostgreSQL1 | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node3 | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | -| Redis - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State2| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis/Sentinel - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` | | Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | @@ -33,7 +31,7 @@ full list of reference architectures, see | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage4 | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | @@ -82,11 +80,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +161,9 @@ To set up GitLab and its components to accommodate up to 25,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs-optional) + to have shared disk storage service for certain GitLab operations (non + Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +184,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.61`: Redis - Queues Primary - `10.6.0.62`: Redis - Queues Replica 1 - `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -794,15 +779,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.61`: Redis - Queues Primary - `10.6.0.62`: Redis - Queues Replica 1 - `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 ### Providing your own Redis instance @@ -814,7 +793,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -832,10 +811,14 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role'] + # Specify server role as 'redis_sentinel_role' 'redis_master_role' + roles(['redis_sentinel_role' 'redis_master_role'] - # IP address pointing to a local IP that the other machines can reach to. + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 + + # IP address pointing to a local IP that the other machines can reach. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make # sure you add extra firewall rules to prevent unauthorized access. @@ -845,8 +828,18 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 + # Port of primary Redis server for Sentinel, uncomment to change to non default. + # Defaults to `6379`. + # redis['master_port'] = 6379 + # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # Must be the same in every Redis node. + redis['master_name'] = 'gitlab-redis-cache' + + # The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -880,10 +873,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -891,14 +880,18 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section replaceing `redis_master_node` with `redis_replica_node`, e.g. ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role'] + roles(['redis_sentinel_role', 'redis_replica_role'] - # IP address pointing to a local IP that the other machines can reach to. - # You can also set bind to '0.0.0.0' which listen in all interfaces. + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = `0.0.0.0` + sentinel['quorum'] = 2 + + # IP address pointing to a local IP that the other machines can reach. + # Set bind to '0.0.0.0' to listen on all interfaces. # If you really need to bind to an external accessible IP, make # sure you add extra firewall rules to prevent unauthorized access. redis['bind'] = '10.6.0.52' @@ -907,16 +900,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + # Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.51' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - # Set the Redis Cache instance as an LRU # 90% of available RAM in MB redis['maxmemory'] = '13500mb' @@ -952,152 +948,23 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. - - - -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - - + -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster -This is the section where we install and set up the new Redis Queues instances. +This is the section where we install and set up the new Redis Persistent instances. Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1107,8 +974,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' + roles ['redis_sentinel_role', 'redis_master_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1120,8 +991,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1145,13 +1027,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). +#### Configure the replica Redis Persistent nodes -#### Configure the replica Redis Queues nodes - -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1160,7 +1038,14 @@ You can specify multiple roles, like sentinel and Redis, as: ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + roles(['redis_sentinel_role', 'redis_replica_role']) + + # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' + roles ['redis_sentinel_role', 'redis_replica_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1172,16 +1057,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1211,15 +1099,6 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. @@ -1229,136 +1108,8 @@ are supported and can be added if needed. -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. - - - -## Configure Gitaly Cluster - -[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended +fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. The recommended cluster setup includes the following components: @@ -1875,30 +1626,30 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly Cluster @@ -2052,9 +1803,9 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] ## Second cluster that will host the queues, shared state, and actionable @@ -2063,19 +1814,19 @@ On each node perform the following: gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2147,7 +1898,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2289,9 +2040,9 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better -in larger setups as object storage is typically much more performant, reliable, -and scalable. +Object storage is also recommended over [NFS](#configure-nfs) and in general +it's better in larger setups as object storage is typically much more performant, +reliable, and scalable. GitLab has been tested on a number of object storage providers: @@ -2359,7 +2110,7 @@ cluster alongside your instance, read how to -## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2427,10 +2178,8 @@ services where applicable): | PostgreSQL1 | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node3 | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | -| Redis - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis/Sentinel - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | @@ -2486,11 +2235,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white -- GitLab From b95c18526a3cd2bf475ed905198ebcab445d54d7 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 22 Sep 2021 12:47:07 +0000 Subject: [PATCH 07/12] Apply 3 suggestion(s) to 1 file(s) --- doc/administration/reference_architectures/25k_users.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index b379afb475b5be..82dacb083d7572 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -31,7 +31,7 @@ full list of reference architectures, see | GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage4 | n/a | n/a | n/a | n/a | n/a | -| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | @@ -161,7 +161,7 @@ To set up GitLab and its components to accommodate up to 25,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) +1. [Configure NFS](#configure-nfs) to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). @@ -1108,7 +1108,9 @@ are supported and can be added if needed. -[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended +## Configure Gitaly Cluster + +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab-provided and recommended fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. -- GitLab From a57b48d89b80e66ff8c3a069cbb045cb78da8a33 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 22 Sep 2021 12:52:35 +0000 Subject: [PATCH 08/12] Apply 9 suggestion(s) to 2 file(s) --- .../reference_architectures/10k_users.md | 8 ++++---- .../reference_architectures/25k_users.md | 13 +++++-------- 2 files changed, 9 insertions(+), 12 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index c6210e089104bb..9444af3f58900c 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -809,7 +809,7 @@ a node and change its status from primary to replica (and vice versa). ```ruby # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' - roles ['redis_sentinel_role', 'redis_master_role'] + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service sentinel['bind'] = '0.0.0.0' @@ -882,7 +882,7 @@ a node and change its status from primary to replica (and vice versa). ```ruby # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' - roles ['redis_sentinel_role', 'redis_replica_role'] + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service sentinel['bind'] = '0.0.0.0' @@ -974,7 +974,7 @@ a node and change its status from primary to replica (and vice versa). ```ruby # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' - roles ['redis_sentinel_role', 'redis_master_role'] + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service sentinel['bind'] = '0.0.0.0' @@ -1037,7 +1037,7 @@ a node and change its status from primary to replica (and vice versa). ```ruby # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' - roles ['redis_sentinel_role', 'redis_replica_role'] + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service sentinel['bind'] = '0.0.0.0' diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 82dacb083d7572..b82dfdbb11623e 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -812,7 +812,7 @@ a node and change its status from primary to replica (and vice versa). ```ruby # Specify server role as 'redis_sentinel_role' 'redis_master_role' - roles(['redis_sentinel_role' 'redis_master_role'] + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service sentinel['bind'] = '0.0.0.0' @@ -880,11 +880,11 @@ a node and change its status from primary to replica (and vice versa). package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section replaceing `redis_master_node` with `redis_replica_node`, e.g. +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`: ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_sentinel_role', 'redis_replica_role'] + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service sentinel['bind'] = `0.0.0.0` @@ -975,7 +975,7 @@ a node and change its status from primary to replica (and vice versa). ```ruby # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' - roles ['redis_sentinel_role', 'redis_master_role'] + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service sentinel['bind'] = '0.0.0.0' @@ -1038,10 +1038,7 @@ a node and change its status from primary to replica (and vice versa). ```ruby # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_sentinel_role', 'redis_replica_role']) - - # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' - roles ['redis_sentinel_role', 'redis_replica_role'] + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service sentinel['bind'] = '0.0.0.0' -- GitLab From 28871b45da60e1dc10583ba574ae04f3039bfa82 Mon Sep 17 00:00:00 2001 From: admiralboom Date: Wed, 22 Sep 2021 15:50:25 -0400 Subject: [PATCH 09/12] 50k reference arch combining sentinel to redis --- .../reference_architectures/25k_users.md | 2 +- .../reference_architectures/50k_users.md | 502 +++++------------- 2 files changed, 125 insertions(+), 379 deletions(-) diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index b82dfdbb11623e..deb6d7f266aac5 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -23,7 +23,7 @@ full list of reference architectures, see | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node3 | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | | Redis/Sentinel - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis/Sentinel - Persistent2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` | +| Redis/Sentinel - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` | | Gitaly | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index b071b48cbd95e1..6e7bcec0eb582c 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -22,10 +22,8 @@ full list of reference architectures, see | PostgreSQL1 | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node3 | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | -| Redis - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Queues / Shared State2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis Sentinel - Cache2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | -| Redis Sentinel - Queues / Shared State2| 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `c5.large` | `A1 v2` | +| Redis/Sentinel - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | `D64s v3` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | @@ -33,7 +31,7 @@ full list of reference architectures, see | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage4 | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | @@ -82,11 +80,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white @@ -137,8 +130,7 @@ our [Sysbench](https://github.com/akopytov/sysbench)-based Due to better performance and availability, for data objects (such as LFS, uploads, or artifacts), using an [object storage service](#configure-the-object-storage) -is recommended instead of using NFS. Using an object storage service also -doesn't require you to provision and maintain a node. +is recommended. It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third-party PostgreSQL database solution will be required. @@ -169,10 +161,8 @@ To set up GitLab and its components to accommodate up to 50,000 users: used for shared data objects. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) - to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). +1. [Configure NFS](#configure-nfs) + to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -193,15 +183,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.61`: Redis - Queues Primary - `10.6.0.62`: Redis - Queues Replica 1 - `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -802,15 +786,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 - `10.6.0.61`: Redis - Queues Primary - `10.6.0.62`: Redis - Queues Replica 1 - `10.6.0.63`: Redis - Queues Replica 2 -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 ### Providing your own Redis instance @@ -822,7 +800,7 @@ optional count argument to SPOP, which is required for [Merge Trains](../../ci/p Note the Redis node's IP address or hostname, port, and password (if required). These will be necessary later when configuring the [GitLab application servers](#configure-gitlab-rails). -### Configure the Redis and Sentinel Cache cluster +### Configure the Redis Cache cluster This is the section where we install and set up the new Redis Cache instances. @@ -840,8 +818,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server role as 'redis_master_role' with Sentinel and enable Consul agent + roles(['redis_sentinel_role', 'redis_master_role', 'consul_role']) + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -853,8 +835,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -888,10 +881,6 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - #### Configure the replica Redis Cache nodes 1. SSH in to the **replica** Redis server. @@ -899,11 +888,15 @@ You can specify multiple roles, like sentinel and Redis, as: package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`, e.g. ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent + roles(['roles_sentinel_role', 'redis_replica_role', 'consul_role']) + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -915,15 +908,19 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 - # The same password for Redis authentication you set up for the primary node. + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - # The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-cache' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 + ## The IP of the primary Redis node. + redis['master_ip'] = '10.6.0.51' # Set the Redis Cache instance as an LRU # 90% of available RAM in MB @@ -952,152 +949,26 @@ You can specify multiple roles, like sentinel and Redis, as: gitlab_rails['auto_migrate'] = false ``` -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus + node you configured and add or replace the file of the same name on this + server. If this is the first Omnibus node you are configuring then you + can skip this step. -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes + to take effect. -1. Go through the steps again for all the other replica nodes, and + 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-cache-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. - - - -#### Configure the Sentinel Cache nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.71`: Sentinel - Cache 1 -- `10.6.0.72`: Sentinel - Cache 2 -- `10.6.0.73`: Sentinel - Cache 3 + Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Cache server: - -1. SSH in to the server that will host Consul/Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-cache' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.51' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.71' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Consul/Sentinel nodes, and - make sure you set up the correct IPs. - - + -### Configure the Redis and Sentinel Queues cluster +### Configure the Redis Persistent cluster This is the section where we install and set up the new Redis Queues instances. @@ -1105,7 +976,7 @@ Both the primary and replica Redis nodes need the same password defined in `redis['password']`. At any time during a failover, the Sentinels can reconfigure a node and change its status from primary to replica (and vice versa). -#### Configure the primary Redis Queues node +#### Configure the primary Redis Persistent node 1. SSH in to the **Primary** Redis server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -1115,8 +986,12 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_master_role' and enable Consul agent - roles(['redis_master_role', 'consul_role']) + # Specify server roles as 'redis_master_role' with Sentinel and enable the Consul agent + roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1128,8 +1003,19 @@ a node and change its status from primary to replica (and vice versa). # machines to connect to it. redis['port'] = 6379 - # Set up password authentication for Redis (use the same password in all nodes). - redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + + # Set up password authentication for Redis and replicas (use the same password in all nodes). + redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_FIRST_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' + + ## The IP of this primary Redis node. + redis['master_ip'] = '10.6.0.61' ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1153,13 +1039,9 @@ a node and change its status from primary to replica (and vice versa). 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -You can specify multiple roles, like sentinel and Redis, as: -`roles ['redis_sentinel_role', 'redis_master_role']`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -#### Configure the replica Redis Queues nodes +#### Configure the replica Redis Persistent nodes -1. SSH in to the **replica** Redis Queue server. +1. SSH in to the **replica** Redis Persistent server. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -1167,8 +1049,12 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_replica_role' and enable Consul agent - roles(['redis_replica_role', 'consul_role']) + # Specify server roles as 'redis_replica_role' with Sentinel and enable Consul agent + roles ['redis_sentinel_role', 'redis_replica_role', 'consul_role'] + + # Set IP bind address and Quorum number for Redis Sentinel service + sentinel['bind'] = '0.0.0.0' + sentinel['quorum'] = 2 # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -1180,16 +1066,20 @@ You can specify multiple roles, like sentinel and Redis, as: # machines to connect to it. redis['port'] = 6379 + ## Port of primary Redis server for Sentinel, uncomment to change to non default. Defaults + ## to `6379`. + #redis['master_port'] = 6379 + # The same password for Redis authentication you set up for the primary node. redis['password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' + + ## Must be the same in every Redis node + redis['master_name'] = 'gitlab-redis-persistent' # The IP of the primary Redis node. redis['master_ip'] = '10.6.0.61' - # Port of primary Redis server, uncomment to change to non default. Defaults - # to `6379`. - #redis['master_port'] = 6379 - ## Enable service discovery for Prometheus consul['monitoring_service_discovery'] = true @@ -1215,144 +1105,7 @@ You can specify multiple roles, like sentinel and Redis, as: 1. Go through the steps again for all the other replica nodes, and make sure to set up the IPs correctly. -You can specify multiple roles, like sentinel and Redis, as: -`roles(['redis_sentinel_role', 'redis_master_role'])`. Read more about -[roles](https://docs.gitlab.com/omnibus/roles/). - -These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-the-sentinel-queues-nodes), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by -the same Sentinels. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) -are supported and can be added if needed. - - - -#### Configure the Sentinel Queues nodes - -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: - -- `10.6.0.81`: Sentinel - Queues 1 -- `10.6.0.82`: Sentinel - Queues 2 -- `10.6.0.83`: Sentinel - Queues 3 - -NOTE: -If you're using an external Redis Sentinel instance, be sure to exclude the -`requirepass` parameter from the Sentinel configuration. This parameter causes -clients to report `NOAUTH Authentication required.`. -[Redis Sentinel 3.2.x doesn't support password authentication](https://github.com/antirez/redis/issues/3279). - -To configure the Sentinel Queues server: - -1. SSH in to the server that will host Sentinel. -1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab - package of your choice. Be sure to both follow _only_ installation steps 1 and 2 - on the page, and to select the correct Omnibus GitLab package, with the same version - and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the contents: - - ```ruby - roles(['redis_sentinel_role', 'consul_role']) - - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis-persistent' - - ## The same password for Redis authentication you set up for the primary node. - redis['master_password'] = 'REDIS_PRIMARY_PASSWORD_OF_SECOND_CLUSTER' - - ## The IP of the primary Redis node. - redis['master_ip'] = '10.6.0.61' - - ## Define a port so Redis can listen for TCP requests which will allow other - ## machines to connect to it. - redis['port'] = 6379 - - ## Port of primary Redis server, uncomment to change to non default. Defaults - ## to `6379`. - #redis['master_port'] = 6379 - - ## Configure Sentinel's IP - sentinel['bind'] = '10.6.0.81' - - ## Port that Sentinel listens on, uncomment to change to non default. Defaults - ## to `26379`. - #sentinel['port'] = 26379 - - ## Quorum must reflect the amount of voting sentinels it take to start a failover. - ## Value must NOT be greater then the amount of sentinels. - ## - ## The quorum can be used to tune Sentinel in two ways: - ## 1. If a the quorum is set to a value smaller than the majority of Sentinels - ## we deploy, we are basically making Sentinel more sensible to primary failures, - ## triggering a failover as soon as even just a minority of Sentinels is no longer - ## able to talk with the primary. - ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are - ## making Sentinel able to failover only when there are a very large number (larger - ## than majority) of well connected Sentinels which agree about the primary being down.s - sentinel['quorum'] = 2 - - ## Consider unresponsive server down after x amount of ms. - #sentinel['down_after_milliseconds'] = 10000 - - ## Specifies the failover timeout in milliseconds. It is used in many ways: - ## - ## - The time needed to re-start a failover after a previous failover was - ## already tried against the same primary by a given Sentinel, is two - ## times the failover timeout. - ## - ## - The time needed for a replica replicating to a wrong primary according - ## to a Sentinel current configuration, to be forced to replicate - ## with the right primary, is exactly the failover timeout (counting since - ## the moment a Sentinel detected the misconfiguration). - ## - ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (REPLICAOF NO ONE yet not - ## acknowledged by the promoted replica). - ## - ## - The maximum time a failover in progress waits for all the replica to be - ## reconfigured as replicas of the new primary. However even after this time - ## the replicas will be reconfigured by the Sentinels anyway, but not with - ## the exact parallel-syncs progression as specified. - #sentinel['failover_timeout'] = 60000 - - ## Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - ## The IPs of the Consul server nodes - ## You can also use FQDNs and intermix them with IPs - consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), - } - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - - # Prevent database migrations from running on upgrade automatically - gitlab_rails['auto_migrate'] = false - ``` - -1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: - - ```shell - sudo touch /etc/gitlab/skip-auto-reconfigure - ``` - - Only the primary GitLab application server should handle migrations. - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the first Omnibus node you configured and add or replace - the file of the same name on this server. If this is the first Omnibus node you are configuring then you can skip this step. - -1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Go through the steps again for all the other Sentinel nodes, and - make sure you set up the correct IPs. +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed.
@@ -1879,30 +1632,30 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actioncable + ## Second cluster that will host the persistent queues, shared state, and actioncable gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Gitaly @@ -2063,30 +1816,30 @@ On each node perform the following: gitlab_rails['redis_cache_instance'] = 'redis://:@gitlab-redis-cache' gitlab_rails['redis_cache_sentinels'] = [ - {host: '10.6.0.71', port: 26379}, - {host: '10.6.0.72', port: 26379}, - {host: '10.6.0.73', port: 26379}, + {host: '10.6.0.51', port: 26379}, + {host: '10.6.0.52', port: 26379}, + {host: '10.6.0.53', port: 26379}, ] - ## Second cluster that will host the queues, shared state, and actionable + ## Second cluster that will host the persistent queues, shared state, and actionable gitlab_rails['redis_queues_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_shared_state_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_actioncable_instance'] = 'redis://:@gitlab-redis-persistent' gitlab_rails['redis_queues_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_shared_state_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] gitlab_rails['redis_actioncable_sentinels'] = [ - {host: '10.6.0.81', port: 26379}, - {host: '10.6.0.82', port: 26379}, - {host: '10.6.0.83', port: 26379}, + {host: '10.6.0.61', port: 26379}, + {host: '10.6.0.62', port: 26379}, + {host: '10.6.0.63', port: 26379}, ] # Set the network addresses that the exporters used for monitoring will listen on @@ -2158,7 +1911,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs-optional): +1. If you're [using NFS](#configure-nfs): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2300,7 +2053,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs-optional) and in general it's better +It's recommended over [NFS](#configure-nfs) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2370,7 +2123,7 @@ cluster alongside your instance, read how to
-## Configure NFS (optional) +## Configure NFS [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. If you intend @@ -2438,10 +2191,8 @@ services where applicable): | PostgreSQL1 | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node3 | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | -| Redis - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Queues / Shared State2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis Sentinel - Cache2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Redis Sentinel - Queues / Shared State2 | 3 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | +| Redis/Sentinel - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Gitaly | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | | Praefect | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | @@ -2497,11 +2248,6 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 - collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - - redis_persistent <.[#FF6347]- redis_persistent_sentinel - redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white -- GitLab From 3896ccba5e868476f032e2ce093463708aea3196 Mon Sep 17 00:00:00 2001 From: admiralboom Date: Wed, 22 Sep 2021 15:55:26 -0400 Subject: [PATCH 10/12] making 10k match verbiage --- .../reference_architectures/10k_users.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 9444af3f58900c..412e9be4b52639 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -22,8 +22,8 @@ full list of reference architectures, see | PostgreSQL1 | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node3 | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Redis - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis/Sentinel - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | @@ -808,7 +808,7 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' + # Specify server roles as 'redis_master_role' with sentinel and the Consul agent roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service @@ -973,7 +973,7 @@ a node and change its status from primary to replica (and vice versa). 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server roles as 'redis_sentinel_role' and 'redis_master_role' + # Specify server roles as 'redis_master_role' with Sentinel and the Consul agent roles ['redis_sentinel_role', 'redis_master_role', 'consul_role'] # Set IP bind address and Quorum number for Redis Sentinel service @@ -2177,8 +2177,8 @@ services where applicable): | PostgreSQL1 | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | | PgBouncer1 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Internal load balancing node3 | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Cache2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis/Sentinel - Persistent2 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | | Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | | Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -- GitLab From d2089e079903a5b3f47c70b30635629d009de507 Mon Sep 17 00:00:00 2001 From: admiralboom Date: Wed, 22 Sep 2021 16:29:28 -0400 Subject: [PATCH 11/12] Gitaly not gitaly --- doc/administration/reference_architectures/50k_users.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 6e7bcec0eb582c..24c8441e3bf3f3 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -31,7 +31,7 @@ full list of reference architectures, see | GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Object storage4 | n/a | n/a | n/a | n/a | n/a | -| NFS server (non-gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | -- GitLab From d70d73a70b0ba38f8795a3c1996d488cc97e15b7 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Mon, 27 Sep 2021 08:27:06 +0000 Subject: [PATCH 12/12] Apply 10 suggestion(s) to 3 file(s) --- .../reference_architectures/10k_users.md | 14 +++++------ .../reference_architectures/25k_users.md | 22 ++++++++--------- .../reference_architectures/50k_users.md | 24 +++++++++---------- 3 files changed, 30 insertions(+), 30 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 412e9be4b52639..0f29fe4a5f2762 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -183,9 +183,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -776,9 +776,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -878,7 +878,7 @@ a node and change its status from primary to replica (and vice versa). package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`, e.g. +1. Edit `/etc/gitlab/gitlab.rb` and add same contents as the primary node in the previous section replacing `redis_master_node` with `redis_replica_node`: ```ruby # Specify server roles as 'redis_sentinel_role' and 'redis_replica_role' diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index deb6d7f266aac5..4d51aaa3030756 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -184,9 +184,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -779,9 +779,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -950,11 +950,11 @@ a node and change its status from primary to replica (and vice versa). Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. - + ### Configure the Redis Persistent cluster diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 24c8441e3bf3f3..166ae311fee77f 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -183,9 +183,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 - `10.6.0.93`: Gitaly 3 @@ -786,9 +786,9 @@ to be used with GitLab. The following IPs will be used as an example: - `10.6.0.51`: Redis - Cache Primary - `10.6.0.52`: Redis - Cache Replica 1 - `10.6.0.53`: Redis - Cache Replica 2 -- `10.6.0.61`: Redis - Queues Primary -- `10.6.0.62`: Redis - Queues Replica 1 -- `10.6.0.63`: Redis - Queues Replica 2 +- `10.6.0.61`: Redis - Persistent Primary +- `10.6.0.62`: Redis - Persistent Replica 1 +- `10.6.0.63`: Redis - Persistent Replica 2 ### Providing your own Redis instance @@ -888,7 +888,7 @@ a node and change its status from primary to replica (and vice versa). package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`, e.g. +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`: ```ruby # Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent @@ -962,11 +962,11 @@ a node and change its status from primary to replica (and vice versa). Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. - + ### Configure the Redis Persistent cluster -- GitLab