From 8c3e86938fa50c39c7d52f8af1c539850cc6165b Mon Sep 17 00:00:00 2001 From: Grant Young Date: Mon, 18 Jan 2021 17:17:03 +0000 Subject: [PATCH 01/18] Update Ref Arch docs with Gitaly Cluster --- .../reference_architectures/10k_users.md | 165 ++++++++++-------- 1 file changed, 88 insertions(+), 77 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index afa53b5efa8232..b9a4567878c015 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -19,93 +19,104 @@ full list of reference architectures, see |--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| | External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| PostgreSQL | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | | PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | | Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 2 (minimum) | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| Gitaly Cluster | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Praefect Postgres | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | - -```mermaid -stateDiagram-v2 - [*] --> LoadBalancer - LoadBalancer --> ApplicationServer - - ApplicationServer --> BackgroundJobs - ApplicationServer --> Gitaly - ApplicationServer --> Redis_Cache - ApplicationServer --> Redis_Queues - ApplicationServer --> PgBouncer - PgBouncer --> Database - ApplicationServer --> ObjectStorage - BackgroundJobs --> ObjectStorage - - ApplicationMonitoring -->ApplicationServer - ApplicationMonitoring -->PgBouncer - ApplicationMonitoring -->Database - ApplicationMonitoring -->BackgroundJobs - - ApplicationServer --> Consul - - Consul --> Database - Consul --> PgBouncer - Redis_Cache --> Consul - Redis_Queues --> Consul - BackgroundJobs --> Consul - - state Consul { - "Consul_1..3" - } - - state Database { - "PG_Primary_Node" - "PG_Secondary_Node_1..2" - } - - state Redis_Cache { - "R_Cache_Primary_Node" - "R_Cache_Replica_Node_1..2" - "R_Cache_Sentinel_1..3" - } - - state Redis_Queues { - "R_Queues_Primary_Node" - "R_Queues_Replica_Node_1..2" - "R_Queues_Sentinel_1..3" - } - - state Gitaly { - "Gitaly_1..2" - } - - state BackgroundJobs { - "Sidekiq_1..4" - } - - state ApplicationServer { - "GitLab_Rails_1..3" - } - - state LoadBalancer { - "LoadBalancer_1" - } - - state ApplicationMonitoring { - "Prometheus" - "Grafana" - } - - state PgBouncer { - "Internal_Load_Balancer" - "PgBouncer_1..3" - } + +```plantuml +@startuml 10k +card "**External Load Balancer**" as elb #6a9be7 +card "**Internal Load Balancer**" as ilb #9370DB + +together { + collections "**GitLab Rails** x3" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} + +together { + collections "**Consul** x3" as consul #e76a9b + card "**Prometheus + Grafana**" as monitor #7FFFD4 +} + +card "Gitaly Cluster" as gitaly_cluster { + collections "**Praefect** x3" as praefect #FF8C00 + collections "**Gitaly Cluster** x3" as gitaly #FF8C00 + card "**Praefect Postgres***" as praefect_postgres #FF8C00 + + praefect -[#FF8C00]-> gitaly + praefect -[#FF8C00]-> praefect_postgres: **//Non fault-tolerant//** +} + +card "Database" as database { + collections "**PGBouncer** x3" as pgbouncer #4EA7FF + card "**Postgres** (Primary)" as postgres_primary #4EA7FF + collections "**Postgres** (Secondary) x2" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]> postgres_secondary +} + +card "redis" as redis { + collections "**Redis Persistent** x3" as redis_persistant #FF6347 + collections "**Redis Cache** x3" as redis_cache #FF6347 + collections "**Redis Persistent Sentinel** x3" as redis_persistant_sentinel #FF6347 + collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 + + redis_persistant <.[#FF6347]- redis_persistant_sentinel + redis_cache <.[#FF6347]- redis_cache_sentinel +} + +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[hidden]-> sidekiq +elb -[#6a9be7]--> monitor + +gitlab -[#32CD32]> sidekiq +gitlab -[#32CD32]--> ilb +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis +gitlab -[hidden]-> monitor +gitlab -[hidden]-> consul + +sidekiq -[#ff8dd1]--> ilb +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis +sidekiq -[hidden]-> monitor +sidekiq -[hidden]-> consul + +ilb -[#9370DB]-> database +ilb -[#9370DB]-> gitaly_cluster +ilb -[norank,hidden]-> redis + +consul .[#e76a9b]u-> gitlab +consul .[#e76a9b]u-> sidekiq +consul .[#e76a9b]> monitor +consul .[#e76a9b]-> database +consul .[#e76a9b]-> gitaly_cluster +consul .[#e76a9b]-> redis + +monitor .[#7FFFD4]u-> gitlab +monitor .[#7FFFD4]u-> sidekiq +monitor .[#7FFFD4]> consul +monitor .[#7FFFD4]-> database +monitor .[#7FFFD4]-> gitaly_cluster +monitor .[#7FFFD4]-> redis +monitor .[#7FFFD4]-> ilb +monitor .[#7FFFD4,norank]u--> elb + +@enduml ``` The Google Cloud Platform (GCP) architectures were built and tested using the -- GitLab From c7edb1c889f7949d1ddd4ad564d39d90021f02aa Mon Sep 17 00:00:00 2001 From: Grant Young Date: Mon, 18 Jan 2021 17:39:57 +0000 Subject: [PATCH 02/18] Tweak new RA diagram --- .../reference_architectures/10k_users.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index b9a4567878c015..43168ba50ab814 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -45,8 +45,8 @@ together { } together { - collections "**Consul** x3" as consul #e76a9b card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b } card "Gitaly Cluster" as gitaly_cluster { @@ -85,35 +85,35 @@ elb -[#6a9be7]--> monitor gitlab -[#32CD32]> sidekiq gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32]--> object_storage +gitlab -[#32CD32,norank]---> redis gitlab -[hidden]-> monitor gitlab -[hidden]-> consul sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1]--> object_storage +sidekiq -[#ff8dd1,norank]---> redis sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul ilb -[#9370DB]-> database ilb -[#9370DB]-> gitaly_cluster -ilb -[norank,hidden]-> redis +ilb -[hidden]-> redis consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq consul .[#e76a9b]> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b]-> redis +consul .[#e76a9b,norank]--> redis monitor .[#7FFFD4]u-> gitlab monitor .[#7FFFD4]u-> sidekiq monitor .[#7FFFD4]> consul monitor .[#7FFFD4]-> database monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4]-> redis -monitor .[#7FFFD4]-> ilb +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4]> ilb monitor .[#7FFFD4,norank]u--> elb @enduml -- GitLab From 33f81e3b9e5bbcf016afbecbb98384dfc097d55d Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 19 Jan 2021 10:33:10 +0000 Subject: [PATCH 03/18] Further tweaks to RA 10k diagram --- .../reference_architectures/10k_users.md | 22 +++++++++---------- 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 43168ba50ab814..88b4495882b44f 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -52,10 +52,10 @@ together { card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 collections "**Gitaly Cluster** x3" as gitaly #FF8C00 - card "**Praefect Postgres***" as praefect_postgres #FF8C00 + card "**Praefect Postgres***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 praefect -[#FF8C00]-> gitaly - praefect -[#FF8C00]-> praefect_postgres: **//Non fault-tolerant//** + praefect -[#FF8C00]> praefect_postgres } card "Database" as database { @@ -68,37 +68,35 @@ card "Database" as database { } card "redis" as redis { - collections "**Redis Persistent** x3" as redis_persistant #FF6347 + collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - collections "**Redis Persistent Sentinel** x3" as redis_persistant_sentinel #FF6347 + collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - redis_persistant <.[#FF6347]- redis_persistant_sentinel + redis_persistent <.[#FF6347]- redis_persistent_sentinel redis_cache <.[#FF6347]- redis_cache_sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[hidden]-> sidekiq elb -[#6a9be7]--> monitor gitlab -[#32CD32]> sidekiq gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]--> object_storage -gitlab -[#32CD32,norank]---> redis +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis gitlab -[hidden]-> monitor gitlab -[hidden]-> consul sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]--> object_storage -sidekiq -[#ff8dd1,norank]---> redis +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> database ilb -[#9370DB]-> gitaly_cluster -ilb -[hidden]-> redis +ilb -[#9370DB]-> database consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -- GitLab From d63dfb73c15e81111eeecab5719cafc44fc86400 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 19 Jan 2021 13:42:54 +0000 Subject: [PATCH 04/18] Add in new sections and Praefect PG guidance --- .../reference_architectures/10k_users.md | 335 +++++++++++++++--- 1 file changed, 277 insertions(+), 58 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 88b4495882b44f..e84a98d5eda9c3 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -12,8 +12,10 @@ full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 10,000 -> - **High Availability:** Yes +> - **High Availability:** Yes* > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS +> +> _\* At this time Praefect requires a third party PostgreSQL database solution to achieve full High Availability. We hope to offer a built in solution for this in the future._ | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| @@ -28,11 +30,12 @@ full list of reference architectures, see | Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | | Gitaly Cluster | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | | Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Praefect Postgres | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | | Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | ```plantuml @startuml 10k @@ -52,7 +55,7 @@ together { card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 collections "**Gitaly Cluster** x3" as gitaly #FF8C00 - card "**Praefect Postgres***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 + card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 praefect -[#FF8C00]-> gitaly praefect -[#FF8C00]> praefect_postgres @@ -60,8 +63,8 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**Postgres** (Primary)" as postgres_primary #4EA7FF - collections "**Postgres** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF + collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary @@ -129,19 +132,25 @@ uploads, or artifacts), using an [object storage service](#configure-the-object- is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. +It's also worth noting that at this time [Praefect requires it's own database server][Gitaly Cluster](../gitaly/praefect.md#postgresql) and +that to achieve full High Availability a third party PostgreSQL database solution will be required. +We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server +can be set up via GitLab Omnibus, which the above specs reflect. + ## Setup components To set up GitLab and its components to accommodate up to 10,000 users: -1. [Configure the external load balancing node](#configure-the-external-load-balancer) +1. [Configure the external load balancer](#configure-the-external-load-balancer) to handle the load balancing of the GitLab application services nodes. +1. [Configure the internal load balancer](#configure-the-internal-load-balancer). + to handle the load balancing of GitLab application internal connections. 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). -1. [Configure the internal load balancing node](#configure-the-internal-load-balancer). 1. [Configure Redis](#configure-redis). -1. [Configure Gitaly](#configure-gitaly), - which provides access to the Git repositories. +1. [Configure Gitaly Cluster](#configure-gitaly-cluster), + provides access to the Git repositories. 1. [Configure Sidekiq](#configure-sidekiq). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails) to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend @@ -185,8 +194,13 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.81`: Sentinel - Queues 1 - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -- `10.6.0.91`: Gitaly 1 -- `10.6.0.92`: Gitaly 2 +- `10.6.0.91`: Gitaly Cluster 1 +- `10.6.0.92`: Gitaly Cluster 2 +- `10.6.0.93`: Gitaly Cluster 3 +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.134`: Praefect 3 +- `10.6.0.141`: Praefect Postgres 1 (non HA) - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -317,6 +331,71 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. +## Configure the internal load balancer + +The Internal Load Balancer is used to balance any internal connections the GitLab environment requires +such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configure-praefect) (Gitaly Cluster). + +Note that it's a separate node from the External Load Balancer and shouldn't have any access externally. + +The following IP will be used as an example: + +- `10.6.0.40`: Internal Load Balancer + +Here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +frontend internal-praefect-tcp-in + bind *:2305 + mode tcp + option tcplog + option clitcpka + + default_backend praefect + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 10.6.0.21:6432 check + server pgbouncer2 10.6.0.22:6432 check + server pgbouncer3 10.6.0.23:6432 check + +backend praefect + mode tcp + option tcp-check + option srvtcpka + + server praefect1 10.6.0.131:2305 check + server praefect2 10.6.0.132:2305 check + server praefect3 10.6.0.133:2305 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + +
+ + Back to setup components + +
+ ## Configure Consul The following IPs will be used as an example: @@ -671,52 +750,6 @@ The following IPs will be used as an example: -### Configure the internal load balancer - -If you're running more than one PgBouncer node as recommended, then at this time you'll need to set -up a TCP internal load balancer to serve each correctly. - -The following IP will be used as an example: - -- `10.6.0.40`: Internal Load Balancer - -Here's how you could do it with [HAProxy](https://www.haproxy.org/): - -```plaintext -global - log /dev/log local0 - log localhost local1 notice - log stdout format raw local0 - -defaults - log global - default-server inter 10s fall 3 rise 2 - balance leastconn - -frontend internal-pgbouncer-tcp-in - bind *:6432 - mode tcp - option tcplog - - default_backend pgbouncer - -backend pgbouncer - mode tcp - option tcp-check - - server pgbouncer1 10.6.0.21:6432 check - server pgbouncer2 10.6.0.22:6432 check - server pgbouncer3 10.6.0.23:6432 check -``` - -Refer to your preferred Load Balancer's documentation for further guidance. - -
- - Back to setup components - -
- ## Configure Redis Using [Redis](https://redis.io/) in scalable environment is possible using a **Primary** x **Replica** @@ -1311,7 +1344,193 @@ To configure the Sentinel Queues server: -## Configure Gitaly +## Configure Gitaly Cluster + +[Gitaly Cluster](../gitaly/praefect.html) is GitLab's recommended fault tolerant solution for storing Git repositories. +In this configuration every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. + +The recommended Cluster setup includes the following components: + +- 3 Gitaly nodes - Replicated storage of Git repositories +- 3 Praefect nodes - Router and transaction manager for Gitaly Cluster +- 1 Praefect PostgreSQL node - Database server for Praefect + - A third party solution is currently required for Praefect database connections to be made HA + +A load balancer is also required for Praefect. In this guide we use the Internal Load Balancer node for this. + +This section will detail how to configure the recommended standard setup in order. For more advanced setups please refer to the [Gitaly Cluster documentation](../gitaly/praefect.html). + +### Configure Praefect PostgreSQL + +Praefect, the routing and transaction manager for Gitaly Cluster, requires a database to store data on Gitaly Cluster status. + +**As noted earlier in this guide [Praefect currently requires a third party PostgreSQL database solution to achieve full High Availability](../gitaly/praefect.md#postgresql)**. +In addition to this it should be noted that Praefect also currently requires it's own separate database server. + +We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server +can be set up via GitLab Omnibus, which this section will detail. + +#### Standalone non HA PostgreSQL using Omnibus GitLab + +The following IPs will be used as an example: + +- `10.6.0.141`: PostgreSQL + +First, make sure to [install](https://about.gitlab.com/install/) +the Linux GitLab package **on each node**. Following the steps, +install the necessary dependencies from step 1, and add the +GitLab package repository from step 2. When installing GitLab +in the second step, do not supply the `EXTERNAL_URL` value. + + +```ruby +# Disable all components except PostgreSQL and Consul +roles ['postgres_role'] +repmgr['enable'] = false +patroni['enable'] = false + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['sql_user_password'] = "" +# Replace XXX.XXX.XXX.XXX/YY with Network Address +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + +gitlab_rails['auto_migrate'] = false + +# Enable service discovery for Prometheus +consul['enable'] = true +consul['monitoring_service_discovery'] = true +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} + +# Set the network addresses that the exporters will listen on for monitoring +node_exporter['listen_address'] = '0.0.0.0:9100' +postgres_exporter['listen_address'] = '0.0.0.0:9187' +``` + +TODO: Copy and paste steps from Gitaly Cluster docs + +1. SSH in to the PostgreSQL node. +1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as ``. +1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default + username of `praefect` (recommended). The command will request the password `` + and confirmation. Use the value that is output by this command in the next + step as the value of ``: + + ```shell + sudo gitlab-ctl pg-password-md5 praefect + ``` + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Consul + roles ['postgres_role'] + repmgr['enable'] = false + patroni['enable'] = false + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['max_connections'] = 200 + + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PRAEFECT_POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = "" + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +1. Connect to the PostgreSQL server with the `praefect` user you created earlier. The database template1 is used because it is created by default on all PostgreSQL servers. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) +are supported and can be added if needed. + + + +#### Configure your own Praefect PostgreSQL server + +TODO: Guidance on how to set up a HA PostgreSQL server for Praefect? + +Input from Gitaly team may be required but any solution must have a static IP address that doesn't change when failover occurs as Praefect's database connection settings +can't be changed automatically. + +Examples could include [Google's Cloud SQL offering](https://cloud.google.com/sql/docs/postgres/high-availability#normal) + +#### Praefect PostgreSQL post-configuration + +After the Praefect PostgreSQL server has been set up you'll then need to configure the User and Database for Praefect to use. + +We recommend the user be named `praefect` and the database `praefect_production` and these can be configured as standard in PostgreSQL. +The password for the user is the same as the one you configured earlier - ``. + +An example of how this would work with a GitLab Omnibus PostgreSQL set up is below. + +1. SSH in to the PostgreSQL node. + +1. Connect to the PostgreSQL server with administrative access. + The `gitlab-psql` user should be used here for this as it's added by default in Omnibus. + The database `template1` is used because it is created by default on all PostgreSQL servers. + + ```shell + /opt/gitlab/embedded/bin/psql -U gitlab-psql -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create the new user `praefect`, replacing ``: + + ```shell + CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD ; + ``` + +1. Reconnect to the PostgreSQL server, this time as the `praefect` user: + + ```shell + /opt/gitlab/embedded/bin/psql -U praefect -d template1 -h POSTGRESQL_SERVER_ADDRESS + ``` + +1. Create a new database `praefect_production`: + + ```shell + CREATE DATABASE praefect_production WITH ENCODING=UTF8; + ``` + + + +### Configure Praefect + +### Configure Gitaly NOTE: [Gitaly Cluster](../gitaly/praefect.md) support -- GitLab From fb448be04722f4a77d3ab42c110e1b3b76c4d4c6 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 19 Jan 2021 15:58:26 +0000 Subject: [PATCH 05/18] First draft of full 10k RA Cluster docs --- .../reference_architectures/10k_users.md | 352 +++++++++++------- 1 file changed, 224 insertions(+), 128 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index e84a98d5eda9c3..2b65cfe7bdca42 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -135,7 +135,7 @@ doesn't require you to provision and maintain a node. It's also worth noting that at this time [Praefect requires it's own database server][Gitaly Cluster](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via GitLab Omnibus, which the above specs reflect. +can be set up via Omnibus GitLab, which the above specs reflect. ## Setup components @@ -200,7 +200,7 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.131`: Praefect 1 - `10.6.0.132`: Praefect 2 - `10.6.0.134`: Praefect 3 -- `10.6.0.141`: Praefect Postgres 1 (non HA) +- `10.6.0.141`: Praefect PostgreSQL 1 (non HA) - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 - `10.6.0.103`: Sidekiq 3 @@ -1346,7 +1346,7 @@ To configure the Sentinel Queues server: ## Configure Gitaly Cluster -[Gitaly Cluster](../gitaly/praefect.html) is GitLab's recommended fault tolerant solution for storing Git repositories. +[Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. In this configuration every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. The recommended Cluster setup includes the following components: @@ -1358,7 +1358,7 @@ The recommended Cluster setup includes the following components: A load balancer is also required for Praefect. In this guide we use the Internal Load Balancer node for this. -This section will detail how to configure the recommended standard setup in order. For more advanced setups please refer to the [Gitaly Cluster documentation](../gitaly/praefect.html). +This section will detail how to configure the recommended standard setup in order. For more advanced setups please refer to the [Gitaly Cluster documentation](../gitaly/praefect.md). ### Configure Praefect PostgreSQL @@ -1368,7 +1368,7 @@ Praefect, the routing and transaction manager for Gitaly Cluster, requires a dat In addition to this it should be noted that Praefect also currently requires it's own separate database server. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via GitLab Omnibus, which this section will detail. +can be set up via Omnibus GitLab, which this section will detail. #### Standalone non HA PostgreSQL using Omnibus GitLab @@ -1382,7 +1382,6 @@ install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. - ```ruby # Disable all components except PostgreSQL and Consul roles ['postgres_role'] @@ -1492,7 +1491,7 @@ After the Praefect PostgreSQL server has been set up you'll then need to configu We recommend the user be named `praefect` and the database `praefect_production` and these can be configured as standard in PostgreSQL. The password for the user is the same as the one you configured earlier - ``. -An example of how this would work with a GitLab Omnibus PostgreSQL set up is below. +An example of how this would work with a Omnibus GitLab PostgreSQL set up is below. 1. SSH in to the PostgreSQL node. @@ -1530,22 +1529,126 @@ An example of how this would work with a GitLab Omnibus PostgreSQL set up is bel ### Configure Praefect -### Configure Gitaly +Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through +it. This section details how to configure it. -NOTE: -[Gitaly Cluster](../gitaly/praefect.md) support -for the Reference Architectures is being -worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified -some Architecture specs will likely change as a result to support the new -and improved designed. +It should note that Praefect requires several secret tokens to secure communications across the Cluster: + +- `praefect_external_token`: Repositories hosted on your Praefect cluster can only be accessed by Gitaly clients that carry this token. +- `praefect_internal_token`: This token is used for replication traffic inside your Praefect cluster. This is distinct from `praefect_external_token` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. + +It's also worth noting that the `praefect_postgresql_password` defined in the previous section is +also required as part of this setup. + +Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains +the details of each Gitaly node that makes up the cluster. Each Storage is also given a name +and this name is used in several areas of config. In this guide the name of the storage will be +`default`. Note that this guide is geared towards new installs, if upgrading an existing environment +to use Gitaly Cluster you may need to use a different name. +Please refer to the [docs](../gitaly/praefect.md#praefect) for more info. + +To configure the Praefect nodes, on each one: + +1. SSH in to the Praefect server. +1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package of your choice. Be sure to follow _only_ installation steps 1 and 2 + on the page. +1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + + ```ruby + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a separate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Praefect Configuration + praefect['enable'] = true + praefect['listen_addr'] = '0.0.0.0:2305' + + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + + # Praefect External Token + # This is needed by clients outside the cluster (like GitLab Shell) to communicate with the Praefect cluster + praefect['auth_token'] = '' + + # Praefect Database Settings + praefect['database_host'] = '10.142.0.141' + praefect['database_port'] = 5432 + praefect['database_host_no_proxy'] = '10.142.0.141' + praefect['database_port_no_proxy'] = 5432 + praefect['database_dbname'] = 'praefect_production' + praefect['database_user'] = 'praefect' + praefect['database_password'] = '' + + # Praefect Virtual Storage config + # Name of storage hash must match storage name in git_data_dirs on GitLab + # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') + praefect['virtual_storages'] = { + 'default' => { + 'gitaly-1' => { + 'address' => 'tcp://10.142.0.91:8075', + 'token' => '', + 'primary' => true + }, + 'gitaly-2' => { + 'address' => 'tcp://10.142.0.92:8075', + 'token' => '' + }, + 'gitaly-3' => { + 'address' => 'tcp://10.142.0.93:8075', + 'token' => '' + }, + } + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + praefect['prometheus_listen_addr'] = '0.0.0.0:9652' + + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + # + # END user configuration + ``` + + 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and + then replace the file of the same name on this server. If that file isn't on + this server, add the file from your Consul server to this server. + + 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -[Gitaly](../gitaly/index.md) server node requirements are dependent on data, -specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5 TB of data. Depending on your -repository storage requirements, you may require additional Gitaly server nodes. +### Configure Gitaly + +The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have +requirements that are dependent on data, specifically the number of projects +and those projects' sizes. It's recommended that a Gitaly Cluster stores +no more than 5 TB of data on each node. Depending on your +repository storage requirements, you may require additional Gitaly Clusters. Due to Gitaly having notable input and output requirements, we strongly -recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs +recommend that all Gitaly Cluster nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for write operations. These IOPS values are initial recommendations, and may be @@ -1553,36 +1656,15 @@ adjusted to greater or lesser values depending on the scale of your environment's workload. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. -Be sure to note the following items: - -- The GitLab Rails application shards repositories into - [repository storage paths](../repository_storage_paths.md). -- A Gitaly server can host one or more storage paths. -- A GitLab server can use one or more Gitaly server nodes. -- Gitaly addresses must be specified to be correctly resolvable for all Gitaly - clients. -- Gitaly servers must not be exposed to the public internet, as Gitaly's network - traffic is unencrypted by default. The use of a firewall is highly recommended - to restrict access to the Gitaly server. Another option is to - [use TLS](#gitaly-tls-support). +Gitaly servers must not be exposed to the public internet, as Gitaly's network +traffic is unencrypted by default. The use of a firewall is highly recommended +to restrict access to the Gitaly server. Another option is to +[use TLS](#gitaly-cluster-tls-support). -NOTE: -The token referred to throughout the Gitaly documentation is an arbitrary -password selected by the administrator. This token is unrelated to tokens -created for the GitLab API or other similar web API tokens. - -This section describes how to configure two Gitaly servers, with the following -IPs and domain names: - -- `10.6.0.91`: Gitaly 1 (`gitaly1.internal`) -- `10.6.0.92`: Gitaly 2 (`gitaly2.internal`) +For configuring Gitaly you should note the following: -Assumptions about your servers include having the secret token be `gitalysecret`, -and that your GitLab installation has three repository storages: - -- `default` on Gitaly 1 -- `storage1` on Gitaly 1 -- `storage2` on Gitaly 2 +- `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node +- `auth_token` should be the same as `praefect_internal_token` On each node: @@ -1592,21 +1674,9 @@ On each node: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: - - ```ruby # /etc/gitlab/gitlab.rb - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the GitLab Rails application setup - gitaly['auth_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false redis['enable'] = false @@ -1635,36 +1705,42 @@ On each node: # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections gitaly['listen_addr'] = "0.0.0.0:8075" + + # Gitaly Auth Token + # Should be the same as praefect_internal_token + gitaly['auth_token'] = '' ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - - On `gitaly1.internal`: + - On Gitaly Cluster node 1: ```ruby git_data_dirs({ - 'default' => { - 'path' => '/var/opt/gitlab/git-data' - }, - 'storage1' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-1" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - - On `gitaly2.internal`: + - On Gitaly Cluster node 2: ```ruby git_data_dirs({ - 'storage2' => { - 'path' => '/mnt/gitlab/git-data' - }, + "gitaly-2" => { + "path" => "/var/opt/gitlab/git-data" + } }) ``` - + - On Gitaly Cluster node 3: + + ```ruby + git_data_dirs({ + "gitaly-3" => { + "path" => "/var/opt/gitlab/git-data" + } + }) + ``` 1. Copy the `/etc/gitlab/gitlab-secrets.json` file from your Consul server, and then replace the file of the same name on this server. If that file isn't on @@ -1674,32 +1750,44 @@ On each node: ### Gitaly TLS support -Gitaly supports TLS encryption. To be able to communicate -with a Gitaly instance that listens for secure connections you will need to use `tls://` URL -scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. +Praefect supports TLS encryption. To communicate with a Praefect instance that listens +for secure connections, you must: -You will need to bring your own certificates as this isn't provided automatically. -The certificate, or its certificate authority, must be installed on all Gitaly -nodes (including the Gitaly node using the certificate) and on all client nodes -that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +- Use a `tls://` URL scheme in the `gitaly_address` of the corresponding storage entry + in the GitLab configuration. +- Bring your own certificates because this isn't provided automatically. The certificate + corresponding to each Praefect server must be installed on that Praefect server. -NOTE: -The self-signed certificate must specify the address you use to access the -Gitaly server. If you are addressing the Gitaly server by a hostname, you can -either use the Common Name field for this, or add it as a Subject Alternative -Name. If you are addressing the Gitaly server by its IP address, you must add it -as a Subject Alternative Name to the certificate. -[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). +Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers +and on all Praefect clients that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). + +Note the following: + +- The certificate must specify the address you use to access the Praefect server. If + addressing the Praefect server by: -It's possible to configure Gitaly servers with both an unencrypted listening -address (`listen_addr`) and an encrypted listening address (`tls_listen_addr`) -at the same time. This allows you to do a gradual transition from unencrypted to -encrypted traffic, if necessary. + - Hostname, you can either use the Common Name field for this, or add it as a Subject + Alternative Name. + - IP address, you must add it as a Subject Alternative Name to the certificate. -To configure Gitaly with TLS: +- You can configure Praefect servers with both an unencrypted listening address + `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. + This allows you to do a gradual transition from unencrypted to encrypted traffic, if + necessary. -1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +- The Internal Load Balancer will also access to the certificates and need to be configured + to allow for TLS passthrough. + Refer to the load balancers documentation on how to configure this. + +To configure Praefect with TLS: + +**For Omnibus GitLab** + +1. Create certificates for Praefect servers. + +1. On the Praefect servers, create the `/etc/gitlab/ssl` directory and copy your key + and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -1708,27 +1796,34 @@ To configure Gitaly with TLS: sudo chmod 644 key.pem cert.pem ``` -1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when - calling into itself: +1. Edit `/etc/gitlab/gitlab.rb` and add: - ```shell - sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ```ruby + praefect['tls_listen_addr'] = "0.0.0.0:3305" + praefect['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + praefect['key_path'] = "/etc/gitlab/ssl/key.pem" ``` -1. Edit `/etc/gitlab/gitlab.rb` and add: +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). - +1. On the Praefect clients (including each Gitaly server), copy the certificates, + or their certificate authority, into `/etc/gitlab/trusted-certs`: - ```ruby - gitaly['tls_listen_addr'] = "0.0.0.0:9999" - gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" - gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ ``` -1. Delete `gitaly['listen_addr']` to allow only encrypted connections. +1. On the Praefect clients (except Gitaly servers), edit `git_data_dirs` in + `/etc/gitlab/gitlab.rb` as follows: + + ```ruby + git_data_dirs({ + "default" => { + "gitaly_address" => 'tls://LOAD_BALANCER_SERVER_ADDRESS:2305', + "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' + } + }) + ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -1814,17 +1909,20 @@ To configure the Sidekiq nodes, on each one: ### Gitaly ### ####################################### + # git_data_dirs get configured for the Praefect virtual storage + # Address is Internal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '' + } }) - gitlab_rails['gitaly_token'] = 'YOUR_TOKEN' ####################################### ### Postgres ### ####################################### - gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP + gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' gitlab_rails['db_adapter'] = 'postgresql' @@ -1897,17 +1995,14 @@ On each node perform the following: ```ruby external_url 'https://gitlab.example.com' - # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests - # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. - # The following two values must be the same as their respective values - # of the Gitaly setup - gitlab_rails['gitaly_token'] = 'gitalysecret' - gitlab_shell['secret_token'] = 'shellsecret' - + # git_data_dirs get configured for the Praefect virtual storage + # Address is Interal Load Balancer for Praefect + # Token is praefect_external_token git_data_dirs({ - 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, - 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + "default" => { + "gitaly_address" => "tcp://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '' + } }) ## Disable components that will not be on the GitLab application server @@ -1967,14 +2062,15 @@ On each node perform the following: ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the +1. If you're using [Gitaly with TLS support](#gitaly-cluster-tls-support), make sure the `git_data_dirs` entry is configured with `tls` instead of `tcp`: ```ruby git_data_dirs({ - 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, - 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + "default" => { + "gitaly_address" => "tls://10.6.0.40:2305", # internal load balancer IP + "gitaly_token" => '' + } }) ``` -- GitLab From bfb9e9b5a0131a3f6ca92afb8dde11afe9d32aef Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 19 Jan 2021 16:19:36 +0000 Subject: [PATCH 06/18] Update 10k RA with 3rd party db guidance --- .../reference_architectures/10k_users.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 2b65cfe7bdca42..29e4edbafe5304 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1408,8 +1408,6 @@ node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' ``` -TODO: Copy and paste steps from Gitaly Cluster docs - 1. SSH in to the PostgreSQL node. 1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as ``. 1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default @@ -1477,12 +1475,14 @@ are supported and can be added if needed. #### Configure your own Praefect PostgreSQL server -TODO: Guidance on how to set up a HA PostgreSQL server for Praefect? +As noted elsewhere in this guide, currently a third party PostgreSQL solution for Praefect's database is recommended if aiming for full High Availability. + +There are many third party solutions for PostgreSQL HA. Note that the solution selected must have the following to work with Praefect: -Input from Gitaly team may be required but any solution must have a static IP address that doesn't change when failover occurs as Praefect's database connection settings -can't be changed automatically. +- For HA the solution must have a static IP for all connections that doesn't change on failover +- [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported -Examples could include [Google's Cloud SQL offering](https://cloud.google.com/sql/docs/postgres/high-availability#normal) +Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). #### Praefect PostgreSQL post-configuration @@ -1593,6 +1593,7 @@ To configure the Praefect nodes, on each one: # Praefect Database Settings praefect['database_host'] = '10.142.0.141' praefect['database_port'] = 5432 + # `no_proxy` settings must always be a direct connection for caching praefect['database_host_no_proxy'] = '10.142.0.141' praefect['database_port_no_proxy'] = 5432 praefect['database_dbname'] = 'praefect_production' @@ -1748,7 +1749,7 @@ On each node: 1. Save the file, and then [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -### Gitaly TLS support +### Gitaly Cluster TLS support Praefect supports TLS encryption. To communicate with a Praefect instance that listens for secure connections, you must: -- GitLab From 3f2451b2819d106081a50fcca51cc0b07a3d86bf Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 26 Jan 2021 15:41:12 +0000 Subject: [PATCH 07/18] Update RA docs with new Git Push info --- doc/administration/reference_architectures/10k_users.md | 2 +- doc/administration/reference_architectures/1k_users.md | 1 + doc/administration/reference_architectures/25k_users.md | 2 +- doc/administration/reference_architectures/2k_users.md | 2 +- doc/administration/reference_architectures/3k_users.md | 2 +- doc/administration/reference_architectures/50k_users.md | 2 +- doc/administration/reference_architectures/5k_users.md | 2 +- doc/administration/reference_architectures/index.md | 3 ++- 8 files changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 95c156c7064bbd..daa5094fcead68 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -13,7 +13,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 10,000 > - **High Availability:** Yes* -> - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS +> - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > > _\* At this time Praefect requires a third party PostgreSQL database solution to achieve full High Availability. We hope to offer a built in solution for this in the future._ diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index a818c68606652e..97cb5c87195271 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -18,6 +18,7 @@ many organizations . > - **Supported users (approximate):** 1,000 > - **High Availability:** No. For a highly-available environment, you can > follow the [3K reference architecture](3k_users.md). +> - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS | Users | Configuration | GCP | AWS | Azure | |--------------|-------------------------|----------------|-----------------|----------------| diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index fca5409200a0ba..dec5d7fc1b90c8 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -13,7 +13,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 25,000 > - **High Availability:** Yes -> - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS +> - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index edefc8549acbaa..ed393c7367ef4d 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -14,7 +14,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can > follow the [3K reference architecture](3k_users.md). -> - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS +> - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|--------|-------------------------|----------------|--------------|---------| diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index b4cae5effea733..218b87d8cf9334 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -21,7 +21,7 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes -> - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git: 6 RPS +> - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 44ebc4f9385e6b..ccd9d8c3fb9722 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -13,7 +13,7 @@ full list of reference architectures, see > - **Supported users (approximate):** 50,000 > - **High Availability:** Yes -> - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS +> - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index a149f2b91ae928..305e10d979cdac 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -20,7 +20,7 @@ costly-to-operate environment by using the > - **Supported users (approximate):** 5,000 > - **High Availability:** Yes -> - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS +> - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index b90b8d67b68e1b..745b0169511f8c 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -29,7 +29,8 @@ per 1,000 users: - API: 20 RPS - Web: 2 RPS -- Git: 2 RPS +- Git (Pull): 2 RPS +- Git (Push): 0.4 RPS (rounded to nearest integer) For GitLab instances with less than 2,000 users, it's recommended that you use the [default setup](#automated-backups) by -- GitLab From 56787d03f399e499d7ed9187cabeaa51617c433b Mon Sep 17 00:00:00 2001 From: Grant Young Date: Wed, 3 Feb 2021 12:04:46 +0000 Subject: [PATCH 08/18] Add in links to issues for Praefect Postgres --- doc/administration/reference_architectures/10k_users.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index daa5094fcead68..a62b48aa566eb4 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -15,7 +15,7 @@ full list of reference architectures, see > - **High Availability:** Yes* > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > -> _\* At this time Praefect requires a third party PostgreSQL database solution to achieve full High Availability. We hope to offer a built in solution for this in the future._ +> _\* At this time Praefect requires a third party PostgreSQL database solution to achieve full High Availability. We hope to offer a built in solution for this in the future. Refer to the following issues for more information: [omnibus-gitlab#5919](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [gitaly#3398](https://gitlab.com/gitlab-org/gitaly/-/issues/3398)_ | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| @@ -135,7 +135,7 @@ doesn't require you to provision and maintain a node. It's also worth noting that at this time [Praefect requires it's own database server][Gitaly Cluster](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [omnibus-gitlab#5919](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [gitaly#3398](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) ## Setup components -- GitLab From 27bf598f3529b0d98fa5b2ef53b4c44f7355fd33 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Wed, 3 Feb 2021 12:05:49 +0000 Subject: [PATCH 09/18] Apply 2 suggestion(s) to 1 file(s) --- doc/administration/reference_architectures/10k_users.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index a62b48aa566eb4..1c7aa45593dbbc 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -132,7 +132,7 @@ uploads, or artifacts), using an [object storage service](#configure-the-object- is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. -It's also worth noting that at this time [Praefect requires it's own database server][Gitaly Cluster](../gitaly/praefect.md#postgresql) and +It's also worth noting that at this time [Praefect requires its own database server][Gitaly Cluster](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [omnibus-gitlab#5919](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [gitaly#3398](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) @@ -1365,7 +1365,7 @@ This section will detail how to configure the recommended standard setup in orde Praefect, the routing and transaction manager for Gitaly Cluster, requires a database to store data on Gitaly Cluster status. **As noted earlier in this guide [Praefect currently requires a third party PostgreSQL database solution to achieve full High Availability](../gitaly/praefect.md#postgresql)**. -In addition to this it should be noted that Praefect also currently requires it's own separate database server. +In addition to this it should be noted that Praefect also currently requires its own separate database server. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which this section will detail. -- GitLab From 25f22e1f27aab46305ddcd88efe60283a17ef997 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Wed, 3 Feb 2021 12:23:01 +0000 Subject: [PATCH 10/18] Quote issue links --- doc/administration/reference_architectures/10k_users.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index a62b48aa566eb4..36298f42fac359 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -15,7 +15,7 @@ full list of reference architectures, see > - **High Availability:** Yes* > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > -> _\* At this time Praefect requires a third party PostgreSQL database solution to achieve full High Availability. We hope to offer a built in solution for this in the future. Refer to the following issues for more information: [omnibus-gitlab#5919](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [gitaly#3398](https://gitlab.com/gitlab-org/gitaly/-/issues/3398)_ +> _\* At this time Praefect requires a third party PostgreSQL database solution to achieve full High Availability. We hope to offer a built in solution for this in the future. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398)_ | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| @@ -135,7 +135,7 @@ doesn't require you to provision and maintain a node. It's also worth noting that at this time [Praefect requires it's own database server][Gitaly Cluster](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [omnibus-gitlab#5919](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [gitaly#3398](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) +can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) ## Setup components -- GitLab From 02a02e8748ea9e6f1d83839543d13b472f84d9b8 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 5 Feb 2021 13:10:30 +0000 Subject: [PATCH 11/18] Apply 1 suggestion(s) to 1 file(s) --- doc/administration/reference_architectures/10k_users.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 1b01db1ab852bc..562f6daad61e95 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1493,7 +1493,7 @@ The password for the user is the same as the one you configured earlier - ` Date: Fri, 5 Feb 2021 13:11:39 +0000 Subject: [PATCH 12/18] Review fixes --- .../reference_architectures/10k_users.md | 34 +++---------------- 1 file changed, 4 insertions(+), 30 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 1b01db1ab852bc..045a4372d383af 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -194,9 +194,9 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.81`: Sentinel - Queues 1 - `10.6.0.82`: Sentinel - Queues 2 - `10.6.0.83`: Sentinel - Queues 3 -- `10.6.0.91`: Gitaly Cluster 1 -- `10.6.0.92`: Gitaly Cluster 2 -- `10.6.0.93`: Gitaly Cluster 3 +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 - `10.6.0.131`: Praefect 1 - `10.6.0.132`: Praefect 2 - `10.6.0.134`: Praefect 3 @@ -1377,37 +1377,11 @@ The following IPs will be used as an example: - `10.6.0.141`: PostgreSQL First, make sure to [install](https://about.gitlab.com/install/) -the Linux GitLab package **on each node**. Following the steps, +the Linux GitLab package the node. Following the steps, install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. -```ruby -# Disable all components except PostgreSQL and Consul -roles ['postgres_role'] -repmgr['enable'] = false -patroni['enable'] = false - -# PostgreSQL configuration -postgresql['listen_address'] = '0.0.0.0' -postgresql['sql_user_password'] = "" -# Replace XXX.XXX.XXX.XXX/YY with Network Address -postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24) - -gitlab_rails['auto_migrate'] = false - -# Enable service discovery for Prometheus -consul['enable'] = true -consul['monitoring_service_discovery'] = true -consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) -} - -# Set the network addresses that the exporters will listen on for monitoring -node_exporter['listen_address'] = '0.0.0.0:9100' -postgres_exporter['listen_address'] = '0.0.0.0:9187' -``` - 1. SSH in to the PostgreSQL node. 1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as ``. 1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default -- GitLab From bd309d434ccfa3b4f26e4f53f598e3b1beb0f36c Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 5 Feb 2021 13:19:58 +0000 Subject: [PATCH 13/18] Update 10k diagram --- doc/administration/reference_architectures/10k_users.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index b08199448dddd5..b47509d8d65892 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -54,7 +54,7 @@ together { card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 - collections "**Gitaly Cluster** x3" as gitaly #FF8C00 + collections "**Gitaly** x3" as gitaly #FF8C00 card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 praefect -[#FF8C00]-> gitaly -- GitLab From c301b3db1924d58aa65c2155755f4148fa10e0e6 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Fri, 5 Feb 2021 13:25:07 +0000 Subject: [PATCH 14/18] Add 2k plantuml diagram --- .../reference_architectures/2k_users.md | 64 ++++++++----------- 1 file changed, 26 insertions(+), 38 deletions(-) diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index ed393c7367ef4d..05c28718f98fa8 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -27,44 +27,32 @@ For a full list of reference architectures, see | Object storage | n/a | n/a | n/a | n/a | n/a | | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | -```mermaid -stateDiagram-v2 - [*] --> LoadBalancer - LoadBalancer --> ApplicationServer - - ApplicationServer --> Gitaly - ApplicationServer --> Redis - ApplicationServer --> Database - ApplicationServer --> ObjectStorage - - ApplicationMonitoring -->ApplicationServer - ApplicationMonitoring -->Redis - ApplicationMonitoring -->Database - - - state Database { - "PG_Node" - } - state Redis { - "Redis_Node" - } - - state Gitaly { - "Gitaly" - } - - state ApplicationServer { - "AppServ_1..2" - } - - state LoadBalancer { - "LoadBalancer" - } - - state ApplicationMonitoring { - "Prometheus" - "Grafana" - } +```plantuml +@startuml 2k +card "**External Load Balancer**" as elb #6a9be7 + +collections "**GitLab Rails** x3" as gitlab #32CD32 +card "**Prometheus + Grafana**" as monitor #7FFFD4 +card "**Gitaly**" as gitaly #FF8C00 +card "**PostgreSQL**" as postgres #4EA7FF +card "**Redis**" as redis #FF6347 +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]--> monitor + +gitlab -[#32CD32]--> gitaly +gitlab -[#32CD32]--> postgres +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]--> redis + +monitor .[#7FFFD4]u-> gitlab +monitor .[#7FFFD4]-> gitaly +monitor .[#7FFFD4]-> postgres +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4,norank]u--> elb + +@enduml ``` The Google Cloud Platform (GCP) architectures were built and tested using the -- GitLab From 41cc2532fda746352a056453a470af757ca886db Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Tue, 9 Feb 2021 20:18:56 +0100 Subject: [PATCH 15/18] Copy edit Gitaly cluster docs --- .../reference_architectures/10k_users.md | 105 ++++++++---------- 1 file changed, 47 insertions(+), 58 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index dfc02a55c3dbe8..0f47d2cc52317b 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -12,10 +12,8 @@ full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). > - **Supported users (approximate):** 10,000 -> - **High Availability:** Yes* +> - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS -> -> _\* At this time Praefect requires a third party PostgreSQL database solution to achieve full High Availability. We hope to offer a built in solution for this in the future. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398)_ | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| @@ -56,7 +54,7 @@ card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 collections "**Gitaly** x3" as gitaly #FF8C00 card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 - + praefect -[#FF8C00]-> gitaly praefect -[#FF8C00]> praefect_postgres } @@ -65,7 +63,7 @@ card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF - + pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary } @@ -75,7 +73,7 @@ card "redis" as redis { collections "**Redis Cache** x3" as redis_cache #FF6347 collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 - + redis_persistent <.[#FF6347]- redis_persistent_sentinel redis_cache <.[#FF6347]- redis_cache_sentinel } @@ -1347,42 +1345,39 @@ To configure the Sentinel Queues server: ## Configure Gitaly Cluster [Gitaly Cluster](../gitaly/praefect.md) is a GitLab provided and recommended fault tolerant solution for storing Git repositories. -In this configuration every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. +In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. -The recommended Cluster setup includes the following components: +The recommended cluster setup includes the following components: -- 3 Gitaly nodes - Replicated storage of Git repositories -- 3 Praefect nodes - Router and transaction manager for Gitaly Cluster -- 1 Praefect PostgreSQL node - Database server for Praefect - - A third party solution is currently required for Praefect database connections to be made HA +- 3 Gitaly nodes: Replicated storage of Git repositories. +- 3 Praefect nodes: Router and transaction manager for Gitaly Cluster. +- 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution + is required for Praefect database connections to be made highly available. +- 1 load balancer: A load balancer is required for Praefect. In this guide we use the internal load balancer node for this. -A load balancer is also required for Praefect. In this guide we use the Internal Load Balancer node for this. - -This section will detail how to configure the recommended standard setup in order. For more advanced setups please refer to the [Gitaly Cluster documentation](../gitaly/praefect.md). +This section will detail how to configure the recommended standard setup in order. +For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md). ### Configure Praefect PostgreSQL -Praefect, the routing and transaction manager for Gitaly Cluster, requires a database to store data on Gitaly Cluster status. +Praefect, the routing and transaction manager for Gitaly Cluster, requires its own database server to store data on Gitaly Cluster status. -**As noted earlier in this guide [Praefect currently requires a third party PostgreSQL database solution to achieve full High Availability](../gitaly/praefect.md#postgresql)**. -In addition to this it should be noted that Praefect also currently requires its own separate database server. +If you want to have a highly available setup, Praefect requires a third-party PostgreSQL database. +A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919). -We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server -can be set up via Omnibus GitLab, which this section will detail. - -#### Standalone non HA PostgreSQL using Omnibus GitLab +#### Praefect non-HA PostgreSQL standalone using Omnibus GitLab The following IPs will be used as an example: - `10.6.0.141`: PostgreSQL First, make sure to [install](https://about.gitlab.com/install/) -the Linux GitLab package the node. Following the steps, +the Linux GitLab package in the Praefect PostgreSQL node. Following the steps, install the necessary dependencies from step 1, and add the GitLab package repository from step 2. When installing GitLab in the second step, do not supply the `EXTERNAL_URL` value. -1. SSH in to the PostgreSQL node. +1. SSH in to the Praefect PostgreSQL node. 1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as ``. 1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default username of `praefect` (recommended). The command will request the password `` @@ -1435,11 +1430,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Connect to the PostgreSQL server with the `praefect` user you created earlier. The database template1 is used because it is created by default on all PostgreSQL servers. - -Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) -are supported and can be added if needed. +1. Follow the [post configuration](#praefect-postgresql-post-configuration). -#### Configure your own Praefect PostgreSQL server +#### Praefect HA PostgreSQL third-party solution -As noted elsewhere in this guide, currently a third party PostgreSQL solution for Praefect's database is recommended if aiming for full High Availability. +[As noted](#configure-praefect-postgresql), a third-party PostgreSQL solution for +Praefect's database is recommended if aiming for full High Availability. -There are many third party solutions for PostgreSQL HA. Note that the solution selected must have the following to work with Praefect: +There are many third-party solutions for PostgreSQL HA. The solution selected must have the following to work with Praefect: -- For HA the solution must have a static IP for all connections that doesn't change on failover -- [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported +- A static IP for all connections that doesn't change on failover. +- [`LISTEN`](https://www.postgresql.org/docs/12/sql-listen.html) SQL functionality must be supported. Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). +Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). + #### Praefect PostgreSQL post-configuration -After the Praefect PostgreSQL server has been set up you'll then need to configure the User and Database for Praefect to use. +After the Praefect PostgreSQL server has been set up, you'll then need to configure the user and database for Praefect to use. -We recommend the user be named `praefect` and the database `praefect_production` and these can be configured as standard in PostgreSQL. -The password for the user is the same as the one you configured earlier - ``. +We recommend the user be named `praefect` and the database `praefect_production`, and these can be configured as standard in PostgreSQL. +The password for the user is the same as the one you configured earlier as ``. -An example of how this would work with a Omnibus GitLab PostgreSQL set up is below. +This is how this would work with a Omnibus GitLab PostgreSQL setup: 1. SSH in to the Praefect PostgreSQL node. - 1. Connect to the PostgreSQL server with administrative access. The `gitlab-psql` user should be used here for this as it's added by default in Omnibus. The database `template1` is used because it is created by default on all PostgreSQL servers. @@ -1506,20 +1499,18 @@ An example of how this would work with a Omnibus GitLab PostgreSQL set up is bel Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. -It should note that Praefect requires several secret tokens to secure communications across the Cluster: +Praefect requires several secret tokens to secure communications across the Cluster: -- `praefect_external_token`: Repositories hosted on your Praefect cluster can only be accessed by Gitaly clients that carry this token. -- `praefect_internal_token`: This token is used for replication traffic inside your Praefect cluster. This is distinct from `praefect_external_token` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. - -It's also worth noting that the `praefect_postgresql_password` defined in the previous section is -also required as part of this setup. +- ``: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. +- ``: Used for replication traffic inside your Gitaly cluster. This is distinct from `praefect_external_token` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. +- ``: The Praefect PostgreSQL password defined in the previous section is also required as part of this setup. Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains -the details of each Gitaly node that makes up the cluster. Each Storage is also given a name -and this name is used in several areas of config. In this guide the name of the storage will be -`default`. Note that this guide is geared towards new installs, if upgrading an existing environment -to use Gitaly Cluster you may need to use a different name. -Please refer to the [docs](../gitaly/praefect.md#praefect) for more info. +the details of each Gitaly node that makes up the cluster. Each storage is also given a name +and this name is used in several areas of the config. In this guide, the name of the storage will be +`default`. Also, this guide is geared towards new installs, if upgrading an existing environment +to use Gitaly Cluster, you may need to use a different name. +Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. To configure the Praefect nodes, on each one: @@ -1528,7 +1519,7 @@ To configure the Praefect nodes, on each one: package of your choice. Be sure to follow _only_ installation steps 1 and 2 on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: - + ```ruby # Avoid running unnecessary services on the Gitaly server postgresql['enable'] = false @@ -1598,7 +1589,7 @@ To configure the Praefect nodes, on each one: # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' praefect['prometheus_listen_addr'] = '0.0.0.0:9652' - + ## The IPs of the Consul server nodes ## You can also use FQDNs and intermix them with IPs consul['configuration'] = { @@ -1623,7 +1614,7 @@ no more than 5 TB of data on each node. Depending on your repository storage requirements, you may require additional Gitaly Clusters. Due to Gitaly having notable input and output requirements, we strongly -recommend that all Gitaly Cluster nodes use solid-state drives (SSDs). These SSDs +recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for write operations. These IOPS values are initial recommendations, and may be @@ -1687,7 +1678,7 @@ On each node: ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: - - On Gitaly Cluster node 1: + - On Gitaly node 1: ```ruby git_data_dirs({ @@ -1697,7 +1688,7 @@ On each node: }) ``` - - On Gitaly Cluster node 2: + - On Gitaly node 2: ```ruby git_data_dirs({ @@ -1707,7 +1698,7 @@ On each node: }) ``` - - On Gitaly Cluster node 3: + - On Gitaly node 3: ```ruby git_data_dirs({ @@ -1757,8 +1748,6 @@ Note the following: To configure Praefect with TLS: -**For Omnibus GitLab** - 1. Create certificates for Praefect servers. 1. On the Praefect servers, create the `/etc/gitlab/ssl` directory and copy your key -- GitLab From 892d69f3a0ada5afc7dcebfd4548873ed90a3deb Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 10 Feb 2021 08:31:12 +0100 Subject: [PATCH 16/18] Fix nodes IPs --- .../reference_architectures/10k_users.md | 26 ++++++++++++++----- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 0f47d2cc52317b..c0db4405836985 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -197,7 +197,7 @@ The following list includes descriptions of each server and its assigned IP: - `10.6.0.93`: Gitaly 3 - `10.6.0.131`: Praefect 1 - `10.6.0.132`: Praefect 2 -- `10.6.0.134`: Praefect 3 +- `10.6.0.133`: Praefect 3 - `10.6.0.141`: Praefect PostgreSQL 1 (non HA) - `10.6.0.101`: Sidekiq 1 - `10.6.0.102`: Sidekiq 2 @@ -1369,7 +1369,7 @@ A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-g The following IPs will be used as an example: -- `10.6.0.141`: PostgreSQL +- `10.6.0.141`: Praefect PostgreSQL First, make sure to [install](https://about.gitlab.com/install/) the Linux GitLab package in the Praefect PostgreSQL node. Following the steps, @@ -1512,6 +1512,12 @@ and this name is used in several areas of the config. In this guide, the name of to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. +The following IPs will be used as an example: + +- `10.6.0.131`: Praefect 1 +- `10.6.0.132`: Praefect 2 +- `10.6.0.133`: Praefect 3 + To configure the Praefect nodes, on each one: 1. SSH in to the Praefect server. @@ -1556,10 +1562,10 @@ To configure the Praefect nodes, on each one: praefect['auth_token'] = '' # Praefect Database Settings - praefect['database_host'] = '10.142.0.141' + praefect['database_host'] = '10.6.0.141' praefect['database_port'] = 5432 # `no_proxy` settings must always be a direct connection for caching - praefect['database_host_no_proxy'] = '10.142.0.141' + praefect['database_host_no_proxy'] = '10.6.0.141' praefect['database_port_no_proxy'] = 5432 praefect['database_dbname'] = 'praefect_production' praefect['database_user'] = 'praefect' @@ -1571,16 +1577,16 @@ To configure the Praefect nodes, on each one: praefect['virtual_storages'] = { 'default' => { 'gitaly-1' => { - 'address' => 'tcp://10.142.0.91:8075', + 'address' => 'tcp://10.6.0.91:8075', 'token' => '', 'primary' => true }, 'gitaly-2' => { - 'address' => 'tcp://10.142.0.92:8075', + 'address' => 'tcp://10.6.0.92:8075', 'token' => '' }, 'gitaly-3' => { - 'address' => 'tcp://10.142.0.93:8075', + 'address' => 'tcp://10.6.0.93:8075', 'token' => '' }, } @@ -1632,6 +1638,12 @@ For configuring Gitaly you should note the following: - `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` +The following IPs will be used as an example: + +- `10.6.0.91`: Gitaly 1 +- `10.6.0.92`: Gitaly 2 +- `10.6.0.93`: Gitaly 3 + On each node: 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab -- GitLab From 90eb4007b7212015711884e8658aa4d0ce6ff1be Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 10 Feb 2021 08:34:53 +0100 Subject: [PATCH 17/18] Link to internal load balancer --- doc/administration/reference_architectures/10k_users.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index c0db4405836985..530419ec9280c2 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1353,7 +1353,8 @@ The recommended cluster setup includes the following components: - 3 Praefect nodes: Router and transaction manager for Gitaly Cluster. - 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution is required for Praefect database connections to be made highly available. -- 1 load balancer: A load balancer is required for Praefect. In this guide we use the internal load balancer node for this. +- 1 load balancer: A load balancer is required for Praefect. The + [internal load balancer](#configure-the-internal-load-balancer) will be used. This section will detail how to configure the recommended standard setup in order. For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md). -- GitLab From c73e3b7b8a7c14bd7da92608ce693c3a77ed5f05 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Fri, 12 Feb 2021 11:53:25 +0000 Subject: [PATCH 18/18] Apply 2 suggestion(s) to 1 file(s) --- doc/administration/reference_architectures/10k_users.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 530419ec9280c2..fd837c7981a492 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -33,7 +33,7 @@ full list of reference architectures, see | GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | ```plantuml @startuml 10k @@ -130,7 +130,7 @@ uploads, or artifacts), using an [object storage service](#configure-the-object- is recommended instead of using NFS. Using an object storage service also doesn't require you to provision and maintain a node. -It's also worth noting that at this time [Praefect requires its own database server][Gitaly Cluster](../gitaly/praefect.md#postgresql) and +It's also worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and that to achieve full High Availability a third party PostgreSQL database solution will be required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398) -- GitLab