From 8855eb3c6df4fcce8c59e8df7aa6fbbe8eaa91ee Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 16 Mar 2023 11:01:45 +0000 Subject: [PATCH 01/16] Expand Ref Arch docs with notes on unsupported and more --- .../reference_architectures/index.md | 80 +++++++++++++------ 1 file changed, 57 insertions(+), 23 deletions(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index e4d07558306ae4..aca064c460f8a9 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -47,9 +47,9 @@ The following Cloud Native Hybrid reference architectures, where select recommen The first choice to consider is whether a Self Managed approach is correct for you and your requirements. -Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more. +Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more. This includes both the initial setup of the environment and the longer term maintenance. -As such, it's recommended that you have a working knowledge of running applications in production when deciding on going down this route. For users who want a more managed solution it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). +As such, it's recommended that you have a working knowledge of running and maintaining applications in production when deciding on going down this route. For users who aren't in this position our [Professional Services](https://about.gitlab.com/services/#implementation-services) team offer implementation services but for those who want a more managed solution long term it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). ## Deciding which architecture to use @@ -89,6 +89,12 @@ In general then, we'd only recommend you employ HA in the following scenarios: - When you have 3,000 or more users. - When GitLab being down would critically impact your workflow. +#### Scaled Down High Availability (HA) approaches + +For users who have determined that they require HA for less than 3,000 users there are some supported modifications available to scale down the smallest 3k Reference Architecture further. + +[Head to this section for more information](3k_users.html#supported-modifications-for-lower-user-counts-ha). + #### Zero Downtime Upgrades [Zero Downtime Upgrades](../../update/zero_downtime.md) are available for standard Reference Architecture environments with HA (Cloud Native Hybrid is not supported at this time). This allows for an environment to stay up during an upgrade, but the process is more complex as a result and has some limitations as detailed in the documentation. @@ -163,7 +169,7 @@ Before implementing a reference architecture, refer to the following requirement These reference architectures were built and tested on Google Cloud Platform (GCP) using the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) -CPU platform as a baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). +CPU platform as a lowest common denominator baseline ([Sysbench benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks)). Newer, similarly-sized CPUs are supported and may have improved performance as a result. For Omnibus GitLab environments, ARM-based equivalents are also supported. @@ -324,7 +330,9 @@ When selecting a database service, it should run a standard, performant, and [su - Read Replicas for [Database Load Balancing](../postgresql/database_load_balancing.md). - Cross Region replication for [GitLab Geo](../geo/index.md). -Several cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: +#### Unsupported database services + +Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. @@ -341,6 +349,51 @@ In addition to the above, you should be aware of the additional specific guidanc - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements. - [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have [performance limits that can impact production use at certain times](https://gitlab.com/gitlab-org/gitlab/-/issues/344861). However, this has only been seen in our largest architectures (25k+) so far. +## Deviating from the suggested reference architectures + +As a general guideline, the further away you move from the Reference Architectures, +the harder it is to get support for it. With any deviation, you're introducing +a layer of complexity that adds challenges to finding out where potential +issues might lie. + +The reference architectures use the official GitLab Linux packages (Omnibus +GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are +installed on separate machines (virtualized or bare metal), with machine hardware +requirements listed in the "Configuration" column and equivalent VM standard sizes listed +in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). + +Running components on Docker (including Compose) with the same specs should be fine, as Docker is well known in terms of support. +However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers. + +### Unsupported Designs + +While we endeavour to try and have a good range of support for GitLab environment designs there are certain approaches we know definitively not to work and as a result are not supported at this time. + +#### Gitaly Cluster in Kubernetes + +Running Gitaly Cluster in Kubernetes is not supported. + +This is a complex area as Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to +significant and complex issues. + +Gitaly Cluster is only supported to be run on VMs. [Refer to this Epic for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127) + +#### Autoscaling of stateful nodes + +As a general guidance only _stateless_ components of GitLab can be run in Autoscaling groups at this time, namely GitLab Rails +and Sidekiq. + +Other components that have state, such as Gitaly, are not supported in this fashion. + +This extends to other third-party components such as Postgres and Redis as given in Omnibus GitLab, but you can explore other third-party solutions for those components as desired unless called out specifically as unsupported. + +#### Spreading one environment over multiple datacenters + +Deploying one GitLab environment over multiple datacenters is not supported due to potential split brain edge cases +if a DC were to go down. For example, several components of the GitLab setup, namely Consul and Praefect require an odd number quorum to function correctly and splitting over DCs can impact this notably. + +For deploying GitLab over multiple datacenters or regions we offer [GitLab Geo](https://about.gitlab.com/solutions/geo/) as a comprehensive solution. + ## Validation and test results The [Quality Engineering team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) @@ -561,22 +614,3 @@ The following table details the cost to run the different reference architecture -## Deviating from the suggested reference architectures - -As a general guideline, the further away you move from the Reference Architectures, -the harder it is to get support for it. With any deviation, you're introducing -a layer of complexity that adds challenges to finding out where potential -issues might lie. - -The reference architectures use the official GitLab Linux packages (Omnibus -GitLab) or [Helm Charts](https://docs.gitlab.com/charts/) to install and configure the various components. The components are -installed on separate machines (virtualized or bare metal), with machine hardware -requirements listed in the "Configuration" column and equivalent VM standard sizes listed -in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). - -Running components on Docker (including Compose) with the same specs should be fine, as Docker is well known in terms of support. -However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers. - -Other technologies, like [Docker swarm](https://docs.docker.com/engine/swarm/) -are not officially supported, but can be implemented at your own risk. In that -case, GitLab Support is not able to help you. -- GitLab From 659c35e431b2524841aefc76efe543cf6b2318db Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 16 Mar 2023 11:24:26 +0000 Subject: [PATCH 02/16] Add section on maintenance tips --- .../reference_architectures/index.md | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index aca064c460f8a9..3c6925f5ba3b60 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -614,3 +614,38 @@ The following table details the cost to run the different reference architecture +## Maintaining a Reference Architecture environment + +Maintaining a Reference Architecture environment is generally the same as any other GitLab environment is generally covered in other sections of this documentation. + +In this section you'll find links to documentation for relevant areas as well as any specific Reference Architecture notes. + +### Upgrades + +Upgrades for a Reference Architecture environment is the same as any other GitLab environment. +The main [Upgrade GitLab](../../update/) section has detailed steps on how to approach this. + +Zero-Downtime Upgrades are also available [as mentioned earlier](#zero-downtime-upgrades). + +NOTE: +You should upgrade a Reference Architecture in the same order as you created it. + +### Scaling an environment + +Scaling a GitLab environment is designed to be as seemless as possible. + +In terms of the Reference Architectures you would look to the next size and adjust accordingly. +Most will only need vertical scaling, but there are some specific areas that can be adjusted depending on the setup: + +- If scaling from a non-HA environment to a HA environment various components are recommended to be deployed in their HA forms: + - Redis to multi-node Redis w/ Redis Sentinel + - Postgres to multi-node Postgres w/ Consul + PgBouncer + - Gitaly to Gitaly Cluster w/ Praefect +- From 10k users and higher Redis is recommended to be split into multiple HA servers as it's single threaded + +### How do I monitor my environment + +There are numerous options available to monitor your GitLab environment. + +[Several of these are bundled with GitLab](../monitoring/), but it's also possible to use third-party +options if desired. -- GitLab From fe555046d36af9daed0bdab6f68d50aac72d8e3b Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 16 Mar 2023 11:25:25 +0000 Subject: [PATCH 03/16] Adjust section to reduce duplication --- doc/administration/reference_architectures/index.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 3c6925f5ba3b60..b5dea34b7880cb 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -76,8 +76,6 @@ High Availability ensures every component in the GitLab setup can handle failure For environments serving 3,000 or more users we generally recommend that a HA strategy is used as at this level outages have a bigger impact against more users. All the architectures in this range have HA built in by design for this reason. -For users who still need to have HA for a lower number of users this can also be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). - #### Do you need High Availability (HA)? As mentioned above, achieving HA does come at a cost. The environment's required are sizable as each component needs to be multiplied, which comes with additional actual and maintenance costs. @@ -91,9 +89,7 @@ In general then, we'd only recommend you employ HA in the following scenarios: #### Scaled Down High Availability (HA) approaches -For users who have determined that they require HA for less than 3,000 users there are some supported modifications available to scale down the smallest 3k Reference Architecture further. - -[Head to this section for more information](3k_users.html#supported-modifications-for-lower-user-counts-ha). +For users who still need to have HA for a lower number of users this can also be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). #### Zero Downtime Upgrades -- GitLab From 18f205892523339b363921eed387850fb5528648 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 16 Mar 2023 11:41:31 +0000 Subject: [PATCH 04/16] Add quorum notes --- .../reference_architectures/10k_users.md | 8 ++++++++ .../reference_architectures/25k_users.md | 8 ++++++++ .../reference_architectures/3k_users.md | 12 ++++++++++-- .../reference_architectures/50k_users.md | 8 ++++++++ .../reference_architectures/5k_users.md | 10 ++++++++-- 5 files changed, 42 insertions(+), 4 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 1934ca6bff0b85..b2baa92c24c9ce 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -417,6 +417,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -1336,6 +1341,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. + Praefect requires several secret tokens to secure communications across the Cluster: - ``: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 08cb6e2cdff62f..f92bb8faf0b2d9 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -434,6 +434,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -1353,6 +1358,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. + Praefect requires several secret tokens to secure communications across the Cluster: - ``: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index c28799ed4595f2..8878c7c4de2bcd 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -640,8 +640,13 @@ are supported and can be added if needed. ## Configure Consul and Sentinel -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: +Now that the Redis servers are all set up, let's configure the Consul + Sentinel +servers. + +NOTE: +To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. + +The following IPs will be used as an example: - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 @@ -1288,6 +1293,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. + Praefect requires several secret tokens to secure communications across the Cluster: - ``: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 3f2771dda29596..ab64189efbbe96 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -426,6 +426,11 @@ spread connections equally in practice. ## Configure Consul +Next, we set up the Consul servers. + +NOTE: +To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. + The following IPs will be used as an example: - `10.6.0.11`: Consul 1 @@ -1349,6 +1354,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. + Praefect requires several secret tokens to secure communications across the Cluster: - ``: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 43d63d4f08e1c2..a80bb3035c017a 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -637,8 +637,11 @@ are supported and can be added if needed. ## Configure Consul and Sentinel -Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs are used as an example: +Now that the Redis servers are all set up, let's configure the Consul + Sentinel +servers. + +NOTE: +To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 @@ -1285,6 +1288,9 @@ This is how this would work with a Omnibus GitLab PostgreSQL setup: Praefect is the router and transaction manager for Gitaly Cluster and all connections to Gitaly go through it. This section details how to configure it. +NOTE: +To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. + Praefect requires several secret tokens to secure communications across the Cluster: - ``: Used for repositories hosted on your Gitaly cluster and can only be accessed by Gitaly clients that carry this token. -- GitLab From 59c6a6ec383bb8cdb586de15cf9bb62f5664bb1b Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 16 Mar 2023 11:43:25 +0000 Subject: [PATCH 05/16] Adjust autoscaling note with link --- doc/administration/reference_architectures/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index b5dea34b7880cb..acd0aef3692ae0 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -379,9 +379,9 @@ Gitaly Cluster is only supported to be run on VMs. [Refer to this Epic for more As a general guidance only _stateless_ components of GitLab can be run in Autoscaling groups at this time, namely GitLab Rails and Sidekiq. -Other components that have state, such as Gitaly, are not supported in this fashion. +Other components that have state, such as Gitaly, are not supported in this fashion (see this [issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2997) for more information). -This extends to other third-party components such as Postgres and Redis as given in Omnibus GitLab, but you can explore other third-party solutions for those components as desired unless called out specifically as unsupported. +This extends to other third-party components such as Postgres and Redis as given in Omnibus GitLab, but you can explore other third-party solutions for those components if desired unless called out specifically as unsupported. #### Spreading one environment over multiple datacenters -- GitLab From 99de85a39208f7e99d633da122e73ed2e499a098 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 16 Mar 2023 11:50:20 +0000 Subject: [PATCH 06/16] Adjust tables --- .../reference_architectures/index.md | 76 ++++++++----------- 1 file changed, 32 insertions(+), 44 deletions(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index acd0aef3692ae0..0b39e9bc1098a4 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -476,34 +476,34 @@ table.test-coverage th { 1k Weekly - - - - + + + + 2k Weekly - - - - + + + + Planned 3k Weekly - - + + Weekly - + 5k Weekly - - - - + + + + 10k @@ -511,29 +511,33 @@ table.test-coverage th { Weekly Weekly Weekly - Ad-Hoc + 25k Weekly - - - - Ad-Hoc + + + + 50k Weekly - - Ad-Hoc (inc Cloud Services) - - + + + + ## Cost to run -The following table details the cost to run the different reference architectures across GCP, AWS, and Azure. Bare-metal costs are not included here as it varies widely depending on each customer configuration. +As a starting point, the following table details initial costs for the different reference architectures across GCP, AWS, and Azure via Omnibus. + +NOTE: +Due to the nature of Cloud Native Hybrid it's not possible to give a static cost calculation. +Bare-metal costs are also not included here as it varies widely depending on each customer configuration. @@ -541,71 +545,55 @@ The following table details the cost to run the different reference architecture - - - + + + - - - - - - - - - - - - - - - -
Reference
Architecture		GCPAWSAzureGCPAWSAzure
OmnibusCloud Native Hybrid OmnibusCloud Native Hybrid Omnibus
1k Calculated cost Calculated cost Calculated cost
2k Calculated cost Calculated cost Calculated cost
3k Calculated cost Calculated cost Calculated cost
5k Calculated cost Calculated cost Calculated cost
10k Calculated cost Calculated cost Calculated cost
25k Calculated cost Calculated cost Calculate cost
50k Calculated cost Calculated cost Calculated cost
-- GitLab From e8bcadce163bf93efb0c527e0cae2d58f2886792 Mon Sep 17 00:00:00 2001 From: Kassandra Svoboda Date: Thu, 16 Mar 2023 17:02:25 +0000 Subject: [PATCH 07/16] Apply 1 suggestion(s) to 1 file(s) --- doc/administration/reference_architectures/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 0b39e9bc1098a4..bd3858947760e0 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -369,7 +369,7 @@ While we endeavour to try and have a good range of support for GitLab environmen Running Gitaly Cluster in Kubernetes is not supported. -This is a complex area as Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to +Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to significant and complex issues. Gitaly Cluster is only supported to be run on VMs. [Refer to this Epic for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127) -- GitLab From fffd8c04da08cf2bd8b20a2d5d9a54ba00f46a51 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 16 Mar 2023 17:34:27 +0000 Subject: [PATCH 08/16] Fix lint errors --- doc/administration/reference_architectures/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index bd3858947760e0..206aaa5315340b 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -607,7 +607,7 @@ In this section you'll find links to documentation for relevant areas as well as ### Upgrades Upgrades for a Reference Architecture environment is the same as any other GitLab environment. -The main [Upgrade GitLab](../../update/) section has detailed steps on how to approach this. +The main [Upgrade GitLab](../../update/index.md) section has detailed steps on how to approach this. Zero-Downtime Upgrades are also available [as mentioned earlier](#zero-downtime-upgrades). @@ -631,5 +631,5 @@ Most will only need vertical scaling, but there are some specific areas that can There are numerous options available to monitor your GitLab environment. -[Several of these are bundled with GitLab](../monitoring/), but it's also possible to use third-party +[Several of these are bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party options if desired. -- GitLab From e587162606c412c006ba51dc295a1c3901ae5429 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 16 Mar 2023 17:54:40 +0000 Subject: [PATCH 09/16] Further updates --- .../reference_architectures/index.md | 60 +++++++++---------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 206aaa5315340b..0311bdf9386d75 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -363,16 +363,16 @@ However, it is still an additional layer and may still add some support complexi ### Unsupported Designs -While we endeavour to try and have a good range of support for GitLab environment designs there are certain approaches we know definitively not to work and as a result are not supported at this time. +While we endeavour to try and have a good range of support for GitLab environment designs there are certain approaches we know definitively not to work and as a result are not supported at this time, which are detailed in this section below. -#### Gitaly Cluster in Kubernetes +#### Stateful components in Kubernetes such as Gitaly Cluster -Running Gitaly Cluster in Kubernetes is not supported. +[Running stateful components in Kubernetes is not supported](https://docs.gitlab.com/charts/installation/). One exception for this is Gitaly Sharded in single node form, but please be aware this is still not recommended. -Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to -significant and complex issues. +Gitaly Cluster is only supported to be run on VMs as Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to significant and complex issues. +[Refer to this Epic for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127) -Gitaly Cluster is only supported to be run on VMs. [Refer to this Epic for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127) +This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported. #### Autoscaling of stateful nodes @@ -381,7 +381,7 @@ and Sidekiq. Other components that have state, such as Gitaly, are not supported in this fashion (see this [issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2997) for more information). -This extends to other third-party components such as Postgres and Redis as given in Omnibus GitLab, but you can explore other third-party solutions for those components if desired unless called out specifically as unsupported. +This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported. #### Spreading one environment over multiple datacenters @@ -476,34 +476,34 @@ table.test-coverage th { 1k Weekly - - - - + + + + 2k Weekly - - - + + + Planned 3k Weekly - - + + Weekly - + 5k Weekly - - - - + + + + 10k @@ -511,23 +511,23 @@ table.test-coverage th { Weekly Weekly Weekly - + 25k Weekly - - - - + + + + 50k Weekly - - - - + + + + -- GitLab From 4e309c356b4e61f1f470812641ef4d7d7547b143 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Tue, 21 Mar 2023 14:50:46 +0000 Subject: [PATCH 10/16] Apply 26 suggestion(s) to 2 file(s) --- .../reference_architectures/5k_users.md | 2 + .../reference_architectures/index.md | 55 +++++++++---------- 2 files changed, 29 insertions(+), 28 deletions(-) diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index a80bb3035c017a..642eb9096a4cd3 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -643,6 +643,8 @@ servers. NOTE: To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. +The following IPs will be used as an example: + - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 - `10.6.0.13`: Consul/Sentinel 3 diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 0311bdf9386d75..99376d8ba2c42e 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -49,7 +49,7 @@ The first choice to consider is whether a Self Managed approach is correct for y Running any application in production is complex, and the same applies for GitLab. While we aim to make this as smooth as possible, there are still the general complexities. This depends on the design chosen, but typically you'll need to manage all aspects such as hardware, operating systems, networking, storage, security, GitLab itself, and more. This includes both the initial setup of the environment and the longer term maintenance. -As such, it's recommended that you have a working knowledge of running and maintaining applications in production when deciding on going down this route. For users who aren't in this position our [Professional Services](https://about.gitlab.com/services/#implementation-services) team offer implementation services but for those who want a more managed solution long term it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). +As such, it's recommended that you have a working knowledge of running and maintaining applications in production when deciding on going down this route. If you aren't in this position, our [Professional Services](https://about.gitlab.com/services/#implementation-services) team offers implementation services, but for those who want a more managed solution long term, it's recommended to instead explore our other offerings such as [GitLab SaaS](../../subscriptions/gitlab_com/index.md) or [GitLab Dedicated](../../subscriptions/gitlab_dedicated/index.md). ## Deciding which architecture to use @@ -89,7 +89,7 @@ In general then, we'd only recommend you employ HA in the following scenarios: #### Scaled Down High Availability (HA) approaches -For users who still need to have HA for a lower number of users this can also be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). +If you still need to have HA for a lower number of users, this can be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). #### Zero Downtime Upgrades @@ -347,7 +347,7 @@ In addition to the above, you should be aware of the additional specific guidanc ## Deviating from the suggested reference architectures -As a general guideline, the further away you move from the Reference Architectures, +As a general guideline, the further away you move from the reference architectures, the harder it is to get support for it. With any deviation, you're introducing a layer of complexity that adds challenges to finding out where potential issues might lie. @@ -358,37 +358,37 @@ installed on separate machines (virtualized or bare metal), with machine hardwar requirements listed in the "Configuration" column and equivalent VM standard sizes listed in GCP/AWS/Azure columns of each [available reference architecture](#available-reference-architectures). -Running components on Docker (including Compose) with the same specs should be fine, as Docker is well known in terms of support. +Running components on Docker (including Docker Compose) with the same specs should be fine, as Docker is well known in terms of support. However, it is still an additional layer and may still add some support complexities, such as not being able to run `strace` easily in containers. -### Unsupported Designs +### Unsupported designs -While we endeavour to try and have a good range of support for GitLab environment designs there are certain approaches we know definitively not to work and as a result are not supported at this time, which are detailed in this section below. +While we endeavour to try and have a good range of support for GitLab environment designs, there are certain approaches we know definitively not to work, and as a result are not supported. Those approaches are detailed in the following sections. -#### Stateful components in Kubernetes such as Gitaly Cluster +#### Stateful components in Kubernetes -[Running stateful components in Kubernetes is not supported](https://docs.gitlab.com/charts/installation/). One exception for this is Gitaly Sharded in single node form, but please be aware this is still not recommended. +[Running stateful components in Kubernetes, such as Gitaly Cluster, is not supported](https://docs.gitlab.com/charts/installation/#configure-the-helm-chart-to-use-external-stateful-data). Gitaly Cluster is only supported to be run on VMs as Git itself doesn't match well with the Kubernetes design and attempting to run it can lead to significant and complex issues. -[Refer to this Epic for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127) +[Refer to epic 6127 for more information](https://gitlab.com/groups/gitlab-org/-/epics/6127). This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported. #### Autoscaling of stateful nodes -As a general guidance only _stateless_ components of GitLab can be run in Autoscaling groups at this time, namely GitLab Rails +As a general guidance, only _stateless_ components of GitLab can be run in Autoscaling groups, namely GitLab Rails and Sidekiq. -Other components that have state, such as Gitaly, are not supported in this fashion (see this [issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2997) for more information). +Other components that have state, such as Gitaly, are not supported in this fashion (for more information, see [issue 2997](https://gitlab.com/gitlab-org/gitaly/-/issues/2997)). This also applies to other third-party stateful components such as Postgres and Redis, but you can explore other third-party solutions for those components if desired such as supported Cloud Provider services unless called out specifically as unsupported. -#### Spreading one environment over multiple datacenters +#### Spreading one environment over multiple data centers -Deploying one GitLab environment over multiple datacenters is not supported due to potential split brain edge cases -if a DC were to go down. For example, several components of the GitLab setup, namely Consul and Praefect require an odd number quorum to function correctly and splitting over DCs can impact this notably. +Deploying one GitLab environment over multiple data centers is not supported due to potential split brain edge cases +if a data center were to go down. For example, several components of the GitLab setup, namely Consul and Praefect require an odd number quorum to function correctly and splitting over multiple data centers can impact this notably. -For deploying GitLab over multiple datacenters or regions we offer [GitLab Geo](https://about.gitlab.com/solutions/geo/) as a comprehensive solution. +For deploying GitLab over multiple data centers or regions we offer [GitLab Geo](https://about.gitlab.com/solutions/geo/) as a comprehensive solution. ## Validation and test results @@ -533,11 +533,11 @@ table.test-coverage th { ## Cost to run -As a starting point, the following table details initial costs for the different reference architectures across GCP, AWS, and Azure via Omnibus. +As a starting point, the following table details initial costs for the different reference architectures across GCP, AWS, and Azure via Omnibus. NOTE: -Due to the nature of Cloud Native Hybrid it's not possible to give a static cost calculation. -Bare-metal costs are also not included here as it varies widely depending on each customer configuration. +Due to the nature of Cloud Native Hybrid, it's not possible to give a static cost calculation. +Bare-metal costs are also not included here as it varies widely depending on each configuration. @@ -609,27 +609,26 @@ In this section you'll find links to documentation for relevant areas as well as Upgrades for a Reference Architecture environment is the same as any other GitLab environment. The main [Upgrade GitLab](../../update/index.md) section has detailed steps on how to approach this. -Zero-Downtime Upgrades are also available [as mentioned earlier](#zero-downtime-upgrades). +[Zero-downtime upgrades](#zero-downtime-upgrades) are also available. NOTE: You should upgrade a Reference Architecture in the same order as you created it. ### Scaling an environment -Scaling a GitLab environment is designed to be as seemless as possible. +Scaling a GitLab environment is designed to be as seamless as possible. -In terms of the Reference Architectures you would look to the next size and adjust accordingly. -Most will only need vertical scaling, but there are some specific areas that can be adjusted depending on the setup: +In terms of the Reference Architectures, you would look to the next size and adjust accordingly. +Most setups would only need vertical scaling, but there are some specific areas that can be adjusted depending on the setup: -- If scaling from a non-HA environment to a HA environment various components are recommended to be deployed in their HA forms: +- If you're scaling from a non-HA environment to an HA environment, various components are recommended to be deployed in their HA forms: - Redis to multi-node Redis w/ Redis Sentinel - Postgres to multi-node Postgres w/ Consul + PgBouncer - Gitaly to Gitaly Cluster w/ Praefect -- From 10k users and higher Redis is recommended to be split into multiple HA servers as it's single threaded +- From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded. -### How do I monitor my environment +### How to monitor your environment -There are numerous options available to monitor your GitLab environment. - -[Several of these are bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party +To monitor your GitLab environment, you can use the tools +[bundled with GitLab](../monitoring/index.md), but it's also possible to use third-party options if desired. -- GitLab From dcc1e1b84a1ceeabefd7a96bb971999460177691 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Tue, 21 Mar 2023 14:51:03 +0000 Subject: [PATCH 11/16] Apply 1 suggestion(s) to 1 file(s) --- doc/administration/reference_architectures/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 99376d8ba2c42e..efbe711ec6b3c6 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -87,7 +87,7 @@ In general then, we'd only recommend you employ HA in the following scenarios: - When you have 3,000 or more users. - When GitLab being down would critically impact your workflow. -#### Scaled Down High Availability (HA) approaches +#### Scaled-down High Availability (HA) approaches If you still need to have HA for a lower number of users, this can be achieved with an adjusted [3K architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). -- GitLab From 94c70fd813006316d09059af860cfd98765c860d Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 21 Mar 2023 15:45:16 +0000 Subject: [PATCH 12/16] Adjust Azure notes --- doc/administration/reference_architectures/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index efbe711ec6b3c6..e82248a1825d21 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -331,7 +331,7 @@ When selecting a database service, it should run a standard, performant, and [su Several database cloud provider services are known not to support the above or have been found to have other issues and aren't recommended: - [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible and not supported. See [14.4.0](../../update/index.md#1440) for more details. -- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is **strongly not recommended** for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. See [Recommendation Notes for Azure](#recommendation-notes-for-azure) for more details. - [Google AlloyDB](https://cloud.google.com/alloydb) and [Amazon RDS Multi-AZ DB cluster](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/multi-az-db-clusters-concepts.html) have not been tested and are not recommended. Both solutions are specifically not expected to work with GitLab Geo. - [Amazon RDS Multi-AZ DB instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZSingleStandby.html) is a separate product and is supported. @@ -341,8 +341,8 @@ Due to performance issues that we found with several key Azure services, we only In addition to the above, you should be aware of the additional specific guidance for Azure: -- **We outright strongly do not recommend [Azure Database for PostgreSQL Single Server](https://learn.microsoft.com/en-us/azure/postgresql/single-server/overview-single-server)** specifically due to significant performance and stability issues found. **For GitLab 14.0 and higher the service is not supported** due to it only supporting up to PostgreSQL 11. - - A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements. +- [Azure Database for PostgreSQL Single Server](https://azure.microsoft.com/en-gb/products/postgresql/#overview) (Single / Flexible) is not supported for use due to notable performance / stability issues or missing functionality. +- A new service, [Azure Database for PostgreSQL Flexible Server](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/) has been released. [Internal testing](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/91) has shown that it does look to perform as expected, but this hasn't been validated in production, so generally isn't recommended at this time. Additionally, as it's a new service, you may find that it's missing some functionality depending on your requirements. - [Azure Blob Storage](https://azure.microsoft.com/en-gb/products/storage/blobs/) has been found to have [performance limits that can impact production use at certain times](https://gitlab.com/gitlab-org/gitlab/-/issues/344861). However, this has only been seen in our largest architectures (25k+) so far. ## Deviating from the suggested reference architectures -- GitLab From ce9415aa5242cd4ef43b75e1f2d76a329bcf5372 Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 21 Mar 2023 17:00:18 +0000 Subject: [PATCH 13/16] Add note about migrations and PgBouncer --- doc/administration/reference_architectures/10k_users.md | 8 +++----- doc/administration/reference_architectures/25k_users.md | 8 +++----- doc/administration/reference_architectures/2k_users.md | 8 +++----- doc/administration/reference_architectures/3k_users.md | 8 +++----- doc/administration/reference_architectures/50k_users.md | 8 +++----- doc/administration/reference_architectures/5k_users.md | 8 +++----- 6 files changed, 18 insertions(+), 30 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index b2baa92c24c9ce..a969561aeee1cd 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -2097,11 +2097,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed you should configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index f92bb8faf0b2d9..5aa9f45fb7120e 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -2116,11 +2116,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed you should configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index d87e8270dcbc7b..1b939d176f15bd 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -774,11 +774,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed you should configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 8878c7c4de2bcd..cf8d779552ee83 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -2081,11 +2081,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed you should configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index ab64189efbbe96..2bdc9c28f52886 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -2115,11 +2115,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). sudo gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed you should configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 642eb9096a4cd3..773cbdb8a607d0 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -2076,11 +2076,9 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). gitlab-rake gitlab:db:configure ``` - If you encounter a `rake aborted!` error message stating that PgBouncer is - failing to connect to PostgreSQL, it may be that your PgBouncer node's IP - address is missing from PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` - on your database nodes. Before proceeding, see - [PgBouncer error `ERROR: pgbouncer cannot connect to server`](../postgresql/replication_and_failover.md#pgbouncer-error-error-pgbouncer-cannot-connect-to-server). + Note that this requires the Rails node to be configured to connect to the primary database + directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). + After migrations have completed you should configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). -- GitLab From 1cfffb3f567367adfbce8a11612df329fbaa498a Mon Sep 17 00:00:00 2001 From: Grant Young Date: Tue, 21 Mar 2023 17:09:56 +0000 Subject: [PATCH 14/16] Add ASG note --- doc/administration/reference_architectures/10k_users.md | 6 ++++-- doc/administration/reference_architectures/25k_users.md | 6 ++++-- doc/administration/reference_architectures/2k_users.md | 4 +++- doc/administration/reference_architectures/3k_users.md | 6 ++++-- doc/administration/reference_architectures/50k_users.md | 6 ++++-- doc/administration/reference_architectures/5k_users.md | 6 ++++-- 6 files changed, 23 insertions(+), 11 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index a969561aeee1cd..d89b59b71f7303 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly5 6 | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | | Praefect5 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq7 | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails7 | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage4 | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). NOTE: diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 5aa9f45fb7120e..8388088a400764 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly5 6 | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | | Praefect5 | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq7 | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails7 | 5 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage4 | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). NOTE: diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 1b939d176f15bd..28f7c4e7c67239 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -26,7 +26,7 @@ For a full list of reference architectures, see | PostgreSQL1 | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | `D2s v3` | | Redis2 | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | `D2s v3` | | Gitaly5 | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| GitLab Rails | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | +| GitLab Rails6 | 2 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | `F8s v2` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Object storage4 | - | - | - | - | - | @@ -45,6 +45,8 @@ For a full list of reference architectures, see 5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +6. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). NOTE: diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index cf8d779552ee83..f45eab59592646 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -40,8 +40,8 @@ For a full list of reference architectures, see | Gitaly5 6 | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | | Praefect5 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | +| Sidekiq7 | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| GitLab Rails7 | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage4 | - | - | - | - | @@ -62,6 +62,8 @@ For a full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). NOTE: diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 2bdc9c28f52886..a62ff8180ccc3c 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -31,8 +31,8 @@ full list of reference architectures, see | Gitaly5 6 | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | | Praefect5 | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | +| Sidekiq7 | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| GitLab Rails7 | 12 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | | Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | | Object storage4 | - | - | - | - | @@ -53,6 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). NOTE: diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 773cbdb8a607d0..29474579d0d48b 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -37,8 +37,8 @@ costly-to-operate environment by using the | Gitaly5 6 | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | | Praefect5 | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Praefect PostgreSQL1 | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | +| Sidekiq7 | 4 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| GitLab Rails7 | 3 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | | Monitoring node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | | Object storage4 | - | - | - | - | @@ -59,6 +59,8 @@ costly-to-operate environment by using the 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. +7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). NOTE: -- GitLab From 6c5e4662a7fb880c4db25e4b020009d23e7d6d11 Mon Sep 17 00:00:00 2001 From: Achilleas Pipinellis Date: Wed, 22 Mar 2023 17:15:41 +0000 Subject: [PATCH 15/16] Apply 12 suggestion(s) to 6 file(s) --- doc/administration/reference_architectures/10k_users.md | 6 +++--- doc/administration/reference_architectures/25k_users.md | 6 +++--- doc/administration/reference_architectures/2k_users.md | 6 +++--- doc/administration/reference_architectures/3k_users.md | 6 +++--- doc/administration/reference_architectures/50k_users.md | 6 +++--- doc/administration/reference_architectures/5k_users.md | 6 +++--- 6 files changed, 18 insertions(+), 18 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index d89b59b71f7303..31c421c3416c93 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -53,8 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. -7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). - - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. NOTE: @@ -2101,7 +2101,7 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). Note that this requires the Rails node to be configured to connect to the primary database directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). - After migrations have completed you should configure the node to pass through PgBouncer again. + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 8388088a400764..4c6fa949aba29b 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -53,8 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. -7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). - - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. NOTE: @@ -2120,7 +2120,7 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). Note that this requires the Rails node to be configured to connect to the primary database directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). - After migrations have completed you should configure the node to pass through PgBouncer again. + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 28f7c4e7c67239..7c1a76a2d5184c 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -45,8 +45,8 @@ For a full list of reference architectures, see 5. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. -6. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). - - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). +6. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. NOTE: @@ -778,7 +778,7 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). Note that this requires the Rails node to be configured to connect to the primary database directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). - After migrations have completed you should configure the node to pass through PgBouncer again. + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index f45eab59592646..3ed9c07d4bfaf5 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -62,8 +62,8 @@ For a full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. -7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). - - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. NOTE: @@ -2085,7 +2085,7 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). Note that this requires the Rails node to be configured to connect to the primary database directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). - After migrations have completed you should configure the node to pass through PgBouncer again. + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index a62ff8180ccc3c..094f837162e577 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -53,8 +53,8 @@ full list of reference architectures, see 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. -7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). - - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. NOTE: @@ -2119,7 +2119,7 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). Note that this requires the Rails node to be configured to connect to the primary database directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). - After migrations have completed you should configure the node to pass through PgBouncer again. + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 29474579d0d48b..03d37c2b11a0de 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -59,8 +59,8 @@ costly-to-operate environment by using the 6. Gitaly has been designed and tested with repositories of varying sizes that follow best practices. However, large repositories or monorepos that don't follow these practices can significantly impact Gitaly requirements. Refer to [Large repositories](index.md#large-repositories) for more information. -7. Can be placed in Auto Scaling Groups (ASGs) as component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). - - Note that however for GitLab Rails certain processes should only be run on one node including [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md). +7. Can be placed in Auto Scaling Groups (ASGs) as the component doesn't store any [stateful data](index.md#autoscaling-of-stateful-nodes). + However, for GitLab Rails certain processes like [migrations](#gitlab-rails-post-configuration) and [Mailroom](../incoming_email.md) should be run on only one node. NOTE: @@ -2080,7 +2080,7 @@ the [HTTPS documentation](https://docs.gitlab.com/omnibus/settings/ssl.html). Note that this requires the Rails node to be configured to connect to the primary database directly, [bypassing PgBouncer](../postgresql/pgbouncer.md#procedure-for-bypassing-pgbouncer). - After migrations have completed you should configure the node to pass through PgBouncer again. + After migrations have completed, you must configure the node to pass through PgBouncer again. 1. [Configure fast lookup of authorized SSH keys in the database](../operations/fast_ssh_key_lookup.md). -- GitLab From 6745d5f45c9c6d2e6a00be445d1244fdfdf18fcb Mon Sep 17 00:00:00 2001 From: Grant Young Date: Thu, 23 Mar 2023 10:37:42 +0000 Subject: [PATCH 16/16] Further tweaks --- .../reference_architectures/10k_users.md | 11 ++++++----- .../reference_architectures/25k_users.md | 11 ++++++----- .../reference_architectures/2k_users.md | 4 +--- .../reference_architectures/3k_users.md | 8 +++----- .../reference_architectures/50k_users.md | 11 ++++++----- .../reference_architectures/5k_users.md | 8 +++----- doc/administration/reference_architectures/index.md | 5 ++++- 7 files changed, 29 insertions(+), 29 deletions(-) diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 31c421c3416c93..09ba9838cf8e40 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -422,7 +422,7 @@ spread connections equally in practice. Next, we set up the Consul servers. NOTE: -To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. +Consul must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. The following IPs will be used as an example: @@ -808,6 +808,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or higher. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -1344,7 +1347,7 @@ Praefect is the router and transaction manager for Gitaly Cluster and all connec it. This section details how to configure it. NOTE: -To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. +Consul must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. Praefect requires several secret tokens to secure communications across the Cluster: @@ -1520,9 +1523,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 4c6fa949aba29b..f55740bdf9dbf2 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -439,7 +439,7 @@ spread connections equally in practice. Next, we set up the Consul servers. NOTE: -To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. +Consul must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. The following IPs will be used as an example: @@ -825,6 +825,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or higher. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -1361,7 +1364,7 @@ Praefect is the router and transaction manager for Gitaly Cluster and all connec it. This section details how to configure it. NOTE: -To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. +Praefect must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. Praefect requires several secret tokens to secure communications across the Cluster: @@ -1537,9 +1540,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 7c1a76a2d5184c..893d9eff21cedc 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -425,9 +425,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS -for write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +for write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Be sure to note the following items: diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 3ed9c07d4bfaf5..47d49ffb437354 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -646,7 +646,7 @@ Now that the Redis servers are all set up, let's configure the Consul + Sentinel servers. NOTE: -To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. +Consul and Redis Sentinel must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. The following IPs will be used as an example: @@ -1296,7 +1296,7 @@ Praefect is the router and transaction manager for Gitaly Cluster and all connec it. This section details how to configure it. NOTE: -To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. +Praefect must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. Praefect requires several secret tokens to secure communications across the Cluster: @@ -1472,9 +1472,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 094f837162e577..b89effb730a6bc 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -431,7 +431,7 @@ spread connections equally in practice. Next, we set up the Consul servers. NOTE: -To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. +Consul must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. The following IPs will be used as an example: @@ -818,6 +818,9 @@ Using [Redis](https://redis.io/) in scalable environment is possible using a **P topology with a [Redis Sentinel](https://redis.io/docs/manual/sentinel/) service to watch and automatically start the failover procedure. +NOTE: +Redis clusters must each be deployed in an odd number of 3 nodes or higher. This is to ensure Redis Sentinel can take votes as part of a quorum. This does not apply when configuring Redis externally, such as a cloud provider service. + Redis requires authentication if used with Sentinel. See [Redis Security](https://redis.io/docs/manual/security/) documentation for more information. We recommend using a combination of a Redis password and tight @@ -1357,7 +1360,7 @@ Praefect is the router and transaction manager for Gitaly Cluster and all connec it. This section details how to configure it. NOTE: -To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. +Praefect must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. Praefect requires several secret tokens to secure communications across the Cluster: @@ -1533,9 +1536,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 03d37c2b11a0de..5d6c650fe50857 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -643,7 +643,7 @@ Now that the Redis servers are all set up, let's configure the Consul + Sentinel servers. NOTE: -To maintain a voting quorum, Consul must be deployed in an odd number of 3 nodes or higher. +Consul and Redis Sentinel must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. The following IPs will be used as an example: @@ -1293,7 +1293,7 @@ Praefect is the router and transaction manager for Gitaly Cluster and all connec it. This section details how to configure it. NOTE: -To maintain a voting quorum, Praefect must be deployed in an odd number of 3 nodes or higher. +Praefect must be deployed in an odd number of 3 nodes or higher. This is to ensure the nodes can take votes as part of a quorum. Praefect requires several secret tokens to secure communications across the Cluster: @@ -1469,9 +1469,7 @@ Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs should have a throughput of at least 8,000 input/output operations per second (IOPS) for read operations and 2,000 IOPS for -write operations. These IOPS values are initial recommendations, and may be -adjusted to greater or lesser values depending on the scale of your -environment's workload. If you're running the environment on a Cloud provider, +write operations. If you're running the environment on a Cloud provider, refer to their documentation about how to configure IOPS correctly. Gitaly servers must not be exposed to the public internet, as Gitaly's network diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index e82248a1825d21..b73740cd1d2623 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -386,7 +386,7 @@ This also applies to other third-party stateful components such as Postgres and #### Spreading one environment over multiple data centers Deploying one GitLab environment over multiple data centers is not supported due to potential split brain edge cases -if a data center were to go down. For example, several components of the GitLab setup, namely Consul and Praefect require an odd number quorum to function correctly and splitting over multiple data centers can impact this notably. +if a data center were to go down. For example, several components of the GitLab setup, namely Consul, Redis Sentinel and Praefect require an odd number quorum to function correctly and splitting over multiple data centers can impact this notably. For deploying GitLab over multiple data centers or regions we offer [GitLab Geo](https://about.gitlab.com/solutions/geo/) as a comprehensive solution. @@ -627,6 +627,9 @@ Most setups would only need vertical scaling, but there are some specific areas - Gitaly to Gitaly Cluster w/ Praefect - From 10k users and higher, Redis is recommended to be split into multiple HA servers as it's single threaded. +Conversely, if you have robust metrics in place that show the environment is over-provisioned you can apply the same process for +scaling downloads. It's recommended to take an iterative approach when scaling downwards however to ensure there are no issues. + ### How to monitor your environment To monitor your GitLab environment, you can use the tools -- GitLab