diff --git a/doc/administration/img/db_load_balancing_postgres_stats.png b/doc/administration/img/db_load_balancing_postgres_stats.png deleted file mode 100644 index 8b311616e7b600ed5191a13821faf25eaafbdf97..0000000000000000000000000000000000000000 Binary files a/doc/administration/img/db_load_balancing_postgres_stats.png and /dev/null differ diff --git a/doc/administration/index.md b/doc/administration/index.md index 53a3c305aab471d5a528e62fb20c28397f94105b..17938d820c8930d6906709e18d5e57efcd4703fc 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -31,8 +31,6 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Installing GitLab - [Install](../install/index.md): Requirements, directory structures, and installation methods. - - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. - - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). - [Reference architectures](reference_architectures/index.md): Add additional resources to support more users. - [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. - [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. @@ -79,6 +77,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Enabling and disabling features flags](feature_flags.md): how to enable and disable GitLab features deployed behind feature flags. - [Application settings cache expiry interval](application_settings_cache.md) +- [Database load balancing](postgresql/database_load_balancing.md): Distribute database queries among multiple database servers. +- [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). #### Customizing GitLab appearance diff --git a/doc/administration/logs.md b/doc/administration/logs.md index bf74a96a627d7e9685ffa125d4e8a19a403341fd..c9b1253f04151034b274b1d967411611a47e2fbb 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -830,7 +830,7 @@ are generated in a location based on your installation method: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15442) in GitLab 12.3. -Contains details of GitLab [Database Load Balancing](database_load_balancing.md). +Contains details of GitLab [Database Load Balancing](postgresql/database_load_balancing.md). Depending on your installation method, this file is located at: - Omnibus GitLab: `/var/log/gitlab/gitlab-rails/database_load_balancing.log` diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index d798feb71a9f7079406db30d49b1ce69a2272347..6a5ddb3435dd6ea95e119037b12891157a57f52e 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -26,10 +26,9 @@ From left to right, the performance bar displays: details for each query: - **In a transaction**: shows up below the query if it was executed in the context of a transaction - - **Role**: shows up when [database load - balancing](../../database_load_balancing.md) is enabled. It shows - which server role was used for the query. "Primary" means that the query - was sent to the read/write primary server. "Replica" means it was sent + - **Role**: shows up when [database load balancing](../../postgresql/database_load_balancing.md) + is enabled. It shows which server role was used for the query. "Primary" means + that the query was sent to the read/write primary server. "Replica" means it was sent to a read-only replica. - **Config name**: shows up only when the `GITLAB_MULTIPLE_DATABASE_METRICS` environment variable is set. This is diff --git a/doc/administration/database_load_balancing.md b/doc/administration/postgresql/database_load_balancing.md similarity index 51% rename from doc/administration/database_load_balancing.md rename to doc/administration/postgresql/database_load_balancing.md index 45f27a8a8f2ecfd5a53dfb2f7015a6474854eaf5..d04e4de1cf32f455cb4e017a4b4e37ad7f0c3fb9 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/postgresql/database_load_balancing.md @@ -8,150 +8,128 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60894) from GitLab Premium to GitLab Free in 14.0. - -Distribute read-only queries among multiple database servers. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334494) for Sidekiq in GitLab 14.1. ## Overview -Database load balancing improves the distribution of database workloads across -multiple computing resources. Load balancing aims to optimize resource use, -maximize throughput, minimize response time, and avoid overload of any single -resource. Using multiple components with load balancing instead of a single -component may increase reliability and availability through redundancy. -[_Wikipedia article_](https://en.wikipedia.org/wiki/Load_balancing_(computing)) - -When database load balancing is enabled in GitLab, the load is balanced using -a simple round-robin algorithm, without any external dependencies such as Redis. - -In the following image, you can see the load is balanced rather evenly among -all the secondaries (`db4`, `db5`, `db6`). Because `SELECT` queries are not -sent to the primary (unless necessary), the primary (`db3`) hardly has any load. - -![DB load balancing graph](img/db_load_balancing_postgres_stats.png) - -## Requirements +With Database Load Balancing, read-only queries can be distributed across +multiple PostgreSQL nodes to increase performance. -For load balancing to work, you need at least PostgreSQL 11 or newer, -[**MySQL is not supported**](../install/requirements.md#database). You also need to make sure that you have -at least 1 secondary in [hot standby](https://www.postgresql.org/docs/11/hot-standby.html) mode. +This functionality is provided natively in GitLab Rails and Sidekiq where +it can be configured to balance the load round robin, without any external dependencies. -Load balancing also requires that the configured hosts **always** point to the -primary, even after a database failover. Furthermore, the additional hosts to -balance load among must **always** point to secondary databases. This means that -you should put a load balancer in front of every database, and have GitLab connect -to those load balancers. +```plantuml +@startuml +card "**Internal Load Balancer**" as ilb #9370DB +skinparam linetype ortho -For example, say you have a primary (`db1.gitlab.com`) and two secondaries, -`db2.gitlab.com` and `db3.gitlab.com`. For this setup, you need to have 3 -load balancers, one for every host. For example: +together { + collections "**GitLab Rails** x3" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} -- `primary.gitlab.com` forwards to `db1.gitlab.com` -- `secondary1.gitlab.com` forwards to `db2.gitlab.com` -- `secondary2.gitlab.com` forwards to `db3.gitlab.com` +collections "**Consul** x3" as consul #e76a9b -Now let's say that a failover happens and db2 becomes the new primary. This -means forwarding should now happen as follows: +card "Database" as database { + collections "**PGBouncer x3**\n//Consul//" as pgbouncer #4EA7FF + + card "**PostgreSQL** //Primary//\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// **x2**\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_secondary #4EA7FF -- `primary.gitlab.com` forwards to `db2.gitlab.com` -- `secondary1.gitlab.com` forwards to `db1.gitlab.com` -- `secondary2.gitlab.com` forwards to `db3.gitlab.com` + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]r-> postgres_secondary +} -GitLab does not take care of this for you, so you need to do so yourself. +gitlab -[#32CD32]-> ilb +gitlab -[hidden]-> pgbouncer +gitlab .[#32CD32,norank]-> postgres_primary +gitlab .[#32CD32,norank]-> postgres_secondary -Finally, load balancing requires that GitLab can connect to all hosts using the -same credentials and port as configured in the -[Enabling load balancing](#enabling-load-balancing) section. Using -different ports or credentials for different hosts is not supported. +sidekiq -[#ff8dd1]-> ilb +sidekiq -[hidden]-> pgbouncer +sidekiq .[#ff8dd1,norank]-> postgres_primary +sidekiq .[#ff8dd1,norank]-> postgres_secondary -## Use cases +ilb -[#9370DB]-> pgbouncer -- For GitLab instances with thousands of users and high traffic, you can use - database load balancing to reduce the load on the primary database and - increase responsiveness, thus resulting in faster page load inside GitLab. +consul -[#e76a9b]r-> pgbouncer +consul .[#e76a9b,norank]r-> postgres_primary +consul .[#e76a9b,norank]r-> postgres_secondary +@enduml +``` -## Enabling load balancing +### Requirements -For the environment in which you want to use load balancing, you'll need to add -the following. This balances the load between `host1.example.com` and -`host2.example.com`. +Note the following requirements for Database Load Balancing: -**In Omnibus installations:** +- A HA Postgres setup that has 1 or more secondary nodes replicating the primary. +- Each Postgres node can be connected to with the same credentials and on the same port. -1. Edit `/etc/gitlab/gitlab.rb` and add the following line: +When using Omnibus GitLab the additional requirements apply when [configuring a multi-node setup](replication_and_failover.md): - ```ruby - gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } - ``` +- PgBouncer configured on each PostgreSQL node to pooling all load balanced connections. -1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +## Configuring Database Load Balancing ---- +Database Load Balancing can be configured in one of two ways: -**In installations from source:** +- Hosts - A list of PostgreSQL hosts. (recommended) +- Service Discovery - A DNS record that returns a list of PostgreSQL hosts. -1. Edit `/home/git/gitlab/config/database.yml` and add or amend the following lines: +Refer below to each section for more details. - ```yaml - production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - hosts: - - host1.example.com - - host2.example.com - ``` +### Hosts -1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +Configuring with a list of hosts is straight forward. For the environment in which you want to use +load balancing, you add the `gitlab_rails['db_load_balancing']` setting into the `gitlab.rb` file in +the GitLab Rails / Sidekiq nodes. -### Load balancing for Sidekiq +For example, on an environment that has PostgreSQL running on the hosts `host1.example.com`, +`host2.example.com` and `host3.example.com` and reachable on the same port configured with +`gitlab_rails['db_port']`: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334494) in GitLab 14.1, load balancing for Sidekick is enabled by default. + +1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following line: -Sidekiq jobs mostly write to the primary database, but there are read-only jobs that can benefit -from the use of Sidekiq load balancing. -These jobs can use load balancing and database replicas to read the application state. -This allows to offload the primary database. +```ruby +gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] } +``` -For Sidekiq, we can define -[data consistency](../development/sidekiq_style_guide.md#job-data-consistency-strategies) -requirements for a specific job. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + -## Service Discovery **(PREMIUM SELF)** +### Service Discovery > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in GitLab 11.0. -Service discovery allows GitLab to automatically retrieve a list of secondary -databases to use, instead of having to manually specify these in the -`database.yml` configuration file. Service discovery works by periodically +Service discovery allows GitLab to automatically retrieve a list of PostgreSQL +hosts to use. Service discovery works by periodically checking a DNS A record, using the IPs returned by this record as the addresses for the secondaries. For service discovery to work, all you need is a DNS server and an A record containing the IP addresses of your secondaries. -To use service discovery you need to change your `database.yml` configuration -file so it looks like the following: - -```yaml -production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - discover: - nameserver: localhost - record: secondary.postgresql.service.consul - record_type: A - port: 8600 - interval: 60 - disconnect_timeout: 120 + +When using Omnibus GitLab the provided [Consul](../consul.md) service works as +a DNS server and will return PostgreSQL addresses via the `postgresql-ha.service.consul` +record. An example of how you would configure this along with details for each +setting after is as follows: + +1. On each GitLab Rails / Sidekiq node, edit `/etc/gitlab/gitlab.rb` and add the following line: + +```ruby +gitlab_rails['db_load_balancing'] = { 'discover' => { + 'nameserver' => 'localhost' + 'record' => 'postgresql-ha.service.consul' + 'record_type' => 'A' + 'port' => '8600' + 'interval' => '60' + 'disconnect_timeout' => '120' + } +} ``` + -Here, the `discover:` section specifies the configuration details to use for -service discovery. - -### Configuration - -The following options can be set: +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. | Option | Description | Default | |----------------------|---------------------------------------------------------------------------------------------------|-----------| @@ -179,45 +157,40 @@ When the list of hosts is updated, it might take a while for the old connections to be terminated. The `disconnect_timeout` setting can be used to enforce an upper limit on the time it takes to terminate all old database connections. -Some nameservers (like [Consul](https://www.consul.io/docs/discovery/dns#udp-based-dns-queries)) can return a truncated list of hosts when -queried over UDP. To overcome this issue, you can use TCP for querying by setting -`use_tcp` to `true`. +### Handling Stale Reads **(PREMIUM SELF)** -## Balancing queries - -Read-only `SELECT` queries balance among all the secondary hosts. -Everything else (including transactions) executes on the primary. -Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. - -## Prepared statements - -Prepared statements don't work well with load balancing and are disabled -automatically when load balancing is enabled. This should have no impact on -response timings. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. -## Primary sticking +To prevent reading from an outdated secondary the load balancer checks if it +is in sync with the primary. If the data is determined to be recent enough the +secondary is used, otherwise it is ignored. To reduce the overhead of +these checks we only perform these checks at certain intervals. -After a write has been performed, GitLab sticks to using the primary for a -certain period of time, scoped to the user that performed the write. GitLab -reverts back to using secondaries when they have either caught up, or after 30 -seconds. +There are three configuration options that influence this behavior: -## Failover handling +| Option | Description | Default | +|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| +| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | +| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | +| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | -In the event of a failover or an unresponsive database, the load balancer -tries to use the next available host. If no secondaries are available the -operation is performed on the primary instead. +The defaults should be sufficient for most users. -If a connection error occurs while writing data, the -operation is retried up to 3 times using an exponential back-off. +An example of configuring these options with a hosts list would be: -When using load balancing, you should be able to safely restart a database server -without it immediately leading to errors being presented to the users. +```ruby +gitlab_rails['db_load_balancing'] = { + 'hosts' => ['host1.example.com', 'host2.example.com', `host3.example.com`] + 'max_replication_difference' => 16777216 # 16 MB + 'max_replication_lag_time' => 30 + 'replica_check_interval' => 30 +} +``` ## Logging The load balancer logs various events in -[`database_load_balancing.log`](logs.md#database_load_balancinglog), such as +[`database_load_balancing.log`](../logs.md#database_load_balancinglog), such as - When a host is marked as offline - When a host comes back online @@ -237,36 +210,35 @@ For example: {"severity":"INFO","time":"2019-09-02T12:12:01.728Z","correlation_id":"abcdefg","event":"host_online","message":"Host came back online","db_host":"111.222.333.444","db_port":null,"tag":"rails.database_load_balancing","environment":"production","hostname":"web-example-1","fqdn":"gitlab.example.com","path":null,"params":null} ``` -## Handling Stale Reads **(PREMIUM SELF)** +## Implementation Details -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in GitLab 10.3. +### Balancing queries -To prevent reading from an outdated secondary the load balancer checks if it -is in sync with the primary. If the data is determined to be recent enough the -secondary is used, otherwise it is ignored. To reduce the overhead of -these checks we only perform these checks at certain intervals. +Read-only `SELECT` queries balance among all the given hosts. +Everything else (including transactions) executes on the primary. +Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. -There are three configuration options that influence this behavior: +### Prepared statements -| Option | Description | Default | -|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| -| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | -| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | -| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | +Prepared statements don't work well with load balancing and are disabled +automatically when load balancing is enabled. This should have no impact on +response timings. -The defaults should be sufficient for most users. Should you want to change them -you can specify them in `config/database.yml` like so: - -```yaml -production: - username: gitlab - database: gitlab - encoding: unicode - load_balancing: - hosts: - - host1.example.com - - host2.example.com - max_replication_difference: 16777216 # 16 MB - max_replication_lag_time: 30 - replica_check_interval: 30 -``` +### Primary sticking + +After a write has been performed, GitLab sticks to using the primary for a +certain period of time, scoped to the user that performed the write. GitLab +reverts back to using secondaries when they have either caught up, or after 30 +seconds. + +### Failover handling + +In the event of a failover or an unresponsive database, the load balancer +tries to use the next available host. If no secondaries are available the +operation is performed on the primary instead. + +If a connection error occurs while writing data, the +operation is retried up to 3 times using an exponential back-off. + +When using load balancing, you should be able to safely restart a database server +without it immediately leading to errors being presented to the users. diff --git a/doc/administration/postgresql/img/pg_ha_architecture.png b/doc/administration/postgresql/img/pg_ha_architecture.png deleted file mode 100644 index 5d2a4a584bfe8ef098628c7897a26b355d5f351c..0000000000000000000000000000000000000000 Binary files a/doc/administration/postgresql/img/pg_ha_architecture.png and /dev/null differ diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index e5fef61540a8422b859ada8f91847af25c0e212c..6c866fba0ab61e6b30f35680584e1ff4af115124 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -17,7 +17,7 @@ through `/etc/gitlab/gitlab.rb`. ## PgBouncer as part of a fault-tolerant GitLab installation -This content has been moved to a [new location](replication_and_failover.md#configuring-the-pgbouncer-node). +This content has been moved to a [new location](replication_and_failover.md#configuring-the-pgbouncer-nodes). ## PgBouncer as part of a non-fault-tolerant GitLab installation diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index 01fe4bf64baa1e188ca011d85b0e199dffd378ba..984495b03d26394c0ea702553b4a443b49689296 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -19,13 +19,54 @@ replication and failover for GitLab. ## Architecture The Omnibus GitLab recommended configuration for a PostgreSQL cluster with -replication and failover requires: +replication, failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An internal load balancer (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](database_load_balancing.md) enabled. + - This feature requires a local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + +```plantuml +@startuml +card "**Internal Load Balancer**" as ilb #9370DB +skinparam linetype ortho + +together { + collections "**GitLab Rails** x3" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 +} + +collections "**Consul** x3" as consul #e76a9b -- A minimum of three database nodes. -- A minimum of three `Consul` server nodes. -- A minimum of one `pgbouncer` service node, but it's recommended to have one per database node. An internal load balancer (TCP) is required when there is more than one `pgbouncer` service node. +card "Database" as database { + collections "**PGBouncer x3**\n//Consul//" as pgbouncer #4EA7FF + + card "**PostgreSQL** //Primary//\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// **x2**\n//Patroni//\n//PgBouncer//\n//Consul//" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]r-> postgres_secondary +} -![PostgreSQL HA Architecture](img/pg_ha_architecture.png) +gitlab -[#32CD32]-> ilb +gitlab -[hidden]-> pgbouncer +gitlab .[#32CD32,norank]-> postgres_primary +gitlab .[#32CD32,norank]-> postgres_secondary + +sidekiq -[#ff8dd1]-> ilb +sidekiq -[hidden]-> pgbouncer +sidekiq .[#ff8dd1,norank]-> postgres_primary +sidekiq .[#ff8dd1,norank]-> postgres_secondary + +ilb -[#9370DB]-> pgbouncer + +consul -[#e76a9b]r-> pgbouncer +consul .[#e76a9b,norank]r-> postgres_primary +consul .[#e76a9b,norank]r-> postgres_secondary +@enduml +``` You also need to take into consideration the underlying network topology, making sure you have redundant connectivity between all Database and GitLab instances @@ -38,13 +79,14 @@ shipped with Omnibus GitLab, and thus Patroni becomes mandatory for replication ### Database node -Each database node runs three services: +Each database node runs four services: - `PostgreSQL`: The database itself. - `Patroni`: Communicates with other Patroni services in the cluster and handles failover when issues with the leader server occurs. The failover procedure consists of: - Selecting a new leader for the cluster. - Promoting the new node to leader. - Instructing remaining servers to follow the new leader node. +- `PgBouncer`: A local pooler for the node. Used for read queries as part of [Database Load Balancing](database_load_balancing.md). - `Consul` agent: To communicate with Consul cluster which stores the current Patroni state. The agent monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. ### Consul server node @@ -62,8 +104,22 @@ Each PgBouncer node runs two services: Each service in the package comes with a set of [default ports](../package_information/defaults.md#ports). You may need to make specific firewall rules for the connections listed below: +There are several connection flows in this setup: + +#### Primary + - Application servers connect to either PgBouncer directly via its [default port](../package_information/defaults.md) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers. -- PgBouncer connects to the primary database servers [PostgreSQL default port](../package_information/defaults.md) +- PgBouncer connects to the primary database server's [PostgreSQL default port](../package_information/defaults.md) + +#### Database Load Balancing + +For read queries against data that haven't been recently changed and are update to date on all database nodes: + +- Application servers connect to the local PgBouncer service via its [default port](../package_information/defaults.md) on each database node in round robin fashion +- Local PgBouncer connects to the local database server's [PostgreSQL default port](../package_information/defaults.md) + +#### Replication + - Patroni actively manages the running PostgreSQL processes and configuration. - PostgreSQL secondaries connect to the primary database servers [PostgreSQL default port](../package_information/defaults.md) - Consul servers and agents connect to each others [Consul default ports](../package_information/defaults.md) @@ -203,8 +259,8 @@ repmgr-specific configuration as well. Especially, make sure that you remove `po Here is an example: ```ruby -# Disable all components except Patroni and Consul -roles(['patroni_role']) +# Disable all components except Patroni, PgBouncer and Consul +roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -245,6 +301,15 @@ patroni['allowlist'] = %w(XXX.XXX.XXX.XXX/YY 127.0.0.1/32) # Replace XXX.XXX.XXX.XXX/YY with Network Address postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY 127.0.0.1/32) +# Local PgBouncer service for Database Load Balancing +pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "PGBOUNCER_USERNAME", + password: 'PGBOUNCER_PASSWORD_HASH' + } +} + # Replace placeholders: # # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z @@ -342,7 +407,7 @@ You can use different certificates and keys for both API server and client on di However, the CA certificate (`patroni['tls_ca_file']`), TLS certificate verification (`patroni['tls_verify']`), and client TLS authentication mode (`patroni['tls_client_mode']`), must each have the same value on all nodes. -### Configuring the PgBouncer node +### Configuring the PgBouncer nodes 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. @@ -480,6 +545,7 @@ attributes set, but the following need to be set. gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' gitlab_rails['auto_migrate'] = false + gitlab_rails['db_load_balancing'] = { 'hosts' => ['POSTGRESQL_NODE_1', 'POSTGRESQL_NODE_2', 'POSTGRESQL_NODE_3'] } ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -595,8 +661,8 @@ An internal load balancer (TCP) is then required to be setup to serve each PgBou On database nodes edit `/etc/gitlab/gitlab.rb`: ```ruby -# Disable all components except Patroni and Consul -roles(['patroni_role']) +# Disable all components except Patroni, PgBouncer and Consul +roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -616,6 +682,15 @@ patroni['postgresql']['max_wal_senders'] = 7 patroni['allowlist'] = = %w(10.6.0.0/16 127.0.0.1/32) postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16 127.0.0.1/32) +# Local PgBouncer service for Database Load Balancing +pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '771a8625958a529132abe6f1a4acb19c' + } +} + # Configure the Consul agent consul['services'] = %w(postgresql) consul['configuration'] = { @@ -650,115 +725,6 @@ After deploying the configuration follow these steps: gitlab-rake gitlab:db:configure ``` -### Example minimal setup - -This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer setup alongside). - -It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. -The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#manual-failover-procedure-for-patroni) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines. - -In this example, we start with all servers on the same 10.6.0.0/16 private network range; they can connect to each freely other on those addresses. - -Here is a list and description of each machine and the assigned IP: - -- `10.6.0.21`: PostgreSQL 1 -- `10.6.0.22`: PostgreSQL 2 -- `10.6.0.23`: PostgreSQL 3 -- `10.6.0.31`: GitLab application - -All passwords are set to `toomanysecrets`. Please do not use this password or derived hashes. - -The `external_url` for GitLab is `http://gitlab.example.com` - -After the initial configuration, if a failover occurs, the PostgresSQL leader node changes to one of the available secondaries until it is failed back. - -#### Example minimal configuration for database servers - -On database nodes edit `/etc/gitlab/gitlab.rb`: - -```ruby -# Disable all components except Patroni and Consul -roles(['patroni_role']) - -# PostgreSQL configuration -postgresql['listen_address'] = '0.0.0.0' -postgresql['hot_standby'] = 'on' -postgresql['wal_level'] = 'replica' - -# Disable automatic database migrations -gitlab_rails['auto_migrate'] = false - -# Configure the Consul agent -consul['services'] = %w(postgresql) - -postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' -postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' - -# Sets `max_replication_slots` to double the number of database nodes. -# Patroni uses one extra slot per node when initiating the replication. -patroni['postgresql']['max_replication_slots'] = 6 - -patroni['username'] = 'PATRONI_API_USERNAME' -patroni['password'] = 'PATRONI_API_PASSWORD' - -# Set `max_wal_senders` to one more than the number of replication slots in the cluster. -# This is used to prevent replication from using up all of the -# available database connections. -patroni['postgresql']['max_wal_senders'] = 7 - -patroni['allowlist'] = = %w(10.6.0.0/16 127.0.0.1/32) -postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16 127.0.0.1/32) - -consul['configuration'] = { - server: true, - retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) -} -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -#### Example minimal configuration for application server - -On the server edit `/etc/gitlab/gitlab.rb`: - -```ruby -external_url 'http://gitlab.example.com' - -gitlab_rails['db_host'] = '127.0.0.1' -gitlab_rails['db_port'] = 6432 -gitlab_rails['db_password'] = 'toomanysecrets' -gitlab_rails['auto_migrate'] = false - -postgresql['enable'] = false -pgbouncer['enable'] = true -consul['enable'] = true - -# Configure PgBouncer -pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - -# Configure Consul agent -consul['watchers'] = %w(postgresql) - -pgbouncer['users'] = { - 'gitlab-consul': { - password: '5e0e3263571e3704ad655076301d6ebe' - }, - 'pgbouncer': { - password: '771a8625958a529132abe6f1a4acb19c' - } -} - -consul['configuration'] = { - retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) -} -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -#### Example minimal setup manual steps - -The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps). - ## Patroni NOTE: @@ -1047,7 +1013,7 @@ Here are a few key facts that you must consider before upgrading PostgreSQL: configured replication method (`pg_basebackup` is the only available option). It might take some time for replica to catch up with the leader, depending on the size of your database. -- An overview of the upgrade procedure is outlined in [Patoni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version). +- An overview of the upgrade procedure is outlined in [Patroni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version). You can still use `gitlab-ctl pg-upgrade` which implements this procedure with a few adjustments. Considering these, you should carefully plan your PostgreSQL upgrade: diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 9c3c33e1fa86e177617eee97408c5dfaa08e9f4b..d42e76248d9decf14e860213be3a2890f4012dc0 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -49,6 +49,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 10k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -73,8 +75,8 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary @@ -83,31 +85,38 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -471,8 +480,8 @@ run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance @@ -488,12 +497,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.21`: PostgreSQL primary @@ -548,8 +570,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -594,6 +616,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -654,9 +685,11 @@ If the 'State' column for any node doesn't say "running", check the -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -1671,8 +1704,8 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1797,6 +1830,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2212,6 +2247,7 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 10k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 @@ -2221,7 +2257,6 @@ card "Kubernetes via Helm Charts" as kubernetes { collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } @@ -2249,37 +2284,33 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 25cafbe667b9bc94dc34c44f0a42cc91a9422e7d..95892dcd1882f55073386f623e6e278b64ce363f 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -49,6 +49,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 25k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -73,8 +75,8 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary @@ -83,31 +85,38 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -474,8 +483,8 @@ run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance @@ -491,12 +500,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.21`: PostgreSQL primary @@ -551,8 +573,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -597,6 +619,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -657,9 +688,11 @@ If the 'State' column for any node doesn't say "running", check the -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -1677,8 +1710,8 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1805,6 +1838,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2212,16 +2247,16 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 25k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 together { - collections "**Webservice** x7" as gitlab #32CD32 + collections "**Webservice** x4" as gitlab #32CD32 collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } @@ -2249,37 +2284,33 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index e619294704fe8a77af0b69b16ac130c3eba426bd..be982712c893669e84f57b01bfda75482435481b 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -41,6 +41,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 2k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 collections "**GitLab Rails** x3" as gitlab #32CD32 @@ -1038,6 +1040,7 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 2k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 @@ -1045,10 +1048,8 @@ card "Kubernetes via Helm Charts" as kubernetes { together { collections "**Webservice** x3" as gitlab #32CD32 collections "**Sidekiq** x2" as sidekiq #ff8dd1 + card "**Supporting Services**" as support } - - card "**Prometheus + Grafana**" as monitor #7FFFD4 - card "**Supporting Services**" as support } card "**Gitaly**" as gitaly #FF8C00 @@ -1057,7 +1058,6 @@ card "**Redis**" as redis #FF6347 cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor gitlab -[#32CD32]--> gitaly gitlab -[#32CD32]--> postgres @@ -1066,14 +1066,8 @@ gitlab -[#32CD32]--> redis sidekiq -[#ff8dd1]--> gitaly sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> postgres -sidekiq -[#ff8dd1]---> redis - -monitor .[#7FFFD4]u-> gitlab -monitor .[#7FFFD4]-> gitaly -monitor .[#7FFFD4]-> postgres -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4,norank]u--> elb +sidekiq -[#ff8dd1]--> postgres +sidekiq -[#ff8dd1]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 9332ae8d27104a63aed5d67190c359dfa05af23c..89ad619b048cecaa31539d9b436afba93cffecd1 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -58,6 +58,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 3k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -66,7 +68,10 @@ together { collections "**Sidekiq** x4" as sidekiq #ff8dd1 } -card "**Prometheus + Grafana**" as monitor #7FFFD4 +together { + card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b +} card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -79,47 +84,45 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary } -card "**Consul + Sentinel**" as consul_sentinel { - collections "**Consul** x3" as consul #e76a9b - collections "**Redis Sentinel** x3" as sentinel #e6e727 -} - card "Redis" as redis { collections "**Redis** x3" as redis_nodes #FF6347 - - redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -769,8 +772,8 @@ run: sentinel: (pid 30098) 76832s; run: log: (pid 29704) 76850s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance @@ -786,12 +789,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.31`: PostgreSQL primary @@ -846,8 +862,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -892,6 +908,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -952,9 +977,11 @@ If the 'State' column for any node doesn't say "running", check the -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.21`: PgBouncer 1 @@ -1613,8 +1640,8 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1773,6 +1800,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2183,25 +2212,21 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 3k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 together { - collections "**Webservice** x2" as gitlab #32CD32 - collections "**Sidekiq** x3" as sidekiq #ff8dd1 + collections "**Webservice** x4" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } card "**Internal Load Balancer**" as ilb #9370DB - -card "**Consul + Sentinel**" as consul_sentinel { - collections "**Consul** x3" as consul #e76a9b - collections "**Redis Sentinel** x3" as sentinel #e6e727 -} +collections "**Consul** x3" as consul #e76a9b card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -2221,41 +2246,33 @@ card "Database" as database { postgres_primary .[#4EA7FF]> postgres_secondary } -card "Redis" as redis { +card "redis" as redis { collections "**Redis** x3" as redis_nodes #FF6347 - - redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index bbdf798d9ad58e41ae7a9521e177b12fcee46114..0fc6073f754e95d095f531644575c5e3313e563b 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -49,6 +49,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 50k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -73,8 +75,8 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary @@ -83,31 +85,38 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -480,8 +489,8 @@ run: node-exporter: (pid 30093) 76833s; run: log: (pid 29663) 76855s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance @@ -497,12 +506,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.21`: PostgreSQL primary @@ -557,8 +579,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -604,6 +626,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -664,9 +695,11 @@ If the 'State' column for any node doesn't say "running", check the -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.31`: PgBouncer 1 @@ -891,7 +924,7 @@ a node and change its status from primary to replica (and vice versa). package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version and type (Community or Enterprise editions) as your current install. -1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the priimary node in the previous section by replacing `redis_master_node` with `redis_replica_node`: +1. Edit `/etc/gitlab/gitlab.rb` and add the same contents as the primary node in the previous section by replacing `redis_master_node` with `redis_replica_node`: ```ruby # Specify server role as 'redis_replica_role' with Sentinel and enable Consul agent @@ -1684,8 +1717,8 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1819,6 +1852,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.21', '10.6.0.22', '10.6.0.23'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2226,16 +2261,16 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 50k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 together { - collections "**Webservice** x16" as gitlab #32CD32 + collections "**Webservice** x4" as gitlab #32CD32 collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } @@ -2263,37 +2298,33 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 + + redis_cache -[hidden]-> redis_persistent } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index a1921f50e4ea305db2e3dc292e4dbeb84910dae9..4f0b333f2b025db8aed36153233982baeb59ce01 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -55,6 +55,8 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 5k +skinparam linetype ortho + card "**External Load Balancer**" as elb #6a9be7 card "**Internal Load Balancer**" as ilb #9370DB @@ -63,7 +65,10 @@ together { collections "**Sidekiq** x4" as sidekiq #ff8dd1 } -card "**Prometheus + Grafana**" as monitor #7FFFD4 +together { + card "**Prometheus + Grafana**" as monitor #7FFFD4 + collections "**Consul** x3" as consul #e76a9b +} card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -76,47 +81,45 @@ card "Gitaly Cluster" as gitaly_cluster { card "Database" as database { collections "**PGBouncer** x3" as pgbouncer #4EA7FF - card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF - collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + card "**PostgreSQL** //Primary//" as postgres_primary #4EA7FF + collections "**PostgreSQL** //Secondary// x2" as postgres_secondary #4EA7FF pgbouncer -[#4EA7FF]-> postgres_primary postgres_primary .[#4EA7FF]> postgres_secondary } -card "**Consul + Sentinel**" as consul_sentinel { - collections "**Consul** x3" as consul #e76a9b - collections "**Redis Sentinel** x3" as sentinel #e6e727 -} - card "Redis" as redis { collections "**Redis** x3" as redis_nodes #FF6347 - - redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]--> monitor +elb -[#6a9be7,norank]--> monitor -gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis +gitlab -[#32CD32,norank]--> ilb +gitlab -[#32CD32]r-> object_storage +gitlab -[#32CD32]----> redis +gitlab .[#32CD32]----> database gitlab -[hidden]-> monitor gitlab -[hidden]-> consul -sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis +sidekiq -[#ff8dd1,norank]--> ilb +sidekiq -[#ff8dd1]r-> object_storage +sidekiq -[#ff8dd1]----> redis +sidekiq .[#ff8dd1]----> database sidekiq -[hidden]-> monitor sidekiq -[hidden]-> consul -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden]--> redis +ilb -[hidden]u-> consul +ilb -[hidden]u-> monitor consul .[#e76a9b]u-> gitlab consul .[#e76a9b]u-> sidekiq -consul .[#e76a9b]> monitor +consul .[#e76a9b]r-> monitor consul .[#e76a9b]-> database consul .[#e76a9b]-> gitaly_cluster consul .[#e76a9b,norank]--> redis @@ -760,8 +763,8 @@ run: sentinel: (pid 30098) 76832s; run: log: (pid 29704) 76850s ## Configure PostgreSQL -In this section, you'll be guided through configuring an external PostgreSQL database -to be used with GitLab. +In this section, you'll be guided through configuring a highly available PostgreSQL +cluster to be used with GitLab. ### Provide your own PostgreSQL instance @@ -777,12 +780,25 @@ If you use a cloud-managed service, or provide your own PostgreSQL: needs privileges to create the `gitlabhq_production` database. 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). +1. For improved performance, configuring [Database Load Balancing](../postgresql/database_load_balancing.md) + with multiple read replicas is recommended. See [Configure GitLab using an external PostgreSQL service](../postgresql/external.md) for further configuration steps. ### Standalone PostgreSQL using Omnibus GitLab +The recommended Omnibus GitLab configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three PostgreSQL nodes. +- A minimum of three Consul server nodes. +- A minimum of three PgBouncer nodes that track and handle primary database reads and writes. + - An [internal load balancer](#configure-the-internal-load-balancer) (TCP) to balance requests between the PgBouncer nodes. +- [Database Load Balancing](../postgresql/database_load_balancing.md) enabled. + + A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. + The following IPs will be used as an example: - `10.6.0.31`: PostgreSQL primary @@ -837,8 +853,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. On every database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: ```ruby - # Disable all components except Patroni and Consul - roles(['patroni_role']) + # Disable all components except Patroni, PgBouncer and Consul + roles(['patroni_role', 'pgbouncer_role']) # PostgreSQL configuration postgresql['listen_address'] = '0.0.0.0' @@ -883,6 +899,15 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Replace 10.6.0.0/24 with Network Address postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/24 127.0.0.1/32) + # Local PgBouncer service for Database Load Balancing + pgbouncer['databases'] = { + gitlabhq_production: { + host: "127.0.0.1", + user: "pgbouncer", + password: '' + } + } + # Set the network addresses that the exporters will listen on for monitoring node_exporter['listen_address'] = '0.0.0.0:9100' postgres_exporter['listen_address'] = '0.0.0.0:9187' @@ -943,9 +968,11 @@ If the 'State' column for any node doesn't say "running", check the -## Configure PgBouncer +### Configure PgBouncer + +Now that the PostgreSQL servers are all set up, let's configure PgBouncer +for tracking and handling reads/writes to the primary database. -Now that the PostgreSQL servers are all set up, let's configure PgBouncer. The following IPs will be used as an example: - `10.6.0.21`: PgBouncer 1 @@ -1604,8 +1631,8 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['db_host'] = '10.6.0.40' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -1764,6 +1791,8 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '' + gitlab_rails['db_load_balancing'] = { 'hosts' => ['10.6.0.31', '10.6.0.32', '10.6.0.33'] } # PostgreSQL IPs + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -2153,25 +2182,21 @@ For all PaaS solutions that involve configuring instances, it is strongly recomm ```plantuml @startuml 5k +skinparam linetype ortho card "Kubernetes via Helm Charts" as kubernetes { card "**External Load Balancer**" as elb #6a9be7 together { - collections "**Webservice** x5" as gitlab #32CD32 - collections "**Sidekiq** x3" as sidekiq #ff8dd1 + collections "**Webservice** x4" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 } - card "**Prometheus + Grafana**" as monitor #7FFFD4 card "**Supporting Services**" as support } card "**Internal Load Balancer**" as ilb #9370DB - -card "**Consul + Sentinel**" as consul_sentinel { - collections "**Consul** x3" as consul #e76a9b - collections "**Redis Sentinel** x3" as sentinel #e6e727 -} +collections "**Consul** x3" as consul #e76a9b card "Gitaly Cluster" as gitaly_cluster { collections "**Praefect** x3" as praefect #FF8C00 @@ -2191,41 +2216,33 @@ card "Database" as database { postgres_primary .[#4EA7FF]> postgres_secondary } -card "Redis" as redis { +card "redis" as redis { collections "**Redis** x3" as redis_nodes #FF6347 - - redis_nodes <.[#FF6347]- sentinel } cloud "**Object Storage**" as object_storage #white elb -[#6a9be7]-> gitlab -elb -[#6a9be7]-> monitor +elb -[hidden]-> sidekiq elb -[hidden]-> support gitlab -[#32CD32]--> ilb -gitlab -[#32CD32]-> object_storage -gitlab -[#32CD32]---> redis -gitlab -[hidden]--> consul +gitlab -[#32CD32]r--> object_storage +gitlab -[#32CD32,norank]----> redis +gitlab -[#32CD32]----> database sidekiq -[#ff8dd1]--> ilb -sidekiq -[#ff8dd1]-> object_storage -sidekiq -[#ff8dd1]---> redis -sidekiq -[hidden]--> consul - -ilb -[#9370DB]-> gitaly_cluster -ilb -[#9370DB]-> database +sidekiq -[#ff8dd1]r--> object_storage +sidekiq -[#ff8dd1,norank]----> redis +sidekiq .[#ff8dd1]----> database -consul .[#e76a9b]-> database -consul .[#e76a9b]-> gitaly_cluster -consul .[#e76a9b,norank]--> redis +ilb -[#9370DB]--> gitaly_cluster +ilb -[#9370DB]--> database +ilb -[hidden,norank]--> redis -monitor .[#7FFFD4]> consul -monitor .[#7FFFD4]-> database -monitor .[#7FFFD4]-> gitaly_cluster -monitor .[#7FFFD4,norank]--> redis -monitor .[#7FFFD4]> ilb -monitor .[#7FFFD4,norank]u--> elb +consul .[#e76a9b]--> database +consul .[#e76a9b,norank]--> gitaly_cluster +consul .[#e76a9b]--> redis @enduml ``` diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 10e6d717a1814480e190cac3babbbbe2910502a9..424173e128a7d79208afcf7a8b6202530b8f20b7 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -29,7 +29,7 @@ project.feature_available?(:feature_symbol) ## Restricting global features (instance) However, for features such as [Geo](../administration/geo/index.md) and -[Load balancing](../administration/database_load_balancing.md), which cannot be restricted +[Load balancing](../administration/postgresql/database_load_balancing.md), which cannot be restricted to only a subset of projects or namespaces, the check is made directly in the instance license. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index cbf3c09b28b336a5285a8ce8b0d2b1af92066ea7..74b1d5680b0966f7f65107a988700430abb2c636 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -160,10 +160,10 @@ query. This in turn makes it much harder for this code to overload a database. ## Use read replicas when possible -In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. +In a DB cluster we have many read replicas and one primary. A classic use of scaling the DB is to have read-only actions be performed by the replicas. We use [load balancing](../administration/postgresql/database_load_balancing.md) to distribute this load. This allows for the replicas to grow as the pressure on the DB grows. By default, queries use read-only replicas, but due to -[primary sticking](../administration/database_load_balancing.md#primary-sticking), GitLab uses the +[primary sticking](../administration/postgresql/database_load_balancing.md#primary-sticking), GitLab uses the primary for some time and reverts to secondaries after they have either caught up or after 30 seconds. Doing this can lead to a considerable amount of unnecessary load on the primary. To prevent switching to the primary [merge request 56849](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56849) introduced the @@ -187,7 +187,7 @@ Internally, our database load balancer classifies the queries based on their mai - Sidekiq background jobs After the above queries are executed, GitLab -[sticks to the primary](../administration/database_load_balancing.md#primary-sticking). +[sticks to the primary](../administration/postgresql/database_load_balancing.md#primary-sticking). To make the inside queries prefer using the replicas, [merge request 59086](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59086) introduced `fallback_to_replicas_for_ambiguous_queries`. This MR is also an example of how we redirected a diff --git a/doc/development/scalability.md b/doc/development/scalability.md index fdae66b7abc8002d76b03802105976bafd5e4526..3fc3d160bc8645e73f244a48979bffc287f5f98d 100644 --- a/doc/development/scalability.md +++ b/doc/development/scalability.md @@ -123,9 +123,8 @@ the read replicas. [Omnibus ships with Patroni](../administration/postgresql/rep #### Load-balancing -GitLab EE has [application support for load balancing using read -replicas](../administration/database_load_balancing.md). This load balancer does -some actions that aren't traditionally available in standard load balancers. For +GitLab EE has [application support for load balancing using read replicas](../administration/postgresql/database_load_balancing.md). +This load balancer does some actions that aren't traditionally available in standard load balancers. For example, the application considers a replica only if its replication lag is low (for example, WAL data behind by less than 100 MB). diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index e28a328888d1d7e07d8f1929aa3f8d10678c5575..4d1f0daf66bcb4a40c44390386ca6e2115f347b4 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -574,7 +574,7 @@ of reading a stale record is non-zero due to replicas potentially lagging behind When the number of jobs that rely on the database increases, ensuring immediate data consistency can put unsustainable load on the primary database server. We therefore added the ability to use -[database load balancing for Sidekiq workers](../administration/database_load_balancing.md#load-balancing-for-sidekiq). +[database load balancing for Sidekiq workers](../administration/postgresql/database_load_balancing.md). By configuring a worker's `data_consistency` field, we can then allow the scheduler to target read replicas under several strategies outlined below.