diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index f04351119ba4d9eb68615be031c51a1b0120b519..5d676dac0001372986e5fc0650b67d79d5b7f7cb 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -188,10 +188,11 @@ To set up GitLab and its components to accommodate up to 10,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs) - to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -2011,7 +2012,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs): +1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2177,7 +2178,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs) and in general it's better +It's recommended over [NFS](#configure-nfs-optional) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2189,7 +2190,6 @@ GitLab has been tested on a number of object storage providers: - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) -- On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. There are two ways of specifying object storage configuration in GitLab: @@ -2199,25 +2199,13 @@ There are two ways of specifying object storage configuration in GitLab: - [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). -Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. - -For configuring object storage in GitLab 13.1 and earlier, or for storage types not -supported by consolidated configuration form, refer to the following guides based -on what features you intend to use: - -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](../uploads.md#using-object-storage) | Yes | -| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | +The consolidated form is used in the following examples when available. + +NOTE: +When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). +The previous [background upload](../../development/uploads/index.md#direct-upload) mode, +which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional). Using separate buckets for each data type is the recommended approach for GitLab. This ensures there are no collisions across the various types of data GitLab stores. @@ -2230,29 +2218,13 @@ in the future. -## Enable incremental logging +### Enable incremental logging GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. -## Configure Advanced Search - -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) -for faster, more advanced code search across your entire GitLab instance. - -Elasticsearch cluster design and requirements are dependent on your specific -data. For recommended best practices about how to set up your Elasticsearch -cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). - -
- - Back to setup components - -
- -## Configure NFS +## Configure NFS (optional) [Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) are recommended over NFS wherever possible for improved performance. @@ -2260,14 +2232,24 @@ are recommended over NFS wherever possible for improved performance. See how to [configure NFS](../nfs.md). WARNING: -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. +Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation) +after the release of GitLab 15.6. No further enhancements are planned for this feature. Read: - [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). +## Configure Advanced Search + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). +
Back to setup components diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index b7ab753eb757843e8cfebba41a3a391f619bed63..423dbc7abfbceac599a856919a5ebc8d9094ac1a 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -188,11 +188,11 @@ To set up GitLab and its components to accommodate up to 25,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs) - to have shared disk storage service for certain GitLab operations (non - Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -2017,7 +2017,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs): +1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2181,9 +2181,9 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -Object storage is also recommended over [NFS](#configure-nfs) and in general -it's better in larger setups as object storage is typically much more performant, -reliable, and scalable. +It's recommended over [NFS](#configure-nfs-optional) and in general it's better +in larger setups as object storage is typically much more performant, reliable, +and scalable. GitLab has been tested on a number of object storage providers: @@ -2193,7 +2193,6 @@ GitLab has been tested on a number of object storage providers: - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) -- On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. There are two ways of specifying object storage configuration in GitLab: @@ -2203,25 +2202,13 @@ There are two ways of specifying object storage configuration in GitLab: - [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). -Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. - -For configuring object storage in GitLab 13.1 and earlier, or for storage types not -supported by consolidated configuration form, refer to the following guides based -on what features you intend to use: - -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](../uploads.md#using-object-storage) | Yes | -| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | +The consolidated form is used in the following examples when available. + +NOTE: +When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). +The previous [background upload](../../development/uploads/index.md#direct-upload) mode, +which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional). Using separate buckets for each data type is the recommended approach for GitLab. This ensures there are no collisions across the various types of data GitLab stores. @@ -2234,12 +2221,28 @@ in the future.
-## Enable incremental logging +### Enable incremental logging GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. +## Configure NFS (optional) + +[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) +are recommended over NFS wherever possible for improved performance. + +See how to [configure NFS](../nfs.md). + +WARNING: +Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation) +after the release of GitLab 15.6. No further enhancements are planned for this feature. + +Read: + +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). +- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). + ## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) @@ -2256,22 +2259,6 @@ cluster alongside your instance, read how to -## Configure NFS - -[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. - -See how to [configure NFS](../nfs.md). - -WARNING: -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. - -Read: - -- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). -- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). - ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) As an alternative approach, you can also run select components of GitLab as Cloud Native diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 265ff9b8882210ddfe30a4d80fa9c5c95fd4d0b0..99cc6d47f6a72ba3d5be4ed60ea4865d5bbfd80f 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -112,11 +112,11 @@ To set up GitLab and its components to accommodate up to 2,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, - more advanced code search across your entire GitLab instance. 1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) to have shared disk storage service as an alternative to Gitaly or object storage. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. ## Configure the external load balancer @@ -887,10 +887,10 @@ running [Prometheus](../monitoring/prometheus/index.md) and ## Configure the object storage -GitLab supports using an object storage service for holding several types of -data, and is recommended over [NFS](#configure-nfs-optional). In general, -object storage services are better for larger environments, as object storage -is typically much more performant, reliable, and scalable. +GitLab supports using an object storage service for holding numerous types of data. +It's recommended over [NFS](#configure-nfs-optional) and in general it's better +in larger setups as object storage is typically much more performant, reliable, +and scalable. GitLab has been tested on a number of object storage providers: @@ -900,7 +900,6 @@ GitLab has been tested on a number of object storage providers: - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) -- On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. There are two ways of specifying object storage configuration in GitLab: @@ -910,29 +909,13 @@ There are two ways of specifying object storage configuration in GitLab: - [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). -Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. - -GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. +The consolidated form is used in the following examples when available. -In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. - -For configuring object storage in GitLab 13.1 and earlier, or for storage types not -supported by consolidated configuration form, refer to the following guides based -on what features you intend to use: - -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](../uploads.md#using-object-storage) | Yes | -| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | +NOTE: +When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). +The previous [background upload](../../development/uploads/index.md#direct-upload) mode, +which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional). Using separate buckets for each data type is the recommended approach for GitLab. This ensures there are no collisions across the various types of data GitLab stores. @@ -945,21 +928,11 @@ in the future. -## Configure Advanced Search **(PREMIUM SELF)** +### Enable incremental logging -You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) -for faster, more advanced code search across your entire GitLab instance. +GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. -Elasticsearch cluster design and requirements are dependent on your specific -data. For recommended best practices about how to set up your Elasticsearch -cluster alongside your instance, read how to -[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). - -
- - Back to setup components - -
+While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. ## Configure NFS (optional) @@ -970,14 +943,30 @@ possible. See how to [configure NFS](../nfs.md). WARNING: -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. +Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation) +after the release of GitLab 15.6. No further enhancements are planned for this feature. Read: - [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). - About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). +## Configure Advanced Search **(PREMIUM SELF)** + +You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) +for faster, more advanced code search across your entire GitLab instance. + +Elasticsearch cluster design and requirements are dependent on your specific +data. For recommended best practices about how to set up your Elasticsearch +cluster alongside your instance, read how to +[choose the optimal cluster configuration](../../integration/advanced_search/elasticsearch.md#guidance-on-choosing-optimal-cluster-configuration). + +
+ + Back to setup components + +
+ ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) As an alternative approach, you can also run select components of GitLab as Cloud Native diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 4d90cac037bf849bea0b1ab61463e07c648fca17..5c227e3dc271aef71d56000393b6d89b8bd40b93 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -194,11 +194,11 @@ To set up GitLab and its components to accommodate up to 3,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, - more advanced code search across your entire GitLab instance. 1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) to have shared disk storage service as an alternative to Gitaly or object storage. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -2130,7 +2130,6 @@ GitLab has been tested on a number of object storage providers: - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) -- On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. There are two ways of specifying object storage configuration in GitLab: @@ -2140,25 +2139,13 @@ There are two ways of specifying object storage configuration in GitLab: - [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). -Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. - -For configuring object storage in GitLab 13.1 and earlier, or for storage types not -supported by consolidated configuration form, refer to the following guides based -on what features you intend to use: - -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](../uploads.md#using-object-storage) | Yes | -| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | +The consolidated form is used in the following examples when available. + +NOTE: +When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). +The previous [background upload](../../development/uploads/index.md#direct-upload) mode, +which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional). Using separate buckets for each data type is the recommended approach for GitLab. This ensures there are no collisions across the various types of data GitLab stores. @@ -2171,12 +2158,28 @@ in the future. -## Enable incremental logging +### Enable incremental logging GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. +## Configure NFS (optional) + +[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) +are recommended over NFS wherever possible for improved performance. + +See how to [configure NFS](../nfs.md). + +WARNING: +Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation) +after the release of GitLab 15.6. No further enhancements are planned for this feature. + +Read: + +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). +- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). + ## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) @@ -2193,22 +2196,6 @@ cluster alongside your instance, read how to -## Configure NFS (optional) - -[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. - -See how to [configure NFS](../nfs.md). - -WARNING: -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. - -Read: - -- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). -- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). - ## Supported modifications for lower user counts (HA) The 3k GitLab reference architecture is the smallest we recommend that achieves High Availability (HA). diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index ddf17b9a7af1c5719dd65ae8efabd978ebbbe074..bddec55ba716563636812fbd44bbe27ba29c74e2 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -188,10 +188,11 @@ To set up GitLab and its components to accommodate up to 50,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. 1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, more advanced code search across your entire GitLab instance. -1. [Configure NFS](#configure-nfs) - to have shared disk storage service for certain GitLab operations (non Gitaly or Object Storage). The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -2033,7 +2034,7 @@ On each node perform the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. If you're [using NFS](#configure-nfs): +1. If you're [using NFS](#configure-nfs-optional): 1. If necessary, install the NFS client utility packages using the following commands: @@ -2197,7 +2198,7 @@ To configure the Monitoring node: ## Configure the object storage GitLab supports using an object storage service for holding numerous types of data. -It's recommended over [NFS](#configure-nfs) and in general it's better +It's recommended over [NFS](#configure-nfs-optional) and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -2209,7 +2210,6 @@ GitLab has been tested on a number of object storage providers: - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) -- On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. There are two ways of specifying object storage configuration in GitLab: @@ -2219,25 +2219,13 @@ There are two ways of specifying object storage configuration in GitLab: - [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). -Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. - -For configuring object storage in GitLab 13.1 and earlier, or for storage types not -supported by consolidated configuration form, refer to the following guides based -on what features you intend to use: - -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](../uploads.md#using-object-storage) | Yes | -| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | +The consolidated form is used in the following examples when available. + +NOTE: +When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). +The previous [background upload](../../development/uploads/index.md#direct-upload) mode, +which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional). Using separate buckets for each data type is the recommended approach for GitLab. This ensures there are no collisions across the various types of data GitLab stores. @@ -2250,12 +2238,28 @@ in the future. -## Enable incremental logging +### Enable incremental logging GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. +## Configure NFS (optional) + +[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) +are recommended over NFS wherever possible for improved performance. + +See how to [configure NFS](../nfs.md). + +WARNING: +Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation) +after the release of GitLab 15.6. No further enhancements are planned for this feature. + +Read: + +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). +- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). + ## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) @@ -2272,22 +2276,6 @@ cluster alongside your instance, read how to -## Configure NFS - -[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. - -See how to [configure NFS](../nfs.md). - -WARNING: -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. - -Read: - -- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). -- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). - ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) As an alternative approach, you can also run select components of GitLab as Cloud Native diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index f79d019dc70bad8ab414c2022cef151dcfc8d094..0e599df7c1f4b683da6b75f8818df0f25c5a0657 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -191,12 +191,11 @@ To set up GitLab and its components to accommodate up to 5,000 users: environment. 1. [Configure the object storage](#configure-the-object-storage) used for shared data objects. -1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, - more advanced code search across your entire GitLab instance. 1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) to have shared disk storage service as an alternative to Gitaly or object - storage. You can skip this step if you're not using GitLab Pages (which - requires NFS). + storage. +1. [Configure Advanced Search](#configure-advanced-search) (optional) for faster, + more advanced code search across your entire GitLab instance. The servers start on the same 10.6.0.0/24 private network range, and can connect to each other freely on these addresses. @@ -2130,7 +2129,6 @@ GitLab has been tested on a number of object storage providers: - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift (S3 compatibility mode)](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) -- On-premises hardware and appliances from various storage vendors. - MinIO. We have [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation. There are two ways of specifying object storage configuration in GitLab: @@ -2140,25 +2138,13 @@ There are two ways of specifying object storage configuration in GitLab: - [Storage-specific form](../object_storage.md#storage-specific-configuration): Every object defines its own object storage [connection and configuration](../object_storage.md#connection-settings). -Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. - -For configuring object storage in GitLab 13.1 and earlier, or for storage types not -supported by consolidated configuration form, refer to the following guides based -on what features you intend to use: - -|Object storage type|Supported by consolidated configuration?| -|-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_gitlab.md#upload-backups-to-a-remote-cloud-storage) | No | -| [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | -| [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | -| [Uploads](../uploads.md#using-object-storage) | Yes | -| [Container Registry](../packages/container_registry.md#use-object-storage) (optional feature) | No | -| [Merge request diffs](../merge_request_diffs.md#using-object-storage) | Yes | -| [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | -| [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | -| [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | -| [Terraform state files](../terraform_state.md#using-object-storage) | Yes | +The consolidated form is used in the following examples when available. + +NOTE: +When using the [storage-specific form](../object_storage.md#storage-specific-configuration) +in GitLab 14.x and earlier, you should enable [direct upload mode](../../development/uploads/index.md#direct-upload). +The previous [background upload](../../development/uploads/index.md#direct-upload) mode, +which was deprecated in 14.9, requires shared storage such as [NFS](#configure-nfs-optional). Using separate buckets for each data type is the recommended approach for GitLab. This ensures there are no collisions across the various types of data GitLab stores. @@ -2171,12 +2157,28 @@ in the future. -## Enable incremental logging +### Enable incremental logging GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared through NFS on any GitLab Rails and Sidekiq nodes. While sharing the job logs through NFS is supported, it's recommended to avoid the need to use NFS by enabling [incremental logging](../job_logs.md#incremental-logging-architecture) (required when no NFS node has been deployed). Incremental logging uses Redis instead of disk space for temporary caching of job logs. +## Configure NFS (optional) + +[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) +are recommended over NFS wherever possible for improved performance. + +See how to [configure NFS](../nfs.md). + +WARNING: +Engineering support for NFS for Git repositories is deprecated, and [technical support is scheduled to be unavailable](../nfs.md#gitaly-and-nfs-deprecation) +after the release of GitLab 15.6. No further enhancements are planned for this feature. + +Read: + +- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). +- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). + ## Configure Advanced Search You can leverage Elasticsearch and [enable Advanced Search](../../integration/advanced_search/elasticsearch.md) @@ -2193,22 +2195,6 @@ cluster alongside your instance, read how to -## Configure NFS (optional) - -[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly) -are recommended over NFS wherever possible for improved performance. - -See how to [configure NFS](../nfs.md). - -WARNING: -Engineering support for NFS for Git repositories is deprecated. Technical support is planned to be -unavailable from GitLab 15.0. No further enhancements are planned for this feature. - -Read: - -- [Gitaly and NFS Deprecation](../nfs.md#gitaly-and-nfs-deprecation). -- About the [correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). - ## Cloud Native Hybrid reference architecture with Helm Charts (alternative) As an alternative approach, you can also run select components of GitLab as Cloud Native