From 9a1f301c7ad3ea1c2f7ab671b220369481484732 Mon Sep 17 00:00:00 2001 From: Clemens Beck Date: Tue, 9 Sep 2025 11:38:45 +0200 Subject: [PATCH 1/3] NGINX proxy request buffering documentation Collect current configuration and known issues, performance impact related to NGINX proxy request buffering on a component level. Relates https://gitlab.com/gitlab-org/charts/gitlab/-/issues/6034 --- doc/development/nginx/_index.md | 9 ++++++ doc/development/{nginx.md => nginx/fork.md} | 2 +- .../nginx/proxy_request_buffering.md | 32 +++++++++++++++++++ 3 files changed, 42 insertions(+), 1 deletion(-) create mode 100644 doc/development/nginx/_index.md rename doc/development/{nginx.md => nginx/fork.md} (99%) create mode 100644 doc/development/nginx/proxy_request_buffering.md diff --git a/doc/development/nginx/_index.md b/doc/development/nginx/_index.md new file mode 100644 index 0000000000..f98cc1475d --- /dev/null +++ b/doc/development/nginx/_index.md @@ -0,0 +1,9 @@ +--- +stage: GitLab Delivery +group: Operate +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +title: NGINX chart developer documentation +--- + +- Developing with our [NGINX fork](fork.md) +- [Proxy Request Buffering Evaluation](proxy_request_buffering.md) diff --git a/doc/development/nginx.md b/doc/development/nginx/fork.md similarity index 99% rename from doc/development/nginx.md rename to doc/development/nginx/fork.md index fe7da72321..e0aede7c00 100644 --- a/doc/development/nginx.md +++ b/doc/development/nginx/fork.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w title: Working with the bundled NGINX --- -## NGINX +## NGINX Fork We provide a fork of NGINX with this chart that we maintain via patch files. This approach was choosen to easier upgrade NGINX chart versions and have diff --git a/doc/development/nginx/proxy_request_buffering.md b/doc/development/nginx/proxy_request_buffering.md new file mode 100644 index 0000000000..a7f5e2c927 --- /dev/null +++ b/doc/development/nginx/proxy_request_buffering.md @@ -0,0 +1,32 @@ +--- +stage: GitLab Delivery +group: Operate +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +title: Working with the bundled NGINX +--- + +## NGINX Proxy Request Buffering + +[NGINXs `proxy_request_buffering`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering) +allows to control the buffering behavior of incoming clients. + +The proxy request buffering is enabled by default, which means NGINX will buffer all incoming +requests and will only forward them to the backend after the full request was received. + +If disabled, the request body is sent to the proxied server immediately as it is received. + +GitLab chart by default uses [NGINX Ingress](https://github.com/kubernetes/ingress-nginx) to +handle incoming traffic. By setting the `nginx.ingress.kubernetes.io/proxy-request-buffering` +annotation on a Ingress the NGINX `proxy_request_buffering` setting can be controlled. +This always impacts the whole Ingress and allows no filtering on a path level. + +| Ingress / traffic type | Status in Omnibus | Status in charts | Owning Team | Target value | Comment | +|-----------------------------|-------------------|----------------------|------------------------------|--------------------------------------------|---------| +| Workhorse (Artifact API) | `off` (filtered) | not set (`on`) | Shared | `off` expected to be more performant (large chunked artifact uploads) | Filtered in Omnibus since [MR 4516](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4516). | +| Workhorse (Project Imports) | `off` (filtered) | not set (`on`) | Shared | `off` expected to be more performant (large chunked project uploads) | Setting `off` is expected to [improve performance](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6841). | +| Workhorse (Other Traffic) | not set (`on`) | not set (`on`) | Shared | `on` expected to be more performant (many small requests) | | +| SSH over HTTP(S) | `off` (filtered) | not set (`on`) | Shared | `off` functional requirement + improved disk performance | Known issues [1](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2262 ) [2](https://gitlab.com/gitlab-org/gitlab/-/issues/454707), [3](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/31871) if proxy request buffering is off. | +| KAS | `on` | not set (`on`) | ~"group::environments" | `on` expected to be more performant (many small requests) | [Omnibus configuration](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/04f29efeec94171e60a63eb06690a41f71e89ab1/files/gitlab-cookbooks/gitlab/templates/default/nginx-gitlab-kas-http.conf.erb#L116) | +| GitLab Pages | not set (`on`) | not set (`on`) | ~"group::knowledge" | `on` expected to be more performant (many small requests) | [First assesment](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/4512#note_2747102518) | +| Container Registry | `off` | `off` (configurable) | ~"group::container registry" | `off` functional requirement | [Known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2848) if `proxy_request_buffering` is on. | +| MinIO | - | not set (`on`) | ~group::operate | unknown | Only bundled for PoC/demo environments. | -- GitLab From ab5f9af75947b294594c65dd2f37dd6738afea30 Mon Sep 17 00:00:00 2001 From: Clemens Beck Date: Mon, 22 Sep 2025 11:59:29 +0200 Subject: [PATCH 2/3] Document client_body_buffer_size and it's impact --- .../nginx/proxy_request_buffering.md | 28 ++++++++++++++++--- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/doc/development/nginx/proxy_request_buffering.md b/doc/development/nginx/proxy_request_buffering.md index a7f5e2c927..f1adb0a3ab 100644 --- a/doc/development/nginx/proxy_request_buffering.md +++ b/doc/development/nginx/proxy_request_buffering.md @@ -8,12 +8,29 @@ title: Working with the bundled NGINX ## NGINX Proxy Request Buffering [NGINXs `proxy_request_buffering`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering) -allows to control the buffering behavior of incoming clients. +controls how incoming incoming client requests are handled before forwarding to +upstream server. -The proxy request buffering is enabled by default, which means NGINX will buffer all incoming -requests and will only forward them to the backend after the full request was received. +### Default Behavior (`on`) -If disabled, the request body is sent to the proxied server immediately as it is received. +By default, NGINX buffers the entire request body before forwarding to the backend. +Requests larger than [`client_body_buffer_size`](https://nginx.org/en/docs/http/ngx_http_core_module.html#client_body_buffer_size) +are written to temporary files, and once the full request was received, the request +will be forwarded as a whole to the backend. + +This can help to reduce the amount of concurrent connections the backend has to keep +open and can protect the backend from slow clients. + +### When disabled (`off`) + +If `proxy_request_buffering` is disabled and a incoming requests exceeds the +`client_body_buffer_size` the request chunk will be immediately forwarded to +the backend. + +This can help to optimize (large) streaming requests and reduces disk I/O +of NGINX. + +### GitLab chart configuration GitLab chart by default uses [NGINX Ingress](https://github.com/kubernetes/ingress-nginx) to handle incoming traffic. By setting the `nginx.ingress.kubernetes.io/proxy-request-buffering` @@ -30,3 +47,6 @@ This always impacts the whole Ingress and allows no filtering on a path level. | GitLab Pages | not set (`on`) | not set (`on`) | ~"group::knowledge" | `on` expected to be more performant (many small requests) | [First assesment](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/4512#note_2747102518) | | Container Registry | `off` | `off` (configurable) | ~"group::container registry" | `off` functional requirement | [Known issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2848) if `proxy_request_buffering` is on. | | MinIO | - | not set (`on`) | ~group::operate | unknown | Only bundled for PoC/demo environments. | + +Note: [MR 4523](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/4523) explores how to +configure `proxy_request_buffering` for some workhorse paths to match our Omnibus configuration. -- GitLab From 6be8f948934ed2f5aac4796efef700623a45edcf Mon Sep 17 00:00:00 2001 From: Clemens Beck Date: Mon, 22 Sep 2025 12:54:02 +0200 Subject: [PATCH 3/3] Apply 1 suggestion(s) to 1 file(s) --- doc/development/nginx/proxy_request_buffering.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/development/nginx/proxy_request_buffering.md b/doc/development/nginx/proxy_request_buffering.md index f1adb0a3ab..93c83e65e6 100644 --- a/doc/development/nginx/proxy_request_buffering.md +++ b/doc/development/nginx/proxy_request_buffering.md @@ -8,7 +8,7 @@ title: Working with the bundled NGINX ## NGINX Proxy Request Buffering [NGINXs `proxy_request_buffering`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_request_buffering) -controls how incoming incoming client requests are handled before forwarding to +controls how incoming client requests are handled before forwarding to the upstream server. ### Default Behavior (`on`) -- GitLab