Git LFS over HTTPS timeout and git clone hang because HTTP authorization header is too long
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
After upgrading GitLab from 17.2.1 to 17.2.2, we encountered critical issues that rendered all our gitlab repositories using LFS (Large File Storage) unusable. This situation has severely impacted our ability to push or pull binary files to our gitlab instance.
This is probably a side-effect of the Add a project scope to LfsTokens changes linked to CVE-2024-3035 (A permission check vulnerability in GitLab CE/EE allowed for LFS tokens to read and write to the user owned repositories) and Privilege Escalation via LFS Tokens Granting Unrestricted Repository Access released on 2024/08/08.
We are experiencing a problem related to the excessively long HTTP header "Authorization" used for LFS objects. This header is required for basic access authentication when retrieving LFS objects from /gitlab-lfs/objects. The issue occurs due to the token present in the JSON response ("header":{"Authorization": "Basic ...") that GitLab returns when the git-lfs client makes a call to /info/lfs/objects/batch. The length of this header is causing significant disruptions in our ability to interact with LFS objects effectively.
> POST /user/test-lfs.git/info/lfs/objects/batch HTTP/1.1
> Host: gitlab.example.com
> Accept: application/vnd.git-lfs+json
> Authorization: Basic * * * * *
> Content-Length: 404
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.5.1 (GitHub; linux amd64; go 1.21.8)
>
{"operation":"download","objects":[{"oid":" ...
...
< HTTP/2.0 200 OK
< Content-Length: 2095
< Content-Type: application/vnd.git-lfs+json; charset=utf-8
< Page-Title: GitLab
< ...
08:51:34.596336 trace git-lfs: HTTP: {"objects":[{"oid":".../user/test-lfs.git/gitlab-lfs/objects/956dce4626b41cfcbc5a50ddeb56407ab99c4f217c52cfcd6c3e3549f8603e02","header":{"Authorization":"Basic XXXXXXXXXXXXXXXXXXXXXXX...}}}}]}
...
> GET /user/test-lfs.git/gitlab-lfs/objects/beb888cf483455dfabb9f2df2ea8fe7d1133fcb107f0a5499055bf6075d04728 HTTP/1.1
> Host: gitlab.example.com
08:51:34.695730 trace git-lfs: xfer: adapter "basic" worker 7 auth signal received
> Authorization: Basic * * * * *
> User-Agent: git-lfs/3.5.1 (GitHub; linux amd64; go 1.21.8)
<BLOCKED BY FORTINET IPS AND TIMEOUT + MULTIPLE RETRIES>We implemented a temporary workaround by upgrading the git-lfs client to 3.5.1, enabling gitlab_shell['lfs_pure_ssh_protocol'] = true, and requiring all users to switch to git over SSH. This was necessary because git over HTTPS was freezing for all GitLab repositories using LFS. Unfortunately, this solution did not work in our GitLab CI/CD pipelines and inside Gitlab Runner (gitlab-runner/gitlab-runner-helper image), which default to using git over HTTPS and cannot easily switch to git over SSH.
The root cause is linked to our Intrusion Prevention System (Fortinet IPS), which blocks Git LFS requests due to a too long HTTP authorization value. This results in an alert categorized as HTTP.Server.Authorization.Buffer.Overflow, which is documented in Fortinet's VID12351. The IPS logs confirm that the excessively long authorization header leads to dropped requests.
Aug 26 10:17:55 fortigate date=2024-08-26 time=09:17:55 type="utm" subtype="ips" eventtype="signature" level="alert" vd="root" severity="critical" action="dropped" proto=6 service="HTTPS" policyid=436 policytype="policy" attack="HTTP.Server.Authorization.Buffer.Overflow" srcport=40788 dstport=443 hostname="gitlab.example.com" url="/user/test-lfs.git/gitlab-lfs/objects/42aa061c50b9ba68dbabbc95d33eea2b72e76fa0280518488b6d5a05ba4b8f38" direction="outgoing" attackid=12351 profile="default" ref="http://www.fortinet.com/ids/VID12351" msg="web_server: HTTP.Server.Authorization.Buffer.Overflow" crscore=50 craction=4096 crlevel="critical"After a long process of debugging git-lfs client, we discovered that the root cause is the Intrusion Prevention System (Fortinet IPS), which blocks git LFS requests due to a too long HTTP authorization header with the long token returned by GitLab server after version 17.2.2 for LFS objects. This issue arises when the authentication token for LFS blobs exceeds the HTTP Basic Authorization buffer limit, triggering a critical alert categorized as HTTP.Server.Authorization.Buffer.Overflow. For further details, please refer to Fortinet's documentation VID12351.
export GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1
git clone https://gitlab.example.com/user/test-lfs.git
GET /user/test-lfs.git/gitlab-lfs/objects/956dce4626b41cfcbc5a50ddeb56407ab99c4f217c52cfcd6c3e3549f8603e02 HTTP/1.1
Host: gitlab.example.com
Authorization: Basic YmXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXZ3eHl6
User-Agent: git-lfs/3.5.1 (GitHub; linux amd64; go 1.21.8)To address the issues with Git LFS, I recommend the following changes in GitLab:
(1) Providing a shorter authorization token for LFS objects as it was not blocked by our IPS before we upgraded GitLab to version 17.2.2.
(2) Introduce a specific config setting in gitlab.rb to toggle between:
- reusing the initial Git authentication credential for both non-LFS and LFS files (possibly utilizing credential.helper cache) if LFS objects are hosted on the same URL as gitlab. (As a PoC, we compiled a special git-lfs client that ignores the token returned by the GitLab LFS batch endpoint and reuses the one for non-LFS, and it worked without any blocking of our IPS)
- using a different mean of authentication by returning a JSON with header Authorization from /lfs/objects/batch but make the token about the same length as the one used by gitlab and not three lines as is the case today.
These adjustments would help mitigate the issues caused by the excessive length of the Basic Authorization token and improve the overall functionality of Git LFS in GitLab.
For more details on this problem and comments from the git-lfs team, see also this issue on GitHub Git LFS over HTTPS Blocked by Fortinet IPS due to Basic Authorization Overflow.
Thank you in advance for fixing this regression and improving LFS support in GitLab.