Explore performance improvements of branches and diff_files actions for Projects::CommitController with extreme monorepos
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
As part of continuing performance test efforts Self Managed Platform have created a new prototype pipeline against the Chromium repository, a high percentile large monorepo, to identify "hairline" performance issues that only appear with significant pressure.
Prototype Chromium performance pipeline details
The prototype Chromium performance pipeline has been set up as follows:
- Testing against a 1000 RPS / 50k users Reference Architecture to remove hardware bottlenecks from the equation as much as possible.
- As part of the setup to ensure the results are accurate, additional runs against Gitaly being deployed on substantially larger hardware specs were also done to confirm that hardware specifically isn't a notable factor here.
- The Chromium repository has over 5 million Git References (branches, tags, commits):
Chromium git-sizer output
Processing blobs: 17632726
Processing trees: 30980353
Processing commits: 23538146
Matching commits to trees: 23538146
Processing annotated tags: 0
Processing references: 5611873
| Name | Value | Level of concern |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size | | |
| * Commits | | |
| * Count | 23.5 M | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Total size | 15.3 GiB | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Trees | | |
| * Count | 31.0 M | ******************** |
| * Total size | 110 GiB | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Total tree entries | 2.57 G | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Blobs | | |
| * Count | 17.6 M | *********** |
| * Total size | 1.63 TiB | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Annotated tags | | |
| * Count | 0 | |
| * References | | |
| * Count | 5.61 M | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| | | |
| Biggest objects | | |
| * Commits | | |
| * Maximum size [1] | 20.0 MiB | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| * Maximum parents [2] | 3 | |
| * Trees | | |
| * Maximum entries [3] | 4.39 k | **** |
| * Blobs | | |
| * Maximum size [4] | 731 MiB | !!!!!!!!!!!!!!!!!!!!!!!!!!!!!! |
| | | |
| History structure | | |
| * Maximum history depth | 1.16 M | ** |
| * Maximum tag depth | 0 | |
| | | |
| Biggest checkouts | | |
| * Number of directories [5] | 36.8 k | ****************** |
| * Maximum path depth [6] | 20 | ** |
| * Maximum path length [7] | 284 B | ** |
| * Number of files [8] | 447 k | ******** |
| * Total size of files [9] | 5.25 GiB | ***** |
| * Number of symlinks [10] | 363 | |
| * Number of submodules [11] | 240 | ** |The Commit details webpage, which calls several actions of the Projects::CommitController looks to struggle notably at this scale against this class of repository with it taking several seconds to return:
█ Results summary
* Environment: 50k_chromium_experimental
* Environment Version: 17.0.0-pre `e783f8adf59`
* Option: 60s_1000rps
* Date: 2024-04-26
* Run Time: 2m 25.58s (Start: 13:51:41 UTC, End: 13:54:07 UTC)
* GPT Version: v2.15.0
❯ Overall Results Score: 56.86%
NAME | RPS | RPS RESULT | TTFB AVG | TTFB P90 | DATA RECEIVED | REQ STATUS | RESULT
------------------------------------------|--------|----------------------|-----------|----------------------|-------------------|----------------|------------------
web_project_commit | 100/s | 14.77/s (>32.00/s) | 6785.54ms | 21858.54ms (<3500ms) | 297 MB (3.7 MB/s) | 100.00% (>99%) | FAILED¹²
█ Web - Commit Details Page
data_received.....................................................: 297 MB 3.7 MB/s
data_sent.........................................................: 1.9 MB 23 kB/s
group_duration....................................................: avg=22234.18ms min=20689.10ms med=21697.32ms max=29812.64ms p(90)=24312.61ms p(95)=26222.86ms
http_req_blocked..................................................: avg=0.21ms min=0.00ms med=0.00ms max=18.60ms p(90)=0.00ms p(95)=2.27ms
http_req_connecting...............................................: avg=0.06ms min=0.00ms med=0.00ms max=2.84ms p(90)=0.00ms p(95)=0.61ms
http_req_duration.................................................: avg=6788.65ms min=81.63ms med=3507.09ms max=29681.78ms p(90)=21858.61ms p(95)=22563.08ms
{ expected_response:true }......................................: avg=6788.65ms min=81.63ms med=3507.09ms max=29681.78ms p(90)=21858.61ms p(95)=22563.08ms
http_req_failed...................................................: 0.00% ✓ 0 ✗ 1193
http_req_receiving................................................: avg=3.07ms min=0.02ms med=0.44ms max=115.16ms p(90)=4.88ms p(95)=17.36ms
http_req_sending..................................................: avg=0.04ms min=0.02ms med=0.03ms max=0.14ms p(90)=0.07ms p(95)=0.08ms
http_req_tls_handshaking..........................................: avg=0.13ms min=0.00ms med=0.00ms max=1.84ms p(90)=0.00ms p(95)=1.56ms
http_req_waiting..................................................: avg=6785.54ms min=81.57ms med=3503.56ms max=29681.68ms p(90)=21858.54ms p(95)=22563.01ms
✗ { controller:Projects::CommitController#branches }..............: avg=22201.88ms min=20641.44ms med=21672.53ms max=29681.68ms p(90)=24301.72ms p(95)=26185.27ms
✗ { controller:Projects::CommitController#diff_files }............: avg=3963.76ms min=3503.56ms med=3883.74ms max=5886.52ms p(90)=4134.96ms p(95)=4905.72ms
✓ { controller:Projects::CommitController#merge_requests.json }...: avg=139.72ms min=81.57ms med=125.58ms max=597.28ms p(90)=207.77ms p(95)=219.96ms
✓ { controller:Projects::CommitController#show }..................: avg=854.86ms min=368.09ms med=979.36ms max=3565.60ms p(90)=1122.33ms p(95)=1160.61ms
✗ http_reqs.........................................................: 1193 14.771165/s
✗ { controller:Projects::CommitController#branches }..............: 298 3.689696/s
✗ { controller:Projects::CommitController#diff_files }............: 298 3.689696/s
✗ { controller:Projects::CommitController#merge_requests.json }...: 298 3.689696/s
✗ { controller:Projects::CommitController#show }..................: 298 3.689696/s
iteration_duration................................................: avg=22164.63ms min=1429.26ms med=21694.08ms max=29812.66ms p(90)=24308.24ms p(95)=26219.61ms
iterations........................................................: 298 3.689696/s
✓ successful_requests...............................................: 100.00% ✓ 1192 ✗ 0
vus...............................................................: 2 min=0 max=100
vus_max...........................................................: 100 min=100 max=100Of note specifically the #branches endpoint specifically looks to struggle the most here. But additionally the #show and #diff_files endpoints are also impacted but they already have issues open for more normal sized datasets so this issue is mainly for #branches.
The endpoint appears to be hitting Gitaly notably but not to exhaustion:
As this is a new pipeline and much more greatly dependent on context we aim to select a severity that takes into account likelihood, impact, and more. In this case since it's for a extreme size repository we'll go with a severity3.
Edited by 🤖 GitLab Bot 🤖