Export gets stuck for large projects

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

In our self-hosted Gitlab instance, we are seeing export getting stuck if the projects are large (~6GB). No error was thrown in logs or UI.

Steps to reproduce

1. find a large project and export either through UI or API

2. check after a few hours and observe the following:

   With API:
   “export_status”:“regeneration_in_progress”}

   In Browser:
   “The file containing the export is not available yet; it may still be transferring. Please try again later.”

3. Receive the download link after 24 hours but get The file containing the export is not available yet; it may still be transferring. Please try again later.

Example Project

What is the current bug behavior?

Can't export large projects

What is the expected correct behavior?

Have a downloadable link for the exported project

Relevant logs and/or screenshots

Currently there is no log indicating there was an error with the background export job. I observe Export archive saved but not Export archive uploaded in the sidekiq log. If I do the steps outlined here manually, I get the following object:

#<Gitlab::ImportExport::Saver:0x00007f7207ce55d8
@archive_file=
"/srv/gitlab/shared/tmp/gitlab_exports/@hashed/e0/5c/e05c358682819538ec657faa9edb9d5f32d5a88ad572276e914ae7dd1fcdc3d3/942d8e19bccee6508865d18dac963754/2023-12-27_15-40-574_<PROJECT_NAME>_export.tar.gz",
@assign_duration_s=0.0018187668174505234,
@compress_duration_s=229.82848361111246,
@exportable=#<Project id:15184 PORJECT_NAME>>,
@shared=
#<Gitlab::ImportExport::Shared:0x00007f7214ee58d0
@archive_path=
"/srv/gitlab/shared/tmp/gitlab_exports/@hashed/e0/5c/e05c358682819538ec657faa9edb9d5f32d5a88ad572276e914ae7dd1fcdc3d3/942d8e19bccee6508865d18dac963754",
@errors=[],
@export_path=
"/srv/gitlab/shared/tmp/gitlab_exports/@hashed/e0/5c/e05c358682819538ec657faa9edb9d5f32d5a88ad572276e914ae7dd1fcdc3d3/942d8e19bccee6508865d18dac963754/9b79ac477503b0d328e6d3450c51e431",
@exportable=#<Project id:15184 PORJECT_NAME>>,
@logger=
#<Gitlab::Import::Logger:0x00007f7214f17a10
@default_formatter=#<Logger::Formatter:0x00007f7214f17970 @datetime_format=nil>,
@formatter=nil,
@level=2,
@logdev=
#<Logger::LogDevice:0x00007f7214f178f8
@binmode=false,
@dev=#<File:/srv/gitlab/log/importer.log>,
@filename=#<Pathname:/srv/gitlab/log/importer.log>,
@mon_data=#<Monitor:0x00007f7214f178d0>,
@mon_data_owner_object_id=15773000,
@shift_age=0,
@shift_period_suffix="%Y%m%d",
@shift_size=1048576>,
@progname=nil>,
@relative_archive_path=
"@hashed/e0/5c/e05c358682819538ec657faa9edb9d5f32d5a88ad572276e914ae7dd1fcdc3d3/942d8e19bccee6508865d18dac963754",
@relative_path=
"@hashed/e0/5c/e05c358682819538ec657faa9edb9d5f32d5a88ad572276e914ae7dd1fcdc3d3/942d8e19bccee6508865d18dac963754/9b79ac477503b0d328e6d3450c51e431">,
@upload_bytes=6786118782,
@upload_duration_s=153.9677494589705>

The tar.gz file doesn't exist in the above path. However, I do see the following in the export_path:

ls -al /srv/gitlab/shared/tmp/gitlab_exports/@hashed/e0/5c/e05c358682819538ec657faa9edb9d5f32d5a88ad572276e914ae7dd1fcdc3d3/942d8e19bccee6508865d18dac963754/9b79ac477503b0d328e6d3450c51e431
total 6667356
drwx------ 5 git git       4096 Dec 27 15:39 .
drwx------ 3 git git       4096 Dec 27 15:44 ..
-rw-r----- 1 git git          0 Dec 27 15:33 GITLAB_REVISION
-rw-r----- 1 git git          6 Dec 27 15:33 GITLAB_VERSION
-rw-r----- 1 git git          5 Dec 27 15:33 VERSION
-rw-r----- 1 git git          2 Dec 27 15:39 lfs-objects.json
-rw-r----- 1 git git 6827287795 Dec 27 15:35 project.bundle
-rw-r----- 1 git git      45002 Dec 27 15:39 project.wiki.bundle
drwx------ 2 git git       4096 Dec 27 15:39 snippets
drwx------ 3 git git       4096 Dec 27 15:37 tree
drwx------ 5 git git       4096 Dec 27 15:39 uploads

I'm suspecting s.send(:save_upload) had some faulty behavior and it's not logged correctly.

Symptom looks the same as https://forum.gitlab.com/t/project-export-never-completes/58809

Output of checks

Results of GitLab environment info

Self hosted Gitlab v16.5.3 on k8s

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of: \\\`sudo gitlab-rake gitlab:check SANITIZE=true\\\`) (For installations from source run and paste the output of: \\\`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true\\\`) (we will only investigate if the tests are passing)

Possible fixes