Export/Import API unreliable

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

Using the GitLab EE export/import API yields inconsistent results.

Steps to reproduce

On the source GitLab (at the time of issue, running 13.6.2-ee):

1. Run housekeeping on the project
2. Using the API, create an export request.
3. Wait for the export request to finish.
4. Download the .tar.gz file the export request creates.
5. Archive the source project

Immediately on the destination GitLab.com (at the time of issue, running 13.8.):

1. Use the import API to POST an import request
2. Wait for the import request to be accepted and return a project ID
3. Monitor the import progress using the API and project ID
4. Wait for the API to report a successful import.
5. When project is imported, compare the source and destination projects for inconsistencies.

Example Project

Unfortunately, the products we're importing have proprietary IP and I'm unable to import them into a public project. I can give examples of projects in our private GitLab group through Zendesk.

What is the current bug behavior?

We have inconsistent results when importing projects into GitLab.com. Details are in ZenDesk ticket 186356.

Examples of issues we're seeing:

• Inconsistent MRs: ** MRs missing or changing states (e.g., closed at source, opened after import) ** MR branch refs missing after import
• Inconsistent branches: ** More branches exist after import that don't exist in the source ** Inconsistent project tags.

Here's an example validation - not all things are mismatched all the time:

ERROR: Merge requests in all state: Source: 219, Destination: 215
ERROR: Merge requests in merged state: Source: 189, Destination: 185
Done comparing merge requests: example_project
Comparing branches: example_project
Mismatched branches for: example_project
Source branches: 42, Destination Branches: 53
Only in destination: {'add-user-name-sort', 'dependancies', 'squash_link_in_environment', 'CORE-425', 'ua-vars-1', 'display_git_sha', 'fix-dps-merchant-number', 'fix-ua-creds', 'patch-1', 'update_squash_url', 'fix-docs'}
Only in source: set()
Done comparing branches: example_project
Comparing tags: example_project
Done comparing tags: example_project
VALIDATE_IMPORT: Elapsed time 34520 ms

What is the expected correct behavior?

The exported and imported projects should be identical for the objects included in the export.

Relevant logs and/or screenshots

Logs and other sensitive information is in ZenDesk issue https://gitlab.zendesk.com/agent/tickets/186356 (internal).

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

This is the information for the source GitLab. GitLab.com is the destination GitLab.

Expand for output related to GitLab environment info

System information
System:		Ubuntu 16.04
Proxy:		no_proxy: 127.0.0.1,localhost,169.254.169.254,company-proxy-a,company-proxy-b,0.0.0.0
		https_proxy: http://proxy:8080
		http_proxy: http://proxy:8080
Current User:	git
Using RVM:	no
Ruby Version:	2.7.2p137
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.1
Redis Version:	5.0.9
Git Version:	2.29.0
Sidekiq Version:5.2.9
Go Version:	unknown
GitLab information
Version:	13.7.2-ee
Revision:	55a11a0db46
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.8
URL:		https://git.example.com
HTTP Clone URL:	https://git.example.com/some-group/some-project.git
SSH Clone URL:	git@git.example.com:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: google_oauth2
GitLab Shell
Version:	13.14.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

This is the information for the source GitLab. GitLab.com is the destination GitLab.

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.14.0 ? ... OK (13.14.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
47/6 ... yes
53/8 ... yes
47/9 ... yes
47/10 ... yes
49/12 ... yes
49/13 ... yes
49/15 ... yes
47/18 ... yes
52/19 ... yes
52/20 ... yes
47/21 ... yes
53/28 ... yes
14/30 ... yes
49/31 ... yes
29/33 ... yes
29/35 ... yes
50/36 ... yes
47/39 ... yes
50/41 ... yes
29/42 ... yes
2/44 ... yes
52/45 ... yes
6/46 ... yes
49/47 ... yes
29/49 ... yes
29/50 ... yes
29/51 ... yes
29/52 ... yes
29/53 ... yes
50/56 ... yes
29/57 ... yes
49/59 ... yes
24/60 ... yes
24/61 ... yes
29/62 ... yes
24/63 ... yes
58/64 ... yes
57/65 ... yes
57/66 ... yes
55/67 ... yes
58/68 ... yes
36/69 ... yes
55/71 ... yes
24/72 ... yes
12/73 ... yes
60/74 ... yes
12/75 ... yes
49/76 ... yes
49/77 ... yes
24/78 ... yes
59/79 ... yes
64/80 ... yes
60/81 ... yes
12/82 ... yes
60/83 ... yes
59/84 ... yes
60/85 ... yes
68/86 ... yes
60/87 ... yes
6/88 ... yes
24/89 ... yes
36/91 ... yes
29/92 ... yes
49/93 ... yes
65/94 ... yes
68/96 ... yes
49/97 ... yes
36/99 ... yes
24/100 ... yes
39/101 ... yes
42/103 ... yes
42/105 ... yes
42/107 ... yes
6/109 ... yes
75/112 ... yes
68/113 ... yes
75/114 ... yes
60/115 ... yes
49/116 ... yes
55/120 ... yes
75/121 ... yes
29/124 ... yes
49/125 ... yes
68/126 ... yes
68/127 ... yes
60/128 ... yes
68/129 ... yes
60/130 ... yes
60/131 ... yes
24/132 ... yes
80/135 ... yes
60/136 ... yes
57/137 ... yes
75/138 ... yes
80/139 ... yes
80/140 ... yes
78/141 ... yes
49/142 ... yes
80/143 ... yes
80/145 ... yes
80/148 ... yes
49/153 ... yes
49/154 ... yes
49/157 ... yes
49/158 ... yes
60/160 ... yes
49/161 ... yes
24/162 ... yes
57/163 ... yes
84/164 ... yes
85/165 ... yes
84/166 ... yes
85/167 ... yes
85/168 ... yes
84/169 ... yes
83/170 ... yes
83/171 ... yes
15/172 ... yes
53/173 ... yes
84/174 ... yes
85/175 ... yes
85/176 ... yes
83/177 ... yes
68/178 ... yes
83/179 ... yes
87/180 ... yes
24/181 ... yes
85/182 ... yes
83/183 ... yes
87/185 ... yes
83/188 ... yes
85/189 ... yes
72/190 ... yes
12/192 ... yes
24/193 ... yes
85/194 ... yes
64/195 ... yes
85/196 ... yes
92/197 ... yes
85/198 ... yes
87/199 ... yes
85/200 ... yes
89/201 ... yes
41/203 ... yes
80/205 ... yes
24/206 ... yes
60/209 ... yes
83/210 ... yes
59/211 ... yes
95/212 ... yes
95/213 ... yes
95/214 ... yes
95/215 ... yes
95/216 ... yes
95/217 ... yes
85/218 ... yes
24/219 ... yes
96/220 ... yes
29/221 ... yes
97/222 ... yes
95/223 ... yes
97/224 ... yes
95/225 ... yes
75/230 ... yes
97/231 ... yes
30/232 ... yes
87/233 ... yes
80/234 ... yes
97/235 ... yes
100/236 ... yes
95/238 ... yes
53/239 ... yes
39/240 ... yes
97/241 ... yes
84/242 ... yes
69/243 ... yes
98/244 ... yes
60/246 ... yes
87/247 ... yes
52/249 ... yes
42/250 ... yes
80/251 ... yes
73/252 ... yes
7/253 ... yes
97/254 ... yes
24/256 ... yes
24/257 ... yes
24/258 ... yes
80/260 ... yes
69/261 ... yes
69/263 ... yes
96/264 ... yes
36/265 ... yes
53/267 ... yes
53/268 ... yes
7/269 ... yes
49/270 ... yes
83/271 ... yes
49/272 ... yes
36/273 ... yes
99/274 ... yes
72/275 ... yes
49/276 ... yes
49/277 ... yes
72/278 ... yes
42/279 ... yes
99/285 ... yes
49/288 ... yes
57/290 ... yes
113/291 ... yes
49/294 ... yes
117/295 ... yes
88/297 ... yes
29/298 ... yes
74/299 ... yes
15/300 ... yes
47/301 ... yes
83/302 ... yes
83/304 ... yes
58/307 ... yes
78/308 ... yes
135/311 ... yes
4/312 ... yes
114/313 ... yes
135/314 ... yes
123/315 ... yes
135/316 ... yes
58/320 ... yes
96/321 ... yes
29/322 ... yes
64/323 ... yes
114/324 ... yes
114/325 ... yes
114/326 ... yes
114/328 ... yes
114/329 ... yes
135/332 ... yes
83/333 ... yes
114/334 ... yes
49/335 ... yes
89/337 ... yes
145/342 ... yes
145/343 ... yes
145/345 ... yes
145/347 ... yes
134/349 ... yes
135/350 ... yes
74/351 ... yes
99/356 ... yes
42/360 ... yes
145/361 ... yes
146/363 ... yes
154/365 ... yes
83/366 ... yes
83/367 ... yes
86/368 ... yes
53/369 ... yes
114/372 ... yes
53/374 ... yes
41/376 ... yes
123/377 ... yes
49/378 ... yes
157/383 ... yes
159/384 ... yes
160/385 ... yes
154/386 ... yes
40/387 ... yes
134/388 ... yes
154/389 ... yes
53/390 ... yes
106/391 ... yes
41/392 ... yes
40/393 ... yes
154/394 ... yes
160/395 ... yes
41/396 ... yes
118/397 ... yes
42/399 ... yes
58/400 ... yes
99/401 ... yes
99/402 ... yes
84/403 ... yes
99/404 ... yes
58/407 ... yes
83/408 ... yes
60/410 ... yes
47/411 ... yes
78/416 ... yes
143/417 ... yes
39/419 ... yes
74/420 ... yes
137/424 ... yes
103/425 ... yes
74/426 ... yes
12/430 ... yes
103/432 ... yes
161/434 ... yes
39/435 ... yes
86/436 ... yes
53/439 ... yes
153/441 ... yes
69/444 ... yes
88/446 ... yes
60/447 ... yes
53/448 ... yes
137/451 ... yes
88/452 ... yes
39/453 ... yes
179/454 ... yes
179/455 ... yes
99/456 ... yes
151/457 ... yes
151/458 ... yes
151/459 ... yes
119/460 ... yes
186/461 ... yes
39/462 ... yes
117/463 ... yes
185/464 ... yes
186/465 ... yes
103/466 ... yes
155/467 ... yes
47/468 ... yes
89/469 ... yes
185/470 ... yes
186/471 ... yes
52/473 ... yes
42/474 ... yes
87/475 ... yes
29/476 ... yes
12/478 ... yes
123/479 ... yes
153/480 ... yes
160/482 ... yes
41/483 ... yes
99/484 ... yes
169/486 ... yes
69/488 ... yes
186/489 ... yes
78/490 ... yes
96/491 ... yes
241/492 ... yes
256/496 ... yes
258/497 ... yes
251/498 ... yes
259/499 ... yes
261/500 ... yes
265/501 ... yes
267/502 ... yes
222/503 ... yes
266/504 ... yes
269/506 ... yes
269/508 ... yes
269/511 ... yes
269/512 ... yes
256/513 ... yes
281/515 ... yes
276/517 ... yes
274/519 ... yes
282/520 ... yes
257/521 ... yes
280/522 ... yes
280/523 ... yes
78/524 ... yes
282/526 ... yes
286/527 ... yes
39/528 ... yes
281/529 ... yes
287/530 ... yes
286/531 ... yes
282/532 ... yes
269/533 ... yes
163/534 ... yes
99/536 ... yes
287/537 ... yes
287/538 ... yes
287/540 ... yes
274/541 ... yes
283/542 ... yes
99/543 ... yes
274/544 ... yes
49/545 ... yes
291/548 ... yes
288/549 ... yes
99/550 ... yes
292/551 ... yes
286/552 ... yes
287/553 ... yes
280/555 ... yes
287/556 ... yes
280/557 ... yes
72/558 ... yes
291/559 ... yes
292/560 ... yes
288/561 ... yes
296/562 ... yes
294/563 ... yes
294/564 ... yes
296/565 ... yes
292/566 ... yes
287/567 ... yes
186/568 ... yes
282/569 ... yes
283/572 ... yes
294/573 ... yes
291/574 ... yes
190/576 ... yes
296/577 ... yes
291/578 ... yes
41/579 ... yes
269/580 ... yes
157/581 ... yes
297/582 ... yes
287/583 ... yes
160/584 ... yes
291/586 ... yes
294/588 ... yes
49/589 ... yes
297/590 ... yes
292/591 ... yes
297/592 ... yes
282/593 ... yes
292/594 ... yes
106/595 ... yes
29/596 ... yes
99/597 ... yes
89/598 ... yes
60/599 ... yes
79/600 ... yes
281/602 ... yes
297/603 ... yes
163/604 ... yes
29/606 ... yes
303/607 ... yes
287/608 ... yes
29/609 ... yes
89/610 ... yes
186/611 ... yes
286/612 ... yes
154/613 ... yes
269/614 ... yes
190/615 ... yes
303/616 ... yes
187/617 ... yes
122/618 ... yes
279/619 ... yes
69/620 ... yes
269/621 ... yes
59/622 ... yes
294/623 ... yes
49/624 ... yes
164/626 ... yes
179/628 ... yes
151/630 ... yes
260/631 ... yes
261/632 ... yes
29/634 ... yes
297/635 ... yes
287/637 ... yes
137/638 ... yes
143/639 ... yes
311/640 ... yes
190/641 ... yes
267/642 ... yes
279/643 ... yes
185/644 ... yes
169/645 ... yes
41/646 ... yes
50/649 ... yes
178/650 ... yes
29/651 ... yes
261/652 ... yes
49/653 ... yes
106/654 ... yes
222/655 ... yes
280/656 ... yes
301/658 ... yes
297/659 ... yes
297/660 ... yes
280/661 ... yes
259/662 ... yes
92/663 ... yes
116/664 ... yes
106/666 ... yes
106/667 ... yes
261/668 ... yes
330/670 ... yes
136/671 ... yes
292/672 ... yes
137/673 ... yes
41/674 ... yes
330/675 ... yes
330/676 ... yes
330/677 ... yes
160/679 ... yes
330/681 ... yes
53/682 ... yes
310/683 ... yes
259/684 ... yes
318/685 ... yes
39/686 ... yes
267/687 ... yes
143/688 ... yes
267/689 ... yes
267/690 ... yes
279/691 ... yes
184/692 ... yes
125/693 ... yes
320/694 ... yes
144/695 ... yes
283/696 ... yes
311/697 ... yes
50/698 ... yes
184/699 ... yes
106/700 ... yes
53/701 ... yes
330/702 ... yes
330/703 ... yes
50/704 ... yes
294/705 ... yes
310/706 ... yes
163/707 ... yes
355/708 ... yes
154/709 ... yes
330/710 ... yes
330/711 ... yes
330/712 ... yes
330/713 ... yes
117/715 ... yes
117/716 ... yes
117/717 ... yes
53/719 ... yes
184/720 ... yes
125/721 ... yes
160/722 ... yes
198/723 ... yes
287/727 ... yes
330/728 ... yes
339/729 ... yes
330/730 ... yes
358/731 ... yes
362/732 ... yes
106/733 ... yes
320/734 ... yes
50/735 ... yes
125/736 ... yes
173/737 ... yes
330/739 ... yes
355/740 ... yes
365/741 ... yes
297/742 ... yes
310/743 ... yes
366/744 ... yes
362/746 ... yes
99/747 ... yes
365/748 ... yes
53/749 ... yes
358/751 ... yes
330/752 ... yes
369/753 ... yes
330/754 ... yes
355/756 ... yes
318/758 ... yes
292/759 ... yes
330/761 ... yes
371/762 ... yes
291/763 ... yes
365/764 ... yes
370/765 ... yes
362/766 ... yes
376/767 ... yes
53/770 ... yes
378/771 ... yes
53/772 ... yes
378/773 ... yes
222/778 ... yes
99/779 ... yes
15/780 ... yes
50/781 ... yes
378/782 ... yes
369/783 ... yes
366/784 ... yes
366/785 ... yes
370/786 ... yes
385/787 ... yes
385/788 ... yes
366/791 ... yes
330/792 ... yes
267/795 ... yes
373/796 ... yes
366/797 ... yes
366/801 ... yes
185/802 ... yes
366/805 ... yes
387/806 ... yes
60/807 ... yes
99/808 ... yes
99/809 ... yes
190/810 ... yes
385/811 ... yes
190/812 ... yes
47/813 ... yes
173/815 ... yes
173/816 ... yes
387/817 ... yes
159/818 ... yes
385/819 ... yes
173/822 ... yes
49/823 ... yes
106/824 ... yes
386/826 ... yes
330/827 ... yes
330/828 ... yes
330/829 ... yes
53/830 ... yes
370/831 ... yes
72/832 ... yes
53/833 ... yes
53/834 ... yes
190/835 ... yes
261/836 ... yes
366/837 ... yes
50/838 ... yes
310/839 ... yes
88/840 ... yes
72/841 ... yes
310/843 ... yes
78/844 ... yes
50/845 ... yes
267/846 ... yes
267/847 ... yes
47/848 ... yes
78/849 ... yes
312/850 ... yes
387/851 ... yes
137/852 ... yes
305/853 ... yes
185/854 ... yes
49/855 ... yes
173/856 ... yes
49/858 ... yes
99/859 ... yes
155/860 ... yes
292/861 ... yes
310/864 ... yes
53/865 ... yes
297/866 ... yes
53/869 ... yes
47/870 ... yes
387/871 ... yes
279/872 ... yes
49/873 ... yes
387/874 ... yes
144/875 ... yes
72/876 ... yes
385/877 ... yes
53/878 ... yes
287/884 ... yes
330/885 ... yes
53/887 ... yes
400/889 ... yes
369/891 ... yes
178/893 ... yes
178/894 ... yes
49/895 ... yes
53/896 ... yes
292/897 ... yes
88/901 ... yes
49/902 ... yes
72/903 ... yes
256/906 ... yes
173/907 ... yes
312/908 ... yes
49/910 ... yes
49/911 ... yes
294/912 ... yes
417/913 ... yes
99/914 ... yes
307/915 ... yes
419/918 ... yes
163/919 ... yes
282/920 ... yes
53/921 ... yes
50/922 ... yes
49/923 ... yes
432/929 ... yes
432/931 ... yes
419/932 ... yes
433/933 ... yes
432/934 ... yes
419/935 ... yes
419/936 ... yes
419/937 ... yes
282/938 ... yes
49/939 ... yes
53/940 ... yes
50/941 ... yes
49/942 ... yes
184/943 ... yes
267/944 ... yes
311/945 ... yes
446/946 ... yes
446/947 ... yes
446/948 ... yes
99/951 ... yes
447/952 ... yes
450/953 ... yes
449/954 ... yes
451/955 ... yes
449/956 ... yes
267/957 ... yes
53/958 ... yes
49/959 ... yes
305/962 ... yes
49/963 ... yes
49/964 ... yes
449/965 ... yes
455/966 ... yes
47/968 ... yes
419/969 ... yes
446/970 ... yes
49/971 ... yes
299/972 ... yes
299/973 ... yes
50/974 ... yes
432/975 ... yes
449/977 ... yes
99/979 ... yes
463/981 ... yes
468/984 ... yes
455/986 ... yes
477/987 ... yes
281/989 ... yes
50/990 ... yes
432/991 ... yes
287/992 ... yes
477/993 ... yes
485/994 ... yes
454/995 ... yes
432/996 ... yes
487/997 ... yes
50/998 ... yes
47/1001 ... yes
53/1002 ... yes
449/1003 ... yes
467/1004 ... yes
286/1005 ... yes
49/1006 ... yes
501/1007 ... yes
501/1008 ... yes
467/1010 ... yes
53/1011 ... yes
501/1012 ... yes
501/1013 ... yes
480/1014 ... yes
501/1017 ... yes
53/1018 ... yes
467/1019 ... yes
477/1020 ... yes
435/1021 ... yes
330/1023 ... yes
330/1024 ... yes
501/1025 ... yes
117/1027 ... yes
467/1028 ... yes
49/1029 ... yes
370/1030 ... yes
501/1031 ... yes
501/1032 ... yes
421/1033 ... yes
49/1034 ... yes
49/1035 ... yes
501/1038 ... yes
501/1039 ... yes
429/1040 ... yes
501/1042 ... yes
53/1043 ... yes
467/1044 ... yes
467/1045 ... yes
479/1046 ... yes
501/1047 ... yes
501/1048 ... yes
499/1049 ... yes
435/1051 ... yes
421/1052 ... yes
386/1053 ... yes
449/1055 ... yes
467/1056 ... yes
501/1058 ... yes
53/1059 ... yes
470/1060 ... yes
478/1061 ... yes
464/1062 ... yes
497/1063 ... yes
394/1064 ... yes
555/1065 ... yes
469/1067 ... yes
454/1068 ... yes
556/1069 ... yes
505/1070 ... yes
53/1073 ... yes
49/1074 ... yes
556/1075 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 170
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
  Try fixing it:
  Please migrate all projects to hashed storage
  as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
  For more information see:
  doc/administration/repository_storage_types.md
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes