[go: up one dir, main page]
Skip to content

Project import fails via API and UI for a large project

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Large project import process fails with "exception.class":"NoMethodError","exception.message":"undefined method 'local?'' for nil:NilClass". The results project has all labels, some part of the issues(about 3k of 20k) and no merge requests.

The issue doesn't happen if the import is executed with import rake task.

The test project has these specs: 3.5 Gb Files, 900k commits, 20k issues, 7k MRs, 1.5k labels - the export file is available here (2.7 Gb). The project was populated with data using API to create a large monorepo that should help us to test performance of the large projects on GitLab(gitlab-org/quality/performance#130 (closed)).

Steps to reproduce

  1. Import a large project via UI or API.
  2. Wait for import results.

Example Project

https://gitlab.com/niskhakova/monorepo

What is the current bug behavior?

Large project import fails via UI and API.

What is the expected correct behavior?

Project import successfully finished.

Relevant logs and/or screenshots

Full error logs
  • '/var/log/gitlab/gitlab-rails/exceptions_json.log:1:{"severity":"ERROR","time":"2020-02-28T09:42:42.763Z","correlation_id":"GJa88Q201p","extra.server":{"os":{"name":"Linux","version":"32-Ubuntu SMP Tue Feb 11 03:55:48 UTC 2020","build":"5.0.0-1031-gcp","kernel_version":"Linux gitlab-qa-2k-sidekiq-2 5.0.0-1031-gcp 32-Ubuntu SMP Tue Feb 11 03:55:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux"},"runtime":{"name":"ruby","version":"ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]"}},"extra.sidekiq":{"retry":false,"args":[119],"class":"RepositoryImportWorker","queue":"repository_import","backtrace":5,"status_expiration":54000,"memory_killer_memory_growth_kb":50,"memory_killer_max_memory_growth_kb":300000,"jid":"184b6eea31041631e8059f0f","created_at":1582881187.7651858,"meta.user":"root","meta.caller_id":"Import::GitlabProjectsController#create","correlation_id":"GJa88Q201p","enqueued_at":1582881187.7661953,"interrupted_count":1},"extra.importer":"Import/Export","extra.exportable_id":119,"extra.exportable_path":"qa-perf-testing/linux9","extra.import_jid":"184b6eea31041631e8059f0f","exception.class":"NoMethodError","exception.message":"undefined method local?'' for nil:NilClass","exception.backtrace":["lib/gitlab/import_export/command_line_util.rb:25:in download_or_copy_upload''","lib/gitlab/import_export/file_importer.rb:67:in copy_archive''","lib/gitlab/import_export/file_importer.rb:28:in import''","lib/gitlab/import_export/file_importer.rb:14:in import''","lib/gitlab/import_export/importer.rb:42:in import_file''","lib/gitlab/import_export/importer.rb:21:in execute''","app/services/projects/import_service.rb:109:in import_data''","app/services/projects/import_service.rb:24:in execute''","app/workers/repository_import_worker.rb:24:in perform''","ee/app/workers/ee/repository_import_worker.rb:9:in perform''","lib/gitlab/sidekiq_daemon/monitor.rb:49:in within_job''"]}'
    • '/var/log/gitlab/gitlab-rails/exceptions_json.log:2:{"severity":"ERROR","time":"2020-02-28T09:42:42.766Z","correlation_id":"GJa88Q201p","extra.server":{"os":{"name":"Linux","version":"32-Ubuntu SMP Tue Feb 11 03:55:48 UTC 2020","build":"5.0.0-1031-gcp","kernel_version":"Linux gitlab-qa-2k-sidekiq-2 5.0.0-1031-gcp 32-Ubuntu SMP Tue Feb 11 03:55:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux"},"runtime":{"name":"ruby","version":"ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-linux]"}},"extra.sidekiq":{"retry":false,"args":[119],"class":"RepositoryImportWorker","queue":"repository_import","backtrace":5,"status_expiration":54000,"memory_killer_memory_growth_kb":50,"memory_killer_max_memory_growth_kb":300000,"jid":"184b6eea31041631e8059f0f","created_at":1582881187.7651858,"meta.user":"root","meta.caller_id":"Import::GitlabProjectsController#create","correlation_id":"GJa88Q201p","enqueued_at":1582881187.7661953,"interrupted_count":1},"extra.project_path":"qa-perf-testing/linux9","extra.importer":"gitlab_project","exception.class":"Projects::ImportService::Error","exception.message":"undefined method local?'' for nil:NilClass","exception.backtrace":["lib/gitlab/import_export/importer.rb:27:in rescue in execute''","lib/gitlab/import_export/importer.rb:26:in execute''","app/services/projects/import_service.rb:109:in import_data''","app/services/projects/import_service.rb:24:in execute''","app/workers/repository_import_worker.rb:24:in perform''","ee/app/workers/ee/repository_import_worker.rb:9:in perform''","lib/gitlab/sidekiq_daemon/monitor.rb:49:in within_job''"]}'
    • '/var/log/gitlab/sidekiq/current:1214:{"severity":"WARN","time":"2020-02-28T09:42:42.780Z","retry":false,"class":"RepositoryImportWorker","queue":"repository_import","backtrace":5,"status_expiration":54000,"memory_killer_memory_growth_kb":50,"memory_killer_max_memory_growth_kb":300000,"jid":"184b6eea31041631e8059f0f","created_at":"2020-02-28T09:13:07.765Z","meta.user":"root","meta.caller_id":"Import::GitlabProjectsController#create","correlation_id":"GJa88Q201p","enqueued_at":"2020-02-28T09:13:07.766Z","interrupted_count":1,"pid":30780,"message":"RepositoryImportWorker JID-184b6eea31041631e8059f0f: fail: 0.069095 sec","job_status":"fail","scheduling_latency_s":1774.944911,"duration":0.069095,"cpu_s":0.029325,"completed_at":"2020-02-28T09:42:42.780Z","error_message":"Error importing repository into qa-perf-testing/linux9 - undefined method `local?'' for nil:NilClass","error_class":"RuntimeError","db_duration":16.099144238978624,"db_duration_s":0.016099144238978624}'
    • /var/log/gitlab/sidekiq/current:1215:{"severity":"WARN","time":"2020-02-28T09:42:42.781Z","error_class":"RuntimeError","error_message":"Error importing repository into qa-perf-testing/linux9 - undefined method local?' for nil:NilClass","context":"Job raised exception","jobstr":"{\"retry\":false,\"args\":[119],\"class\":\"RepositoryImportWorker\",\"queue\":\"repository_import\",\"backtrace\":5,\"status_expiration\":54000,\"memory_killer_memory_growth_kb\":50,\"memory_killer_max_memory_growth_kb\":300000,\"jid\":\"184b6eea31041631e8059f0f\",\"created_at\":1582881187.7651858,\"meta.user\":\"root\",\"meta.caller_id\":\"Import::GitlabProjectsController#create\",\"correlation_id\":\"GJa88Q201p\",\"enqueued_at\":1582881187.7661953,\"interrupted_count\":1}","retry":false,"args":[119],"class":"RepositoryImportWorker","queue":"repository_import","backtrace":5,"status_expiration":54000,"memory_killer_memory_growth_kb":50,"memory_killer_max_memory_growth_kb":300000,"jid":"184b6eea31041631e8059f0f","created_at":"2020-02-28T09:13:07.765Z","meta.user":"root","meta.caller_id":"Import::GitlabProjectsController#create","correlation_id":"GJa88Q201p","enqueued_at":"2020-02-28T09:13:07.766Z","interrupted_count":1,"error_backtrace":["app/workers/repository_import_worker.rb:33:in perform'","ee/app/workers/ee/repository_import_worker.rb:9:in perform'","lib/gitlab/sidekiq_daemon/monitor.rb:49:in within_job'"]}

Output of checks

This bug happens on GitLab.com

Edited by 🤖 GitLab Bot 🤖