Project with huge merge requests fails to import with the unclear error: Incorrect JSON format

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

Project with huge merge requests fails to import with the unclear error: Incorrect JSON format.

Steps to reproduce

• Create a project with huge MR. The contents should exceed 50Mb.
• Export this project. Unarchive the export file and make sure that merge_requests.ndjson is bigger than 50Mb (you may manually modify it by adding some random contents, but make sure it is a valid JSON file).
• Try to import this project via the import rake task.

What is the current bug behavior?

The import process will fail with Incorrect JSON format error.

What is the expected correct behavior?

The import error should be clearer about the reason of the failure or it should be completed successfully.

Relevant logs and/or screenshots

In the log /var/log/gitlab/gitlab-rails/exceptions_json.log, you should be able to see a bit more details about the error. For instance:

"exception.message": "quoted string not terminated (merge_request_diff.merge_request_diff_commits[51370].message) at line 1, column 52428511 [parse.",

"exception.message": "Incorrect JSON format in 
/var/opt/gitlab/gitlab-rails/shared/tmp/gitlab_exports/@hashed/dd/26/dd2610d5c3431be01e725020c85c7b565d3e1184c81f8860695480c3ed57df2b/3805fc2c90e30f0768867d413282703e/55e44ac4739284febbe4b26181e54bbd/tree/project/merge_requests.ndjson
on line 3",

The first error points to the part of the file merge_requests.ndjson (line 1, column 52428511). If we trim the file at this place, we may see that it's size is around ~50Mb. In https://gitlab.com/gitlab-org/gitlab/-/blob/v15.1.0-ee/lib/gitlab/import_export/json/ndjson_reader.rb#L7 we define the constant MAX_JSON_DOCUMENT_SIZE:

    module Json
      class NdjsonReader
        MAX_JSON_DOCUMENT_SIZE = 50.megabytes

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:   Debian 10
Proxy:    no
Current User: git
Using RVM:  no
Ruby Version: 2.7.5p203
Gem Version:  3.1.4
Bundler Version:2.3.15
Rake Version: 13.0.6
Redis Version:  6.2.7
Sidekiq Version:6.4.0
Go Version: unknown

GitLab information
Version:  15.1.0-ee
Revision: 31c24d2d864
Directory:  /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.10
URL:    https://gitlabhost.tld
HTTP Clone URL: https://gitlabhost.tld/some-group/some-project.git
SSH Clone URL:  git@gitlabhost.tld:some-group/some-project.git
Elasticsearch:  no
Geo:    no
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:  14.7.4
Repository storage paths:
- default:  /var/opt/gitlab/git-data/repositories
GitLab Shell path:    /opt/gitlab/embedded/service/gitlab-shell

Possible fixes

Increase MAX_JSON_DOCUMENT_SIZE or make it configurable via UI/API/environment variable