[go: up one dir, main page]
Skip to content

Project export/import (ndjson) - child pipeline jobs not imported

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When exporting a project that uses parent/child pipelines and / or dynamic parent child pipelines, the jobs for the children are not there after export and import.

Investigation below suggests that the job data is missing from the ndjson export; I don't know how to fabricate the missing JSON to check whether the importer would cope if given the chance ;)

A customer reported that import missed a number of things, not all of which reproduced on a simple test case.

This was the main issue they were concerned about.

this could be a feature rather than a bug. I also tested (13.0) by setting Feature.enabled?("project_export_as_ndjson") so I could could get an old-format export. Child pipelines are also missing when that's imported.

Steps to reproduce

  • Create a project with parent/child pipelines.
  • Run pipelines
  • Export the project
  • Import it.

Example Project

I built this on a self-managed 13.0 instance. The exports are here: and they include the source code with CI that uses parent/child pipelines AND dynamic parent/child pipelines - so it's a canned project that you could import, drive, and then export to reproduce this.

Note: This project contains a pipeline schedule.

testprojectexport_ndjson.tar.gz

testprojectexport_oldjson.tar.gz

ndjson analysis

I looked through the export to work out if it was an export issue, or an import issue.

Three pipelines for the post recent activity, commit 644e6b70.

image

$ grep   644e6b70 ci_pipelines.ndjson | jq . | grep '^  "ref":'
  "ref": "progress3",
  "ref": "refs/merge-requests/7/head",
  "ref": "test",

The progress3 job:

image image image

$ grep   644e6b70 ci_pipelines.ndjson | head -1 | jq . | egrep '"name"|"id"'
  "id": 284,
      "id": 392,
      "name": "baseline",
          "id": 1131,
          "name": "child2config",
          "id": 1130,
          "name": "pipelinesource",
      "id": 393,
      "name": "one",
          "id": 1132,
          "name": "_one",
          "id": 1133,
          "name": "child1",
      "id": 394,
      "name": "two",
          "id": 1134,
          "name": "_two",
          "id": 1135,
          "name": "child2",

What is the current bug behavior?

The pipelines and the main jobs are imported, but the child pipeline jobs are not, and no aspect of the child pipelines appear in the UI.

What is the expected correct behavior?

All the jobs are imported

Relevant logs and/or screenshots

source

image image

destination

image

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.26.2
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	13.0.5-ee
Revision:	260c6231ed0
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://gitlab-gltest1.watertower
HTTP Clone URL:	https://gitlab-gltest1.watertower/some-group/some-project.git
SSH Clone URL:	git@gitlab-gltest1.watertower:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.2.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 

sudo gitlab-rake gitlab:env:info


System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.26.2
Sidekiq Version:5.2.7
Go Version:	unknown


GitLab information
Version:	13.0.5-ee
Revision:	260c6231ed0
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://gitlab-gltest1.watertower
HTTP Clone URL:	https://gitlab-gltest1.watertower/some-group/some-project.git
SSH Clone URL:	git@gitlab-gltest1.watertower:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	13.2.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git [root@terix sysctl.d]# sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.2.0 ? ... OK (13.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes 5/2 ... yes 5/3 ... yes 5/4 ... yes 6/5 ... yes 6/6 ... yes 6/7 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... yes (2.26.2) Git user has default SSH configuration? ... yes Active users: ... 2 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished

Checking GitLab subtasks ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Edited by 🤖 GitLab Bot 🤖