[go: up one dir, main page]

Tech Evaluation: Upload assets to the generic package registry before creating a release

The objective is ti figure out how much work it needs to be done on the release-cli in order to upload binary assets to the generic package registry introduced in gitlab#235492 (closed).

Proposal

  1. Generate binaries in a job previous to release and save them to the CI artifacts artifacts:paths
  2. Require artifacts in the release job so that they are made available to the release-cli
  3. Add a flag to the release-cli to indicate it should upload a directory before creating a release, for example --upload-dir=/path/to/artifacts
  4. release-cli will upload the files to the generic package registry using the API, may need to use a feature branch
  5. Once files are uploaded, release-cli should create a release attaching permanent links to the release

Sample .gitlab-ci.yml configuration

build:
  script:
    - make build # output goes to bin/binary_*_.ext
  artifacts:
    paths:
      - bin

release:
  image: registry.gitlab.com/gitlab-org/release-cli
  needs:
    - job: build
      artifacts: true
  script:
    - echo "upload binaries and create release"
    - >
      release-cli --name "Release $CI_COMMIT_TAG" --tag-name $CI_COMMIT_TAG --upload-dir=bin/

Considerations

  1. We depend on a previous job to be executed first, meaning the files will be uploaded to the artifacts storage, downloaded into the job runner and uploaded again to the package registry
  2. maybe it can be done in the same step as release where we have the script run before release-cli is called 🤔
  3. If gitlab#17838 (comment 406384884) is on the table then we might just need to add the -upload-dir flag to the release-cli

Results from !59 (closed)

Following the Generic Packages API format /api/v4/:project_id/packages/generic/:package_name/:package_version/:filename I've added a couple of flags to support uploading binaries.

  • --upload-dir=./bin/ -> only supports first level files and skips subdirectories
  • --package-name=$CI_PROJECT_NAME -> this is required by the generic packages API

For now, when --upload-dir is defined:

  • Ignore --assets-link
  • Files need to exists, for example building in a prior stage and sharing them as artifacts in the release job
  • Iterate over regular files inside the --upload-dir
  • Skip subdirectories inside of --upload-dir
  • Upload each file present in the directory and prepare a direct asset link to be passed to the releases API. (type is ignored as the default value is other and filepath is also ignored)
  • Use --package-name to match the API requirement, this is defaulting to the project name right now
  • Use each file inside --upload-dir as the :filename
  • Use a "lean" version of the $CI_COMMIT_TAG to only keep the required format e.g. if $CI_COMMIT_TAG=v1.2.3-rc then only 1.2.3 is used as the :package_version

Here is a sample project (requires staging access) with CI config. Using the release-cli directly:

release-from-tag:
        #image: registry.gitlab.com/gitlab-org/release-cli
  image: registry.gitlab.com/jaime/release-cli:edge
  stage: release
  needs:
    - job: prepare
      artifacts: true
    - job: build  # build job and artifacts required
      artifacts: true
  rules:
    - if: $CI_COMMIT_TAG                                            # Run this job when a tag is created manually
  script:
    - echo "running release from $CI_COMMIT_TAG"
    - >
      release-cli create --name "Release $CI_COMMIT_TAG" --description "Notes: $EXTRA_DESCRIPTION"
      --tag-name $CI_COMMIT_TAG --ref $CI_COMMIT_TAG
      --milestone "m1" --milestone "m2"
      --upload-dir "bin/"

The created release with the list of assets https://staging.gitlab.com/jaimem88/hello/-/releases/v1.10.3-rc1

Screen_Shot_2020-09-23_at_4.00.12_pm

And the contents of the package registry

Screen_Shot_2020-09-23_at_4.08.43_pm

What's next

If we want to pursue this option I can create some follow-up issues.

  1. Upload assets prior to creating a release (weight 3) #59 (closed)

    1. Add API call to generic packages to upload files.
    2. Add flags --upload-dir and --package-name. Only support root level items and ignore subdirectories.
    3. Update documentation and examples on the CI reference.

  2. Extend the release: keyword to support upload_dir and package_name (unsure about weight) gitlab#255271. For example

    release:
  name: "Release $CI_COMMIT_TAG"
  tag_name: "$CI_COMMIT_TAG"
  package_name: "myawesomepackage"
  upload_dir: "./bin/"

  3. Nice-to-have: Extend the packages API to allow using different versioning for :package_version gitlab#255234 (closed)

  4. Nice-to-have: Support subdirectories via --upload-dir #60 (closed)

Issues found

  1. The generic packages API :package_version only accepts text or versions of the format 1.2.3. Values like v1.2.3 or 1.2.3-rc-1 fail from the API. I'll post this feedback on gitlab#235492 (closed).
  2. Because of the above, we can't use $CI_COMMIT_TAG as the value of :package_version right now because the git tag can be anything.
  3. Could not test on GitLab.com yet so did all my testing on staging.
  4. Could not test using the GDK as the packages API returns a 500 error but I didn't investigate further as it works on staging and it's most likely related to my local setup.
Edited by Jaime Martinez