Make GitLab file based import idempotent

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Currently, GitLab file-based import sidekiq workers are not idempotent. This is not ideal, as multiple runs of such jobs can cause unexpected side effects, as duplicate object creation, uncaught exceptions, etc.

As per the sidekiq guidelines, such jobs should ideally be idempotent and transactional https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/sidekiq_style_guide.md#L109-109

Considerations

Consider the case of workers running more than once perhaps through deployments where sidekiq reliable fetch is invoked

• somehow track the position of the import as it proceeds and only starts where it left off(in case of rerun)
  • position = filename and line number
  • record this position in Redis and only save the position again when the next line starts.
• consider upsert if not too expensive of an operation instead of tracking the position. See - #285111 (comment 685766705)