Concern duplication ::Gitlab::Import::Metrics vs ::Gitlab::GithubImport::ObjectCounter
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Motivation
The code review !74206 (closed) made explicit that the importers metrics normalization is more complex than originally estimated.
Problem to Solve
While creating ::Gitlab::GithubImport::ObjectCounter1 to improve the visibility of the Github Importer, some logic duplication was created with ::Gitlab::Import::Metrics. Currently individual objects, like issues or merge requests, in the Github Importer are being counted with ::Gitlab::GithubImport::ObjectCounter, but similarly the Bitbucket Importer is counting some objects with ::Gitlab::Import::Metrics.
Besides that, there's some confusion about how to tag some of the counters2, either with the project or the importer type (or maybe both?!)
Proposed solution
Merge the way objects are being counted under the same class to normalize dashboards and increase the maintainability of the code by removing the duplication. For that, I propose picking the best of both existing classes, like:
-
Move ::Gitlab::GithubImport::ObjectCounterlogic to within the::Gitlab::Import::Metrics-
::Gitlab::GithubImport::ObjectCountercounts individual objects on both prometheus and redis, the former for dashboard and the latter to provide a finishing log with the stats of the import; -
::Gitlab::Import::Metrics, on the other hand, tracks count of projects imported (started, finished, failed) and a duration metric on prometheus, which is a good data to include on grafana dashboards;
-
-
Replace ::Gitlab::GithubImport::ObjectCountercalls by::Gitlab::Import::Metrics -
Add the stats log that Github Importer have to Bitbucket Importer -
Add the project start/failure that Bitbucket Importer have to Github Importer -
Add the duration panel from Bitbucket Importer dashboard to Github Importer dashboard -
Use the Github Importer dashboard as template to other importers, like Bitbucket Importer