Draft: POC Deduplicate intrinsic data into Ci::JobInfo
What does this MR do and why?
This is a POC for moving immutable, intrinsic data into a new model Ci::JobInfo.
Details:
-
Creates
ci_job_infosand adds the columnjob_info_idtoci_builds. -
Moves
ci_builds.scheduling_typeandci_builds.name-->ci_job_infosas normalized columns.- Attempted moving other columns from
ci_buildsand these were the results:-
--> Cannot move. This field is mutated when job transitions to failed.allow_failure -
--> Cannot move. This field is mutated in Ci::ProcessBuildService.when scheduling_type-
--> Indexed column; impractical to move.stage_idx -
name--> Indexed column; impractical to move. Decided to keep it inci_buildsfor now but still copy data over toci_job_infosto dropci_build_names. See !211540 (comment 2878147699).
-
- Costs/benefits:
- Deduplicates
scheduling_typevalue. - We can eventually drop
ci_builds.scheduling_typecolumn.
- Deduplicates
- Attempted moving other columns from
-
Adds
search_vectorcolumn toci_job_infos.- This combined with copying over
ci_builds.nametoci_job_infosreplaces the need forci_build_names. - Costs/benefits:
- Deduplicates
nameandsearch_vectorvalues. - We can eventually drop
ci_build_names.
- Deduplicates
- This combined with copying over
-
Moves
ci_build_needs-->ci_job_infos.config.- Instead of reading
job.needsdirectly, we readneeds_attributesfromjob_info.configand represent it as a collection of needs usingCi::JobNeeds::Collection. This allows us to work with the data in a readable and consistent way. - The only potential blocker is the object hierarchy query in !211540 (diffs).
- Costs/benefits:
- We can either completely drop
ci_build_needsor at least drop the data after pipeline archival.
- We can either completely drop
- Instead of reading
-
Moves
ci_build_sources-->ci_job_infos.source.- Costs/benefits:
- Deduplicates
ci_build_sources.sourcevalues. - We can eventually drop
ci_build_sources.
- Deduplicates
- Costs/benefits:
References
- Spike: Deduplicate intrinsic immutable data fro... (#577211 - closed)
- Spike: Deduplicate `ci_build_names` into one of... (#567704 - closed)
- Spike: Deduplicate `ci_build_needs` (#565821 - closed)
- Spike: Deduplicate `ci_build_sources` (#565806)
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #577211 (closed)
Edited by Leaminn Ma