Create offline transfer pipeline for BulkImports::Common::Pipelines::UploadsPipeline
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
As part of leveraging our existing direct transfer service for offline transfers, we will need to add new file-based pipelines for pipelines in direct transfer that make API requests to the source instance during any part of the ETL pipeline. This new pipeline should not replace the existing pipeline and it should only be executed during offline transfer.
This issue is to create a pipeline compatible with offline transfer for BulkImports::Common::Pipelines::UploadsPipeline. Uploads are already exported in a file. However, this class doesn't use the shared NDJSON extractor because this relation can't be exported as NDJSON, so it implements its own extract to fetch the exported uploads file. The new offline transfer transfer needs to do the same extraction from the configured object storage location instead of the source instance.