Replace sidechannel with gRPC buffer pooling
Sidechannel was put in place as a performance optimization around gRPC. gRPC's Go implementation was inefficient for large file transfer due to the amount of memory allocations that led to GC churn. Sidechannel adds fair amount of complexity as it opens an alternative transport stream along the main gRPC stream. Features such as retries do not automatically work, as don't application level load balancers.
gRPC released support for buffer pooling in v1.66. This would potentially solve the memory allocation related performance issues originating from file transfer that led to sidechannel. We should look into replacing sidechannel with pure gRPC now that we can better control the memory allocations.
In practice, we'd implement a codec that handles the buffer pooling: https://pkg.go.dev/google.golang.org/grpc@v1.69.2/encoding#CodecV2
Alternative approach if buffer pooling doesn't work: https://docs.gitlab.com/ee/architecture/blueprints/gitaly_handle_upload_pack_in_http2_server/.