gitaly warnings: "no alternates directory exists"

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue
• Close this issue

We have a lot of warnings in our gitaly log which say "no alternates directory exists", e.g.

{
"correlation_id":"14091312e37f7c854ff9f618aa89ed8d",
"error":"no alternates directory exists",
"grpc.meta.auth_version":"v2",
"grpc.meta.deadline_type":"none",
"grpc.method":"GetObjectPool",
"grpc.request.fullMethod":"/gitaly.ObjectPoolService/GetObjectPool",
"grpc.request.glProjectPath":"",
"grpc.request.glRepository":"",
"grpc.request.repoPath":"@hashed/85/c5/85c525b08f8384df6ec754acae1007d543aa91c0bfe32705eda1db48fcc8ae15.git",
"grpc.request.repoStorage":"gitaly-1.x",
"grpc.request.topLevelGroup":"@hashed",
"grpc.service":"gitaly.ObjectPoolService",
"grpc.start_time":"2020-09-22T14:17:16Z",
"level":"warning",
"msg":"alternates file does not point to valid git repository",
"peer.address":"y:34432",
"pid":26525,
"span.kind":"server",
"storage":"@hashed/85/c5/85c525b08f8384df6ec754acae1007d543aa91c0bfe32705eda1db48fcc8ae15.git",
"system":"grpc",
"time":"2020-09-22T14:17:16.641Z"
}

There are 655 entries with this correlation_id and 54 of these are like the above, but for different repoPath/storage.

I traced the correlation_id back to the praefect servers it belongs to a replication_job

{
"component":"replication_manager",
"correlation_id":"14091312e37f7c854ff9f618aa89ed8d",
"level":"info",
"msg":"processing replication job",
"pid":20594,
"replication_job_id":9779369,
"time":"2020-09-22T14:17:15.79Z",
"virtual_storage":"default"
}

or to sidekiqs:

{
"severity":"INFO",
"time":"2020-09-22T14:17:11.232Z",
"class":"RepositoryUpdateMirrorWorker",
"retry":0,
"queue":"repository_update_mirror",
"version":0,
"status_expiration":54000,
"jid":"d08adfa79bdf291929409912",
"created_at":"2020-09-22T14:17:11.195Z",
"meta.project":"x",
"meta.root_namespace":"git",
"meta.caller_id":"ProjectImportScheduleWorker",
"correlation_id":"14091312e37f7c854ff9f618aa89ed8d",
"enqueued_at":"2020-09-22T14:17:11.203Z",
"pid":13736,
"message":"RepositoryUpdateMirrorWorker JID-d08adfa79bdf291929409912: start",
"job_status":"start",
"scheduling_latency_s":0.029
}

or

{
"severity":"INFO",
"time":"2020-09-22T14:17:11.282Z",
"class":"ProjectImportScheduleWorker",
"retry":0,
"queue":"project_import_schedule",
"version":0,
"jid":"8e4043a17777be69e9f020c8",
"created_at":"2020-09-22T14:17:10.816Z",
"enqueued_at":"2020-09-22T14:17:10.855Z",
"meta.project":"x",
"meta.root_namespace":"git",
"meta.caller_id":"UpdateAllMirrorsWorker",
"correlation_id":"14091312e37f7c854ff9f618aa89ed8d",
"pid":13736,
"message":"ProjectImportScheduleWorker JID-8e4043a17777be69e9f020c8: done: 0.246333 sec",
"job_status":"done",
"scheduling_latency_s":0.180584,
"redis_calls":5,
"redis_duration_s":0.042581,
"redis_read_bytes":170,
"redis_write_bytes":1157,
"redis_cache_calls":1,
"redis_cache_duration_s":0.010337,
"redis_cache_read_bytes":109,
"redis_cache_write_bytes":51,
"redis_queues_calls":3,
"redis_queues_duration_s":0.024878,
"redis_queues_read_bytes":60,
"redis_queues_write_bytes":1077,
"redis_shared_state_calls":1,
"redis_shared_state_duration_s":0.007366,
"redis_shared_state_read_bytes":1,
"redis_shared_state_write_bytes":29,
"db_count":8,
"db_write_count":2,
"db_cached_count":0,
"duration_s":0.246333,
"cpu_s":0.016955,
"completed_at":"2020-09-22T14:17:11.282Z",
"db_duration_s":0.145114
}

or

{
"severity":"INFO",
"time":"2020-09-22T14:17:14.966Z",
"class":"UpdateAllMirrorsWorker",
"retry":0,
"queue":"cronjob:update_all_mirrors",
"version":0,
"queue_namespace":"cronjob",
"jid":"f5b54c085afab0cf833adbd3",
"created_at":"2020-09-22T14:17:14.950Z",
"meta.caller_id":"UpdateAllMirrorsWorker",
"correlation_id":"14091312e37f7c854ff9f618aa89ed8d",
"enqueued_at":"2020-09-22T14:17:14.951Z",
"pid":13736,
"message":"UpdateAllMirrorsWorker JID-f5b54c085afab0cf833adbd3: done: 0.013561 sec",
"job_status":"done",
"scheduling_latency_s":0.000973,
"redis_calls":5,
"redis_duration_s":0.000715,
"redis_read_bytes":5,
"redis_write_bytes":492,
"redis_queues_calls":2,
"redis_queues_duration_s":0.000266,
"redis_queues_read_bytes":2,
"redis_queues_write_bytes":195,
"redis_shared_state_calls":3,
"redis_shared_state_duration_s":0.000449,
"redis_shared_state_read_bytes":3,
"redis_shared_state_write_bytes":297,
"db_count":1,
"db_write_count":0,
"db_cached_count":0,
"duration_s":0.013561,
"cpu_s":0.006561,
"completed_at":"2020-09-22T14:17:14.966Z",
"db_duration_s":0.00163
}

1. I really don't know about the correlation_id, it is all over the place
2. these warnings are weird, these repos were not forked, so there shouldn't be an alternates file (the repositories were all imported via git protocol and are a pull mirrors of other git repositories outside of our gitlab instance)

version: 13.3.6-ee