[META] Improve GitLab Performance

`[META]` Let's Make GitLab Faster

NB: Work in Progress

Priority items marked with a ⭐️

Monitoring

Add metrics instrumentation to GitLab Internals. #28465
⭐️ Create a publically visible dashboard of key metrics. #948
- Easy to understand! -- everyone in the team should immediately be able to comprehend how fast things are running

Caching

Consider RoR Caching Techniques

Caching with Rails

Fragment and Russian doll Caching
- This may not be useful, since most models will need to be cached per-user.
- Per-user, per-model caching ➡️ Low hit-rates
SQL Caching
- Investigate RoR/Redis cache tuning and performance
  - How well is the cache performing?
  - How big is it? Are keys being evicted early

Improve Conditional HTTP Support in the Application

Rails has fantastic support for generating Etag headers, let's take advantage of it. Unfortunately at the moment, the Etag is being generated from the response

Take Control of Your HTTP Caching in Rails - Jan 2015
How key-based cache expiration works - DHH, Feb 2012

Polling and Realtime

Two separate issues here, but they are closely related:

Realtime Issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/26396

From the client side, we want to get realtime updates
CI runners (potentially tens of thousands concurrently) listening to GitLab for build jobs to run.

Application Performance

⭐️ Find badly performing routes
- n + 1 queries
  - Count number of postgres and redis transactions per request and monitor w/ prometheus
Slowlogs
- postgres slowlog ➡️ kibana
  - pg_stat_statements
- redis slowlog ➡️ kibana
- slowlog
⭐️ Disable seqscan in postgres query planner in tests, so that any queries causing a scan will cause test failure

Git Access

Gitaly

File System Access

Git Storage
- ⭐️ NFS optimisation
- Migration to gitaly
Attachments
- Need to find out more information about where attachments are stored
- Ideally in an object store
- minio already in use?

Frontend Performance

Frontend Performance Issue #23213