[go: up one dir, main page]

Skip to content

Improve the performance of the Projects::FetchStatisticsIncrementService

The workhorse_auth_api Apdex (This is the indicator that has had some huge drops lately, for example:

image

git dashboard

Looking at the logs for the same timeframe, we can see that p99 database duration-peaks seem to correlate for the git-upload-pack endpoint:

image

https://log.gprd.gitlab.net/goto/3e4a237bbdbe6c76da4fcfffd5502203

@andrewn was sampling postgres to investigate database locks on the project_daily_statistics, using the info from marginalia to tie the queries to endpoints and noticed that there were a lot of queries for git_upload_pack marked as idle: gdoc with raw data

Which makes us think we should improve the way we track these fetch statistics

Proposals

  1. Remove the tracker all together: we only increment this counter for git-https, but not over SSH. Which means the counter doesn't paint an entirely accurate picture. So perhaps this counter, that is only accessible through this API endpoint, could be removed?

  2. Only periodically increment a counter in the database, while keeping a live counter in Redis. When incrementing the counter, we schedule a sidekiq job with a delay of a few minutes that can be deduplicated (using deduplicate :until_executing, including_scheduled: true.