Investigate root cause of slow requests in the Settings Sync API

The problem

The apdex index for Settings Sync API endpoints is below the threshold in the IDE group error bucket. Looking at the production logs, we identified the requests that are taking more than 1s to complete over the last 7 days. For some of these requests, there is a correlation between the database access duration and the request duration.

/api/:version/vscode/settings_sync/v1/manifest

https://log.gprd.gitlab.net/app/r/s/1P3MU

GET /api/:version/vscode/settings_sync/v1/resource/:resource_name/:id

https://log.gprd.gitlab.net/app/r/s/tVnQ6

In the logs, for some requests there is a correlation between the duration_s and the db_duration_s fields. I interpret this correlation as the database being the cause for the request duration. For other requests, there is a correlation between the duration_s and the cpu_s fields which might indicate some task that takes considerable time to complete.

So far, we don't have certainty about the root cause for these slow requests because we don't understand the reason behind these numbers.