Inconsistency on listing errors from tenant and cluster level in Error Tracking App

Describe the bug

In the last 24 hours we observed an increase on production alerts caused by ErrorTrackingReadPathIsDown. Below you can see the unsuccessful attempts of the error tracking read path.

The errortracking smoketests probes the write path by writing an error using the Go Sentry SDK and the Opstrace project Sentry DSN. Then for the reading path the tests will attempt to read that error by reaching to https://observe.gitlab.com/v2/errortracking/14485840/projects/32149347/errors. This endpoint corresponds to the tenant level error tracking application. To be more precise this endpoint calls the ListErrorsV2 controller function.

The problem is that when calling https://observe.gitlab.com/v2/errortracking/14485840/projects/32149347/errors, the error is missing. At the same time the error is visible in the Rails UI which is using the cluster level Error Tracking app. The Rails UI is using the corresponding endpoint which maps to https://observe.gitlab.com/errortracking/api/v1/projects/32149347/errors. This request is handled by ListErrors controller function.

To Reproduce

I don't know how to reproduce the bug. But when the bug takes place you can observe the wrong behavior by calling:

curl --location --request GET 'https://observe.gitlab.com/v2/errortracking/14485840/projects/32149347/errors' \
--header 'Authorization: Bearer <GRAFANA_API_TOKEN>'

and then

curl --location --request GET 'https://observe.gitlab.com/errortracking/api/v1/projects/32149347/errors' \
--header 'Gitlab-Error-Tracking-Token: <INTERNAL_ENDPOINT_TOKEN>'

Expected behavior

The expected behaviour is that we get the same errors independently which endpoint we call (tenant or cluster level)