Investigate slowdown of GET /api/:version/user_counts

Overview

The groupfoundations error budget has been dipping due to infrequent, but severe slowdowns of the /api/:version/user_counts endpoint.

This week, there were ~2500 requests over the 5s threshold with some having a json.duration_s as high as 17s.

json.meta.caller_id.keyword : "GET /api/:version/user_counts" and json.duration_s > 5

Possible Causes

Since this endpoint is heavily cached, it's possible that the dips are due to cache repopulation. I'm not sure how to confirm this, or what to do about it however. This will need more investigation.

Proposal

1. Investigate the dips and confirm is this is a cache invalidation/repopulation issue
2. Fix that issue
3. ???

Reference

• Foundations Error budget
• Problematic endpoints (Kibana)
• endpoint details (kibana)
• User Counts endpoint documentation