[go: up one dir, main page]

Skip to content

Add telemetry to MR Reviewer sidebar assignment

(outcome of previous Spike)

Telemetry

We need regular feedback to help us tweak the utility of this sort. If users are not typically choosing the first 1-3 suggested reviewers, we're probably not making the right suggestions.

Overview

When a user is selected to be the reviewer, we should send an internal event with a custom payload indicating their position in the list.
We should not do this if the user has manually searched for users by name/username.

Details

Presumably, as soon as a user searches for a specific string, we would stop doing any of the custom filtering/sorting. It wouldn't make sense to report which position is selected if they've manually searched, as it's likely to always be position 1.

We may want to indicate that a user needed to scroll down before selecting a user. This could be done with an actual check of scrolling down (but not back up) or we could infer it from the position of the selected reviewer (4+ is likely to require a scroll). If the user is having to scroll the list to find the right reviewer, our suggestions aren't accurate enough.

It's not likely that we would need to track reviewer unassignments in the context of this feature: If the user selected position 1 and then later unassigns them, it's probably not a negative signal to the suggested reviewers except in the case of egregious defects (like we failed to filter out a reviewer who was marked "busy", etc.). If we do need to track unassigns, that means we'll probably also want much more robust information about that reviewer from the time they were displayed in the list to when they were unassigned. For example:

  • What position was this reviewer in when they were selected?
  • Why were they placed in that position? (e.g. a compact data code indicating our filter decisions, so we could inspect why they may have been a bad suggestion)
  • What position is this user placed in right now? (e.g. Were they position 1 before but now they don't appear at all because they're marked busy? Our suggestion system didn't fail. Were they position 1 before but their timezone places them already into their weekend? Our suggestion system failed.)
Implementation

At the most basic level, we will send an internal event like code_review_suggested_reviewer_selected with a custom payload like { position: 1 }.

If we need to track unassigns, we'll need to develop a tracking system that's fairly unique at GitLab that remembers previous information to be tracked later. We could send an event like code_review_suggested_reviewer_unassigned with a custom payload like { position: null, original: 1, sort_algorithm: 'assigned:0|tz:-4|busy:0'}.

Note that this will require significant work:

  • Need to store what position the reviewer was originally displayed at alongside their assignment beyond user logouts and page refreshes
  • Need to store the factors that went into their original positioning beyond user logouts and page refreshes
  • Need a system to compare current position to previous position without user interaction querying the list