Investigate desired workhorse socket shutdown behaviour
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
Investigate and determine desired shutdown behaviour for workhorse when dealing with websockets.
Details
In #363096 (comment 980570509) we were running into production alerts when workhorse was terminiating too quickly while puma still had not shut down gracefully. To workaround in gitlab-com/gl-infra/k8s-workloads/gitlab-com!1853 (merged) the shutdown times for puma were adjusted to be much faster, but this hasn't completely eliminated the issue, and it suspected that the instant terminations of the websockets can still be resulting in undesired behaviour in the UI if content was
gitlab-org/charts/gitlab#3367 (comment 1055930809) Workhorse terminating connections is expected and this is fine from the client as they'll simply reconnect decently quickly. But if data was actively being transferred, the client looses this information prematurely. Depending on the data this may present a poor user experience, and as the product puts more items into websockets, this will be increasingly worrisome. There's been decent amount of work done to websockets on the configuration side lately, so I think a new investigation would be beneficial to understand the workflow these days.
Expected Outcome
- Based on a investigation, decide on how we want this to behave
- Determine if we still want/need to add a delay to charts instead to deal with this: gitlab-org/charts/gitlab#3367 (closed)
- Log any additional issues for implementing the solution