Add a smoke test of Duo functionality on a Desktop Editor Extension to the GitLab monolith pipeline

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem to solve

As a developer of the GitLab monolith, when I change an API, I want assurance that I have not degraded any functionality on a GitLab Desktop Editor Extension.

The Editor Extensions depend on APIs within GitLab and the AI Gateway. When these APIs change on .com, the extensions can unexpectedly break.

We're missing an early warning system to tell when a change might affect the Desktop extensions. It is not uncommon that a changes is spotted after .com has been deployed.

Recent examples of issues that affected the extensions and were discovered on .com:

• 2024-12-10: Users unable to use VSCode or JetBr... (gitlab-com/gl-infra/production#18980 - closed)
• gitlab-org/editor-extensions/gitlab-lsp#511 (comment 2252920803)

Existing tests

We have a smoke test for the Web IDE which runs a version of the VS Code Extension, however that differs from the desktop extensions in a number of ways:

• Has a different build process.
• Half-monthly regular updates plus ad-hoc releases, versus monthly for the Web IDE.
• The Web IDE does not use the GitLab Language Server (an internal extension-side component that provides auth, webviews, an API client and more), all the desktop extensions do.
• Differing auth mechanisms, able to access the file system, different network libraries (e.g. sockets, nodejs fetch vs browser fetch).

We have automated e2e tests in VS Code Extension project pipeline here: https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/tree/main/test/e2e. These could be used as a starting point.

Proposal

1. Decide which desktop editor to use.

The main consideration would be ease of automation, and feature coverage. VS Code and JetBrains are the most feature complete.

2. Introduce an e2e smoke test for a desktop extension.

Example using VS Code as the base:

• Add a pipeline that runs on a monolith branch.
  • Grouped with the existing e2e test suite.
• Download and run the latest stable VS Code Extension release.
• Transition period: Allow these new tests to fail. This will let us gather data on stability without impacting GitLab master.
• Add corresponding e2e tests in the VS Code Extension, to ensure minimal chance of the monolith tests breaking due to an extension release.
  • [VSCode] Cover Agentic Chat with E2E tests (gitlab-vscode-extension#1714)
• Consider locking the vscode version (as opposed to always downloading the latest) to reduce the chance that a VS Code release breaks the monolith tests.
  • Set up renovate to update the version - will force the tests to be confirmed passing before each dependency update.

Other info

It would be great to cover off corrective action for 2024-12-10: Users unable to use VSCode or JetBr... (gitlab-com/gl-infra/production#18980 - closed) (comment here), so, if we want to choose an initial smoke test, we could include the use case that a user authenticates with OAuth and successfully streams a Duo Chat message.