Convert existing Workspaces internal script logic to postStart hooks (!182354) · Merge requests · GitLab.org / GitLab

Issue: Convert existing Workspaces internal script log... (#520867 - closed) • Chad Woolley • 18.0

What does this MR do and why?

See Convert existing Workspaces internal script log... (#520867 - closed) for details.

References

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Example of exec into workspace and view poststart logs

gitlab (caw-ws-refactor-component-logic-7 *=)$ kubens gl-rd-ns-4-1-0oqdp7
Context "rancher-desktop" modified.
Active namespace is "gl-rd-ns-4-1-0oqdp7".
gitlab (caw-ws-refactor-component-logic-7 *=)$ PODNAME=$(kubectl get po -o name | cut -d/ -f2) && CONTAINER_NAME=$(kubectl get pod $PODNAME -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{range .env[*]}{.name}{","}{end}{"\n"}{end}' | grep GL_TOOLS_DIR | cut -f 1) && kubectl exec $PODNAME -c $CONTAINER_NAME -it -- /bin/bash
gitlab-workspaces@workspace-4-1-0oqdp7-78c8b8675b-r4nfh:~$ cd /tmp/workspace-logs
gitlab-workspaces@workspace-4-1-0oqdp7-78c8b8675b-r4nfh:/tmp/workspace-logs$ tail -n1000 poststart-stdout.log
2025-04-22T19:41:52+00:00: Running /workspace-scripts/gl-start-sshd-command...
2025-04-22T19:41:52+00:00: ----------------------------------------
2025-04-22T19:41:52+00:00: Starting sshd if it is found...
2025-04-22T19:41:52+00:00: Starting /usr/sbin/sshd on port 60022 with output written to /projects/workspace-logs/start-sshd.log
2025-04-22T19:41:52+00:00: Finished starting sshd if it is found.
2025-04-22T19:41:52+00:00: Running /workspace-scripts/gl-init-tools-command...
2025-04-22T19:41:52+00:00: ----------------------------------------
2025-04-22T19:41:52+00:00: Running /projects/.gl-tools/init_tools.sh with output written to /projects/workspace-logs/init-tools.log...
2025-04-22T19:41:53+00:00: Finished running /projects/.gl-tools/init_tools.sh.
2025-04-22T19:41:53+00:00: Running /workspace-scripts/gl-sleep-until-container-is-running-command...
2025-04-22T19:41:53+00:00: ----------------------------------------
2025-04-22T19:41:53+00:00: Sleeping until workspace is running...
2025-04-22T19:41:53+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:41:58+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:03+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:08+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:13+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:18+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:23+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:28+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:33+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:38+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:43+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:48+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:53+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:42:58+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:43:03+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:43:09+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:43:14+00:00: Workspace state is 'Starting' from status file '/.workspace-data/variables/file/gl_workspace_reconciled_actual_state.txt'. Blocking remaining postStart events execution for 5 seconds until state is 'Running'...
2025-04-22T19:43:19+00:00: Workspace state is now 'Running', continuing postStart hook execution.
2025-04-22T19:43:19+00:00: Finished sleeping until workspace is running.

How to set up and validate locally

Start a workspace
Change to the workspace's namespace via kubens
Exec into the workspace once the tooling-container starts: PODNAME=$(kubectl get po -o name | cut -d/ -f2) && CONTAINER_NAME=$(kubectl get pod $PODNAME -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{range .env[*]}{.name}{","}{end}{"\n"}{end}' | grep GL_TOOLS_DIR | cut -f 1) && kubectl exec $PODNAME -c $CONTAINER_NAME -it -- /bin/bash
View /tmp/workspace-logs/poststart-stdout.log
Modify one of the poststart scripts to print some output to stderr
Start a new workspace and exec into its container using the command above
View /tmp/workspace-logs/poststart-stderr.log, see that it contains your output to stderr

Test with multi-container

Example local (MacOS) devfile:

schemaVersion: 2.2.0
components:
  - name: tooling-container
    attributes:
      gl/inject-editor: true
    container:
      image: registry.gitlab.com/gitlab-org/workspaces/testing/example-sshd-http-app:latest-amd64
      #endpoints:
      #  - name: http-8000
      #    targetPort: 8000
  - name: second-container
    container:
      image: registry.gitlab.com/gitlab-org/workspaces/testing/example-sshd-http-app:latest-amd64
      endpoints:
        - name: http-8000
          targetPort: 8000

See that this container does not have startup scripts run (logs dir not symlinked into /tmp):

PODNAME=$(kubectl get po -o name | cut -d/ -f2) && CONTAINER_NAME=$(kubectl get pod $PODNAME -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{range .env[*]}{.name}{","}{end}{"\n"}{end}' | grep second-container | cut -f 1) && kubectl exec $PODNAME -c $CONTAINER_NAME -it -- /bin/bash

Test with main container with no entrypoint

Use registry.gitlab.com/gitlab-org/workspaces/testing/example-sshd-http-app:latest-amd64-no-entrypoint-or-command
Prove that the container has no entrypoint and cmd:

$  docker inspect --format="ENTRYPOINT: {{.Config.Entrypoint}}\nCMD: {{.Config.Cmd}}" registry.gitlab.com/gitlab-org/workspaces/testing/example-sshd-http-app:latest-amd64-no-entrypoint-or-command
ENTRYPOINT: []\nCMD: []

Use it in a devfile to start a workspace:

schemaVersion: 2.2.0
components:
  - name: tooling-container
    attributes:
      gl/inject-editor: true
    container:
      image: registry.gitlab.com/gitlab-org/workspaces/testing/example-sshd-http-app:latest-amd64-no-entrypoint-or-command

Verify workspace starts successfully and stays running, and has correct poststart logs using commands shown above.

Test with secondary container with no entrypoint (should fail)

Use same no-entrypoint-no-cmd container as above for both primary and secondary containers in devfile:

schemaVersion: 2.2.0
components:
  - name: tooling-container
    attributes:
      gl/inject-editor: true
    container:
      image: registry.gitlab.com/gitlab-org/workspaces/testing/example-sshd-http-app:latest
  - name: second-container
    container:
      image: registry.gitlab.com/gitlab-org/workspaces/testing/example-sshd-http-app:latest

Should fail with this in pod events:

Warning  Failed            4s                kubelet            Error: failed to generate container "6cd890f3e09a72d97017e15e339b4f157dbd833181beb6499542a601c12d9390" spec: failed to generate spec: no command specified

Rollout plan

Verify in production
Monitor Sentry
- https://new-sentry.gitlab.net/organizations/gitlab/issues/ - is:unresolved feature_category:workspaces
Monitor logs for errors: https://log.gprd.gitlab.net/app/r/s/qXhmR

Edited May 07, 2025 by Chad Woolley

Convert existing Workspaces internal script logic to postStart hooks