Investigate into logging architecture for workspace
MR: Pending
Description
Update 13th Aug
After the investigation issue for workspace events, we have identified below 5 aspects below.
- Sending data reliably from Agentk to Rails.
- Rails processing this data and generating GitLab Events data.
- GitLab Events data getting persisted in the DB.
- Backend APIs for fetching the persisted data from the DB.
- Frontend integration
The current plan is to analyze design and implementation for top 3 points first. Then look into the Rails API and Frontend.
Initial description
After some investigation, we have had thoughts on how industry implement logging from workspace to its UI. In this issue we will explore and decide what should be the architecture of workspace loggings.
For logs, we generally category them into three types,
- workspace building/creation logs
- workspace agent logs when workspace is created successfully
- audit logs is for user actions, and are generally saved into DB (we have some issues for this, but this is quite unrelated to our current epic)
Our major focus for the current epic should be 1 (or maybe 2 as well?).
Some rough thoughts so far,
- Logs could be structured entities from our scripts (like,
fetching projects from ...,SSH is ready), and also unstructured if we would like to expose tools logs as well (like git clone logs from git command). - Those logs should be stored into persistent storage in k8s, and not be persistent in GitLab backend DB.
- We could use GitLab backend as a bridge (for example when we have workspace from GitLab runner and k8s), and directly expose API from workspace, for getting log contents.
- Web Socket and long-lived HTTPS is both used in some cases.
Will dig deeper and try bring up some solutions for discussions.
References
Acceptance criteria
TODO: Fill out (required)
-
[Describe what must be achieved to complete this issue.] -
[If applicable, please provide design specifications for this feature/enhancement.] -
[If applicable, please list any technical requirements (performance, security, database, etc.)]
Implementation plan
TODO: Fill out or delete (optional)
[Provide a high-level plan for implementation of this issue, including relevant technical and/or design details.]