From 865c80b0fe56291f313d8f58db0073e9c946f94e Mon Sep 17 00:00:00 2001 From: Sean Carroll Date: Thu, 30 Sep 2021 14:20:40 +0200 Subject: [PATCH 1/3] reorganise README --- README.md | 76 +++++++++++++++++++++++++-------------------- doc/architecture.md | 31 ++++++++++++++++++ doc/gitlab-sshd.md | 29 +++++++++++++++++ 3 files changed, 103 insertions(+), 33 deletions(-) create mode 100644 doc/architecture.md create mode 100644 doc/gitlab-sshd.md diff --git a/README.md b/README.md index a45b30d7..75374f51 100644 --- a/README.md +++ b/README.md @@ -1,28 +1,49 @@ +[![pipeline status](https://gitlab.com/gitlab-org/gitlab-shell/badges/main/pipeline.svg)](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main) +[![coverage report](https://gitlab.com/gitlab-org/gitlab-shell/badges/main/coverage.svg)](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main) +[![Code Climate](https://codeclimate.com/github/gitlabhq/gitlab-shell.svg)](https://codeclimate.com/github/gitlabhq/gitlab-shell) + # GitLab Shell -## GitLab Shell handles git SSH sessions for GitLab +GitLab Shell is a SSH server, configured to handing git SSH sessions for GitLab. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh. + +The purpose of GitLab Shell is to limit shell access to specific `git` commands, and provide authorization and transport for these commands. + -GitLab Shell handles git SSH sessions for GitLab and modifies the list of authorized keys. -GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh. +## Handling `git` SSH sessions When you access the GitLab server over SSH then GitLab Shell will: -1. Limit you to predefined git commands (git push, git pull). +1. Limit you to `git push` and `git pull`, `git fetch` commands only 1. Call the GitLab Rails API to check if you are authorized, and what Gitaly server your repository is on 1. Copy data back and forth between the SSH client and the Gitaly server -If you access a GitLab server over HTTP(S) you end up in [gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). +### `git pull` over SSH -An overview of the four cases described above: +git pull over SSH -> gitlab-shell -> API call to gitlab-rails (Authorization) -> accept or decline -> establish Gitaly session -1. git pull over SSH -> gitlab-shell -> API call to gitlab-rails (Authorization) -> accept or decline -> establish Gitaly session -1. git push over SSH -> gitlab-shell (git command is not executed yet) -> establish Gitaly session -> (in Gitaly) gitlab-shell pre-receive hook -> API call to gitlab-rails (authorization) -> accept or decline push +### `git push` over SSH -## Code status +git push over SSH -> gitlab-shell (git command is not executed yet) -> establish Gitaly session -> (in Gitaly) gitlab-shell pre-receive hook -> API call to gitlab-rails (authorization) -> accept or decline push -[![pipeline status](https://gitlab.com/gitlab-org/gitlab-shell/badges/main/pipeline.svg)](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main) -[![coverage report](https://gitlab.com/gitlab-org/gitlab-shell/badges/main/coverage.svg)](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main) -[![Code Climate](https://codeclimate.com/github/gitlabhq/gitlab-shell.svg)](https://codeclimate.com/github/gitlabhq/gitlab-shell) +For more details see [Architecture](doc/architecture.md) + +### Modifies `authorized_keys` + +GitLab Shell modifies the `authorized_keys` file on the client machine. + +- TODO some details needed here. + +### Runs on Port 22 + +GitLab Shell runs on `port 22` on an Omnibus installation. A "regular" SSH service would need to be configured to run on an alternative port. + +### Accessing GitLab using `https` + +If you access a GitLab server over HTTP(S) you end up in [gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). + +## `gitlab-sshd` + +See [gitlab-sshd](doc/gitlab-sshd) ## Requirements @@ -34,19 +55,19 @@ Download and install the current version of Go from https://golang.org/dl/ We follow the [Golang Release Policy](https://golang.org/doc/devel/release.html#policy) of supporting the current stable version and the previous two major versions. -## Check +### Check Checks if GitLab API access and redis via internal API can be reached: make check -## Compile +### Compile Builds the `gitlab-shell` binaries, placing them into `bin/`. make compile -## Install +### Install Builds the `gitlab-shell` binaries and installs them onto the filesystem. The default location is `/usr/local`, but can be controlled by use of the `PREFIX` @@ -54,7 +75,7 @@ and `DESTDIR` environment variables. make install -## Setup +### Setup This command is intended for use when installing GitLab from source on a single machine. In addition to compiling the gitlab-shell binaries, it ensures that @@ -64,14 +85,14 @@ it unless instructed to by your installation method documentation. make setup -## Testing +### Testing Run tests: bundle install make test -Run gofmt: +Run `gofmt`: make verify @@ -94,24 +115,13 @@ If no `GITALY_CONNECTION_INFO` is set, the test suite will still run, but any tests requiring Gitaly will be skipped. They will always run in the CI environment. -## Git LFS remark +## References -Starting with GitLab 8.12, GitLab supports Git LFS authentication through SSH. +- [Using the GitLab Shell chart](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/#using-the-gitlab-shell-chart) -## Logging Guidelines - -In general, it should be possible to determine the structure, but not content, -of a gitlab-shell or gitlab-sshd session just from inspecting the logs. Some -guidelines: +## Git LFS remark -- We use [`gitlab.com/gitlab-org/labkit/log`](https://pkg.go.dev/gitlab.com/gitlab-org/labkit/log) - for logging functionality -- **Always** include a correlation ID -- Log messages should be invariant and unique. Include accessory information in - fields, using `log.WithField`, `log.WithFields`, or `log.WithError`. -- Log success cases as well as error cases -- Logging too much is better than not logging enough. If a message seems too - verbose, consider reducing the log level before removing the message. +GitLab supports Git LFS authentication through SSH. ## Releasing diff --git a/doc/architecture.md b/doc/architecture.md new file mode 100644 index 00000000..78f60d56 --- /dev/null +++ b/doc/architecture.md @@ -0,0 +1,31 @@ +# Architecture + +```mermaid +sequenceDiagram + participant Git on client + participant SSH server + participant AuthorizedKeysCommand + participant GitLab Shell + participant Rails + participant Gitaly + participant Git on server + + Note left of Git on client: git fetch + Git on client->>+SSH server: ssh git fetch-pack request + SSH server->>+AuthorizedKeysCommand: gitlab-shell-authorized-keys-check git AAAA... + AuthorizedKeysCommand->>+Rails: GET /internal/api/authorized_keys?key=AAAA... + Note right of Rails: Lookup key ID + Rails-->>-AuthorizedKeysCommand: 200 OK, command="gitlab-shell upload-pack key_id=1" + AuthorizedKeysCommand-->>-SSH server: command="gitlab-shell upload-pack key_id=1" + SSH server->>+GitLab Shell: gitlab-shell upload-pack key_id=1 + GitLab Shell->>+Rails: GET /internal/api/allowed?action=upload_pack&key_id=1 + Note right of Rails: Auth check + Rails-->>-GitLab Shell: 200 OK, { gitaly: ... } + GitLab Shell->>+Gitaly: SSHService.SSHUploadPack request + Gitaly->>+Git on server: git upload-pack request + Note over Git on client,Git on server: Bidirectional communication between Git client and server + Git on server-->>-Gitaly: git upload-pack response + Gitaly -->>-GitLab Shell: SSHService.SSHUploadPack response + GitLab Shell-->>-SSH server: gitlab-shell upload-pack response + SSH server-->>-Git on client: ssh git fetch-pack response +``` diff --git a/doc/gitlab-sshd.md b/doc/gitlab-sshd.md new file mode 100644 index 00000000..f4d8175f --- /dev/null +++ b/doc/gitlab-sshd.md @@ -0,0 +1,29 @@ +# gitlab-sshd + +`gitlab-sshd` is a binary in https://gitlab.com/gitlab-org/gitlab-shell which runs as a persistent SSH daemon. It will replace `OpenSSH` on GitLab SaaS and eventually other cloud-native environments. Instead of running an `sshd` process, we would run a `gitlab-sshd` process that does the same job, in a more focused manner. + +## Architecture + +```mermaid +sequenceDiagram + participant Git on client + participant GitLab SSHD + participant Rails + participant Gitaly + participant Git on server + + Note left of Git on client: git fetch + Git on client->>+GitLab SSHD: ssh git fetch-pack request + GitLab SSHD->>+Rails: GET /internal/api/authorized_keys?key=AAAA... + Note right of Rails: Lookup key ID + Rails-->>-GitLab SSHD: 200 OK, command="gitlab-shell upload-pack key_id=1" + GitLab SSHD->>+Rails: GET /internal/api/allowed?action=upload_pack&key_id=1 + Note right of Rails: Auth check + Rails-->>-GitLab SSHD: 200 OK, { gitaly: ... } + GitLab SSHD->>+Gitaly: SSHService.SSHUploadPack request + Gitaly->>+Git on server: git upload-pack request + Note over Git on client,Git on server: Bidirectional communication between Git client and server + Git on server-->>-Gitaly: git upload-pack response + Gitaly -->>-GitLab SSHD: SSHService.SSHUploadPack response + GitLab SSHD-->>-Git on client: ssh git fetch-pack response +``` \ No newline at end of file -- GitLab From 0bd9ce8c5de962f17d30e02b10ea375f4b6ebbba Mon Sep 17 00:00:00 2001 From: Amy Qualls Date: Wed, 9 Feb 2022 06:38:39 -0800 Subject: [PATCH 2/3] Minor revisions to SSHD page Line wraps, present tense, and removing a subheading we don't need. --- doc/gitlab-sshd.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/gitlab-sshd.md b/doc/gitlab-sshd.md index f4d8175f..77100b6a 100644 --- a/doc/gitlab-sshd.md +++ b/doc/gitlab-sshd.md @@ -1,8 +1,9 @@ # gitlab-sshd -`gitlab-sshd` is a binary in https://gitlab.com/gitlab-org/gitlab-shell which runs as a persistent SSH daemon. It will replace `OpenSSH` on GitLab SaaS and eventually other cloud-native environments. Instead of running an `sshd` process, we would run a `gitlab-sshd` process that does the same job, in a more focused manner. - -## Architecture +`gitlab-sshd` is a binary in [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) +which runs as a persistent SSH daemon. It will replace `OpenSSH` on GitLab SaaS, +and eventually other cloud-native environments. Instead of running an `sshd` process, +we run a `gitlab-sshd` process that does the same job, in a more focused manner: ```mermaid sequenceDiagram @@ -26,4 +27,4 @@ sequenceDiagram Git on server-->>-Gitaly: git upload-pack response Gitaly -->>-GitLab SSHD: SSHService.SSHUploadPack response GitLab SSHD-->>-Git on client: ssh git fetch-pack response -``` \ No newline at end of file +``` -- GitLab From ad1d7536ceed5d34c651aa145709fb3c268c8060 Mon Sep 17 00:00:00 2001 From: Amy Qualls Date: Wed, 9 Feb 2022 06:39:04 -0800 Subject: [PATCH 3/3] Major README revisions for tone and style Shift commands from one-per-subheading to an unordered list, which is easier to scan. Line wraps. Capitalization, spelling, word use. Move the requirements section closer to the top of the page; we shouldn't bury this information as the page gets longer. --- README.md | 125 +++++++++++++++++++++++++----------------------------- 1 file changed, 58 insertions(+), 67 deletions(-) diff --git a/README.md b/README.md index 75374f51..a37cd96f 100644 --- a/README.md +++ b/README.md @@ -4,26 +4,37 @@ # GitLab Shell -GitLab Shell is a SSH server, configured to handing git SSH sessions for GitLab. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh. +GitLab Shell is a SSH server, configured to handle Git SSH sessions for GitLab. +GitLab Shell is not a Unix shell, nor a replacement for Bash or Zsh: -The purpose of GitLab Shell is to limit shell access to specific `git` commands, and provide authorization and transport for these commands. +- It limits shell access to specific `git` commands. +- It provides authorization and transport for these commands. +## Requirements + +GitLab Shell is written in Go, and needs a Go compiler to build. It still requires +Ruby to build and test, but not to run. + +Download and install the current version of Go from [golang.org](https://golang.org/dl/) + +We follow the [Golang Release Policy](https://golang.org/doc/devel/release.html#policy) +of supporting the current stable version and the previous two major versions. ## Handling `git` SSH sessions -When you access the GitLab server over SSH then GitLab Shell will: +When you access the GitLab server over SSH, GitLab Shell: -1. Limit you to `git push` and `git pull`, `git fetch` commands only -1. Call the GitLab Rails API to check if you are authorized, and what Gitaly server your repository is on -1. Copy data back and forth between the SSH client and the Gitaly server +1. Limits you to `git push`, `git pull`, and `git fetch` commands only. +1. Calls the GitLab Rails API to check if you are authorized, and what Gitaly server your repository is on. +1. Copies data back and forth between the SSH client and the Gitaly server. ### `git pull` over SSH -git pull over SSH -> gitlab-shell -> API call to gitlab-rails (Authorization) -> accept or decline -> establish Gitaly session +Git pull over SSH -> gitlab-shell -> API call to gitlab-rails (Authorization) -> accept or decline -> establish Gitaly session ### `git push` over SSH -git push over SSH -> gitlab-shell (git command is not executed yet) -> establish Gitaly session -> (in Gitaly) gitlab-shell pre-receive hook -> API call to gitlab-rails (authorization) -> accept or decline push +Git push over SSH -> gitlab-shell (git command is not executed yet) -> establish Gitaly session -> (in Gitaly) gitlab-shell pre-receive hook -> API call to gitlab-rails (authorization) -> accept or decline push For more details see [Architecture](doc/architecture.md) @@ -33,87 +44,67 @@ GitLab Shell modifies the `authorized_keys` file on the client machine. - TODO some details needed here. -### Runs on Port 22 +### Runs on port 22 -GitLab Shell runs on `port 22` on an Omnibus installation. A "regular" SSH service would need to be configured to run on an alternative port. +GitLab Shell runs on `port 22` on an Omnibus installation. To use a regular SSH +service, configure it on an alternative port. -### Accessing GitLab using `https` +### Access GitLab with `https` -If you access a GitLab server over HTTP(S) you end up in [gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). +If you access a GitLab server over HTTP(S) you end up in +[`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). ## `gitlab-sshd` -See [gitlab-sshd](doc/gitlab-sshd) - -## Requirements - -GitLab Shell is written in Go, and needs a Go compiler to build. It still requires -Ruby to build and test, but not to run. - -Download and install the current version of Go from https://golang.org/dl/ - -We follow the [Golang Release Policy](https://golang.org/doc/devel/release.html#policy) -of supporting the current stable version and the previous two major versions. - -### Check - -Checks if GitLab API access and redis via internal API can be reached: - - make check - -### Compile - -Builds the `gitlab-shell` binaries, placing them into `bin/`. - - make compile - -### Install - -Builds the `gitlab-shell` binaries and installs them onto the filesystem. The -default location is `/usr/local`, but can be controlled by use of the `PREFIX` -and `DESTDIR` environment variables. - - make install - -### Setup - -This command is intended for use when installing GitLab from source on a single -machine. In addition to compiling the gitlab-shell binaries, it ensures that -various paths on the filesystem exist with the correct permissions. Do not run -it unless instructed to by your installation method documentation. +See [`gitlab-sshd`](doc/gitlab-sshd). - make setup +## Commands +- `make check`: Checks if GitLab API access and Redis (via internal API) can be reached +- `make compile`: Builds the `gitlab-shell` binaries, placing them into `bin/`. +- `make install`: Builds the `gitlab-shell` binaries and installs them onto the + file system. The default location is `/usr/local`, but you can change it with the `PREFIX` + and `DESTDIR` environment variables. +- `make setup`: Don't run this command unless instructed to by your installation method + documentation. Used when installing GitLab from source on a single machine. Compiles + the `gitlab-shell` binaries, and ensures that file system paths exist and contain + correct permissions. ### Testing Run tests: - bundle install - make test +```shell +bundle install +make test +``` Run `gofmt`: - make verify +```shell +make verify +``` Run both test and verify (the default Makefile target): - bundle install - make validate +```shell +bundle install +make validate +``` ### Gitaly Some tests need a Gitaly server. The -[`docker-compose.yml`](./docker-compose.yml) file will run Gitaly on -port 8075. To tell the tests where Gitaly is, set -`GITALY_CONNECTION_INFO`: +[`docker-compose.yml`](docker-compose.yml) file runs Gitaly on port 8075. +To tell the tests the location of Gitaly, set `GITALY_CONNECTION_INFO`: - export GITALY_CONNECTION_INFO='{"address": "tcp://localhost:8075", "storage": "default"}' - make test +```plaintext +export GITALY_CONNECTION_INFO='{"address": "tcp://localhost:8075", "storage": "default"}' +make test +``` -If no `GITALY_CONNECTION_INFO` is set, the test suite will still run, but any -tests requiring Gitaly will be skipped. They will always run in the CI -environment. +If no `GITALY_CONNECTION_INFO` is set, the test suite still runs, but any +tests requiring Gitaly are skipped. These tests always run in the CI environment. ## References @@ -125,12 +116,12 @@ GitLab supports Git LFS authentication through SSH. ## Releasing -See [PROCESS.md](./PROCESS.md) +See [PROCESS.md](PROCESS.md) ## Contributing -See [CONTRIBUTING.md](./CONTRIBUTING.md). +See [CONTRIBUTING.md](CONTRIBUTING.md). ## License -See [LICENSE](./LICENSE). +See [LICENSE](LICENSE). -- GitLab