diff --git a/doc/gitlab-geo/README.md b/doc/gitlab-geo/README.md index 61782e6eda19a131b70a3f1aa556a44a84a8b047..83535c3ef0930368dbbef296046354aeb0267348 100644 --- a/doc/gitlab-geo/README.md +++ b/doc/gitlab-geo/README.md @@ -7,13 +7,12 @@ We recommend you use with at least GitLab 8.6 EE. GitLab Geo allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational version. - - - - - [Overview](#overview) - [Setup instructions](#setup-instructions) + - [Database Replication](database.md) + - [Configuration](configuration.md) - [Current limitations](#current-limitations) +- [Disaster Recovery](disaster-recovery.md) - [Frequently Asked Questions](#frequently-asked-questions) - [Can I use Geo in a disaster recovery situation?](#can-i-use-geo-in-a-disaster-recovery-situation) - [What data is replicated to a secondary node?](#what-data-is-replicated-to-a-secondary-node) @@ -54,7 +53,7 @@ Geo instances. Follow the steps below in the order that they appear: 1. Install GitLab Enterprise Edition on the server that will serve as the secondary Geo node -1. [Setup a database replication](./database.md) in `primary <-> secondary (read-only)` topology +1. [Setup a database replication](database.md) in `primary <-> secondary (read-only)` topology 1. [Configure GitLab](configuration.md) and set the primary and secondary nodes After you set up the database replication and configure the GitLab Geo nodes, there are a few things to consider: @@ -69,6 +68,7 @@ there are a few things to consider: - You cannot push code to secondary nodes - Git LFS is not supported yet - Git Annex is not supported yet +- Primary node has to be online for OAuth login to happen (existing sessions and git are not affected) ## Frequently Asked Questions @@ -78,6 +78,9 @@ There are limitations to what we replicate (see Current limitations). In an extreme data-loss situation you can make a secondary Geo into your primary, but this is not officially supported yet. +If you still want to proceed, see our step-by-step instructions on how to +manually [promote a secondary node](disaster-recovery.md) into primary. + ### What data is replicated to a secondary node? We currently replicate project repositories and the whole database. This diff --git a/doc/gitlab-geo/configuration.md b/doc/gitlab-geo/configuration.md index 9779e9b5b28cada5318ec6bef9ae91a9c9690292..182965060f8e9d0016dc9c89679599d75ea7dcaf 100644 --- a/doc/gitlab-geo/configuration.md +++ b/doc/gitlab-geo/configuration.md @@ -56,8 +56,13 @@ primary node (**Admin Area > Geo Nodes**) when adding a new one: sudo -u git -H ssh-keygen ``` -The public key for Omnibus installations will be at `/var/opt/gitlab/.ssh/id_rsa.pub`, -whereas for installation from source it will be at `/home/git/.ssh/id_rsa.pub`. +Remember to add your primary node to the `known_hosts` file of your `git` user. + +You can find ssh key files and `know_hosts` at `/var/opt/gitlab/.ssh/` in +Omnibus installations or at `/home/git/.ssh/` when following the source +installation guide. + + If for any reason you generate the key using a different name from the default `id_rsa`, or you want to generate an extra key only for the repository @@ -93,11 +98,11 @@ add any secondary servers as well**. In the following table you can see what all these settings mean: -| Setting | Description | -| ------- | ----------- | -| Primary | This marks a Geo Node as primary. There can be only one primary, make sure that you first add the primary node and then all the others. -| URL | Your instance's full URL, in the same way it is configured in `gitlab.yml` (source based installations) or `/etc/gitlab/gitlab.rb` (omnibus installations). | -|Public Key | The SSH public key of the user that your GitLab instance runs on (unless changed, should be the user `git`). That means that you have to go in each Geo Node separately and create an SSH key pair. See the [SSH key creation](#create-ssh-key-pairs-for-geo-nodes) section. +| Setting | Description | +| --------- | ----------- | +| Primary | This marks a Geo Node as primary. There can be only one primary, make sure that you first add the primary node and then all the others. | +| URL | Your instance's full URL, in the same way it is configured in `gitlab.yml` (source based installations) or `/etc/gitlab/gitlab.rb` (omnibus installations). | +|Public Key | The SSH public key of the user that your GitLab instance runs on (unless changed, should be the user `git`). That means that you have to go in each Geo Node separately and create an SSH key pair. See the [SSH key creation](#create-ssh-key-pairs-for-geo-nodes) section. | First, add your primary node by providing its full URL and the public SSH key you created previously. Make sure to check the box 'This is a primary node' @@ -147,3 +152,37 @@ gitlab-rake gitlab:shell:setup # For source installations sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production ``` + +## Troubleshooting + +Setting up Geo requires careful attention to details and sometimes it's easy to +miss a step. + +Here is a checklist of questions you should ask to try to detect where you have +to fix (all commands and path locations are for Omnibus installs): + +- Is Postgres replication working? +- Are my nodes pointing to the correct database instance? + - You should make sure your primary Geo node points to the instance with + writting permissions. + - Any secondary nodes should point only to read-only instances. +- Can Geo detect my current node correctly? + - Geo uses your defined node from `Admin > Geo` screen, and tries to match + with the value defined in `/etc/gitlab/gitlab.rb` configuration file. + The relevant line looks like: `external_url "http://gitlab.example.com"`. + - To check if node on current machine is correctly detected type: + `sudo gitlab-rails runner "Gitlab::Geo.current_node"`, + expect something like: `#` + - By running the command above, `primary` should be `true` when executed in + the primary node, and `false` on any secondary +- Did I defined the correct SSH Key for the node? + - You must create an SSH Key for `git` user + - This key is the one you have to inform at `Admin > Geo` +- Can primary node communicate with secondary node by HTTP/HTTPS ports? +- Can secondary nodes communicate with primary node by HTTP/HTTPS/SSH ports? +- Can secondary nodes execute a succesfull git clone using git user's own + SSH Key to primary node repository? + +> This list is an atempt to document all the moving parts that can go wrong. +We are working into getting all this steps verified automatically in a +rake task in the future. :) diff --git a/doc/gitlab-geo/database.md b/doc/gitlab-geo/database.md index 7edcd1e3b86feffd32068eb4a2801cbe035a752c..3f5d8d6775e6513844d4cdd6aac3abaf5c83ee2a 100644 --- a/doc/gitlab-geo/database.md +++ b/doc/gitlab-geo/database.md @@ -220,6 +220,4 @@ When prompted, enter the password you set up for the `gitlab_replicator` user. ## MySQL replication -TODO - -[reconfigure gitlab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +We don't support MySQL replication for GitLab Geo. diff --git a/doc/gitlab-geo/disaster-recovery.md b/doc/gitlab-geo/disaster-recovery.md new file mode 100644 index 0000000000000000000000000000000000000000..0c933e5dfd655ac8fb8bb7ddf4dc018cd1a4f9cd --- /dev/null +++ b/doc/gitlab-geo/disaster-recovery.md @@ -0,0 +1,62 @@ +# GitLab Geo Disaster Recovery + +> **Note:** +This is not officially supported yet, please don't use as your only +Disaster Recovery strategy as you may lose data. + +GitLab Geo replicates your database and your Git repositories. We will +support and replicate more data in the future, that will enable you to +fail-over with minimal effort, in a disaster situation. + +See [current limitations](README.md#current-limitations) +for more information. + + +## Promoting a secondary node + +We don't provide yet an automated way to promote a node and do fail-over, +but you can do it manually if you have `root` access to the machine. + +You must make the changes in the exact specific order: + +1. Take down your primary node (or make sure it will not go up during this + process or you may lose data) +2. Wait for any database replication to finish +3. Promote the Postgres in your secondary node as primary +4. Log-in to your secondary node with a user with `sudo` permission +5. Open the interactive rails console: `sudo gitlab-rails console` and execute: + * List your primary node and note down it's id: + + ```ruby + Gitlab::Geo.primary_node + ``` + * Turn your primary into a secondary: + + ```ruby + Gitlab::Geo.primary_node.update(primary: false) + ``` + * List your secondary nodes and note down the id of the one you want to promote: + + ```ruby + Gitlab::Geo.secondary_nodes + ``` + * To promote a node with id `2` execute: + + ```ruby + GeoNode.find(2).update!(primary: true) + ``` + * Now you have to cleanup your new promoted node by running: + + ```ruby + Gitlab::Geo.primary_node.oauth_application.destroy! + Gitlab::Geo.primary_node.system_hook.destroy! + ``` + * And refresh your old primary node to behave correctly as secondary (assuming id is `1`) + + ```ruby + GeoNode.find(1).save! + ``` + * To exit the interactive console, type: `exit` + +6. Rsync everything in `/var/opt/gitlab/gitlab-rails/uploads` and + `/var/opt/gitlab/gitlab-rails/shared` from your old node to the new one.