v2.3 docs on etcd

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Data Directory

Lifecycle

When first started, etcd stores its configuration into a data directory specified by the data-dir configuration parameter. Configuration is stored in the write ahead log and includes: the local member ID, cluster ID, and initial cluster configuration. The write ahead log and snapshot files are used during member operation and to recover after a restart.

Having a dedicated disk to store wal files can improve the throughput and stabilize the cluster. It is highly recommended to dedicate a wal disk and set --wal-dir to point to a directory on that device for a production cluster deployment.

Authentication Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Authentication -- having users and roles in etcd -- was added in etcd 2.1. This guide will help you set up basic authentication in etcd.

etcd before 2.1 was a completely open system; anyone with access to the API could change keys. In order to preserve backward compatibility and upgradability, this feature is off by default.

For a full discussion of the RESTful API, see the authentication API documentation

Special Users and Roles

There is one special user, root, and there are two special roles, root and guest.

Backward Compatibility

Mon, 01 Jan 0001 00:00:00 +0000

The main goal of etcd 2.0 release is to improve cluster safety around bootstrapping and dynamic reconfiguration. To do this, we deprecated the old error-prone APIs and provide a new set of APIs.

The other main focus of this release was a more reliable Raft implementation, but as this change is internal it should not have any notable effects to users.

Command Line Flags Changes

The major flag changes are to mostly related to bootstrapping. The initial-* flags provide an improved way to specify the required criteria to start the cluster. The advertised URLs now support a list of values instead of a single value, which allows etcd users to gracefully migrate to the new set of IANA-assigned ports (2379/client and 2380/peers) while maintaining backward compatibility with the old ports.

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Starting an etcd cluster statically requires that each member knows another in the cluster. In a number of cases, you might not know the IPs of your cluster members ahead of time. In these cases, you can bootstrap an etcd cluster with the help of a discovery service.

Once an etcd cluster is up and running, adding or removing members is done via runtime reconfiguration. To better understand the design behind runtime reconfiguration, we suggest you read the runtime configuration design document.

Configuration Flags

Mon, 01 Jan 0001 00:00:00 +0000

etcd is configurable through command-line flags and environment variables. Options set on the command line take precedence over those from the environment.

The format of environment variable for flag --my-flag is ETCD_MY_FLAG. It applies to all flags.

The official etcd ports are 2379 for client requests, and 2380 for peer communication. Some legacy code and documentation still references ports 4001 and 7001, but all new etcd use and discussion should adopt the assigned ports.

Design of Runtime Reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two Phase Config Changes Keep you Safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member you need to first inform cluster of new configuration and then start the new member.

Discovery Service Protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

The protocol uses a new discovery token to bootstrap one unique etcd cluster. Remember that one discovery token can represent only one etcd cluster. As long as discovery protocol on this token starts, even if it fails halfway, it must not be used to bootstrap another etcd cluster.

Error Code

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the error code used in key space ‘/v2/keys’. Feel free to import ‘https://github.com/etcd-io/etcd/tree/v2.3.8/error' to use.

It’s categorized into four groups:

Command Related Error

name	code	strerror
EcodeKeyNotFound	100	“Key not found”
EcodeTestFailed	101	“Compare failed”
EcodeNotFile	102	“Not a file”
EcodeNotDir	104	“Not a directory”
EcodeNodeExist	105	“Key already exists”
EcodeRootROnly	107	“Root is read only”
EcodeDirNotEmpty	108	“Directory not empty”

Post Form Related Error

name	code	strerror
EcodePrevValueRequired	201	“PrevValue is Required in POST form”
EcodeTTLNaN	202	“The given TTL in POST form is not a number”
EcodeIndexNaN	203	“The given index in POST form is not a number”
EcodeInvalidField	209	“Invalid field”
EcodeInvalidForm	210	“Invalid POST form”

Raft Related Error

name	code	strerror
EcodeRaftInternal	300	“Raft Internal Error”
EcodeLeaderElect	301	“During Leader Election”

etcd Related Error

name	code	strerror
EcodeWatcherCleared	400	“watcher is cleared due to etcd recovery”
EcodeEventIndexCleared	401	“The event in requested index is outdated and cleared”

etcd API

Mon, 01 Jan 0001 00:00:00 +0000

Running a Single Machine Cluster

These examples will use a single member cluster to show you the basics of the etcd REST API. Let’s start etcd:

./bin/etcd

This will bring up etcd listening on the IANA assigned ports and listening on localhost. The IANA assigned ports for etcd are 2379 for client communication and 2380 for server-to-server communication.

Getting the etcd version

The etcd version of a specific instance can be obtained from the /version endpoint.

etcd v3 API

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The etcd v3 API is designed to give users a more efficient and cleaner abstraction compared to etcd v2. There are a number of semantic and protocol changes in this new API.

To prove out the design of the v3 API the team has also built a number of example recipes, there is a video discussing these recipes too.

Design

Flatten binary key-value space
Keep the event history until compaction
- access to old version of keys
- user controlled history compaction
Support range query

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

1) Why can an etcd client read an old version of data when a majority of the etcd cluster members are down?

In situations where a client connects to a minority, etcd favors by default availability over consistency. This means that even though data might be “out of date”, it is still better to return something versus nothing.

In order to confirm that a read is up to date with a majority of the cluster, the client can use the quorum=true parameter on reads of keys. This means that a majority of the cluster is checked on reads before returning the data, otherwise the read will timeout and fail.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Node

Node is an instance of raft state machine.

It has a unique identification, and records other nodes’ progress internally when it is the leader.

Member

Member is an instance of etcd. It hosts a node, and provides service to clients.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

Libraries and Tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-backup - A powerful command line utility for dumping/restoring etcd - Supports v2
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcd-rest - Create generic REST API in Go using etcd as a backend with validation using JSON schema
etcdsh - A command line client with support of command history and tab completion. Supports v2

Go libraries

Members API

Mon, 01 Jan 0001 00:00:00 +0000

List members

Return an HTTP 200 OK response code and a representation of all members in the etcd cluster.

Request

GET /v2/members HTTP/1.1

Example

curl http://10.0.0.10:2379/v2/members

{
 "members": [
 {
 "id": "272e204152",
 "name": "infra1",
 "peerURLs": [
 "http://10.0.0.10:2380"
 ],
 "clientURLs": [
 "http://10.0.0.10:2379"
 ]
 },
 {
 "id": "2225373f43",
 "name": "infra2",
 "peerURLs": [
 "http://10.0.0.11:2380"
 ],
 "clientURLs": [
 "http://10.0.0.11:2379"
 ]
 },
 ]
}

Add a member

Returns an HTTP 201 response code and the representation of added member with a newly generated a memberID when successful. Returns a string describing the failure condition when unsuccessful.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

The naming of metrics follows the suggested Prometheus best practices. A metric name has an etcd or etcd_debugging prefix as its namespace and a subsystem prefix (for example wal and etcdserver).

Miscellaneous APIs

Mon, 01 Jan 0001 00:00:00 +0000

Getting the etcd version

The etcd version of a specific instance can be obtained from the /version endpoint.

curl -L http://127.0.0.1:2379/version

etcd 2.0.12

Checking health of an etcd member node

etcd provides a /health endpoint to verify the health of a particular member.

curl http://10.0.0.10:2379/health

{"health":"true"}

Production Users

Mon, 01 Jan 0001 00:00:00 +0000

This document tracks people and use cases for etcd in production. By creating a list of production use cases we hope to build a community of advisors that we can reach out to with experience using various etcd applications, operation environments, and cluster sizes. The etcd development team may reach out periodically to check-in on your experience and update this list.

discovery.etcd.io

Application: https://github.com/etcd-io/discovery.etcd.io
Launched: Feb. 2014
Cluster Size: 5 members, 5 discovery proxies
Order of Data Size: 100s of Megabytes
Operator: CoreOS, brandon.philips@coreos.com
Environment: AWS
Backups: Periodic async to S3

discovery.etcd.io is the longest continuously running etcd backed service that we know about. It is the basis of automatic cluster bootstrap and was launched in Feb. 2014: https://coreos.com/blog/etcd-0.3.0-released/.

Proxy

Mon, 01 Jan 0001 00:00:00 +0000

etcd can run as a transparent proxy. Doing so allows for easy discovery of etcd within your infrastructure, since it can run on each machine as a local service. In this mode, etcd acts as a reverse proxy and forwards client requests to an active etcd cluster. The etcd proxy does not participate in the consensus replication of the etcd cluster, thus it neither increases the resilience nor decreases the write performance of the etcd cluster.

Reporting Bugs

Mon, 01 Jan 0001 00:00:00 +0000

If you find bugs or documentation mistakes in the etcd project, please let us know by opening an issue. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.

To make your bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. You can also attach etcd log (the starting log with etcd configuration is especially important).

Running etcd under Docker

Mon, 01 Jan 0001 00:00:00 +0000

The following guide will show you how to run etcd under Docker using the static bootstrap process.

Running etcd in standalone mode

In order to expose the etcd API to clients outside of the Docker host you’ll need use the host IP address when configuring etcd.

export HostIP="192.168.12.50"

The following docker run command will expose the etcd client API over ports 4001 and 2379, and expose the peer port over 2380.

This will run the latest release version of etcd. You can specify version if needed (e.g. quay.io/coreos/etcd:v2.2.0).

Runtime Reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Reconfiguration requests can only be processed when the majority of the cluster members are functioning. It is highly recommended to always have a cluster size greater than two in production. It is unsafe to remove a member from a two member cluster. The majority of a two member cluster is also two. If there is a failure during the removal process, the cluster might not able to make progress and need to restart from majority failure.

Security Model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports SSL/TLS as well as authentication through client certificates, both for clients to server as well as peer (server to server / cluster) communication.

To get up and running you first need to have a CA certificate and a signed key pair for one member. It is recommended to create and sign a new key pair for every member in a cluster.

For convenience, the cfssl tool provides an easy interface to certificate generation, and we provide an example using the tool here. You can also examine this alternative guide to generating self-signed key pairs.

Snapshot Migration

Mon, 01 Jan 0001 00:00:00 +0000

You can migrate a snapshot of your data from a v0.4.9+ cluster into a new etcd 2.2 cluster using a snapshot migration. After snapshot migration, the etcd indexes of your data will change. Many etcd applications rely on these indexes to behave correctly. This operation should only be done while all etcd applications are stopped.

To get started get the newest data snapshot from the 0.4.9+ cluster:

curl http://cluster.example.com:4001/v2/migration/snapshot > backup.snap

Now, import the snapshot into your new cluster:

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

The default settings in etcd should work well for installations on a local network where the average network latency is low. However, when using etcd across multiple data centers or over networks with high latency you may need to tweak the heartbeat interval and election timeout settings.

The network isn’t the only source of latency. Each request and response may be impacted by slow disks on both the leader and follower. Each of these timeouts represents the total time from request to successful response from the other machine.

Upgrade etcd from 2.1 to 2.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.0 to 2.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.0 processes and replace them with etcd v2.1 processes
after you are running all v2.1 processes, new features in v2.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade Checklists

Upgrade Requirements

To upgrade an existing etcd deployment to 2.1, you must be running 2.0. If you’re running a version of etcd before 2.0, you must upgrade to 2.0 before upgrading to 2.1.

Upgrade etcd from 2.1 to 2.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.1 to 2.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.1 processes and replace them with etcd v2.2 processes
after you are running all v2.2 processes, new features in v2.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade Checklists

Upgrade Requirement

To upgrade an existing etcd deployment to 2.2, you must be running 2.1. If you’re running a version of etcd before 2.1, you must upgrade to 2.1 before upgrading to 2.2.

Upgrade etcd from 2.2 to 2.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.2 to 2.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.2 processes and replace them with etcd v2.3 processes
after running all v2.3 processes, new features in v2.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade Checklists

Upgrade Requirements

To upgrade an existing etcd deployment to 2.3, the running cluster must be 2.2 or greater. If it’s before 2.2, please upgrade to 2.2 before upgrading to 2.3.

v2 Auth and Security

Mon, 01 Jan 0001 00:00:00 +0000

etcd Resources

There are three types of resources in etcd

permission resources: users and roles in the user store
key-value resources: key-value pairs in the key-value store
settings resources: security settings, auth settings, and dynamic etcd cluster settings (election/heartbeat)

Permission Resources

Users

A user is an identity to be authenticated. Each user can have multiple roles. The user has a capability (such as reading or writing) on the resource if one of the roles has that capability.

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

Goal: We want to be able to upgrade an individual peer in an etcd cluster to a newer version of etcd. The process will take the form of individual followers upgrading to the latest version until the entire cluster is on the new version.

Immediate need: etcd is moving too fast to version the internal API right now. But, we need to keep mixed version clusters from being started by a rolling upgrade process (e.g. the CoreOS developer alpha).