etcd benchmarks on etcd

etcd v2.1.0-alpha benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

etcd v2.2.0 benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Bootstrap another machine, outside of the etcd cluster, and run the hey HTTP benchmark tool with a connection reuse patch to send requests to each etcd cluster member. See the benchmark instructions for the patch and the steps to reproduce our procedures.

etcd v2.2.0-rc benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

etcd v2.2.0-rc-memory benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
At most cases, etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, you need to decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

etcd v3-demo benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Two components of etcd storage consume physical memory. The etcd process allocates an in-memory index to speed key lookup. The process’s page cache, managed by the operating system, stores recently-accessed data from disk for quick re-use.

The in-memory index holds all the keys in a B-tree data structure, along with pointers to the on-disk data (the values). Each key in the B-tree may contain multiple pointers, pointing to different versions of its values. The theoretical memory consumption of the in-memory index can hence be approximated with the formula:

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: The watch features are under active development, and their memory usage may change as that development progresses. We do not expect it to significantly increase beyond the figures stated below.

A primary goal of etcd is supporting a very large number of watchers doing a massively large amount of watching. etcd aims to support O(10k) clients, O(100K) watch streams (O(10) streams per client) and O(10M) total watchings (O(100) watching per stream). The memory consumed by each individual watching accounts for the largest portion of etcd’s overall usage, and is therefore the focus of current and future optimizations.