Testing in Go: philosophy and tools

The Go programming language comes with tools for writing and running tests: the standard library's testing package, and the go test command to run test suites. Like the language itself, Go's philosophy for writing tests is minimalist: use the lightweight testing package along with helper functions written in plain Go. The idea is that tests are just code, and since a Go developer already knows how to write Go using its abstractions and types, there's no need to learn a quirky domain-specific language for writing tests.

The Go Programming Language by Brian Kernighan and Alan Donovan summarizes this philosophy. From chapter 11 on testing:

To see this in practice, here's a simple test of the absolute value function Abs() using the testing package and plain Go:

    func TestAbs(t *testing.T) {
        got := Abs(-1)
        if got != 1 {
            t.Errorf("Abs(-1) = %d; want 1", got)
        }
    }

Contrast that with the following version, written using the popular (though I would argue non-idiomatic) Ginkgo library that provides a means to write RSpec-style tests for Go:

    Describe("Abs", func() {
        It("returns correct abs value for -1", func() {
            got := Abs(-1)
            Expect(got).To(Equal(1))
        })
    })

The functions Describe, Expect, etc, make the test "read like English", but means that there is suddenly a whole new sub-language to learn. The thinking of Go contributors such as Donovan is that there are already tools like == and != built into the language, so why is To(Equal(x)) needed?

That said, Go doesn't stop developers from using such libraries, so developers coming from other languages often find using them more familiar than vanilla testing. One relatively lightweight library is testify/assert, which adds common assertion functions like assert.Equal(), and testify/suite, which adds test-suite utilities like setup and teardown. The "Awesome Go" website provides an extensive list of such third-party packages.

One useful testing tool that's not part of the testing package is reflect.DeepEqual(), which is a standard library function that uses reflection to determine "deep equality", that is, equality after following pointers and recursing into maps, arrays, and so on. This is helpful when tests compare things like JSON objects or structs with pointers in them. Two libraries that build on this are Google's go-cmp package and Daniel Nichter's deep, which are like DeepEqual but produce a human-readable diff of what's not equal rather just returning a boolean. For example, here's a (deliberately broken) test of a MakeUsers() function using go-cmp:

    func TestMakeUser(t *testing.T) {
        got := MakeUser("Bob Smith", "bobby@example.com", 42)
        want := &User{
            Name:  "Bob Smith",
            Email: "bob@example.com",
            Age:   42,
        }
        if diff := cmp.Diff(want, got); diff != "" {
            t.Errorf("MakeUser() mismatch (-want +got):\n%s", diff)
        }
    }

And the human-readable output is:

    user_test.go:16: MakeUser() mismatch (-want +got):
          &main.User{
            Name:  "Bob Smith",
        -   Email: "bob@example.com",
        +   Email: "bobby@example.com",
            Age:   42,
          }

Built-in testing features

The built-in testing package contains various functions to log information and report failures, skip tests at runtime, or only run tests in "short" mode. Short mode provides a way to skip tests that are long running or have a lot of setup, which can be helpful during development. It is enabled using the -test.short command line argument.

Go's test runner executes tests sequentially by default, but there's an opt-in Parallel() function to allow running explicitly-marked tests at the same time across multiple cores.

In Go 1.14, the testing package added a Cleanup() function that registers a function to be called when the test completes. This is a built-in way to simplify teardown, for example to delete database tables after a test finishes:

    func createDatabase(t *testing.T) {
        // ... code to create a test database
        t.Cleanup(func() {
            // ... code to delete the test database
            // runs when the test finishes (success or failure)
        })
    }

    func TestFetchUser(t *testing.T) {
        createDatabase(t) // creates database and registers cleanup
        user, err := FetchUser("bob@example.com")
        if err != nil {
            t.Fatalf("error fetching user: %v", err)
        }
        expected := &User{"Bob Smith", "bob@example.com", 42}
        if !reflect.DeepEqual(user, expected) {
            t.Fatalf("expected user %v, got %v", expected, user)
        }
    }

Go 1.15 is adding a test helper, TempDir(), that creates (and cleans up) a temporary directory for the current test. There's a high bar for adding to the testing package, but Russ Cox on the core Go team gave his approval for this addition: "It seems like temporary directories do come up in a large enough variety of tests to be part of testing proper."

Table-driven tests

A common idiom in Go to avoid repetition when testing various edge cases is called "table-driven tests". This technique iterates over the test cases in a "slice" (Go's term for a view into a resizable array), reporting any failures for each iteration:

    func TestAbs(t *testing.T) {
        tests := []struct {
            input    int
            expected int
        }{
            {1, 1},
            {0, 0},
            {-1, 1},
            {-maxInt, maxInt},
            {maxInt, maxInt},
        }
        for _, test := range tests {
            actual := Abs(test.input)
            if actual != test.expected {
                t.Errorf("Abs(%d) = %d; want %d", test.input, actual, test.expected)
            }
        }
    }

The t.Errorf() calls report the failure but do not stop the execution of the test, so multiple failures can be reported. This style of table-driven test is common throughout the standard library tests (for example, the fmt tests). Subtests, a feature introduced in Go 1.7, gives the ability to run individual sub-tests from the command line, as well as better control over failures and parallelism.

Mocks and interfaces

One of Go's well-known language features is its structurally-typed interfaces, sometimes referred to as "compile-time duck typing". "Interfaces in Go provide a way to specify the behavior of an object: if something can do this, then it can be used here." Interfaces are important whenever there is a need to vary behavior at runtime, which of course includes testing. For example, as Go core contributor Andrew Gerrand said in the slides for his 2014 "Testing Techniques" talk, a file-format parser should not have a concrete file type passed in like this:

    func Parse(f *os.File) error { ... }

Instead, Parse() should simply take a small interface that only implements the functionality needed. In cases like this, the ubiquitous io.Reader is a good choice:

    func Parse(r io.Reader) error { ... }

That way, the parser can be fed anything that implements io.Reader, which includes files, string buffers, and network connections. It also makes it much easier to test (probably using a strings.Reader).

If the tests only use a small part of a large interface, for example one method from a multi-method API, a new struct type can be created that embeds the interface to fulfill the API contract, and only overrides the method being called. A full example of this technique is shown in this Go Playground code.

There are various third party tools, such as GoMock and mockery, that autogenerate mock code from interface definitions. However, Gerrand prefers hand-written fakes:

Testable examples

Go's package documentation is generated from comments in the source code. Unlike Javadoc or C#'s documentation system, which make heavy use of markup in code comments, Go's approach is that comments in source code should still be readable in the source, and not sprinkled with markup.

It takes a similar approach with documentation examples: these are runnable code snippets that are automatically executed when the tests are run, and then included in the generated documentation. Much like Python's doctests, testable examples write to standard output, and the output is compared against the expected output, to avoid regressions in the documented examples. Here's a testable example of an Abs() function:

    func ExampleAbs() {
        fmt.Println(Abs(5))
        fmt.Println(Abs(-42))
        // Output:
        // 5
        // 42
    }

Example functions need to be in a *_test.go file and prefixed with Example. When the test runner executes, the Output: comment is parsed and compared against the actual output, giving a test failure if they differ. These examples are included in the generated documentation as runnable Go Playground snippets, as shown in the strings package, for example.

Benchmarking

In addition to tests, the testing package allows you to run timed benchmarks. These are used heavily throughout the standard library to ensure there are not regressions in execution speed. Benchmarks can be run automatically using go test with the -bench= option. Popular Go author Dave Cheney has a good summary in his article "How to write benchmarks in Go".

As an example, here's the standard library's benchmark for the strings.TrimSpace() function (note the table-driven approach and the use of b.Run() to create sub-benchmarks):

    func BenchmarkTrimSpace(b *testing.B) {
        tests := []struct{ name, input string }{
            {"NoTrim", "typical"},
            {"ASCII", "  foo bar  "},
            {"SomeNonASCII", "    \u2000\t\r\n x\t\t\r\r\ny\n \u3000    "},
            {"JustNonASCII", "\u2000\u2000\u2000☺☺☺☺\u3000\u3000\u3000"},
        }
        for _, test := range tests {
            b.Run(test.name, func(b *testing.B) {
                for i := 0; i < b.N; i++ {
                    TrimSpace(test.input)
                }
            })
        }
    }

The go test tool will report the numbers; a program like benchstat can be used to compare the before and after timings. Output from benchstat is commonly included in Go's commit messages showing the performance improvement. For example, from change 152917:

    name                      old time/op  new time/op  delta
    TrimSpace/NoTrim-8        18.6ns ± 0%   3.8ns ± 0%  -79.53%  (p=0.000 n=5+4)
    TrimSpace/ASCII-8         33.5ns ± 2%   6.0ns ± 3%  -82.05%  (p=0.008 n=5+5)
    TrimSpace/SomeNonASCII-8  97.1ns ± 1%  88.6ns ± 1%   -8.68%  (p=0.008 n=5+5)
    TrimSpace/JustNonASCII-8   144ns ± 0%   143ns ± 0%     ~     (p=0.079 n=4+5)

This shows that the ASCII fast path for TrimSpace made ASCII-only inputs about five times as fast, though the "SomeNonASCII" sub-test slowed down by about 9%.

To diagnose where something is running slowly, the built-in profiling tools can be used, such as the -cpuprofile option when running tests. The built-in go tool pprof displays profile output in a variety of formats, including flame graphs.

The go test command

Go is opinionated about where tests should reside (in files named *_test.go) and how test functions are named (they must be prefixed with Test). The advantage of being opinionated, however, is that the go test tool knows exactly where to look and how to run the tests. There's no need for a makefile or metadata describing where the tests live — if files and functions are named in the standard way, Go already knows where to look.

The go test command is simple on the surface, but it has a number of options for running and filtering tests and benchmarks. Here are some examples:

    go test             # run tests in current directory
    go test package     # run tests for given package
    go test ./...       # run tests for current dir and all sub-packages
    go test -run=foo    # run tests matching regex "foo"
    go test -cover      # run tests and output code coverage
    go test -bench=.    # also run benchmarks
    go test -bench=. -cpuprofile cpu.out
                        # run benchmarks, record profiling info

Go test's -cover mode produces code coverage profiles that can be viewed as HTML using go tool cover -html=coverage.out. When explaining how Go's code coverage tool works, Go co-creator Rob Pike said:

Summing up

Go's testing library is simple but extendable, and the go test runner is a good complement with its test execution, benchmarking, profiling, and code-coverage reporting. You can go a long way with the vanilla testing package — I find Go's minimalist approach to be a forcing function to think differently about testing and to get the most out of native language features, such as interfaces and struct composition. But if you need to pull in third party libraries, they're only a go get away.

func TestFoo(t *testing.T) {
  f := NewFoo()
  defer f.Close()
  for _, c := testCases {
    c := c
    t.Run(c.name, func(t *testing.T) {
      t.Parallel()
      c.func(t, f)
    })
  }