BenchmarkTools makes performance tracking of Julia code easy by supplying a framework for writing and running groups of benchmarks as well as comparing benchmark results. This package is used to write and run the benchmarks found in BaseBenchmarks.jl. The CI infrastructure for automated performance testing of the Julia language is not in this package but can be found in Nanosoldier.jl. Our story begins with two packages, "Benchmarks" and "BenchmarkTrackers". The Benchmarks package implemented an execution strategy for collecting and summarizing individual benchmark results, while BenchmarkTrackers implemented a framework for organizing, running, and determining regressions of groups of benchmarks. Under the hood, BenchmarkTrackers relied on Benchmarks for actual benchmark execution.
Features
- BenchmarkTools is a Julia Language package
- Documentation available
- Individual sample noise could significantly change the execution strategy used to collect further samples
- The estimates used to characterize benchmark results and to detect regressions were statistically vulnerable to noise
- Different benchmarks have different noise tolerances, but there was no way to tune this parameter on a per-benchmark basis
- Running benchmarks took a long time - an order of magnitude longer than theoretically necessary for many functions
- Using the system in the REPL (for example, to reproduce regressions locally) was often cumbersome
Project Samples
