Home

HSBencher Overview

The HSBencher suite of tools are written in Haskell and designed to run and aggregate benchmark data from a distributed execution environment. All the packages can be installed from Hackage.. Different packages provides different components that work together in the complete benchmarking system:

• Benchmark-running (hsbencher pkg)
• Benchmark data uploading (hsbencher-fusion,hsbencher-codespeed)
• Benhcmark data fetching and plotting (hsbencher-tool,hsbencher-graph)

Most use cases break down into (1) those which use HSBencher as a harness to run benchmarks, and (2) those which run benchmarks by some other means, but use HSBencher to upload the data post-facto to a fusion table and enable using the rest of the HSBencher tools.

In the future, other database backends should be supported, but right now the focus is on using Google Fusion Tables to store, retrieve, and view benchmark data.

Case 1: Running benchmarks with a custom harness

In this scenario, the user creates a benchmark suite which after each benchmarking run outputs a file in one of two formats:

• Criterion report -- a binary file containing Criterion benchmark. data. Upload with the hsbencher-fusion-upload-criterion command.
• CSV file -- typically using a subset/superset of the standard HSBencher benchmark schema. Upload with the hsbencher-fusion-upload-csv command.

Both commands should be run in the git repository that contains the benchmarks just ran, and on the machine that ran them. Because the above files (especially Criterion reports) do not contain complete metadata about the benchmarking environment, these upload commands harvest that data from the current machine and the current directory before uploading, so they must be run in the right place.

In both cases, it is good to read about the HSBencherSchemaStyleGuide to see a recommended way to use the different columns in the benchmark data schema.

Case 2: Running benchmarks with HSBencher Harnesses

The goal of an HSBencher-based harness is to run a bunch of benchmarks, each as a single process and as the only thing running on a machine, and then repeat this process, varying across a parameter space. Properties of these runs (e.g. time elapsed) are recorded and then stored in some kind of database (see "backends" below), where they can be retrieved.

HSBencher works well with using systems like Jenkins to launch benchmark jobs on many machines, while relying on HSBencher to aggregate the benchmark data back to a single place.

Currently, adding an HSBencher script to a project requires writing Haskell code to enumerate the benchmarks and describe their configuration space. In the future we may support using simple text files to describe benchmark configurations (as we did in a much earlier version), but for now Haskell coding is required if you want HSBencher to run your benchmarks. Thus, the remainder of this page is split into two sections: to help you either (1) use an existing HSBencher setup (no Haskell coding required), or (2) create a new HSBencher setup.

How to use existing HSBencher suites.

• Create your own Fusion table
• Build the benchmark harness executable (e.g. run-benchmarks.hs)
• Run the benchmark executable

How to write run-benchmarks.hs

Backends

• DribbleBackend -- a simple backend for outputting to a local CSV file.
• FusionTableBackend -- Upload data to a Google Fusion Table. Corresponds to module HSBencher.Backend.Fusion, package hsbencher-fusion.
• BigQueryBackend -- This doesn't exist yet, but is planned for Tableau integration.

Build Methods

• BuiltInBuildMethods -- rudimentary cabal, ghc, and make BuildMethods that ship with the core hsbencher package.
• CustomBuildMethod_HowTo --