Implement a Benchmark with a Custom Output with Statistics

Implement a Benchmark with a Custom Output with Statistics#

This section shows how to implement a benchmark with a custom output with statistics using cpp-stat-bench library.

Hint

If your custom output doesn’t change in each iteration, you should implement a benchmark with a custom output without statistics.

Prerequisites#

Sample Code with Explanation#

Example of a benchmark with a custom output with statistics.#
#include <random>

#include <stat_bench/benchmark_macros.h>

#include "stat_bench/current_invocation_context.h"

[[nodiscard]] double approx_pi(int num_points);

// Define a fixture to define parameters.
class ApproxPiFixture : public stat_bench::FixtureBase {
public:
    ApproxPiFixture() {
        // Define a parameter.
        add_param<int>("num_points")
            // Add values to be tested.
            ->add(100)
            ->add(1000)
            ->add(10000);
    }
};

// Define a benchmark with a fixture.
// The first argument is the fixture class,
// the second argument is the name of the group of benchmarks,
// and the third argument is the name of the case of the benchmark.
STAT_BENCH_CASE_F(ApproxPiFixture, "ApproxPi", "approx_pi") {
    const int num_points =
        // Get the value of the parameter from the current context object.
        stat_bench::current_invocation_context().get_param<int>("num_points");

    constexpr double expected = 3.14159265358979323846;

    // Define a custom output with statistics.
    // Without options, the output is the mean value.
    auto error_output =
        stat_bench::current_invocation_context().add_custom_stat("error");

    // Measure the processing time with indices of threads and samples.
    // Thread index is always zero in this case,
    // because this benchmark has no configuration for multi-threading.
    STAT_BENCH_MEASURE_INDEXED(
        thread_index, sample_index, /*iteration_index*/) {
        const double approx = approx_pi(num_points);
        const double error = std::abs(approx - expected);
        // Set the value to the custom output.
        error_output->add(thread_index, sample_index, error);
    };
}

// Define main function. (Required once in an executable.)
STAT_BENCH_MAIN

double approx_pi(int num_points) {
    std::mt19937_64 engine(std::random_device{}());
    std::uniform_real_distribution<double> dist(-1.0, 1.0);

    int num_inside = 0;
    for (int i = 0; i < num_points; ++i) {
        const double x = dist(engine);
        const double y = dist(engine);
        if (x * x + y * y <= 1.0) {
            ++num_inside;
        }
    }

    return 4.0 * static_cast<double>(num_inside) /
        static_cast<double>(num_points);
}

Execution#

You can build the above code and run the resulting executable without arguments to see an output similar to the one below.

Example output to the console.#
cpp-stat-bench 0.21.0

Benchmark start at 2025-02-13T14:03:35.480786+0000

Time resolution: 1.000e-09 sec.

========================================================================================================================
Processing Time
========================================================================================================================

>> ApproxPi
                                                                                Time [us]
                                                            Iterations Samples       Mean Std. Err.       Max Custom Outputs (mean)
------------------------------------------------------------------------------------------------------------------------
approx_pi (num_points=100)                                           1     100     5.3556    0.0630   10.0730 error=1.170e-01,
approx_pi (num_points=1000)                                          1     100    12.6330    0.2983   35.7360 error=3.656e-02,
approx_pi (num_points=10000)                                         1     100    98.4151    0.1109       104 error=1.194e-02,

========================================================================================================================
Mean Processing Time
========================================================================================================================

>> ApproxPi
                                                                                Time [us]
                                                            Iterations Samples       Mean Std. Err.       Max Custom Outputs (mean)
------------------------------------------------------------------------------------------------------------------------
approx_pi (num_points=100)                                        7198      30     4.5389    0.0165    4.8675 error=1.317e-01,
approx_pi (num_points=1000)                                       2884      30    12.4647    0.0464   12.9911 error=4.141e-02,
approx_pi (num_points=10000)                                       378      30    89.6876    0.4341   97.2853 error=1.304e-02,

Benchmark finished at 2025-02-13T14:03:39.836305+0000

Further Reading#