Hi!

I managed to get started with the feature. There is work to be done, but I open the pull request in case you want to do an early review so I can change my approach if needed. Note that this branch depends on the changes in #54

Here's the progress:

• [x] Benchmark writers can declare one or more parameters of any streamable type.
• [x] Benchmark parameters can be set to a value from the command line.
• [x] The benchmark runner can be told from the command line try a whole range (linear or exponential) of range values for one (and only one) parameter.
• [x] Improve the handling of parameter types.
• [x] Testing (I know, one should do that first, but the areas of the code I needed to touch did not have any tests yet so I just hacked my way through.)
• [x] Update the HTML reporter to show a graph with the different means and variances for the test run with different parameters.

Nice to have that I might not do:

• [ ] Update documentation mentioning this feature.
• [ ] Do something special for other reporters when trying a parameter range.
• [ ] Improve parameter parsing on the command line so one can escape the separator : in string parameters.
• [ ] Add runs through ranges of values over multiple parameters, using support for 3D and/or contour plots from Plotly.
• [ ] Explicitly defining benchmark suites (capturing parameters at suite level for brevity)

RFC @rmartinho

This allows a custom main() function to call nonius::main(), which is useful if some initialization code needs to run before the benchmarks. Previously, main.h++ couldn't be included because main() would then be defined twice.

I want to ignore in the measurement the destruction of an object. I know the special objects: nonius::storage_for & nonius::destructable_object

But I think it will not help me, I have following case:

MyObject obj = method_generate_myobject()

The method method_generate_myobject() will generate an instance of MyObject with automatica storage. The method has access to a private ctor which I cannot invoke directly. How can I ignore with nonius the destruction of MyObject during the benchmark run?

On ubuntu 16:

cd examples g++ -Wall -std=c++11 example1.cpp -I../include/nonius/ -I../include/ -o example1 -pthread /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crt1.o: In function _start': (.text+0x20): undefined reference tomain' collect2: error: ld returned 1 exit status

Adding #include <nonius/main.h++> to the file include/nonius.h++ does fix that but well....

This pull request tries to improve the parameters API. It is a continuation of #56. Following up on the discussion on that pull-request, I try here to make the parameters API more terse and robust. The last commit includes more details.

This branch also has a commit addressing #10. It could be extracted, but I just happened to do it in this branch.

I've created a Conan-package for Nonius here. It works for me on Mac OS X, but I have no way of testing it on Windows.

Also, in order to configure the Boost dependency better, it would be useful to know for which versions of Visual Studio specifically it is required.

I'm happy for any feedback. If you have no objections, once it's confirmed working on Windows, I'd publish it in the Conan central repository.

Plotly (https://plot.ly/javascript/) does not have the licensing problems discussed in #53. Furthermore, it has more features and plot styles, which will be useful later, like in #36.

In this branch I tried to keep the styling similar to the one with Highlights. I made small changes, that I consider improvements, like making the plot adapt automatically to the window size -- including on resize.

An example of the new look of the reports can be seen here:

https://sinusoid.es/misc/nonius-example2-plotly.html

Hey,

I've tried to compile the examples but I'm getting undefined reference to main git log shows : 78df23a7ed94a7311dfa10db531fe2a4c810f479

g++ -o example2 example2.c++ --std=c++14 -I/home/rsoeldner/custom/nonius/include -I/home/rsoeldner/custom/nonius/include/nonius -lpthread
/usr/lib/../lib64/crt1.o: In function `_start':
/home/abuild/rpmbuild/BUILD/glibc-2.19/csu/../sysdeps/x86_64/start.S:118: undefined reference to `main'
collect2: error: ld returned 1 exit status

I've added the ability to filter which benchmarks you want to run as a command line option.

the --filter (and -f) option allows you to define a regex pattern that benchmark names must match to be executed. The default pattern is ".*" which will run all tests.

examples where not linking due to missing main since they probably should only demonstrate the most simple solution, it seems good to use the provided main routine.

Replace's boost::variant with a std::function removing the dependency on Boost.Variant.

Caveat: using a std::shared_ptr is not the same as using an std::unique_ptr but the overhead is on construction only. Since std::function requires CopyConstructible, this is the simplest thing I found that works.

I also updated the header of some examples again to include nonius_single.h++ since they were missing the main function.

If you follow the releases link in the readme at https://github.com/libnonius/nonius/ it leads to https://github.com/rmartinho/nonius/releases . All of the releases there are present as full source only with no single-header versions. The correct link that has releases with single-header versions is https://github.com/libnonius/nonius/releases

I think this is causing some confusion among potential users, as evidenced by other bug reports.

is it possible to have a feature to disable boost with a macro?

#include <boost/math/distributions/normal.hpp>

for a header only library , having a boost dependency seems counter intuitive

to include in a cmake project, it seems 2 header locations are needed ?

include_directories(nonius-1.1.2/include/nonius)
include_directories(nonius-1.1.2/include)

The HTML report's Javascript incorrectly assumes that the benchmarks in each run are sorted in the same order. That's not the case (maybe it should be?). Instead, you can often have something like this:

var data = { runs: [
    { benchmarks: [ { name: 'a', mean: 1 }, { name: 'b', mean: 2 } ] },
    { benchmarks: [ { name: 'b', mean: 3 }, { name: 'a', mean: 4 } ] },
]};

In the above example, the summary plot would assume that the first benchmark is named 'a', and it would draw a line for series 'a' connecting points [1, 3]. It should instead draw a line connecting points [1, 4].

This change updates the javascript to search all the runs in the benchmark for one with the target name.

Fixes #108.

Create a param named X and run a test with multiple param values, like benchmark --reporter=html --param=X:+:0:1:10. The HTML report will show a nice summary page that shows a line plot of how the benchmarks fare on each param value.

Unfortunately, the data is just wrong. The data is correct only for the first (leftmost) column; all other columns have the data mixed up between the various series.

Here's an example:

Note how the orange and blue series seem to trade off on looking bad. That's not backed up by the raw data -- in the raw data, the orange series should always be bad, and the blue should always be okay.

A corrected graph would look like this: