UT: C++20 μ(micro)/Unit Testing Framework

Version Linux MacOs Windows Coveralls Codacy Badge Try it online

"If you liked it then you "should have put a"_test on it", Beyonce rule

UT / μt

| Motivation | Quick Start | Overview | Tutorial | Examples | User Guide | FAQ | Benchmarks |

C++ single header/single module, macro-free μ(micro)/Unit Testing Framework

Motivation

Testing is a very important part of the Software Development, however, C++ doesn't provide any good testing facilities out of the box, which often leads into a poor testing experience for develops and/or lack of tests/coverage in general.

One should treat testing code as production code!

Additionally, well established testing practises such as Test Driven Development (TDD)/Behaviour Driven Development (BDD) are often not followed due to the same reasons.

The following snippet is a common example of testing with projects in C++.

int main() {
  // should sum numbers
  {
    assert(3 == sum(1, 2));
  }
}

There are quite a few problems with the approach above

  • No names for tests (Hard to follow intentions by further readers)
  • No automatic registration of tests (No way to run specific tests)
  • Hard to debug (Assertions don't provide any information why it failed)
  • Hard to scale (No easy path forward for parameterized tests, multiple suites, parallel execution, etc...)
  • Hard to integrate (No easy way to have a custom output such as XML for CI integration)
  • Easy to make mistakes (With implicit casting, floating point comparison, pointer comparison for strings, etc...)
  • Hard to follow good practises such as TDD/BDD (Lack of support for sections and declarative expressions)
  • ...

UT is trying to address these issues by simplifying testing experience with a few simple steps:

And you good to go!

Okay, great, but why I would use UT over other/similar testing frameworks already available in C++?

Great question! There are a few unique features which makes UT worth trying

  • Firstly, it supports all the basic Unit Testing Framework features (automatic registration of tests, assertions, suites, etc...)
  • It's easy to integrate (it's just one header/module)
  • It's macro free which makes testing experience that much nicer (it uses modern C++ features instead, macros are opt-in rather than being compulsory - Can I still use macros?)
  • It's flexible (all parts of the framework such as: runner, reporter, printer can be customized, basically most other Unit Testing Frameworks can be implemented on top of UT primitives)
  • It has smaller learning curve (just a few simple concepts (expect, test, suite))
  • It leverages C++ features to support more complex testing (parameterized)
  • It's faster to compile and execute than similar frameworks which makes it suitable for bigger projects without additional hassle (Benchmarks)
  • It supports TDD/BDD workflows
  • It supports Gherkin specification
  • It supports Spec
  • ...

Sounds intriguing/interesting? Learn more at

Quick Start

https://bit.ly/ut-quick-start (slides)

Overview

Tutorial

    Step 0: Get it...

Get the latest latest header/module from here!

Include/Import

// #include <boost/ut.hpp> // single header
// import boost.ut;        // single module (C++20)

int main() { }

Compile & Run

$CXX main.cpp && ./a.out
All tests passed (0 assert in 0 test)

[Optional] Install it

cmake -Bbuild -H.
cd build && make         # run tests
cd build && make install # install

[Optional] CMake integration

This project provides a CMake config and target. Just load ut with find_package to import the boost::ut target. Linking against this target will add the necessary include directory for the single header file. This is demonstrated in the following example.

find_package(ut REQUIRED)
add_library(my_test my_test.cpp)
target_link_libraries(my_test PRIVATE boost::ut)

[Optional] Conan integration

The boost-ext-ut package is available from Conan Center. Just include it in your project's Conanfile with boost-ext-ut/1.1.8.

    Step 1: Expect it...

Let's write our first assertion, shall we?

int main() {
  boost::ut::expect(true);
}
All tests passed (1 asserts in 0 test)

https://godbolt.org/z/vfx-eB

Okay, let's make it fail now?

int main() {
  boost::ut::expect(1 == 2);
}
main.cpp:4:FAILED [false]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/7qTePx

Notice that expression 1 == 2 hasn't been printed. Instead we got false?

Let's print it then?

int main() {
  using namespace boost::ut;
  expect(1_i == 2);
}
main.cpp:4:FAILED [1 == 2]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/7MXVzu

Okay, now we have it! 1 == 2 has been printed as expected. Notice the User Defined Literal (UDL) 1_i was used. _i is a compile-time constant integer value

  • It allows to override comparison operators 👍
  • It disallow comparison of different types 👍

See the User-guide for more details.

Alternatively, a terse notation (no expect required) can be used.

int main() {
  using namespace boost::ut::literals;
  using namespace boost::ut::operators::terse;

  1_i == 2; // terse notation
}
main.cpp:7:FAILED [1 == 2]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/s77GSm

Other expression syntaxes are also available.

expect(1_i == 2);       // UDL syntax
expect(1 == 2_i);       // UDL syntax
expect(that % 1 == 2);  // Matcher syntax
expect(eq(1, 2));       // eq/neq/gt/ge/lt/le
main.cpp:6:FAILED [1 == 2]
main.cpp:7:FAILED [1 == 2]
main.cpp:8:FAILED [1 == 2]
main.cpp:9:FAILED [1 == 2]
===============================================================================
tests:   0 | 0 failed
asserts: 4 | 0 passed | 4 failed

https://godbolt.org/z/QbgGtc

Okay, but what about the case if my assertion is fatal. Meaning that the program will crash unless the processing will be terminated. Nothing easier, let's just add >> fatal after the expected expression to make it fatal.

expect((1 == 2_i) >> fatal); // fatal assertion
expect(1_i == 2);            // not executed
main.cpp:6:FAILED [1 == 2]
===============================================================================
tests:   1 | 1 failed
asserts: 2 | 0 passed | 2 failed

https://godbolt.org/z/WMe8Y1

But my expression is more complex than just simple comparisons. Not a problem, logic operators are also supported in the expect 👍 .

expect(42l == 42_l and 1 == 2_i); // compound expression
main.cpp:5:FAILED [(42 == 42 and 1 == 2)]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/aEhX4t

Can I add a custom message though? Sure, expect calls are streamable!

int main() {
  expect(42l == 42_l and 1 == 2_i) << "additional info";
}
main.cpp:5:FAILED [(42 == 42 and 1 == 2)] additional info
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/v2PDuU

    Step 2: Group it...

Assertions are great, but how to combine them into more cohesive units? Test cases are the way to go! They allow to group expectations for the same functionality into coherent units.

"hello world"_test = [] { };

Alternatively test("hello world") = [] {} can be used.

All tests passed (0 asserts in 1 tests)

https://godbolt.org/z/Bh-EmY

Notice 1 tests but 0 asserts.

Let's make our first end-2-end test case, shall we?

int main() {
  "hello world"_test = [] {
    int i = 43;
    expect(42_i == i);
  };
}
Running "hello world"...
  main.cpp:8:FAILED [42 == 43]
FAILED
===============================================================================
tests:   1 | 1 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/Y43mXz

👍 We are done here!

I'd like to nest my tests, though and share setup/tear-down. With lambdas used to represents tests/sections we can easily achieve that. Let's just take a look at the following example.

int main() {
  "[vector]"_test = [] {
    std::vector<int> v(5);

    expect((5_ul == std::size(v)) >> fatal);

    should("resize bigger") = [v] { // or "resize bigger"_test
      mut(v).resize(10);
      expect(10_ul == std::size(v));
    };

    expect((5_ul == std::size(v)) >> fatal);

    should("resize smaller") = [=]() mutable { // or "resize smaller"_test
      v.resize(0);
      expect(0_ul == std::size(v));
    };
  }
}
All tests passed (4 asserts in 1 tests)

https://godbolt.org/z/XWAdYt

Nice! That was easy, but I'm a believer into Behaviour Driven Development (BDD). Is there a support for that? Yes! Same example as above just with the BDD syntax.

int main() {
  "vector"_test = [] {
    given("I have a vector") = [] {
      std::vector<int> v(5);
      expect((5_ul == std::size(v)) >> fatal);

      when("I resize bigger") = [=] {
        mut(v).resize(10);

        then("The size should increase") = [=] {
          expect(10_ul == std::size(v));
        };
      };
    };
  };
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/dnvxsE

On top of that, feature/scenario aliases can be leveraged.

int main() {
  feature("vector") = [] {
    scenario("size") = [] {
      given("I have a vector") = [] {
        std::vector<int> v(5);
        expect((5_ul == std::size(v)) >> fatal);

        when("I resize bigger") = [=] {
          mut(v).resize(10);

          then("The size should increase") = [=] {
            expect(10_ul == std::size(v));
          };
        };
      };
    };
  };
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/T4cWss

Can I use Gherkin? Yeah, let's rewrite the example using Gherkin specification

int main() {
  bdd::gherkin::steps steps = [](auto& steps) {
    steps.feature("Vector") = [&] {
      steps.scenario("*") = [&] {
        steps.given("I have a vector") = [&] {
          std::vector<int> v(5);
          expect((5_ul == std::size(v)) >> fatal);

          steps.when("I resize bigger") = [&] {
            v.resize(10);
          };

          steps.then("The size should increase") = [&] {
            expect(10_ul == std::size(v));
          };
        };
      };
    };
  };

  "Vector"_test = steps |
    R"(
      Feature: Vector
        Scenario: Resize
          Given I have a vector
           When I resize bigger
           Then The size should increase
    )";
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/jb1d8P

Nice, is Spec notation supported as well?

int main() {
  describe("vector") = [] {
    std::vector<int> v(5);
    expect((5_ul == std::size(v)) >> fatal);

    it("should resize bigger") = [v] {
      mut(v).resize(10);
      expect(10_ul == std::size(v));
    };
  };
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/6jKKzT

That's great, but how can call the same tests with different arguments/types to be DRY (Don't Repeat Yourself)? Parameterized tests to the rescue!

int main() {
  for (auto i : std::vector{1, 2, 3}) {
    test("parameterized " + std::to_string(i)) = [i] { // 3 tests
      expect(that % i > 0); // 3 asserts
    };
  }
}
All tests passed (3 asserts in 3 tests)

https://godbolt.org/z/Utnd6X

That's it 😮 ! Alternatively, a convenient test syntax is also provided 👍

int main() {
  "args"_test = [](const auto& arg) {
    expect(arg > 0_i) << "all values greater than 0";
  } | std::vector{1, 2, 3};
}
All tests passed (3 asserts in 3 tests)

https://godbolt.org/z/6FHtpq

Check Examples for further reading.

    Step 3: Scale it...

Okay, but my project is more complex than that. How can I scale? Test suites will make that possible. By using suite in translation units tests defined inside will be automatically registered 👍

suite errors = [] {
  "exception"_test = [] {
    expect(throws([] { throw 0; })) << "throws any exception";
  };

  "failure"_test = [] {
    expect(aborts([] { assert(false); }));
  };
};

int main() { }
All tests passed (2 asserts in 2 tests)

https://godbolt.org/z/_ccGwZ


What's next?

Examples

    Assertions

// operators
expect(0_i == sum());
expect(2_i != sum(1, 2));
expect(sum(1) >= 0_i);
expect(sum(1) <= 1_i);
// message
expect(3_i == sum(1, 2)) << "wrong sum";
// expressions
expect(0_i == sum() and 42_i == sum(40, 2));
expect(0_i == sum() or 1_i == sum()) << "compound";
// matchers
expect(that % 0 == sum());
expect(that % 42 == sum(40, 2) and that % (1 + 2) == sum(1, 2));
expect(that % 1 != 2 or 2_i > 3);
// eq/neq/gt/ge/lt/le
expect(eq(42, sum(40, 2)));
expect(neq(1, 2));
expect(eq(sum(1), 1) and neq(sum(1, 2), 2));
expect(eq(1, 1) and that % 1 == 1 and 1_i == 1);
// floating points
expect(42.1_d == 42.101) << "epsilon=0.1";
expect(42.10_d == 42.101) << "epsilon=0.01";
expect(42.10000001 == 42.1_d) << "epsilon=0.1";
// constant
constexpr auto compile_time_v = 42;
auto run_time_v = 99;
expect(constant<42_i == compile_time_v> and run_time_v == 99_i);
// failure
expect(1_i == 2) << "should fail";
expect(sum() == 1_i or 2_i == sum()) << "sum?";
assertions.cpp:53:FAILED [1 == 2] should fail
assertions.cpp:54:FAILED [(0 == 1 or 2 == 0)] sum?
===============================================================================
tests:   0  | 0 failed
asserts: 20 | 18 passed | 2 failed

https://godbolt.org/z/E1c7G5

    Tests

        Run/Skip/Tag

"run UDL"_test = [] {
  expect(42_i == 42);
};

skip / "don't run UDL"_test = [] {
  expect(42_i == 43) << "should not fire!";
};
All tests passed (1 asserts in 1 tests)
1 tests skipped
test("run function") = [] {
  expect(42_i == 42);
};

skip / test("don't run function") = [] {
  expect(42_i == 43) << "should not fire!";
};
All tests passed (1 asserts in 1 tests)
1 tests skipped
tag("nightly") / tag("slow") /
"performance"_test= [] {
  expect(42_i == 42);
};

tag("slow") /
"run slowly"_test= [] {
  expect(42_i == 43) << "should not fire!";
};
cfg<override> = {.tag = {"nightly"}};
All tests passed (1 asserts in 1 tests)
1 tests skipped

https://godbolt.org/z/X3_kG4

        Sections

"[vector]"_test = [] {
  std::vector<int> v(5);

  expect((5_ul == std::size(v)) >> fatal);

  should("resize bigger") = [=] { // or "resize bigger"_test
    mut(v).resize(10);
    expect(10_ul == std::size(v));
  };

  expect((5_ul == std::size(v)) >> fatal);

  should("resize smaller") = [=]() mutable { // or "resize smaller"_test
    v.resize(0);
    expect(0_ul == std::size(v));
  };
};
All tests passed (4 asserts in 1 tests)

https://godbolt.org/z/cE91bj

        Behavior Driven Development (BDD)

"Scenario"_test = [] {
  given("I have...") = [] {
    when("I run...") = [] {
      then("I expect...") = [] { expect(1_i == 1); };
      then("I expect...") = [] { expect(1 == 1_i); };
    };
  };
};
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/mNBySr

        Gherkin

int main() {
  bdd::gherkin::steps steps = [](auto& steps) {
    steps.feature("*") = [&] {
      steps.scenario("*") = [&] {
        steps.given("I have a number {value}") = [&](int value) {
          auto number = value;
          steps.when("I add {value} to it") = [&](int value) {
            number += value;
          };
          steps.then("I expect number to be {value}") = [&](int value) {
            expect(that % number == value);
          };
        };
      };
    };
  };

  "Gherkin"_test = steps |
    R"(
      Feature: Number
        Scenario: Addition
          Given I have a number 40
           When I add 2 to it
           Then I expect number to be 42
    )";
}
All tests passed (1 asserts in 1 tests)

https://godbolt.org/z/BP3hyt

        Spec

int main() {
  describe("equality") = [] {
    it("should be equal")     = [] { expect(0_i == 0); };
    it("should not be equal") = [] { expect(1_i != 0); };
  };
}
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/BXYJ3a

        Parameterized

for (auto i : std::vector{1, 2, 3}) {
  test("parameterized " + std::to_string(i)) = [i] {
    expect(that % i > 0);
  };
}

"args"_test =
   [](auto arg) {
      expect(arg >= 1_i);
    }
  | std::vector{1, 2, 3};

"types"_test =
    []<class T> {
      expect(std::is_integral_v<T>) << "all types are integrals";
    }
  | std::tuple<bool, int>{};

"args and types"_test =
    []<class TArg>(TArg arg) {
      expect(std::is_integral_v<TArg> >> fatal);
      expect(42_i == arg or "is true"_b == arg);
      expect(type<TArg> == type<int> or type<TArg> == type<bool>);
    }
  | std::tuple{true, 42};
All tests passed (14 asserts in 10 tests)

https://godbolt.org/z/4xGGdo

    Suites

namespace ut = boost::ut;

ut::suite errors = [] {
  using namespace ut;

  "throws"_test = [] {
    expect(throws([] { throw 0; }));
  };

  "doesn't throw"_test = [] {
    expect(nothrow([]{}));
  };
};

int main() { }
All tests passed (2 asserts in 2 tests)

https://godbolt.org/z/CFbTP9

    Misc

        Logging

"logging"_test = [] {
  log << "pre";
  expect(42_i == 43) << "message on failure";
  log << "post";
};
Running "logging"...
pre
  logging.cpp:8:FAILED [42 == 43] message on failure
post
FAILED

===============================================================================

tests:   1 | 1 failed
asserts: 1 | 0 passed | 1 failed

https://godbolt.org/z/26fPSY

        Matchers

"matchers"_test = [] {
  constexpr auto is_between = [](auto lhs, auto rhs) {
    return [=](auto value) {
      return that % value >= lhs and that % value <= rhs;
    };
  };

  expect(is_between(1, 100)(42));
  expect(not is_between(1, 100)(0));
};
All tests passed (2 asserts in 1 tests)

https://godbolt.org/z/4qwrCi

        Exceptions/Aborts

"exceptions/aborts"_test = [] {
  expect(throws<std::runtime_error>([] { throw std::runtime_error{""}; }))
    << "throws runtime_error";
  expect(throws([] { throw 0; })) << "throws any exception";
  expect(nothrow([]{})) << "doesn't throw";
  expect(aborts([] { assert(false); }));
};
All tests passed (4 asserts in 1 tests)

https://godbolt.org/z/A2EehK

    Config

        Runner

namespace ut = boost::ut;

namespace cfg {
  class runner {
   public:
    template <class... Ts> auto on(ut::events::test<Ts...> test) { test(); }
    template <class... Ts> auto on(ut::events::skip<Ts...>) {}
    template <class TExpr>
    auto on(ut::events::assertion<TExpr>) -> bool { return true; }
    auto on(ut::events::fatal_assertion) {}
    template <class TMsg> auto on(ut::events::log<TMsg>) {}
  };
} // namespace cfg

template<> auto ut::cfg<ut::override> = cfg::runner{};

https://godbolt.org/z/jdg687

        Reporter

namespace ut = boost::ut;

namespace cfg {
  class reporter {
   public:
    auto on(ut::events::test_begin) -> void {}
    auto on(ut::events::test_run) -> void {}
    auto on(ut::events::test_skip) -> void {}
    auto on(ut::events::test_end) -> void {}
    template <class TMsg> auto on(ut::events::log<TMsg>) -> void {}
    template <class TExpr>
    auto on(ut::events::assertion_pass<TExpr>) -> void {}
    template <class TExpr>
    auto on(ut::events::assertion_fail<TExpr>) -> void {}
    auto on(ut::events::fatal_assertion) -> void {}
    auto on(ut::events::exception) -> void {}
    auto on(ut::events::summary) -> void {}
  };
}  // namespace cfg

template <>
auto ut::cfg<ut::override> = ut::runner<cfg::reporter>{};

https://godbolt.org/z/gsAPKg

        Printer

namespace ut = boost::ut;

namespace cfg {
struct printer : ut::printer {
  template <class T>
  auto& operator<<(T&& t) {
    std::cerr << std::forward<T>(t);
    return *this;
  }
};
}  // namespace cfg

template <>
auto ut::cfg<ut::override> = ut::runner<ut::reporter<cfg::printer>>{};

int main() {
  using namespace ut;
  "printer"_test = [] {};
}

https://godbolt.org/z/XCscF9

User Guide

    API

export module boost.ut; /// __cpp_modules

namespace boost::inline ext::ut::inline v1_1_8 {
  /**
   * Represents test suite object
   */
  struct suite final {
    /**
     * Creates and executes test suite
     * @example suite _ = [] {};
     * @param suite test suite function
     */
    constexpr explicit(false) suite(auto suite);
  };

  /**
   * Creates a test
   * @example "name"_test = [] {};
   * @return test object to be executed
   */
  constexpr auto operator""_test;

  /**
   * Creates a test
   * @example test("name") = [] {};
   * @return test object to be executed
   */
  constexpr auto test = [](const auto name);

  /**
   * Creates a test
   * @example should("name") = [] {};
   * @return test object to be executed
   */
  constexpr auto should = [](const auto name);

  /**
   * Behaviour Driven Development (BDD) helper functions
   * @param name step name
   * @return test object to be executed
   */
  constexpr auto given = [](const auto name);
  constexpr auto when  = [](const auto name);
  constexpr auto then  = [](const auto name);

  /**
   * Evaluates an expression
   * @example expect(42 == 42_i and 1 != 2_i);
   * @param expr expression to be evaluated
   * @param source location https://en.cppreference.com/w/cpp/utility/source_location
   * @return stream
   */
  constexpr OStream& expect(
    Expression expr,
    const std::source_location& location = std::source_location::current()
  );

  struct {
    /**
     * @example (that % 42 == 42);
     * @param expr expression to be evaluated
     */
    [[nodiscard]] constexpr auto operator%(Expression expr) const;
  } that{};

  inline namespace literals {
    /**
     * User defined literals to represent constant values
     * @example 42_i, 0_uc, 1.23_d
     */
    constexpr auto operator""_i;  /// int
    constexpr auto operator""_s;  /// short
    constexpr auto operator""_c;  /// char
    constexpr auto operator""_l;  /// long
    constexpr auto operator""_ll; /// long long
    constexpr auto operator""_u;  /// unsigned
    constexpr auto operator""_uc; /// unsigned char
    constexpr auto operator""_us; /// unsigned short
    constexpr auto operator""_ul; /// unsigned long
    constexpr auto operator""_f;  /// float
    constexpr auto operator""_d;  /// double
    constexpr auto operator""_ld; /// long double

    /**
     * Represents dynamic values
     * @example _i(42), _f(42.)
     */
    constexpr auto _b(bool);
    constexpr auto _c(char);
    constexpr auto _s(short);
    constexpr auto _i(int);
    constexpr auto _l(long);
    constexpr auto _ll(long long);
    constexpr auto _u(unsigned);
    constexpr auto _uc(unsigned char);
    constexpr auto _us(unsigned short);
    constexpr auto _ul(unsigned long);
    constexpr auto _f(float);
    constexpr auto _d(double);
    constexpr auto _ld(long double);

    /**
     * Logical representation of constant boolean (true) value
     * @example "is set"_b     : true
     *          not "is set"_b : false
     */
    constexpr auto operator ""_b;
  } // namespace literals

  inline namespace operators {
    /**
     * Comparison functions to be used in expressions
     * @example eq(42, 42), neq(1, 2)
     */
    constexpr auto eq(Operator lhs, Operator rhs);  /// ==
    constexpr auto neq(Operator lhs, Operator rhs); /// !=
    constexpr auto gt(Operator lhs, Operator rhs);  /// >
    constexpr auto ge(Operator lhs, Operator rhs);  /// >=
    constexpr auto lt(Operator lhs, Operator rhs);  /// <
    constexpr auto le(Operator lhs, Operator rhs);  /// <=

    /**
     * Overloaded comparison operators to be used in expressions
     * @example (42_i != 0)
     */
    constexpr auto operator==;
    constexpr auto operator!=;
    constexpr auto operator>;
    constexpr auto operator>=;
    constexpr auto operator<;
    constexpr auto operator<=;

    /**
     * Overloaded logic operators to be used in expressions
     * @example (42_i != 0 and 1 == 2_i)
     */
    constexpr auto operator and;
    constexpr auto operator or;
    constexpr auto operator not;

    /**
     * Executes parameterized tests
     * @example "parameterized"_test = [](auto arg) {} | std::tuple{1, 2, 3};
     */
    constexpr auto operator|;

    /**
     * Creates tags
     * @example tag("slow") / tag("nightly") / "perf"_test = []{};
     */
    constexpr auto operator/;

    /**
     * Creates a `fatal_assertion` from an expression
     * @example (42_i == 0) >> fatal
     */
    constexpr auto operator>>;
  } // namespace operators

  /**
   * Creates skippable test object
   * @example skip / "don't run"_test = [] { };
   */
  constexpr auto skip = tag("skip");

  struct {
    /**
     * @example log << "message!";
     * @param msg stringable message
     */
    auto& operator<<(Msg msg);
  } log{};

  /**
   * Makes object mutable
   * @example mut(object)
   * @param t object to be mutated
   */
  template<class T> auto mut(const T& t) -> T&;

  /**
   * Default execution flow policy
   */
  class runner {
   public:
    /**
     * @example cfg<override> = {
        .filter  = "test.section.*",
        .colors  = { .none = "" },
        .dry__run = true
       };
     * @param options.filter {default: "*"} runs all tests which names
                                            matches test.section.* filter
     * @param options.colors {default: {
                               .none = "\033[0m",
                               .pass = "\033[32m",
                               .fail  = "\033[31m"
              } if specified then overrides default color values
     * @param options.dry_run {default: false} if true then print test names to be
                                               executed without running them
     */
    auto operator=(options);

    /**
     * @example suite _ = [] {};
     * @param suite() executes suite
     */
    template<class TSuite>
    auto on(ut::events::suite<TSuite>);

    /**
     * @example "name"_test = [] {};
     * @param test.type ["test", "given", "when", "then"]
     * @param test.name "name"
     * @param test.arg parameterized argument
     * @param test() executes test
     */
    template<class... Ts>
    auto on(ut::events::test<Ts...>);

    /**
     * @example skip / "don't run"_test = []{};
     * @param skip.type ["test", "given", "when", "then"]
     * @param skip.name "don't run"
     * @param skip.arg parameterized argument
     */
    template<class... Ts>
    auto on(ut::events::skip<Ts...>);

    /**
     * @example file.cpp:42: expect(42_i == 42);
     * @param assertion.expr 42_i == 42
     * @param assertion.location { "file.cpp", 42 }
     * @return true if expr passes, false otherwise
     */
    template <class TExpr>
    auto on(ut::events::assertion<TExpr>) -> bool;

    /**
     * @example expect((2_i == 1) >> fatal)
     * @note triggered by `fatal`
     *       should std::exit
     */
    auto on(ut::events::fatal_assertion);

    /**
     * @example log << "message"
     * @param log.msg "message"
     */
    template<class TMsg>
    auto on(ut::events::log<TMsg>);

    /**
     * Explicitly runs registered test suites
     * If not called directly test suites are executed with run's destructor
     * @example return run({.report_errors = true})
     * @param run_cfg.report_errors {default: false} if true it prints the summary after runnig
     */
    auto run(run_cfg);

    /**
     * Runs registered test suites if they haven't been explicilty executed already
     */
    ~run();
  };

  /**
   * Default reporter policy
   */
  class reporter {
   public:
    /**
     * @example file.cpp:42: "name"_test = [] {};
     * @param test_begin.type ["test", "given", "when", "then"]
     * @param test_begin.name "name"
     * @param test_begin.location { "file.cpp", 42 }
     */
    auto on(ut::events::test_begin) -> void;

    /**
     * @example "name"_test = [] {};
     * @param test_run.type ["test", "given", "when", "then"]
     * @param test_run.name "name"
     */
    auto on(ut::events::test_run) -> void;

    /**
     * @example "name"_test = [] {};
     * @param test_skip.type ["test", "given", "when", "then"]
     * @param test_skip.name "name"
     */
    auto on(ut::events::test_skip) -> void;

    /**
     * @example "name"_test = [] {};
     * @param test_end.type ["test", "given", "when", "then"]
     * @param test_end.name "name"
     */
    auto on(ut::events::test_end) -> void;

    /**
     * @example log << "message"
     * @param log.msg "message"
     */
    template<class TMsg>
    auto on(ut::events::log<TMsg>) -> void;

    /**
     * @example file.cpp:42: expect(42_i == 42);
     * @param assertion_pass.expr 42_i == 42
     * @param assertion_pass.location { "file.cpp", 42 }
     */
    template <class TExpr>
    auto on(ut::events::assertion_pass<TExpr>) -> void;

    /**
     * @example file.cpp:42: expect(42_i != 42);
     * @param assertion_fail.expr 42_i != 42
     * @param assertion_fail.location { "file.cpp", 42 }
     */
    template <class TExpr>
    auto on(ut::events::assertion_fail<TExpr>) -> void;

    /**
     * @example expect((2_i == 1) >> fatal)
     * @note triggered by `fatal`
     *       should std::exit
     */
    auto on(ut::events::fatal_assertion) -> void;

    /**
     * @example "exception"_test = [] { throw std::runtime_error{""}; };
     */
    auto on(ut::events::exception) -> void;

    /**
     * @note triggered on destruction of runner
     */
    auto on(ut::events::summary) -> void;
  };

  /**
   * Used to override default running policy
   * @example template <> auto cfg<override> = runner<reporter>{};
   */
  struct override {};

  /**
   * Default UT execution policy
   * Can be overwritten with override
   */
  template <class = override> auto cfg = runner<reporter>{};
}

    Configuration

Option Description Example
BOOST_UT_VERSION Current version 1'1'8

FAQ

    How does it work?

suite

/**
 * Reperesents suite object
 * @example suite _ = []{};
 */
struct suite final {
  /**
   * Assigns and executes test suite
   */
  [[nodiscard]] constexpr explicit(false) suite(Suite suite) {
    suite();
  }
};

test

/**
 * Creates named test object
 * @example "hello world"_test
 * @return test object
 */
[[nodiscard]] constexpr Test operator ""_test(const char* name, std::size_t size) {
  return test{{name, size}};
}
/**
 * Represents test object
 */
struct test final {
  std::string_view name{}; /// test case name

  /**
   * Assigns and executes test function
   * @param test function
   */
  constexpr auto operator=(const Test& test) {
    std::cout << "Running... " << name << '\n';
    test();
  }
};

expect

/**
 * Evaluates an expression
 * @example expect(42_i == 42);
 * @param expr expression to be evaluated
 * @param source location https://en.cppreference.com/w/cpp/utility/source_location
 * @return stream
 */
constexpr OStream& expect(
  Expression expr,
  const std::source_location& location = std::source_location::current()
) {
  if (not static_cast<bool>(expr) {
    std::cerr << location.file()
              << ':'
              << location.line()
              << ":FAILED: "
              << expr
              << '\n';
  }

  return std::cerr;
}
/**
 * Creates constant object for which operators can be overloaded
 * @example 42_i
 * @return integral constant object
 */
template <char... Cs>
[[nodiscard]] constexpr Operator operator""_i() -> integral_constant<int, value<Cs...>>;
/**
 * Overloads comparison if at least one of {lhs, rhs} is an Operator
 * @example (42_i == 42)
 * @param lhs Left-hand side operator
 * @param rhs Right-hand side operator
 * @return Comparison object
 */
[[nodiscard]] constexpr auto operator==(Operator lhs, Operator rhs) {
  return eq{lhs, rhs};
}
/**
 * Comparison Operator
 */
template <Operator TLhs, Opeartor TRhs>
struct eq final {
  TLhs lhs{}; // Left-hand side operator
  TRhs rhs{}; // Right-hand side operator

  /**
   * Performs comparison operatation
   * @return true if expression is succesful
   */
  [[nodiscard]] constexpr explicit operator bool() const {
    return lhs == rhs;
  }

  /**
   * Nicely prints the operation
   */
  friend auto operator<<(OStream& os, const eq& op) -> Ostream& {
    return (os << op.lhs << " == " << op.rhs);
  }
};

Sections

/**
 * Convenient aliases for creating test named object
 * @example should("return true") = [] {};
 */
constexpr auto should = [](const auto name) { return test{name}; };

Behaviour Driven Development (BDD)

/**
 * Convenient aliases for creating BDD tests
 * @example feature("Feature") = [] {};
 * @example scenario("Scenario") = [] {};
 * @example given("I have an object") = [] {};
 * @example when("I call it") = [] {};
 * @example then("I should get") = [] {};
 */
constexpr auto feature  = [](const auto name) { return test{name}; };
constexpr auto scenario = [](const auto name) { return test{name}; };
constexpr auto given    = [](const auto name) { return test{name}; };
constexpr auto when     = [](const auto name) { return test{name}; };
constexpr auto then     = [](const auto name) { return test{name}; };

https://godbolt.org/z/6Nk5Mi

Spec

/**
 * Convenient aliases for creating Spec tests
 * @example describe("test") = [] {};
 * @example it("should...") = [] {};
 */
constexpr auto describe = [](const auto name) { return test{name}; };
constexpr auto it       = [](const auto name) { return test{name}; };

Example implementation

Try it online

    Fast compilation times (Benchmarks)?

Implementation

  • Leveraging C++20 features

  • Avoiding unique types for lambda expressions

  template <class Test>
    requires not std::convertible_to<Test, void (*)()>>
  constexpr auto operator=(Test test);

vs

  // Compiles 5x faster because it doesn't introduce a new type for each lambda
  constexpr auto operator=(void (*test)());
  • Type-name erasure (allows types/function memoization)
  eq<integral_constant<42>, int>{ {}, 42 }

vs

  // Can be memoized - faster to compile
  eq<int, int>{42, 42}
  • Limiting preprocessor work

    • Single header/module
    • Minimal number of include files
  • Simplified versions of

    • std::function
    • std::string_view

    C++20 features?

    C++2X integration?

Parameterized tests with Expansion statements (https://wg21.link/P1306r1)

template for (auto arg : std::tuple<int, double>{}) {
  test("types " + std::to_string(arg)) = [arg] {
    expect(type(arg) == type<int> or type(arg) == type<double>);
  };
}
All tests passed (2 asserts in 2 tests)

https://cppx.godbolt.org/z/dMmqmM

    Is standardization an option?

Personally, I believe that C++ standard could benefit from common testing primitives (expect, ""_test) because

  • It lowers the entry-level to the language (no need for third-party libraries)
  • It improves the education aspect (one standard way of doing it)
  • It makes the language more coherent/stable (consistent design with other features, stable API)
  • It makes the testing a first class citizen (shows that the community cares about this aspect of the language)
  • It allows to publish tests for the Standard Library (STL) in the standard way (coherency, easier to extend)
  • It allows to act as additional documentation as a way to verify whether a particular implementation is conforming (quality, self-verification)
  • It helps with establishing standard vocabulary for testing (common across STL and other projects)

    Can I still use macros?

Sure, although please notice that there are negatives of using macros such as

  • Error messages might be not clear and/or point to the wrong line
  • Global scope will be polluted
  • Type safety will be ignored
#define EXPECT(...) ::boost::ut::expect(::boost::ut::that % __VA_ARGS__)
#define SUITE       ::boost::ut::suite _ = []
#define TEST(name)  ::boost::ut::detail::test{"test", name} = [=]() mutable

SUITE {
  TEST("suite") {
    EXPECT(42 == 42);
  };
};

int main() {
  TEST("macro") {
    EXPECT(1 != 2);
  };

  TEST("vector") {
    std::vector<int> v(5);

   EXPECT((5u == std::size(v)) >> fatal) << "fatal";

    TEST("resize bigger") {
      v.resize(10);
      EXPECT(10u == std::size(v));
    };
  };
}
All tests passed (4 asserts in 3 tests)

https://godbolt.org/z/WcEKTr

    What about Mocks/Stubs/Fakes?

Consider using one of the following frameworks

    What about Microbenchmarking?

Example benchmark

Consider using one of the following frameworks

    Related materials/talks?

    How to contribute?

CONTRIBUTING

Benchmarks

Framework Version Standard License Linkage Test configuration
Boost.Test 1.71.0 C++03 Boost 1.0 single header/library static library
GoogleTest 1.10.0 C++11 BSD-3 library static library
Catch 2.10.2 C++11 Boost 1.0 single header CATCH_CONFIG_FAST_COMPILE
Doctest 2.3.5 C++11 MIT single header DOCTEST_CONFIG_SUPER_FAST_ASSERTS
UT 1.1.0 C++20 Boost 1.0 single header/module
Include / 0 tests, 0 asserts, 1 cpp file
Assert / 1 test, 1'000'000 asserts, 1 cpp file
Test / 1'000 tests, 0 asserts, 1 cpp file
Suite / 10'000 tests, 0 asserts, 100 cpp files
Suite+Assert / 10'000 tests, 40'000 asserts, 100 cpp files
Suite+Assert+STL / 10'000 tests, 20'000 asserts, 100 cpp files
Incremental Build - Suite+Assert+STL / 1 cpp file change (1'000 tests, 20'000 asserts, 100 cpp files)
Suite+Assert+STL / 10'000 tests, 20'000 asserts, 100 cpp files
(Headers vs Precompiled headers vs C++20 Modules)

https://github.com/cpp-testing/ut-benchmark


Disclaimer UT is not an official Boost library.

Owner
boost::ext
Modern C++ libraries (not officially in Boost)
boost::ext
Comments
  • Source-file-location fails on macOS, giving 'unknown'

    Source-file-location fails on macOS, giving 'unknown'

    Expected Behavior

    The ApprovalTests.cpp UT integration tests pass with the latest version of this project on macOS.

    This combination passes:

    Actual Behavior

    On macOS, with both g++9 homebrew and clang-9 MacPorts, using 515c0803f1ffc2016a84dcfbb737d0ec8075bed4, the UT tests all fail with:

    *****************************************************************************
    *                                                                           *
    * Welcome to Approval Tests.
    *
    * There seems to be a problem with your build configuration.
    * We cannot find the test source file at:
    *   unknown
    *
    * For details on how to fix this, please visit:
    * https://github.com/approvals/ApprovalTests.cpp/blob/master/doc/TroubleshootingMisconfiguredBuild.md
    *                                                                           *
    *****************************************************************************
    

    Steps to Reproduce the Problem

    • Clone ApprovalTest.cpp: 63acddcf2d0833e5e757f801f3f3c821f110c581
    • Copy ut header 515c0803f1ffc2016a84dcfbb737d0ec8075bed4 over ApprovalTests.cpp/third_party/ut/include/boost/ut.hpp
    • Build on Mac
    • Run UT_Tests

    Specifications

    It's failing for me with:

    CMAKE_VERSION = 3.16.2
    CMAKE_GENERATOR = Unix Makefiles
    CMAKE_SOURCE_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp
    CMAKE_CURRENT_SOURCE_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp
    CMAKE_CURRENT_BINARY_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/cmake-build-debug-g-9-brew
    CMAKE_CXX_COMPILER = /usr/local/Cellar/gcc/9.2.0_2/bin/g++-9
    CMAKE_CXX_COMPILER_ID = GNU
    CMAKE_CXX_COMPILER_VERSION = 9.2.0
    CMAKE_UNITY_BUILD = 
    

    and

    CMAKE_VERSION = 3.16.2
    CMAKE_GENERATOR = Unix Makefiles
    CMAKE_SOURCE_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp
    CMAKE_CURRENT_SOURCE_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp
    CMAKE_CURRENT_BINARY_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/cmake-build-debug-clang-mp-90-brew
    CMAKE_CXX_COMPILER = /opt/local/bin/clang++-mp-9.0
    CMAKE_CXX_COMPILER_ID = Clang
    CMAKE_CXX_COMPILER_VERSION = 9.0.1
    CMAKE_UNITY_BUILD = 
    

    Likely cause

    There are not many differences between 8004290d195b08eda531eb015ddcda2e11d41ce9 and 515c0803f1ffc2016a84dcfbb737d0ec8075bed4

    I suspect that the cause is this, as in both compiler cases, __APPLE__ is defined:

    @@ -211,10 +211,10 @@ namespace reflection {
     class source_location {
      public:
       [[nodiscard]] static constexpr auto current(
    -#if (__GNUC__ >= 9 or __clang_major__ >= 9)
    +#if ((__GNUC__ >= 9 or __clang_major__ >= 9) and not defined(__APPLE__))
           const char* file = __builtin_FILE(), int line = __builtin_LINE()
     #else
    -      const char* file = {}, int line = {}
    +      const char* file = "unknown", int line = {}
     #endif
               ) noexcept {
         source_location sl{};
    
  • Errors popped up for Visual Studio 16.10 Preview 3

    Errors popped up for Visual Studio 16.10 Preview 3

    I'm posting this because I'm not sure if it's a problem with Visual Studio or Boost.ut or if it's me.

    Expected Behavior

    These worked in the 16.9.x versions of Visual Studio.

    Actual Behavior

    Errors popped up for Visual Studio 16.10 Preview 3

    Steps to Reproduce the Problem

    1. Install Visual Studio 16.10 Preview 3
    2. Try to compile
    3. Error

    Specifications

    • Version: boost.ut trunk
    • Platform: Windows 10 x64
    • Subsystem: ?
    ====================[ Build | all | Debug Windows ]=============================
    "C:\Program Files\JetBrains\CLion 2020.2\bin\cmake\win\bin\cmake.exe" --build D:\dev\ToolsLibrary\cmake-build-debug-windows --target all
    Scanning dependencies of target ToolsLibrary_random_test
    [  7%] Building CXX object tests/src/CMakeFiles/ToolsLibrary_random_test.dir/random_test.cpp.obj
    random_test.cpp
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): error C3378: a declaration can be exported only from a module interface unit
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): error C2230: could not find module 'boost.ut'
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(10): error C3378: a declaration can be exported only from a module interface unit
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(10): error C7635: a module import declaration cannot appear in an 'export' declaration sequence
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(16): warning C5244: '#include <ciso646>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <ciso646>;'.
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(16): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(55): warning C5244: '#include <array>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <array>;'.
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(55): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(56): warning C5244: '#include <iostream>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <iostream>;'.
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(56): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(57): warning C5244: '#include <sstream>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <sstream>;'.
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(57): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(58): warning C5244: '#include <string_view>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <string_view>;'.
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(58): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(60): warning C5244: '#include <vector>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <vector>;'.
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(60): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(70): error C3378: a declaration can be exported only from a module interface unit
    D:\dev\ToolsLibrary\src\tl/algorithm.hpp(9): warning C5244: '#include <algorithm>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <algorithm>;'.
    D:\dev\ToolsLibrary\src\tl/algorithm.hpp(9): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\src\tl/algorithm.hpp(10): warning C5244: '#include <cassert>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <cassert>;'.
    D:\dev\ToolsLibrary\src\tl/algorithm.hpp(10): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\src\tl/algorithm.hpp(11): warning C5244: '#include <numeric>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <numeric>;'.
    D:\dev\ToolsLibrary\src\tl/algorithm.hpp(11): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    D:\dev\ToolsLibrary\src\tl/random.hpp(8): warning C5244: '#include <random>' in the purview of module 'boost.ut' appears erroneous.  Consider moving that directive before the module declaration, or replace the textual inclusion with 'import <random>;'.
    D:\dev\ToolsLibrary\src\tl/random.hpp(8): note: to simplify migration, consider the temporary use of /Wv:18 flag with the version of the compiler with which you used to build without warnings
    D:\dev\ToolsLibrary\cmake-build-debug-windows\_deps\ut-src\include\boost/ut.hpp(9): note: see module 'boost.ut' declaration
    NMAKE : fatal error U1077: 'C:\PROGRA~2\MICROS~1\2019\Preview\VC\Tools\MSVC\1429~1.300\bin\Hostx86\x86\cl.exe' : return code '0x2'
    Stop.
    NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30035\bin\HostX86\x86\nmake.exe"' : return code '0x2'
    Stop.
    NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30035\bin\HostX86\x86\nmake.exe"' : return code '0x2'
    Stop.
    
  • Recent CMake changes in this project have broken builds of dependent projects

    Recent CMake changes in this project have broken builds of dependent projects

    Expected Behavior

    1. boost.ut target seems to no longer be created

    That the target boost.ut is still provided, to link against, when this project is obtained via add_subdirectory(), FetchContent() or similar...

    2. include file no longer found: #include <boost/ut.hpp>

    Example code - this is how I obtain the boost.ut project - UtVersion is master:

    https://github.com/claremacrae/ApprovalTests.cpp.CMakeSamples/blob/main/dev_approvals_fetch_content/CMakeLists.txt#L130-L133

    FetchContent_Declare(boost.ut
            GIT_REPOSITORY https://github.com/boost-ext/ut.git
            GIT_TAG ${UtVersion})
    FetchContent_MakeAvailable(boost.ut)
    

    The repo goes on to use the target boost.ut.

    This directory and GitHub action exists to help the ApprovalTests.cpp project have early detection of any changes in test frameworks that might affect our users, when we do our next release.

    This has been working for some months, and was still working on the 24th of Feb.

    Actual Behavior

    The next build, on 27th Feb at 01:10 GMT gave these errors

    1. in clang builds:

    Fetching boost.ut...
    CMake Error at CMakeLists.txt:137 (target_compile_options):
      Cannot specify compile options for target "boost.ut" which is not built by
      this project.
    

    2. in gcc builds:

    Building CXX object approvaltests.cpp_build/tests/UT_Tests/CMakeFiles/UT_Tests.dir/UTApprovalTestTests.cpp.o
    In file included from /home/runner/work/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp/ApprovalTests/ApprovalTests.hpp:68,
                     from /home/runner/work/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp/tests/UT_Tests/UTApprovalTestTests.cpp:3:
    /home/runner/work/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp/ApprovalTests/../ApprovalTests/integrations/ut/UTApprovals.h:14:10: fatal error: boost/ut.hpp: No such file or directory
       14 | #include <boost/ut.hpp>
          |          ^~~~~~~~~~~~~~
    compilation terminated.
    approvaltests.cpp_build/tests/UT_Tests/CMakeFiles/UT_Tests.dir/build.make:81: recipe for target 'approvaltests.cpp_build/tests/UT_Tests/CMakeFiles/UT_Tests.dir/UTApprovalTestTests.cpp.o' failed
    

    Steps to Reproduce the Problem

    Run this on a unix box with a recent version of either linux or gcc installed:

    git clone https://github.com/claremacrae/ApprovalTests.cpp.CMakeSamples
    cd ApprovalTests.cpp.CMakeSamples
    git clone https://github.com/approvals/ApprovalTests.cpp.git
    cd dev_approvals_fetch_content
    ./build.sh
    

    Wait until all the dependencies have been cloned, and then one of the errors will occur - depending on whether you are using gcc or clang.

    Specifications

    The failure was detected in a GitHub Actions build. The failing builds can be seen here:

    https://github.com/claremacrae/ApprovalTests.cpp.CMakeSamples/actions/runs/604457152

    • Linux Clang 6, 8, 9, and 10
    • Linux gcc 9 and 10
    • macOS Xcode 10.3, 11.7 and 12.

    It's likely that the ones that passed are skipping the building of boost.ut

  • Conan Center

    Conan Center

    What do you think about making a [Boost-ext].UT package available in ConanCenter? I would like to able to handle the dependency through Conan, and having it available in ConanCenter seems the easiest way to do so.

    Their instructions for doing so are available here. I found an example recipe for a header-only library on ConanCenter here.

  • Approval Test Integration Bug

    Approval Test Integration Bug

    Hi,

    I really like the [Boost].UT framework. It's clean and easy to use and does away with macros. Thanks for developing it!

    Anyways, I want to use approval tests (https://github.com/approvals/ApprovalTests.cpp) in a project I'm working on, and I'm implementing the integration between UT <> ApprovalTests.cpp. I followed the guidelines (https://github.com/approvals/ApprovalTests.cpp/blob/master/doc/SupportingNewTestFramework.md#top) and the UT reporter example (https://godbolt.org/z/gsAPKg). I made this simple example:

    #include <string>
    
    #include "ut.hpp"
    #include "ApprovalTests.hpp"
    
    namespace ut = boost::ut;
    
    namespace ApprovalTests
    {
        namespace cfg {
            class reporter : public ut::reporter<ut::printer> {
                private:
                    TestName currentTest;
    
                public:
                    auto on(ut::events::test_begin test_begin) -> void {
                        std::string name = test_begin.name;
                        currentTest.sections.emplace_back(name);
                        ut::reporter<ut::printer>::on(test_begin);
                    }
                    auto on(ut::events::test_run test_run) -> void { 
                        ut::reporter<ut::printer>::on(test_run);
                    }
                    auto on(ut::events::test_skip test_skip) -> void { ut::reporter<ut::printer>::on(test_skip); }
                    auto on(ut::events::test_end test_end) -> void {
                        while (!currentTest.sections.empty()) {
                            currentTest.sections.pop_back();
                        }
                        ut::reporter<ut::printer>::on(test_end);
                    }
                    template <class TMsg>
                    auto on(ut::events::log<TMsg>) -> void {}
                    template <class TLocation, class TExpr>
                    auto on(ut::events::assertion_pass<TLocation, TExpr> location) -> void {
                        currentTest.setFileName(location.location.file_name());
                        ApprovalTestNamer::currentTest(&currentTest);
                        ut::reporter<ut::printer>::on(location);
                    }
                    template <class TLocation, class TExpr>
                    auto on(ut::events::assertion_fail<TLocation, TExpr> fail) -> void { ut::reporter<ut::printer>::on(fail); }
                    auto on(ut::events::fatal_assertion fatal) -> void { ut::reporter<ut::printer>::on(fatal); }
                    auto on(ut::events::exception exception) -> void { ut::reporter<ut::printer>::on(exception); }
                    auto on(ut::events::summary summary) -> void { ut::reporter<ut::printer>::on(summary); }
            };
        }  // namespace cfg
    }
    
    template <>
    auto ut::cfg<ut::override> = ut::runner<ApprovalTests::cfg::reporter>{};
    
    int main()
    {
        using namespace ut;
        using namespace ApprovalTests;
    
    	"Approval"_test = []() {
            //expect(true);
            expect(nothrow([] { Approvals::verify("Approval Tests can verify text via the golder master method"); }));
    	};
    
    }
    

    The issue is unless the line containing expect(true); is uncommented it doesn't work. It complains that currentTest was not configured. This seems more like a UT issue than an ApprovalTests issue.

    Can anyone help? I'm not familiar enough with the UT code logic to track this issue. Or am I wrong and this is an ApprovalTests.cpp issue?

    I appreciate any help.

  • fix cmake install config package

    fix cmake install config package

    use CPM.cmake to install the project cleanup cmake file format cmake file fix compile errors fix some clang-tidy warnings add some Notes to source

    Problem:

    • dit not compile on OSX with gcc-10, clang-11, AppleClangV12

    Solution:

    • use target_link_library()
    • and change some compiler flags
    • correct source code

    Issue: this fix https://github.com/boost-ext/ut/issues/420 too

    Reviewers: @

  • :bug: :art: [exception] Tests that throw unexpected exceptions appear…

    :bug: :art: [exception] Tests that throw unexpected exceptions appear…

    … to pass, by returning 0 exit status, Fix #299

    Problem:

    • fails counter is not properly incremented in the case when an exception is thrown.
    • There are unused boolean exception variables.

    Solution:

    • Update fails count in exception handling.
    • Remove them.
  • Improve readability and glancability of docs in README.txt

    Improve readability and glancability of docs in README.txt

    Expected Behavior

    That it's easy to search text, and see the whole structure, of the docs on the front page:

    https://github.com/boost-experimental/ut

    Actual Behavior

    • Most sections on the front page are collapsed, and some require two levels of expansion of triangles to see the content
    • Searching in the browser (Ctrl+F) doesn't find text in collapsed sections
    • I've seen someone who is more familiar with UT than me still click on 2 or 3 triangles before finding the code example they wanted to show me

    Specifications

    • Testing in both Firefox and Chrome (Windows 10)

    Suggestions

    • Perhaps provide a button at the top that opens all the triangles all the way down
    • Or just remove the triangles and collapsing altogether
  • Conan version 1.1.8 is broken on newer msvc.

    Conan version 1.1.8 is broken on newer msvc.

    I don't think Conan will push out a version with out a new numbered release. Since 1.1.8 is from Aug 2020. I was wondering if you could push out a new build using the current version of the code. So that Conan could release it.

    We needed a way to turn Modules off. And it only exists in the current code it doesn't exist in the current released build.

  • Build errors on Visual Studio with clang-cl compiler when using boost.ut's CMakeLists.txt

    Build errors on Visual Studio with clang-cl compiler when using boost.ut's CMakeLists.txt

    Expected Behavior

    Boost.ut builds and tests pass with VS2019 MSVC with clang-cl compiler

    I know this, as we have been building that with the Approval Tests integration, which works file with this compiler.

    Actual Behavior

    Builds fail, with the error:

      Building Custom Rule D:/a/ApprovalTests.cpp.Builds/ApprovalTests.cpp.Builds/_deps/ut-src/example/CMakeLists.txt
    clang-cl : error : unknown argument ignored in clang-cl: '-pedantic' [-Werror,-Wunknown-argument] [D:\a\ApprovalTests.cpp.Builds\ApprovalTests.cpp.Builds\_deps\ut-build\example\BDD.vcxproj]
    clang-cl : error : unknown argument ignored in clang-cl: '-pedantic-errors' [-Werror,-Wunknown-argument] [D:\a\ApprovalTests.cpp.Builds\ApprovalTests.cpp.Builds\_deps\ut-build\example\BDD.vcxproj]
    

    For an example failure, see https://github.com/claremacrae/ApprovalTests.cpp.Builds/runs/370748554

    Steps to Reproduce the Problem

    1. cmake -G "Visual Studio 16 2019" -T "clangcl"
    2. build all of Boost.ut, including examples

    Specifications

    • Version: 8004290d195b08eda531eb015ddcda2e11d41ce9
    • Platform: 'The CXX compiler identification is Clang 9.0.0 with MSVC-like command-line'
    • Subsystem: ?

    Nature of fix

    See https://github.com/approvals/ApprovalTests.cpp/blob/v.7.0.0/tests/UT_Tests/CMakeLists.txt#L17 for the type of check to use...

    Someone previously reported this problem with ApprovalTests.cpp on this environment, and provided a fix which used windows-style compiler warnings if using clang-cl...

  • Print type name in failed templated test

    Print type name in failed templated test

    It would be really nice if a failing template test prints the type name for which it failed.

    So for this code,

    "test"_test = []<class T>{
      expect(std::same_as<bool, T>);
    } | std::tuple<bool, int>{};
    

    ut should print something like

    Running "test<bool>"...PASSED
    Running "test<int>"...
      test.cpp:2:FAILED [false]
    FAILED
    
  • bus error exception with msgpack header

    bus error exception with msgpack header

    Expected Behavior

    exit ok without exception

    Actual Behavior

    Bus error***Exception occurs

    Steps to Reproduce the Problem

    the following empty test will yield Bus error***Exception

    #include <msgpack.hpp>
    #include <boost/ut.hpp>
    
    int main() {
        boost::ut::test("bus_error") = [] {};
    }
    

    note that, move boost/ut.hpp header to top workaround this issue

    #include <boost/ut.hpp> // now it is fine
    #include <msgpack.hpp>
    
    int main() {
        boost::ut::test("bus_error") = [] {};
    }
    

    removing the empty test case also suppress the issue

    #include <msgpack.hpp>
    #include <boost/ut.hpp>
    
    int main() {
        // no exception now
        // boost::ut::test("bus_error") = [] {};
    }
    
    

    Specifications

    • Version: boost-ext-ut/1.1.9 from conan
    • Platform: mac os Apple m1 and Linux amd64
    • Subsystem: g++ 11 arm and amd64
  • Operator ambiguity error with MSVC STL ranges

    Operator ambiguity error with MSVC STL ranges

    Expected Behavior

    ut should not interfere with usage of STL.

    Actual Behavior

    With MSVC STL, inclusion of ut.hpp alongside <ranges> results in operator | overload ambiguity errors.

    Steps to Reproduce the Problem

    The following is sufficient to reproduce:

    #define BOOST_UT_DISABLE_MODULE
    #include <boost/ut.hpp>
    #include <ranges>
    #include <functional>
    
    using namespace boost::ut;
    
    auto const foo = std::ranges::views::iota(0, 10) | std::ranges::views::transform(std::identity{});
    

    [Compiler Explorer] Experimenting with the above link, GCC succeeds, Clang succeeds with libc++ but fails with libstdc++ as well as with MSVC's implementation.

    Specifications

    • Version: v1.1.9
    • Platform: Windows
    • Compiler: MSVC (14.31.31103) & Clang (15.0 from source)
  • Provide a name for test suites

    Provide a name for test suites

    A nice feature to have would be to have a name for a test suite that shows up in the error messages when a test fails. Maybe something like this?

    Starting test suite "Serial Communication Utilities"
    ==========
    Running "[success] read<Length>"...PASSED
    Running "[failure bytes_available] read<Length>"...PASSED
    Running "[failure read] read<Length>"...PASSED
    Running "[success] write_then_read"...PASSED
    Running "[failure bytes_available] write_then_read"...PASSED
    Running "[failure read] write_then_read"...PASSED
    Running "[failure on timeout] read"...
      util.test.cpp:306:FAILED [nullptr == 0x16d6a5bf0]
      util.test.cpp:307:FAILED [0 == 1024]
    FAILED
    Running "[failure on write] write_then_read"...PASSED
    Running "[failure on timeout] write_then_read"...
      util.test.cpp:354:FAILED [nullptr == 0x16d6a5bc0]
      util.test.cpp:355:FAILED [0 == 1024]
    FAILED
    ===========
    Ending test suite "Serial Communication Utilities"
    

    I'm open to any way of presenting this. Could even put the test suite name before the test name.

  • `throws` with message

    `throws` with message

    It would be nice if ut permitted to check that an exception's message (.what()) matches some expected pattern, like with GoogleTest's ThrowsMessage matcher.

    I was thinking about something along these lines:

    // Add new throws_message function
    "exception_message"_test = [] {
    	expect(throws_message([] { throw std::runtime_error("feature_request"); }, "feature_request"));
    };
    
    // Add an overload to the existing throws function
    "exception_message"_test = [] {
    	expect(throws([] { throw std::runtime_error("feature_request"); }, "feature_request"));
    };
    
  • Custom reporter to only show errors

    Custom reporter to only show errors

    Expected Behavior

    Ability to customize reporter

    Actual Behavior

    Summary called twice

    Steps to Reproduce the Problem

    Using your exemple in https://github.com/boost-ext/ut/blob/master/example/cfg/reporter.cpp I tried to create a reporter that would show only the tests with failures I'm getting a crash because the summay is called twice, first time from my own call to run(), and the second time from the destructor of runner that I can't use because of std containers used in functions beeing tested.

    1.https://godbolt.org/z/9Ej6EsKq6

    Specifications

    • Version:
    • Platform:
    • Subsystem:

    Thank you very much Nelson

UT: C++20 μ(micro)/Unit Testing Framework
UT: C++20 μ(micro)/Unit Testing Framework

"If you liked it then you "should have put a"_test on it", Beyonce rule UT / μt | Motivation | Quick Start | Overview | Tutorial | Examples | User Gui

Sep 28, 2022
A micro unit-testing library for C/C++

µ-test A micro unit testing framework for C/C++ to get you up and running with unit-testing ASAP (even without libc). Usage Simply include the C and h

Dec 8, 2021
A complete unit testing framework in a header

liblittletest A complete unit testing framework in a header liblittletest is an easy to use all-in-an-header testing framework; all you have to do in

Nov 11, 2021
Modern c++17 unit testing framework on Microsoft Windows, Apple macOS, Linux, iOS and android.
Modern c++17 unit testing framework on Microsoft Windows, Apple macOS, Linux, iOS and android.

tunit Modern c++17 unit testing framework on Windows, macOS, Linux, iOS and android. Continuous Integration build status Operating system Status Windo

Apr 5, 2022
Kernel-mode C++ unit testing framework in BDD-style

There is a lack of unit testing frameworks that work in OS kernel. This library closes that gap and is targeted for windows driver developers.

Jun 11, 2022
A C++ micro-benchmarking framework

Nonius What is nonius? Nonius is an open-source framework for benchmarking small snippets of C++ code. It is very heavily inspired by Criterion, a sim

Apr 13, 2022
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)

Catch2 v3 is being developed! You are on the devel branch, where the next major version, v3, of Catch2 is being developed. As it is a significant rewo

Sep 26, 2022
A modern, C++11-native, single-file header-only, tiny framework for unit-tests, TDD and BDD (includes C++98 variant)

lest – lest errors escape testing This tiny C++11 test framework is based on ideas and examples by Kevlin Henney [1,2] and on ideas found in the CATCH

Sep 19, 2022
Various Framework to do Unit Test in C++
Various Framework to do Unit Test in C++

Unit Test in C++ There are many frameworks to performs unit test in C++, we will present the most popular ones and show how to use them. The testing f

Nov 18, 2021
Upp11 - C++11 lightweight single header unit test framework

upp11 Lightweight C++11 single header unit test framework To use framework: Copy upp11.h in you project dir. Create unit test source files or modify e

Apr 4, 2019
The fastest feature-rich C++11/14/17/20 single-header testing framework
The fastest feature-rich C++11/14/17/20 single-header testing framework

master branch Windows All dev branch Windows All doctest is a new C++ testing framework but is by far the fastest both in compile times (by orders of

Sep 26, 2022
Googletest - Google Testing and Mocking Framework

GoogleTest OSS Builds Status Announcements Release 1.10.x Release 1.10.x is now available. Coming Soon Post 1.10.x googletest will follow Abseil Live

Sep 24, 2022
C++ xUnit-like testing framework without macros

tst C++ testing framework. Installation, documentation, tutorials See WiKi. Features xUnit-like concepts minimal use of preprocessor macros declarativ

Sep 26, 2022
c++ testing framework

iutest iutest - iris unit test framework Welcome to the iutest iutest is framework for writing C++ tests. Features An XUnit test framework. Header onl

Sep 12, 2022
Simple C testing framework

MrTest Simple C testing framework Usage Copy the mrtest.c and mrtest.h file into your project. In order to use the mrtest main: create a .c file that

Jul 20, 2022
xtest is a C++ testing framework inspired by googletest.
xtest is a C++ testing framework inspired by googletest.

xtest C++ testing framework inspired by googletest Explore the docs » Wiki · Report Bug · Request Feature Contents xtest Commence Prerequisites Ubuntu

Jul 9, 2022
A minimal testing framework for C/C++
A minimal testing framework for C/C++

mtest About mtest is a minimal testing framework for C++. Requirements Windows or UNIX-like host A compiler supporting C++11 Usage To include mtest in

Jan 10, 2022
The fastest feature-rich C++11/14/17/20 single-header testing framework
The fastest feature-rich C++11/14/17/20 single-header testing framework

master branch dev branch doctest is a new C++ testing framework but is by far the fastest both in compile times (by orders of magnitude) and runtime c

Sep 29, 2022
A dynamic mock tool for C/C++ unit test on Linux&MacOS X86_64

lmock 接口 替换一个函数,修改机器指令,用新函数替换旧函数,支持全局函数(包括第三方和系统函数)、成员函数(包括静态和虚函数)

Sep 20, 2022