Native ApprovalTests for C++ on Linux, Mac and Windows

Approval Tests for C++

Build status Actions Status Python tests Documentation Status License Contributor Covenant ConanCenter package

⬇️ Download the latest version (v.10.8.0) of the single header file here.

📖 Read the Docs

Contents

What are Approval Tests?

Also known as Golden Master Tests or Snapshot Testing, Approval Tests are an alternative to asserts. If you are unfamiliar with them, we have an overview and a tutorial.

They are great for testing objects with lots of fields, or lists of objects.

Requirements

Getting Started

ToString (ostream insertion)

Often, you will need to create functions to allow objects to print their state. This is commonly done with an ostream << operator. You can find examples here: To String

What's new?

Check out the Features page or upcoming release notes to see what we've been working on lately, or browse the past release notes.

Feedback

If you have any comment or suggestion on this documentation, please email Llewellyn or Clare via the details in the Contributing page.

Comments
  • Document Ninja Generator Issue

    Document Ninja Generator Issue

    Documentation is buried and lacking about a critical bug when using the Ninja generator. Any test framework integration, such as Catch2, that uses the __FILE__ macro to locate Approval Test source files will fail when using Ninja. This is not only for MSVC as referenced in the current documentation but for any time the Ninja generator is used. I recently started to use the Ninja generator for all of my builds, so was caught unaware of this issue when using LLVM's Clang with Ninja on macOS. It should also be noted that the Ninja generator is not a problem when using Boost ut because it uses std::source_location instead of the __FILE__ macro. I tested this successfully just to be sure. If this bug is documented before the troubleshooting page, perhaps in the pages on the effected test frameworks, user's may avoid this issue entirely.

  • Tests fail when using Ninja generator (in VS2019)

    Tests fail when using Ninja generator (in VS2019)

    CMake in VS2019 seems to default to using the Ninja generator.

    If I build with this and run the tests I get exceptions thrown.

    1: Catch1_Tests.exe is a Catch v1.12.1 host application. 1: Run with -? for options 1: 1: ------------------------------------------------------------------------------- 1: YouCanVerifyText 1: ------------------------------------------------------------------------------- 1: ..\..\..\tests\Catch1_Tests\ApprovalsTests.cpp(6) 1: ............................................................................... 1: 1: ..\..\..\tests\Catch1_Tests\ApprovalsTests.cpp(6): FAILED: 1: due to unexpected exception with message: 1: Unable to write file: ..\..\..\tests\Catch1_Tests\ApprovalsTests. 1: YouCanVerifyText.received.txt 1: 1: =============================================================================== 1: test cases: 1 | 1 failed 1: assertions: 1 | 1 failed

    If I use the Visual Studio 2019 generator then all seems fine

  • Provide a way to select the Reporter at run-time

    Provide a way to select the Reporter at run-time

    I’d really like to remove the need to edit and recompile to change the reporter used, probably by providing a command-line argument to do so.

    We could base the option names on the reporter classes, with the Reporter suffix removed.

    Examples that spring to mind:

    -r BeyondCompareReporter
    --reporter Quiet
    -r Clipboard
    -r AutoApprove
    -r ‘C:\path to my program\...exe’
    -r ‘mydifftool -1 {received} -2 {approved} &’
    -r ‘someconsolediff {received} {approved}’
    

    We would want to control whether the tool runs in the foreground or background, if it launches another process.

  • Move only disposer

    Move only disposer

    Description

    I came across this problem when writing an utility method, which registers the same comparator for multiple file extensions: link.

    In linked code, Disposer class wraps a vector of ComparatorDisposer objects. Each ComparatorDisposer is created by FileApprover::registerComparatorForExtension, copied to a vector and destructed and at that point the comparator is immediately disposed.

    Expected behavior is that the comparator is disposed once the copy in the vector is destructed. Destructor of the original ComparatorDisposer instance should not dispose the comparator.

    The solution

    I've added a flag isActive, which is initially set to true. The flag is set to false when a ComparatorDisposer is moved-from, to indicate that it's destructor should not restore previousComparator.

    Additionally, I've declared copy constructor as deleted.

  • Change Catch2 includes to catch2/catch.hpp

    Change Catch2 includes to catch2/catch.hpp

    From the current maintainer of Catch2, after we announced we had changed from Catch.hpp to catch.cpp:

    "I strongly recommend going with #include "catch2/catch.hpp"

    Because if you use Catch2 via CMake or having it installed, the include path will be <catch2/catch.hpp>"

  • Increase the number of collections in `verifyAllCombinations` beyond 9

    Increase the number of collections in `verifyAllCombinations` beyond 9

    Currently, the number of collections in verifyAllCombinations is limited to 9, and is extensible by code by the user of the library. Some legacy code has functions with more than 9 parameters, sometimes many more. It would be more convenient if the user didn't have to write the extension themselves. This issue suggests to increase the number of collections that the library can support natively, ideally without limit, by using variadic templates. If I'm not mistaken, Clare already mentioned this could be an evolution for the library. This issue is then to log this request. Keep up the amazing work! :)

  • Try speeding up CI builds

    Try speeding up CI builds

    Feedback is painfully slow from our Travis and AppVeyor CI builds now:

    • Travis: 40 - 45 minutes elapsed time
      • 13 configurations - each taking 3 to 4 minutes
    • AppVeyor - 18 minutes
      • Cygwin: 11 minutes
      • Mingw 7.5 minutes
      • Some of the pain here is the queue time, which is probably outside our control
  • Naming of targets in third_party is non-standard

    Naming of targets in third_party is non-standard

    If the Catch2 project is included via CMake's add_subdirectory() or FetchContent, then the following targets are created, as far as I can tell:

    • Catch2
    • Catch2::Catch2

    Unfortunately I didn't appreciate the significance of this when creating third_party/catch2/CMakeLists.txt - which creates the target catch

    The name Catch2::Catch2 is preferred, as if that is missing, a warning is issued when CMake runs, making it easier to track down missing dependencies.

    I think it should be possible to create aliases in the CMake files in third_party to retain the old naming, in case any users are already depending on the third_party target-names I created earlier.

  • F!! [Boost].UT test framework integration support

    F!! [Boost].UT test framework integration support

    Hi,

    This is the initial implementation of the [Boost].UT test framework integration support, developed alongside @claremacrae I see that I still have to build the tests for the approval tests of the integration with the test framework :) But please take a look to see if anything else is missing and if there are changes needed for what's been done.

  • Project does not compile with Clang

    Project does not compile with Clang

    When I try to compile the ApprovalTests.cpp project with Clang, I receive this error message:

    In file included from /Users/arh/code/ApprovalTests.cpp/src/ApprovalTests_Catch2_Tests/CombinationTests.cpp:8:
    /Users/arh/code/ApprovalTests.cpp/src/ApprovalTests/../ApprovalTests/CombinationApprovals.h:102:58: error: call to function 'operator<<' that is neither visible in the template definition nor found by argument-dependent lookup
                                                s << ") => " << result << '\n';
                                                             ^
    

    Clang is actually correct here, and the problem is that gcc and msvc accept incorrect code. The situation is described in detail at http://clang.llvm.org/compatibility.html#dep_lookup and the second example describes exactly the case in ApprovalTests.

    There are two obvious ways to make the problem go away quickly:

    1. Put the operator<< overload in CombinationTests.cpp inside the std namespace, so that it can be found by ADL in the verifyAllCombinations function template. Unfortunately this is illegal, as we're only allowed to put full specializations in to the std namespace.
    2. Predeclare the operator<< overload before including the CombinationApprovals.h header.

    This Godbolt example contains a minimal example: https://godbolt.org/z/nReFWZ

    I'm probably going outside the scope of this Issue now, but I believe a better approach would be to provide a customization point that allows users to provide custom string conversions for their own types, or for types that do not easily support a stream insertion operator. Sometimes third-party types do provide a stream insertion operator, but its output isn't acceptable for some reason. In those cases, it's helpful to have a separate mechanism for stringifying the type.

    The Catch2 library has a nice mechanism for this: https://github.com/catchorg/Catch2/blob/master/docs/tostring.md The Boost unit test library does it too: https://www.boost.org/doc/libs/1_68_0/libs/test/doc/html/boost_test/test_output/test_tools_support_for_logging/testing_tool_output_disable.html

    Neither of those implementations could be described as 'trivial', but I think that some form of customization point that goes beyond requiring an operator<< overload would be advantageous for ApprovalTests.cpp.

    Whilst looking in to this, I found a useful reference on best practices for providing customization points: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4381.html

  • Use namespace instead of class Approvals

    Use namespace instead of class Approvals

    All classes should be put inside namespace. And the class Approvals should be removed since it is just a collection of static functions. They should be free functions inside the namespace.

    The name of the namespace should be ideally ApprovalTests. Downstream users would then use

    ApprovalTests::verify(...);
    
  • Bump DoozyX/clang-format-lint-action from 0.14 to 0.15

    Bump DoozyX/clang-format-lint-action from 0.14 to 0.15

    Bumps DoozyX/clang-format-lint-action from 0.14 to 0.15.

    Release notes

    Sourced from DoozyX/clang-format-lint-action's releases.

    v0.15

    Commits
    • c3b2c94 Merge pull request #48 from kalabukdima/master
    • 7cd7220 Merge pull request #49 from SK83RJOSH/master
    • acc83cb Support clang-format 15
    • a69aedf Colorize output in github action
    • 33bcef1 Merge pull request #46 from heisencoder/master
    • 2e526c4 Update examples in README.md with latest version
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump actions/checkout from 2 to 3.1.0

    Bump actions/checkout from 2 to 3.1.0

    Bumps actions/checkout from 2 to 3.1.0.

    Release notes

    Sourced from actions/checkout's releases.

    v3.1.0

    What's Changed

    New Contributors

    Full Changelog: https://github.com/actions/checkout/compare/v3.0.2...v3.1.0

    v3.0.2

    What's Changed

    Full Changelog: https://github.com/actions/checkout/compare/v3...v3.0.2

    v3.0.1

    v3.0.0

    • Updated to the node16 runtime by default
      • This requires a minimum Actions Runner version of v2.285.0 to run, which is by default available in GHES 3.4 or later.

    v2.4.2

    What's Changed

    Full Changelog: https://github.com/actions/checkout/compare/v2...v2.4.2

    v2.4.1

    • Fixed an issue where checkout failed to run in container jobs due to the new git setting safe.directory

    v2.4.0

    • Convert SSH URLs like org-<ORG_ID>@github.com: to https://github.com/ - pr

    v2.3.5

    Update dependencies

    v2.3.4

    v2.3.3

    ... (truncated)

    Changelog

    Sourced from actions/checkout's changelog.

    v3.1.0

    v3.0.2

    v3.0.1

    v3.0.0

    v2.3.1

    v2.3.0

    v2.2.0

    v2.1.1

    • Changes to support GHES (here and here)

    v2.1.0

    v2.0.0

    v2 (beta)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Catch2 integration does not handle Generators inside sections

    Catch2 integration does not handle Generators inside sections

    Per Catch2 documentation:

    GENERATE can be seen as an implicit SECTION, that goes from the place GENERATE is used, to the end of the scope. [...] The fact that GENERATE introduces a virtual SECTION can also be used to make a generator replay only some SECTIONs, without having to explicitly add a SECTION

    I looked into the Catch2 integration and it seems like the Catch2 API does not include generator information as part of the sector:

    // Approvals/integrations/catch/Cathc2Approvals.h
    
    struct Catch2TestCommitRevert : Catch::TestEventListenerBase
    {
        ...
        virtual void sectionStarting(Catch::SectionInfo const& sectionInfo) override
        {
            currentTest.sections.push_back(sectionInfo.name); // Name here does not include the generator value
        }
        ...
    };
    

    In fact, if you look at the Catch::SectionInfo type it only includes name and source location. I'm no expert on the Catch2 internals but at first glance, I would say Catch2 generators are handled more like part of the body of a section rather than variants of the parent section (Like generated tests cases in other frameworks).

    The result of this behavior is that if you run an approval inside a section with a generator, the approval is uniquely registered for the section rather than each generated variant of the section, resulting in approval failures following the execution of the different variants of the section:

    enum class Keyword
    {
        Unknown, Foo, Bar, Quux
    };
    
    Keyword parseKeyword(const std::string_view word);
    
    SECTION("Verify our parsing algorithm works with the most relevant words in the vocabulary of a programmer")
    {
         const std::string_view word = GENERATE("foo", "bar", "quux");
         Approvals::verify(parseKeyword(word)); // If we approve "foo", it fails in the next iteration of the generator
    }
    
  • with CppUTest: `ApprovalMismatchException` suppresses other tests output

    with CppUTest: `ApprovalMismatchException` suppresses other tests output

    see example here

    the example shows a test file that has two failing tests - one of them is using approvals, the other is a non-approvals test.

    expected

    both failures should be shown

    actual

    the non-approvals test output is not seen in the terminal

  • "Unable to create directory" - unable to run test build with mingw provided by qt-installer

    Hello, I want to use approval test for some legacy embedded C code. Thus my c++/cmake knowledge is very limited.

    My Setup:

    • Windows 10
    • ApprovalTests 10.12.2
    • CMake 3.21.1 -- provided by qt-installer
    • MinGw64 11.2.0 -- provided by qt-installer for qt 6.3.0
    • QT Creator 7.0.1

    I get the same error message as mentioned in #195 I took his example files and created a new cmake project via QT Creator

    __FILE__ == C:/Users/florian.fischer/Documents/inmach/src/approvalTest_problem/path_problem_demo/test.cpp
    
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    path_problem_demo.exe is a Catch v2.13.9 host application.
    Run with -? for options
    
    -------------------------------------------------------------------------------
    MyTestCase
    -------------------------------------------------------------------------------
    C:/Users/florian.fischer/Documents/inmach/src/approvalTest_problem/path_problem_demo/test.cpp:5
    ...............................................................................
    
    C:/Users/florian.fischer/Documents/inmach/src/approvalTest_problem/path_problem_demo/test.cpp:5: FAILED:
    due to unexpected exception with message:
      Unable to create directory: 
    
    ===============================================================================
    test cases: 1 | 1 failed
    assertions: 1 | 1 failed
    
    

    I "solved" it by modifying

    std::string SystemUtils::getDirectorySeparator()
    {
        //return isWindowsOs() ? "\\" : "/";
        return "/";
    }
    

    But then it fails when i want to use AutoApproveIfMissingReporter because copy does not like the paths with / instead of \

A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)

Catch2 v3 is being developed! You are on the devel branch, where the next major version, v3, of Catch2 is being developed. As it is a significant rewo

Nov 28, 2022
A modern, C++11-native, single-file header-only, tiny framework for unit-tests, TDD and BDD (includes C++98 variant)

lest – lest errors escape testing This tiny C++11 test framework is based on ideas and examples by Kevlin Henney [1,2] and on ideas found in the CATCH

Nov 26, 2022
testing joystick under Linux environment, support monitoring disconnection state and auto recovery mode

qjoystick This qjoystick class is rewritten based on the library: https://github.com/drewnoakes/joystick. Please look at this library if you want to s

Oct 30, 2021
A dynamic mock tool for C/C++ unit test on Linux&MacOS X86_64

lmock 接口 替换一个函数,修改机器指令,用新函数替换旧函数,支持全局函数(包括第三方和系统函数)、成员函数(包括静态和虚函数)

Nov 22, 2022
TestFrame - This is a test framework that uses Raylib and ImGui together to help test and develop concepts
TestFrame - This is a test framework that uses Raylib and ImGui together to help test and develop concepts

This is a test framework that uses Raylib and ImGui together to help test and develop concepts. It is made using C++ because it uses classes for windows and views.

May 13, 2022
C++ Testing using spies and fakes for isolation and simulation
C++ Testing using spies and fakes for isolation and simulation

ELFspy is a library for linux for writing tests using fakes and spies in C++. For very large call graphs, the testing of the higher nodes can be diffi

Nov 22, 2022
Googletest - Google Testing and Mocking Framework

GoogleTest OSS Builds Status Announcements Release 1.10.x Release 1.10.x is now available. Coming Soon Post 1.10.x googletest will follow Abseil Live

Nov 27, 2022
Practical mutation testing tool for C and C++

Mull Mull is a tool for Mutation Testing based on LLVM/Clang with a strong focus on C and C++ languages. For installation and usage please refer to th

Nov 15, 2022
An area to test reading in ATLAS xAOD format and writing out to Parquet

xaod_to_parquet An area to test reading in ATLAS xAOD format and writing out to Parquet Getting the Code Clone the repository with the --recursive fla

Nov 19, 2021
Xerus - A general purpose library for numerical calculations with higher order tensors, Tensor-Train Decompositions / Matrix Product States and other Tensor Networks

About The xerus library is a general purpose library for numerical calculations with higher order tensors, Tensor-Train Decompositions / Matrix Produc

Apr 20, 2021
A simple, cross-platform, and continuously integrated C++14 project template

Project Name A simple, cross-platform, and continuously integrated C++14 project template. Making cross platform C++ projects is widely known to be a

Jul 27, 2022
The Hoard Memory Allocator: A Fast, Scalable, and Memory-efficient Malloc for Linux, Windows, and Mac.

The Hoard Memory Allocator Copyright (C) 1998-2020 by Emery Berger The Hoard memory allocator is a fast, scalable, and memory-efficient memory allocat

Nov 20, 2022
Audacity is an easy-to-use, multi-track audio editor and recorder for Windows, Mac OS X, GNU/Linux and other operating systems
Audacity is an easy-to-use, multi-track audio editor and recorder for Windows, Mac OS X, GNU/Linux and other operating systems

Audacity is an easy-to-use, multi-track audio editor and recorder for Windows, Mac OS X, GNU/Linux and other operating systems. Audacity is open source software licensed under GPL, version 2 or later.

Nov 30, 2022
Sneedacity (formerly Audacity) is an easy-to-use, multi-track audio editor and recorder for Windows, Mac OS X, GNU/Linux and other operating systems.
Sneedacity (formerly Audacity) is an easy-to-use, multi-track audio editor and recorder for Windows, Mac OS X, GNU/Linux and other operating systems.

Sneedacity (formerly Audacity) is an easy-to-use, multi-track audio editor and recorder for Windows, Mac OS X, GNU/Linux and other operating systems. Sneedacity is open source software licensed under GPL, version 2 or later.

Nov 28, 2022
Visualization Library is a C++ middleware for high-performance 2D and 3D graphics applications based on OpenGL 1.x-4.x supporting Windows, Linux and Mac OS X.

Visualization Library 2.2 Gallery About Visualization Library is a C++ middleware for high-performance 2D and 3D graphics applications based on the in

Nov 8, 2022
Oxygine is C++ engine and framework for 2D games on iOS, Android, Windows, Linux and Mac

BUILD AND RUN See oxygine-framework/readme/ folder. It has instructions on how to build and run oxygine on different platforms. Wiki available at http

Nov 24, 2022
GB Studio is a quick and easy to use retro adventure game creator for Game Boy available for Mac, Linux and Windows
GB Studio is a quick and easy to use retro adventure game creator for Game Boy available for Mac, Linux and Windows

GB Studio is a quick and easy to use retro adventure game creator for Game Boy available for Mac, Linux and Windows

Nov 30, 2022
Alpha Plot is a free application for Scientific Data Analysis and Visualization for Windows, Linux and Mac OS X
Alpha Plot is a free application for Scientific Data Analysis and Visualization for Windows, Linux and Mac OS X

Alpha Plot is a free application for Scientific Data Analysis and Visualization for Windows, Linux and Mac OS X (probably BSD also). Web Link Website

Nov 20, 2022
Slate is a bitmap editor available for Linux, Windows and Mac.
Slate is a bitmap editor available for Linux, Windows and Mac.

Slate is a bitmap editor available for Linux, Windows and Mac.

Nov 25, 2022
Serial Data Monitor is a multiplatform (Windows, Linux, Mac, ...) tool to interactively receive/edit/monitor data and send commands to an embedded system via the serial bus
Serial Data Monitor is a multiplatform (Windows, Linux, Mac, ...) tool to interactively receive/edit/monitor data and send commands to an embedded system via the serial bus

See wiki for full documentation Serial Data Monitor Description Serial Data Monitor is a multiplatform (Windows, Linux, Mac, ...) tool to interactivel

Oct 29, 2021