C++17 library for all your binary de-/serialization needs

blobify

blobify is a header-only C++17 library to handle binary de-/serialization in your project. Given a user-defined C++ struct, blobify can encode its data into a compact binary stream and, conversely, load a structure from its binary encoding. Unlike other serialization frameworks, blobify requires no boilerplate code and instead infers the serialized layout from the structure definition alone. Customizations to the default behavior are enabled by specifying properties using an embedded domain specific language.

Common applications of blobify include parsing file formats, network communication, and device drivers.

What is it?

Consider bitmap loading: You'd typically represent bitmap header elements using C++ structs and enums, such as:

enum class Compression : uint32_t {
    None      = 0,
    RLE8      = 1,
    RLE4      = 2,
    // ...
};

struct BMPHeader {
  uint16_t signature;

  uint32_t size_bytes;
  uint32_t reserved;
  uint32_t data_offset;

  // (other members omitted for simplicity)

  Compression compression;
};

Parsing a .bmp file into a such a BMPHeader value is as simple as it gets with blobify:

std::ifstream file("/path/to/bitmap.bmp");
blob::istream_storage storage { file };

auto header = blob::load<BMPHeader>(storage);

The first two lines open a file and prepare it for input to blobify, and the last line performs the actual read. Not only does this provide a convenient interface, but blobify takes care of the subtle details for you too, such as removing compiler-inserted padding bytes between struct members.

More elaborate usage examples can be found in the examples directory.

Customization via properties

Reliable binary deserialization often requires extensive data validation: Your code might expect version fields to have a certain value, and enum-types should often only have one of the enumerated values. Some data might require postprocessing steps, such as fields encoded in different endianness or different units than the one used at runtime. Blobify allow the user to specify such properties and validation requirements using a built-in DSL.

Consider the signature member of BMPHeader, which identifies a bitmap file as such. If it's not 0x4d42 (an encoding of the string "BM"), a validation error should be triggered. In blobify this can be implemented by defining a properties function:

constexpr auto properties(blob::tag<BMPHeader>) {
    blob::properties_t<BMPHeader> props { };

    // Throw exception if the loaded signature is not 0x4d42
    props.member<&BMPHeader::signature>().expected_value = uint16_t { 0x4d42 };

    // Throw exception if the loaded compression value is none of None, RLE4, or RLE8
    props.member<&BMPHeader::compression>().validate_enum = true;

    return props;
}

Crucially, the properties function must be constexpr and must reside in the same namespace as the definition of BMPHeader. If no such function is defined, blobify will apply a set of default properties that describe a compact binary encoding in native endianness without any validation.

Error handling

Coarse error handling can be done by catching blobify's base exception type blob::exception:

try {
    auto header = blob::load<BMPHeader>(storage);
} catch (blob::exception&) {
    std::cerr << "Failed to load BMPHeader" << std::endl;
}

However, especially in data validation you might want to dynamically handle errors depending on the specific error condition. Hence, blobify's exception hierarchy allows to distinguish between different error causes (end-of-file/unexpected value/invalid enum/...) as well as the specific data member that triggered the error.

try {
    auto header = blob::load<BMPHeader>(storage);
} catch (blob::storage_exhausted_exception&) {
    std::cerr << "Unexpected early end-of-file" << std::endl;
} catch (blob::unexpected_value_exception_for<BMPHeader::signature>&) {
    std::cerr << "Invalid BMP signature" << std::endl;
} catch (blob::invalid_enum_value_exception_for<BMPHeader::compression>&) {
    std::cerr << "Invalid compression value" << std::endl;
}

Usage

Setting up blobify is easiest if your project is built on CMake. Just put a copy of blobify in your project tree (e.g. using a Git submodule) and add_subdirectory it from your main CMakeLists.txt. This will register the blobify target that you can target_link_libraries against.

If your project does not use CMake, setting up blobify is still easy: Just point your compiler to blobify's main include directory as well as the magic_get and magic_enum header paths and you should be good to go.

Credits

blobify's API significantly benefits from the marvelous work done by Antony Polukhin and Daniil Goncharov on their respective libraries PFR (aka magic_get) and magic_enum.

Support

blobify is an offspring of Mikage, a 3DS emulator for Android devices. The library was created out of the critical need in game console emulation to detect invalid deserialization inputs reliably with minimal boilerplate, since failure to do so may cause unimplemented features or subtle emulation bugs to go unnoticed.

If you want to show your appreciation for blobify, the best way of doing so is to support me on Patreon.

Owner
Tony Wasserka
Independent consultant・embedded systems, device drivers, firmware, kernels, etc・C++ and Haskell・Wii and 3DS emulator developer
Tony Wasserka
Comments
  • Error compiling with MSVC

    Error compiling with MSVC

    When I try to compile the code with MSVC I keep getting these errors:

    D:\source\blobify\include\blobify\detail/pmd_traits.hpp(36,39): error C3520: 'PointersToMember': parameter pack must be expanded in this context [D:\source\blobify\build-msvc\blobify-associated-sources.vcxproj]
    D:\source\blobify\include\blobify\detail/pmd_traits.hpp(36,8): error C3520: 'PointersToMember': parameter pack must be expanded in this context [D:\source\blobify\build-msvc\blobify-associated-sources.vcxproj]
    

    Checked for MSVC version 19.28.29337, earlier versions don't seem to be working either

  • Compile Errors when building

    Compile Errors when building

    When running make on the repository itself or when it is included in another project these errors are received. This is using the latest stable version of cmake(3.16.5) and version 7.5.0 of gcc which has full cpp 17 support.

    In file included from blobify/include/blobify/exceptions.hpp:4:0, from blobify/include/blobify/load.hpp:5, from blobify/include/blobify/blobify.hpp:4, from blobify/src/blobify.cpp:2: blobify/include/blobify/detail/pmd_traits.hpp: In substitution of ‘template<class Data, auto ...PointersToMember> using pointed_member_type = std::remove_reference_t<decltype ((PointersToMember .* ... .(declval)()))> [with Data = typename blob::detail::pmd_traits_t::parent_type; auto ...PointersToMember = {PointerToMember1, PointersToMember ...}]’: blobify/include/blobify/store.hpp:154:102: required from here blobify/include/blobify/detail/pmd_traits.hpp:46:113: error: ‘PointerToMember1’ cannot be used as a member pointer, since it is of type ‘auto’ using pointed_member_type = std::remove_reference_t<decltype((std::declval() . ... .* PointersToMember))>; ^ In file included from blobify/include/blobify/modify.hpp:5:0, from blobify/include/blobify/blobify.hpp:5, from blobify/src/blobify.cpp:2: blobify/include/blobify/store.hpp: In function ‘constexpr void blob::lens_store(Storage&&, const Value&, blob::tag)’: blobify/include/blobify/store.hpp:157:34: error: ‘SpecificValueType’ was not declared in this scope detail::lens_store_to_offset<SpecificValueType, ConstructionPolicy, PointerToMember1, PointersToMember...>(storage, 0, value); ^~~~~~~~~~~~~~~~~ CMakeFiles/blobify-associated-sources.dir/build.make:62: recipe for target 'CMakeFiles/blobify-associated-sources.dir/src/blobify.cpp.o' failed make[2]: *** [CMakeFiles/blobify-associated-sources.dir/src/blobify.cpp.o] Error 1 CMakeFiles/Makefile2:75: recipe for target 'CMakeFiles/blobify-associated-sources.dir/all' failed make[1]: *** [CMakeFiles/blobify-associated-sources.dir/all] Error 2 Makefile:129: recipe for target 'all' failed make: *** [all] Error 2

  • Debug Symbols cause Compilation Error

    Debug Symbols cause Compilation Error

    When including

    set(CMAKE_CXX_FLAGS "-Wall -g -D_GLIBCXX_DEBUG")
    

    in the CMakeLists.txt, the compiler produces this error:

    In file included from blobify/include/blobify/blobify.hpp:4,
                     from blobify/examples/bmp.cpp:1:
    blobify/include/blobify/load.hpp: In instantiation of ‘constexpr decltype(auto) blob::detail::validate_element(Member&&) [with auto member_props = (& blob::detail::member_properties_for<BMP::SecondaryHeaderV4, 5>); Member = BMP::Compression]’:
    blobify/include/blobify/load.hpp:100:46:   required from ‘constexpr Member blob::detail::load_element(Storage&) [with Member = BMP::Compression; auto member_props = (& blob::detail::member_properties_for<BMP::SecondaryHeaderV4, 5>); Storage = blob::istream_storage; ConstructionPolicy = blob::detail::default_construction_policy]’
    blobify/include/blobify/load.hpp:136:109:   required from ‘Data blob::detail::load_helper_t<Storage, ConstructionPolicy, Data, std::tuple<Members ...> >::operator()(Storage&, std::index_sequence<I ...>) const [with long unsigned int ...Idxs = {0, 1, 2, 3, 4, 5, 6}; Storage = blob::istream_storage; ConstructionPolicy = blob::detail::default_construction_policy; Data = BMP::SecondaryHeaderV4; Members = {unsigned int, unsigned int, unsigned int, short unsigned int, short unsigned int, BMP::Compression, std::__debug::array<unsigned char, 88>}; std::index_sequence<I ...> = std::integer_sequence<long unsigned int, 0, 1, 2, 3, 4, 5, 6>]’
    blobify/include/blobify/load.hpp:148:87:   required from ‘constexpr Data blob::detail::do_load(Storage&, blob::tag<ConstructionPolicy>) [with Data = BMP::SecondaryHeaderV4; Storage = blob::istream_storage; ConstructionPolicy = blob::detail::default_construction_policy]’
    blobify/include/blobify/load.hpp:207:70:   required from here
    blobify/include/blobify/load.hpp:74:58: error: ‘constexpr std::pair<_FIter, _FIter> std::minmax_element(_FIter, _FIter, _Compare) [with _FIter = const BMP::Compression*; _Compare = blob::detail::validate_element(Member&&) [with auto member_props = (& blob::detail::member_properties_for<BMP::SecondaryHeaderV4, 5>); Member = BMP::Compression]::<lambda(auto:3, auto:4)>]’ called in a constant expression
       74 |         constexpr auto minmax_value = std::minmax_element(values_begin, values_end, enum_less);
          |                                       ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    In file included from /usr/include/c++/9/functional:65,
                     from blobify/magic_get/include/boost/pfr/detail/functional.hpp:12,
                     from blobify/magic_get/include/boost/pfr/precise/functors.hpp:12,
                     from blobify/magic_get/include/boost/pfr/precise.hpp:13,
                     from blobify/include/blobify/detail/pmd_traits.hpp:4,
                     from blobify/include/blobify/exceptions.hpp:4,
                     from blobify/include/blobify/load.hpp:5,
                     from blobify/include/blobify/blobify.hpp:4,
                     from blobify/examples/bmp.cpp:1:
    /usr/include/c++/9/bits/stl_algo.h:3425:5: note: ‘constexpr std::pair<_FIter, _FIter> std::minmax_element(_FIter, _FIter, _Compare) [with _FIter = const BMP::Compression*; _Compare = blob::detail::validate_element(Member&&) [with auto member_props = (& blob::detail::member_properties_for<BMP::SecondaryHeaderV4, 5>); Member = BMP::Compression]::<lambda(auto:3, auto:4)>]’ is not usable as a ‘constexpr’ function because:
     3425 |     minmax_element(_ForwardIterator __first, _ForwardIterator __last,
          |     ^~~~~~~~~~~~~~
    In file included from /usr/include/c++/9/debug/debug.h:87,
                     from /usr/include/c++/9/bits/stl_algobase.h:69,
                     from /usr/include/c++/9/bits/char_traits.h:39,
                     from /usr/include/c++/9/string:40,
                     from /usr/include/c++/9/stdexcept:39,
                     from /usr/include/c++/9/array:39,
                     from /usr/include/c++/9/tuple:39,
                     from blobify/magic_get/include/boost/pfr/detail/stdtuple.hpp:13,
                     from blobify/magic_get/include/boost/pfr/precise/core.hpp:16,
                     from blobify/magic_get/include/boost/pfr/precise.hpp:12,
                     from blobify/include/blobify/detail/pmd_traits.hpp:4,
                     from blobify/include/blobify/exceptions.hpp:4,
                     from blobify/include/blobify/load.hpp:5,
                     from blobify/include/blobify/blobify.hpp:4,
                     from blobify/examples/bmp.cpp:1:
    /usr/include/c++/9/bits/stl_algo.h:3433:7: error: call to non-‘constexpr’ function ‘bool __gnu_debug::__valid_range(_InputIterator, _InputIterator) [with _InputIterator = const BMP::Compression*]’
     3433 |       __glibcxx_requires_valid_range(__first, __last);
          |       ^
    make[2]: *** [lib/blobify/examples/CMakeFiles/example-bmp.dir/build.make:82: lib/blobify/examples/CMakeFiles/example-bmp.dir/bmp.cpp.o] Error 1
    make[1]: *** [CMakeFiles/Makefile2:306: lib/blobify/examples/CMakeFiles/example-bmp.dir/all] Error 2
    make: *** [Makefile:149: all] Error 2
    

    I am using g++-9 right now but clang-10 also produces this error. Is there any way to avoid this without stripping debug symbols?

  • Minor Fixes

    Minor Fixes

    I mostly wanted to fix this warning:

    CMake Warning (dev) at /usr/local/Cellar/cmake/3.18.1/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:273 (message):
      The package name passed to `find_package_handle_standard_args` (MagicEnum)
      does not match the name of the calling package (magic_enum).  This can lead
      to problems in calling code that expects `find_package` result variables
      (e.g., `_FOUND`) to follow a certain pattern.
    

    But I thought I would also fix a few minor things as well.

  • Missing Header/hpp file?

    Missing Header/hpp file?

    Your example that's included on the readme and the bmp example that you provide use a hpp file called "stream_storage.hpp". I was trying to recreate your example for myself but I can't find this file anywhere. Is memory_storage.hpp a replacement for it, or do we need stream_storage.hpp for these examples to work properly?

  • Compile error on clang 13.0.0

    Compile error on clang 13.0.0

    blobify/magic_enum/include/magic_enum.hpp:165:52: fatal error: instantiating fold expression with 257 arguments exceeded expression nesting limit of 256
      constexpr auto num_valid = ((valid[I] ? 1 : 0) + ...);
                                 ~~~~~~~~~~~~~~~~~~~~~~^~~~
    

    This is a known issue fixed on Magic Enum v0.7.2

    Please update this dependency to the latest version Magic Enum v0.7.3

    Thank you

  • cannot compile bmp.cpp on vs2019

    cannot compile bmp.cpp on vs2019

    When building using Visual Studio 2019, it bumps into errors. The first one is [build] blobify/load.hpp(74,1): error C3861: “minmax_element”: identifier not found [examples\example-bmp.vcxproj]

    I solved it by adding #include <algorithm> in blobify/load.hpp.

  • Unable to Use C Style Arrays

    Unable to Use C Style Arrays

    Hello, I am currently playing around with blobify and I noticed that using a C arrays causes errors to appear (error.log). Here is an example below:

    struct headerSFAT {
        char indentifier[4];
        uint16_t length;
        uint16_t byteOrder;
        uint32_t fileSize;
        uint32_t dataOffset;
        uint16_t versionNumber;
        uint16_t reserved;
    };
    

    I was wondering if this is normal or not?

Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications

Zmeya Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serializ

Jun 6, 2022
Fast Binary Encoding is ultra fast and universal serialization solution for C++, C#, Go, Java, JavaScript, Kotlin, Python, Ruby, Swift

Fast Binary Encoding (FBE) Fast Binary Encoding allows to describe any domain models, business objects, complex data structures, client/server request

Jun 16, 2022
Binary Serialization

Binn Binn is a binary data serialization format designed to be compact, fast and easy to use. Performance The elements are stored with their sizes to

Jun 13, 2022
Cap'n Proto serialization/RPC system - core tools and C++ library
Cap'n Proto serialization/RPC system - core tools and C++ library

Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except

Jun 23, 2022
A C++11 library for serialization
A C++11 library for serialization

cereal - A C++11 library for serialization cereal is a header-only C++11 serialization library. cereal takes arbitrary data types and reversibly turns

Jun 23, 2022
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Jun 20, 2022
Simple C++ 20 Serialization Library that works out of the box with aggregate types!

BinaryLove3 Simple C++ 20 Serialization Library that works out of the box with aggregate types! Requirements BinaryLove3 is a c++20 only library.

Dec 23, 2021
CppSerdes is a serialization/deserialization library designed with embedded systems in mind
CppSerdes is a serialization/deserialization library designed with embedded systems in mind

A C++ serialization/deserialization library designed with embedded systems in mind

Jun 3, 2022
Header-only library for automatic (de)serialization of C++ types to/from JSON.

fuser 1-file header-only library for automatic (de)serialization of C++ types to/from JSON. how it works The library has a predefined set of (de)seria

Jun 4, 2022
Yet another JSON/YAML/BSON serialization library for C++.
Yet another JSON/YAML/BSON serialization library for C++.

ThorsSerializer Support for Json Yaml Bson NEW Benchmark Results Conformance mac linux Performance max linux For details see: JsonBenchmark Yet anothe

Jun 15, 2022
Cista is a simple, high-performance, zero-copy C++ serialization & reflection library.

Simple C++ Serialization & Reflection. Cista++ is a simple, open source (MIT license) C++17 compatible way of (de-)serializing C++ data structures. Si

Jun 11, 2022
Jun 16, 2022
Yet Another Serialization
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

Jun 20, 2022
An implementation of the MessagePack serialization format in C / msgpack.org[C]

CMP CMP is a C implementation of the MessagePack serialization format. It currently implements version 5 of the MessagePack Spec. CMP's goal is to be

May 27, 2022
MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]

Introduction MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is: Simple and easy to use Secure agai

May 24, 2022
Serialization framework for Unreal Engine Property System that just works!

DataConfig Serialization framework for Unreal Engine Property System that just works! Unreal Engine features a powerful Property System which implemen

Jun 12, 2022
Yet Another Serialization
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

Sep 7, 2021
universal serialization engine

A Universal Serialization Engine Based on compile-time Reflection iguana is a modern, universal and easy-to-use serialization engine developed in c++1

Jun 20, 2022
A lightweight C++20 serialization framework

zpp::bits A modern C++20 binary serialization library, with just one header file. This library is a successor to zpp::serializer. The library tries to

Jun 9, 2022