Your binary serialization library

Bitsery

Build Status Join the chat at https://gitter.im/bitsery/Lobby

Header only C++ binary serialization library. It is designed around the networking requirements for real-time data delivery, especially for games.

All cross-platform requirements are enforced at compile time, so serialized data do not store any meta-data information and is as small as possible.

bitsery is looking for your feedback on gitter

Features

  • Cross-platform compatible.
  • Optimized for speed and space.
  • No code generation required: no IDL or metadata, just use your types directly.
  • Configurable runtime error checking on deserialization.
  • Can read/write from any source: stream (file, network stream. etc... ), or buffer (vector, c-array, etc...).
  • Don't pay for what you don't use! - customize your serialization via extensions. Some notable extensions allow:
    • fine-grained bit-level serialization control.
    • forward/backward compatibility for your types.
    • smart and raw pointers with allocators support and customizable runtime polymorphism.
  • Easily extendable for any type.
  • Allows brief (similar to cereal) or/and verbose syntax for better serialization control.
  • Configurable endianness support.
  • No macros.

Why use bitsery

Look at the numbers and features list, and decide yourself.

library data size serialize deserialize
bitsery 6913B 959ms 927ms
bitsery_compress 4213B 1282ms 1115ms
boost 11037B 9826ms 8313ms
cereal 10413B 6324ms 5698ms
flatbuffers 14924B 5129ms 2142ms
protobuf 10018B 11966ms 13919ms
yas 10463B 1908ms 1217ms

benchmarked on Ubuntu with GCC 8.3.0, more details can be found here

If still not convinced read more in library motivation section.

Usage example

#include <bitsery/bitsery.h>
#include <bitsery/adapter/buffer.h>
#include <bitsery/traits/vector.h>

enum class MyEnum:uint16_t { V1,V2,V3 };
struct MyStruct {
    uint32_t i;
    MyEnum e;
    std::vector<float> fs;
};

template <typename S>
void serialize(S& s, MyStruct& o) {
    s.value4b(o.i);
    s.value2b(o.e);
    s.container4b(o.fs, 10);
}

using Buffer = std::vector<uint8_t>;
using OutputAdapter = bitsery::OutputBufferAdapter<Buffer>;
using InputAdapter = bitsery::InputBufferAdapter<Buffer>;

int main() {
    MyStruct data{8941, MyEnum::V2, {15.0f, -8.5f, 0.045f}};
    MyStruct res{};

    Buffer buffer;

    auto writtenSize = bitsery::quickSerialization<OutputAdapter>(buffer, data);
    auto state = bitsery::quickDeserialization<InputAdapter>({buffer.begin(), writtenSize}, res);

    assert(state.first == bitsery::ReaderError::NoError && state.second);
    assert(data.fs == res.fs && data.i == res.i && data.e == res.e);
}

For more details go directly to quick start tutorial.

How to use it

This documentation comprises these parts:

documentation is in progress, most parts are empty, but contributions are welcome.

Requirements

Works with C++11 compiler, no additional dependencies, include <bitsery/bitsery.h> and you're done.

some bitsery extensions might require higher C++ standard (e.g. StdVariant)

Platforms

This library was tested on

  • Windows: Visual Studio 2015, MinGW (GCC 5.2)
  • Linux: GCC 5.4, Clang 3.9
  • OS X Mavericks: AppleClang 8

There is a patch that allows using bitsery with non-fully compatible C++11 compilers.

  • CentOS 7 with gcc 4.8.2.

License

bitsery is licensed under the MIT license.

Owner
Mindaugas Vinkelis
Mindaugas Vinkelis
https://github.com/fraillt/bitsery
Comments
  • Error on Centos7

    Error on Centos7

    I am trying to use Bitsery5.0.1 on a Centos7 witg gcc4.8.2. I got this error:

    /usr/include/c++/4.8.2/bits/hashtable.h: In instantiation of 'class std::_Hashtable<long unsigned int, std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > >, bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > >, std::__detail::_Select1st, std::equal_to<long unsigned int>, std::hash<long unsigned int>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >':
/usr/include/c++/4.8.2/bits/unordered_map.h:100:18:   required from 'class std::unordered_map<long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> >, std::hash<long unsigned int>, std::equal_to<long unsigned int>, bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > > >'
/__w/OpenGeode/OpenGeode/build/third_party/bitsery/install/include/bitsery/ext/utils/polymorphism_utils.h:182:19:   required from here
/usr/include/c++/4.8.2/bits/hashtable.h:194:47: error: no type named 'pointer' in 'class bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > >'
       typedef typename _Alloc::pointer        pointer;
                                               ^
/usr/include/c++/4.8.2/bits/hashtable.h:195:55: error: no type named 'const_pointer' in 'class bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > >'
       typedef typename _Alloc::const_pointer          const_pointer;
                                                       ^
/usr/include/c++/4.8.2/bits/hashtable.h:196:55: error: no type named 'reference' in 'class bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > >'
       typedef typename _Alloc::reference              reference;
                                                       ^
/usr/include/c++/4.8.2/bits/hashtable.h:197:55: error: no type named 'const_reference' in 'class bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > >'
       typedef typename _Alloc::const_reference        const_reference;
                                                       ^
/usr/include/c++/4.8.2/bits/hashtable.h:317:8: error: no class template named 'rebind' in 'class bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > >'
        _Node_allocator_type;
        ^
/usr/include/c++/4.8.2/bits/hashtable.h:319:8: error: no class template named 'rebind' in 'class bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > >'
        _Bucket_allocator_type;
        ^
/usr/include/c++/4.8.2/bits/hashtable.h:321:75: error: no class template named 'rebind' in 'class bitsery::ext::pointer_utils::StdPolyAlloc<std::pair<const long unsigned int, std::vector<long unsigned int, bitsery::ext::pointer_utils::StdPolyAlloc<long unsigned int> > > >'
       using __before_begin = __detail::_Before_begin<_Node_allocator_type>;
                                                                           ^

    Do you have an idea of the origin of this error?

  • Serializing forward-declared types

    Serializing forward-declared types

    I have a bunch of forward declares for types used in my classes and in order to use bitsery it looks like I need to include the headers for them (forward declaring doesn't work).

    More of a detail: I use forward declares because I hold raw pointers to my types, but I don't actually use bitsery's raw pointer features because I manage the memory/hierarchy myself. Instead I serialize them dereferenced.

    Anyway, is there a way to avoid having the includes? Using forward declares is better practice.

    Thanks, great library :)

  • Read Serialized Buffers in Place

    Read Serialized Buffers in Place

    not sure the effort that would be required to implement this, but, given the memory layout of variables and their data in serialized buffers must (?) be deterministic, it is certainly possible to read buffers in place.

    right now I use flatbuffers instead of bitsery to serialize/deserialize the information I exchange between my app and server solely because of the ability to read flatbuffers in place-- in my architecture philosophy of minimizing the usage of server resources and time by executing any work on the client possible. so even though the sum total work to serialize/deserialize with bitsery is much less than with flatbuffers, it's still the less optimal choice.

    adding this ability would instantly make bitsery the absolutely optimal choice for networking!

  • SegFault while deserializing

    SegFault while deserializing

    I am trying to upgrade my code to bitsery 5. No issue during the porting. However, I have a seg fault in my tests.

    Here is the valgrind log of one of them:

    valgrind ./test-vertex-identifier 
==26526== Memcheck, a memory error detector
==26526== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==26526== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==26526== Command: ./test-vertex-identifier
==26526== 
==26526== Invalid read of size 8
==26526==    at 0x5306FE2: void bitsery::ext::pointer_utils::PolyAllocWithTypeId::deallocate<std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2> >(std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2>*, unsigned long, unsigned long) const (memory_resource.h:86)
==26526==    by 0x5306B42: bitsery::ext::pointer_utils::StdPolyAlloc<std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2> >::deallocate(__gnu_cxx::_Lock_policy*, unsigned long) (memory_resource.h:147)
==26526==    by 0x530680D: std::allocator_traits<bitsery::ext::pointer_utils::StdPolyAlloc<std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2>&, __gnu_cxx::_Lock_policy*, unsigned long) (alloc_traits.h:328)
==26526==    by 0x53062AF: std::__allocated_ptr<bitsery::ext::pointer_utils::StdPolyAlloc<std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() (allocated_ptr.h:73)
==26526==    by 0x5308249: std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (shared_ptr_base.h:476)
==26526==    by 0x723AB28: bitsery::ext::pointer_utils::StdPolyAlloc<bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> > >::deallocate(bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*, unsigned long) (memory_resource.h:147)
==26526==    by 0x72249B6: void bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>::addToMap<bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >(std::integral_constant<bool, false>)::{lambda(bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*)#1}::operator()(bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*) const (polymorphism_utils.h:150)
==26526==    by 0x72DE8B1: std::_Sp_counted_deleter<bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*, void bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>::addToMap<bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >(std::integral_constant<bool, false>)::{lambda(bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> > >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:470)
==26526==    by 0x5277691: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:154)
==26526==    by 0x5277342: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:684)
==26526==    by 0x5298AA9: std::__shared_ptr<bitsery::ext::PolymorphicHandlerBase, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:1123)
==26526==    by 0x5298AC5: std::shared_ptr<bitsery::ext::PolymorphicHandlerBase>::~shared_ptr() (shared_ptr.h:93)
==26526==  Address 0x8 is not stack'd, malloc'd or (recently) free'd
==26526== 
==26526== 
==26526== Process terminating with default action of signal 11 (SIGSEGV)
==26526==  Access not within mapped region at address 0x8
==26526==    at 0x5306FE2: void bitsery::ext::pointer_utils::PolyAllocWithTypeId::deallocate<std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2> >(std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2>*, unsigned long, unsigned long) const (memory_resource.h:86)
==26526==    by 0x5306B42: bitsery::ext::pointer_utils::StdPolyAlloc<std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2> >::deallocate(__gnu_cxx::_Lock_policy*, unsigned long) (memory_resource.h:147)
==26526==    by 0x530680D: std::allocator_traits<bitsery::ext::pointer_utils::StdPolyAlloc<std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2>&, __gnu_cxx::_Lock_policy*, unsigned long) (alloc_traits.h:328)
==26526==    by 0x53062AF: std::__allocated_ptr<bitsery::ext::pointer_utils::StdPolyAlloc<std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() (allocated_ptr.h:73)
==26526==    by 0x5308249: std::_Sp_counted_deleter<geode::VariableAttribute<unsigned int>*, bitsery::ext::smart_ptr_details::SmartPtrOwnerManager<std::shared_ptr<geode::VariableAttribute<unsigned int> > >::createSharedPolymorphic(bitsery::ext::smart_ptr_details::SharedPtrSharedState&, std::shared_ptr<geode::VariableAttribute<unsigned int> >&, bitsery::ext::MemResourceBase*, std::shared_ptr<bitsery::ext::PolymorphicHandlerBase> const&)::{lambda(geode::VariableAttribute<unsigned int>*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<geode::VariableAttribute<unsigned int> >, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (shared_ptr_base.h:476)
==26526==    by 0x723AB28: bitsery::ext::pointer_utils::StdPolyAlloc<bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> > >::deallocate(bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*, unsigned long) (memory_resource.h:147)
==26526==    by 0x72249B6: void bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>::addToMap<bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >(std::integral_constant<bool, false>)::{lambda(bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*)#1}::operator()(bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*) const (polymorphism_utils.h:150)
==26526==    by 0x72DE8B1: std::_Sp_counted_deleter<bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*, void bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>::addToMap<bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >(std::integral_constant<bool, false>)::{lambda(bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> >*)#1}, bitsery::ext::pointer_utils::StdPolyAlloc<bitsery::ext::PolymorphicHandler<bitsery::ext::StandardRTTI, bitsery::Deserializer<bitsery::BasicInputStreamAdapter<char, bitsery::DefaultConfig, std::char_traits<char> >, std::tuple<bitsery::ext::PolymorphicContext<bitsery::ext::StandardRTTI>, bitsery::ext::PointerLinkingContext, bitsery::ext::InheritanceContext> >, geode::VariableAttribute<unsigned int>, geode::VariableAttribute<unsigned int> > >, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:470)
==26526==    by 0x5277691: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:154)
==26526==    by 0x5277342: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:684)
==26526==    by 0x5298AA9: std::__shared_ptr<bitsery::ext::PolymorphicHandlerBase, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:1123)
==26526==    by 0x5298AC5: std::shared_ptr<bitsery::ext::PolymorphicHandlerBase>::~shared_ptr() (shared_ptr.h:93)
==26526==  If you believe this happened as a result of a stack
==26526==  overflow in your program's main thread (unlikely but
==26526==  possible), you can try to increase the size of the
==26526==  main thread stack using the --main-stacksize= flag.
==26526==  The main thread stack size used in this run was 8388608.
==26526== 
==26526== HEAP SUMMARY:
==26526==     in use at exit: 23,006 bytes in 393 blocks
==26526==   total heap usage: 2,459 allocs, 2,066 frees, 199,049 bytes allocated
==26526== 
==26526== LEAK SUMMARY:
==26526==    definitely lost: 0 bytes in 0 blocks
==26526==    indirectly lost: 0 bytes in 0 blocks
==26526==      possibly lost: 0 bytes in 0 blocks
==26526==    still reachable: 23,006 bytes in 393 blocks
==26526==         suppressed: 0 bytes in 0 blocks
==26526== Rerun with --leak-check=full to see details of leaked memory
==26526== 
==26526== For counts of detected and suppressed errors, rerun with: -v
==26526== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Erreur de segmentation (core dumped)

    Do you have an idea of the issue and its origin?

  • Recreate Pointer Objects w/ Custom Allocator During Deserialization

    Recreate Pointer Objects w/ Custom Allocator During Deserialization

    title is pretty self explanatory :)

    i have some objects i'm currently using new to allocate, but want to switch over to using a custom allocator. i serialize/deserialize these objects with bitsery, and so would need the custom allocator to be triggered upon deserialization.

    maybe i missed it but i didn't see any hooks to allocate memory when deserializing a pointer.

  • Vector of bit-packed structs not measuring correctly?

    Vector of bit-packed structs not measuring correctly?

    Hello, me again with a measurement problem. This time, I've learned my lesson and boiled it down to a simple example before bringing it here :D

    I have the following code, where an EntityUpdate struct contains a vector of Input structs. Input contains an array that gets bit-packed on serialization.

    #include "bitsery/bitsery.h"
#include "bitsery/adapter/buffer.h"
#include "bitsery/adapter/stream.h"
#include "bitsery/adapter/measure_size.h"
#include "bitsery/traits/vector.h"
#include "bitsery/ext/value_range.h"

struct Input {
public:
    enum Type : uint8_t { XUp, XDown, YUp, YDown, ZUp, ZDown, NumTypes, None };
    enum State : uint8_t { Released = 0, Pressed = 1 };

    typedef std::array<State, Type::NumTypes> StateArr;
    StateArr inputStates{};
};

struct EntityUpdate {
    std::vector<Input> inputs;
};

template<typename S>
void serialize(S& serializer, EntityUpdate& entityUpdate)
{
    serializer.container(entityUpdate.inputs,
                         static_cast<std::size_t>(1000));
}

template<typename S>
void serialize(S& serializer, Input& input)
{
    serializer.enableBitPacking([&input](typename S::BPEnabledType& sbp) {
        sbp.container(input.inputStates, [](typename S::BPEnabledType& sbp,
                                            Input::State& inputState) {
            constexpr bitsery::ext::ValueRange<Input::State> range{
                Input::State::Released, Input::State::Pressed};
            sbp.ext(inputState, range);
        });
    });
}

using OutputAdapter = bitsery::OutputBufferAdapter<std::vector<uint8_t>>;

int main()
{
    EntityUpdate entityUpdate{};
    for (unsigned int i = 0; i < 4; ++i) {
        entityUpdate.inputs.push_back(Input{});
    }

    // Measure the size.
    std::size_t measuredSize{bitsery::quickSerialization(
        bitsery::MeasureSize{}, entityUpdate)};

    // Serialize and get the real size.
    std::vector<uint8_t> buffer{};
    std::size_t finalSize{bitsery::quickSerialization<OutputAdapter>(
        buffer, entityUpdate)};

    assert(measuredSize == finalSize);

    return 0;
}

    The issue is, when the array is filled with more than 4 inputs, I'm getting different values between the measured size and the serialized size. In the example, I'm getting 4B for measured and 5B for final. 5B makes sense, since each Input can fit in 1B and the array will use 1B for its size.

    Am I doing something incorrectly, or is there a bug?

  • serialization of 3rd party datastructures

    serialization of 3rd party datastructures

    Is it possible to serialize e.g. glm datastructures (glm::vec2, glm::vec3, ...)? The examples show only serialization of classes and struct defined by the user, and simply defining

    #include <glm/vec3.hpp>

template<typename S>
void serialize(S& s, glm::vec3& vec) {
    s.value4b(vec.x);
    s.value4b(vec.y);
    s.value4b(vec.z);
}

    somewhere where the serialization is done was apparently not the correct way.

  • Why does the buffer uses resize instead of reserve?

    Why does the buffer uses resize instead of reserve?

    https://github.com/fraillt/bitsery/blob/master/include/bitsery/traits/core/std_defaults.h#L85

    Wouldn't be the reserve method a better solution instead of resize? Wasn't that possible?

  • Serializer value() size auto-deduction

    Serializer value() size auto-deduction

    Probably discussed somewhere before, but.

    Serializer have

    template<size_t VSIZE, typename T>
void value(const T &v)

    why this is impossible:

    template<typename T>
void value(const T &v)
{
    value<sizeof(T)>(v);
}

    ?

    Why value size must be always specified?

  • Dynamic polymorphic class registration

    Dynamic polymorphic class registration

    I would like to know how to dynamically extend bitsery polymorphism. Let's take your example and assume this code is in a library A:

    template<>
struct PolymorphicBaseClass<Shape> : PolymorphicDerivedClasses<Circle, Rectangle> {
};

    Now, in another library B, I would like to add another class Square derived from Shape. Is it possible to register this new relationship between Square and Shape without modifying the library A which does not know Square?

  • How to serialize a vector without size before data?

    How to serialize a vector without size before data?

    Hello, I have a serialization format that whenever a std::vector needs to be (de)serialized, the amount of data is defined by two other members in the struct that are (de)serialized prior the std::vector, something like that: uint32_t lowBound; uint32_t highBound; std::vector<int32_t> data;

    so within data there are highBound - lowBound + 1 elements, but the length is not being (de)serialized (e.g. only lowBound, highBound and the data itself is (de)serialized). How can I achieve this with bitsery? Thanks,

  • pass cursor and needed byte space to BufferAdapterTraits::increaseBufferSize

    pass cursor and needed byte space to BufferAdapterTraits::increaseBufferSize

    two insights seen when resizing buffers...

    1. when inserting an object much larger than the current buffer size, the current implementation will loop many times until the buffer grows large enough, burning several unnecessary allocations and memcpy-s in the worst case. so inform the implementation how many bytes we need.

    2. when implementing an alloc + memcpy, since the buffer's length isn't updated until after the serialization, we're forced to copy the entire capacity. though this capacity is insufficient to accommodate the new addition, we still might be copying excess bytes. very mildly cleaner this way. wouldn't warrant a PR without 1) though.

  • Vector<String *>

    Vector

    i'm trying to serialize a vector of string pointers for the first time...

    class Contact {
public:
    
    String name;
    std::vector<String *> bits;
};

template <typename S>
static void serialize(S& s, Contact& contact)
{
    s.text1b(contact.name, UINT8_MAX);
    
    s.container(contact.bits, UINT16_MAX, [](S& s1, auto& bit) {
        
        PointerOwner{}.serialize(s1, bit, [](S& s2, auto &v) { s2.text1b(v, UINT16_MAX); });
    });
}

    thoughts on this?

    In file included from /Users/victorstewart/xxx/Data/Database.cpp:9:
In file included from /Users/victorstewart/xxx/Data/Database.hpp:9:
In file included from /Users/victorstewart/xxx/all.h:40:
In file included from /Users/victorstewart/Dropbox/nametag/nametag/engines/nametag.engines.bitsery.v0.h:1:
In file included from /Users/victorstewart/Dropbox/nametag/nametag/libraries/include/bitsery/bitsery.h:40:
In file included from /Users/victorstewart/Dropbox/nametag/nametag/libraries/include/bitsery/serializer.h:27:
In file included from /Users/victorstewart/Dropbox/nametag/nametag/libraries/include/bitsery/ext/../details/serialization_common.h:29:
/Users/victorstewart/Dropbox/nametag/nametag/libraries/include/bitsery/ext/../details/adapter_common.h:87:28: error: no member named 'writeBytes' in 'bitsery::InputBufferAdapter<String, FastConfig>'
                w.template writeBytes<1>(static_cast<uint8_t>(size));
                ~          ^
In file included from /Users/victorstewart/xxx/Data/Database.cpp:9:
In file included from /Users/victorstewart/xxx/Data/Database.hpp:9:
In file included from /Users/victorstewart/xxx/all.h:40:
In file included from /Users/victorstewart/Dropbox/nametag/nametag/engines/nametag.engines.bitsery.v0.h:3:
In file included from /Users/victorstewart/Dropbox/nametag/nametag/libraries/include/bitsery/ext/pointer.h:28:
/Users/victorstewart/Dropbox/nametag/nametag/libraries/include/bitsery/ext/utils/pointer_utils.h:297:34: note: in instantiation of function template specialization 'bitsery::details::writeSize<bitsery::InputBufferAdapter<String, FastConfig>>' requested here
                        details::writeSize(ser.adapter(), ptrInfo.id);
                                 ^
/Users/victorstewart/xxx/Data/Database.cpp:95:24: note: in instantiation of function template specialization 'bitsery::ext::pointer_utils::PointerObjectExtensionBase<pointer_details::PtrOwnerManager, PolymorphicContext, bitsery::ext::StandardRTTI>::serialize<bitsery::Deserializer<bitsery::InputBufferAdapter<String, FastConfig>, std::tuple<bitsery::ext::PointerLinkingContext>>, String *, (lambda at /Users/victorstewart/xxx/Data/Database.cpp:95:43)>' requested here
        PointerOwner{}.serialize(s1, bit, [](S& s2, auto &v) { s2.text1b(v, UINT16_MAX); });

    there also isn't really an ergonomic way to do this currently.

    trying to map it through .object() with s1.ext(bit, PointerOwner{});and defining a serialize function for my custom String type triggers a static assert finding a free and member function named serialize because i have this in my class.

    template <typename T>
uint32_t serialize(T&& object);

    i could change it's name, but maybe we need to check if the return type of the serialize function is void?

    but regardless i think the correct way would be something like exttext1b. but of course that's pretty ugly and adds ever more functions.

  • Is bitsery faster than capnp?

    Is bitsery faster than capnp?

    I'm using capnp which is fast but just like everybody else I'm always looking for a faster library. I'm curious why capnp is not on the current benchmark list?

  • ContainerTraits::size and BufferAdapterTraits::increaseBufferSize

    ContainerTraits::size and BufferAdapterTraits::increaseBufferSize

    thoughts on renaming size to remainingCapacity? because it really requests remainingCapacity not size. just debugged an infinite loop where i'd implemented these as size when really the intention was remainingCapacity.

  • io_uring based stream adapter

    io_uring based stream adapter

    not sure if you've heard about io_uring on Linux but it basically allows for syscall-less asynchronous operations and IO through the kernel. It's incredibly fast. I can show you a generic read/write file object i wrote to do DirectIO.

    anyway this would obviously be the most performant way to stream to/from disk.

    not sure how you'd feel about including a new stream adapter into the library that had a dependency on https://github.com/axboe/liburing + Linux.

    even if not in the library, i assume if I wrote a new BasicBufferedOutputStreamAdapter say IOuringBufferedOutputStreamAdapter, and the same for input, and mimicked all the function calls etc, that it would "just work"?

    I have a situation where I'm serializing very large (possible into the GBs) objects then writing them to disk, and don't want to blow up memory, thrash the page cache (hence DirectIO), nor waste precious seconds getting the data onto disk.

Related tags
Serialization serialization cpp14 cpp11 binary-serialization
Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications

Zmeya Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serializ

Jun 6, 2022
Fast Binary Encoding is ultra fast and universal serialization solution for C++, C#, Go, Java, JavaScript, Kotlin, Python, Ruby, Swift

Fast Binary Encoding (FBE) Fast Binary Encoding allows to describe any domain models, business objects, complex data structures, client/server request

Jun 16, 2022
Binary Serialization

Binn Binn is a binary data serialization format designed to be compact, fast and easy to use. Performance The elements are stored with their sizes to

Jun 13, 2022
Cap'n Proto serialization/RPC system - core tools and C++ library
 Cap'n Proto serialization/RPC system - core tools and C++ library

Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except

Jun 23, 2022
A C++11 library for serialization
 A C++11 library for serialization

cereal - A C++11 library for serialization cereal is a header-only C++11 serialization library. cereal takes arbitrary data types and reversibly turns

Jun 23, 2022
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Jun 20, 2022
Simple C++ 20 Serialization Library that works out of the box with aggregate types!

BinaryLove3 Simple C++ 20 Serialization Library that works out of the box with aggregate types! Requirements BinaryLove3 is a c++20 only library.

Dec 23, 2021
CppSerdes is a serialization/deserialization library designed with embedded systems in mind
 CppSerdes is a serialization/deserialization library designed with embedded systems in mind

A C++ serialization/deserialization library designed with embedded systems in mind

Jun 3, 2022
Header-only library for automatic (de)serialization of C++ types to/from JSON.

fuser 1-file header-only library for automatic (de)serialization of C++ types to/from JSON. how it works The library has a predefined set of (de)seria

Jun 4, 2022
Yet another JSON/YAML/BSON serialization library for C++.
 Yet another JSON/YAML/BSON serialization library for C++.

ThorsSerializer Support for Json Yaml Bson NEW Benchmark Results Conformance mac linux Performance max linux For details see: JsonBenchmark Yet anothe

Jun 15, 2022
Cista is a simple, high-performance, zero-copy C++ serialization & reflection library.

Simple C++ Serialization & Reflection. Cista++ is a simple, open source (MIT license) C++17 compatible way of (de-)serializing C++ data structures. Si

Jun 11, 2022
Bond is a cross-platform framework for working with schematized data. It supports cross-language de/serialization and powerful generic mechanisms for efficiently manipulating data. Bond is broadly used at Microsoft in high scale services.

Bond Bond is an open-source, cross-platform framework for working with schematized data. It supports cross-language serialization/deserialization and

Jun 16, 2022
Yet Another Serialization
 Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

Jun 20, 2022
An implementation of the MessagePack serialization format in C / msgpack.org[C]

CMP CMP is a C implementation of the MessagePack serialization format. It currently implements version 5 of the MessagePack Spec. CMP's goal is to be

May 27, 2022
MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]

Introduction MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is: Simple and easy to use Secure agai

May 24, 2022
Serialization framework for Unreal Engine Property System that just works!

DataConfig Serialization framework for Unreal Engine Property System that just works! Unreal Engine features a powerful Property System which implemen

Jun 12, 2022
Yet Another Serialization
 Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

Sep 7, 2021
universal serialization engine

A Universal Serialization Engine Based on compile-time Reflection iguana is a modern, universal and easy-to-use serialization engine developed in c++1

Jun 20, 2022
A lightweight C++20 serialization framework

zpp::bits A modern C++20 binary serialization library, with just one header file. This library is a successor to zpp::serializer. The library tries to

Jun 9, 2022