The Leap Motion cross-format, cross-platform declarative serialization library

Introduction to LeapSerial

LeapSerial is a cross-format, declarative, serialization and deserialization library written and maintained by Leap Motion. This library is built with CMake, and makes heavy use of C++11.

LeapSerial is mainly intended to provide users with a way to describe how their types should be serialized without being too concerned about the wire format serialization should take. Users should be able to directly annotate and serialize/deserialize their business objects rather than having to convert to and from DTOs generated by tools like protoc.

The following four output formats are currently supported:

  • JSON
  • Protobuf
  • Flatbuffers
  • LeapSerial Proprietary

LeapSerial also provides a simple stream concept, some wrapper classes, and a few standard implementations. These streams may be composed with one another similarly to how it's done with Protobuf streams.

  • Wrapper for std::istream and std::ostream
  • AES-256 encryption
  • Zlib compression
  • BZip2 compression
  • Memory stream

Users may also write their own.

Quick Start

Here's how you mark the fields to be serialized:

#include <LeapSerial/LeapSerial.h>

class MyClass {
  int my_member;

  static leap::descriptor GetDescriptor() {
    return {
      &MyClass::my_member
    };
  }
};

Serialization one-liner:

MyClass myClass;
std::stringstream os;
leap::Serialize(os, myClass);

Deserialization is also a one-liner:

std::shared_ptr<MyClass> a = leap::Deserialize<MyClass>(ss, myClass);

If your type doesn't use native pointers (either directly or transitively), you can also deserialize in-place.

MyClass b;
leap::Deserialize(ss, b);

Alternative Archivers

LeapSerial has a few output formats that are well supported. The leap::Serialize call, by default, will use the internal LeapSerial archiver, which formats data in a custom bitstream format. You can use other formats, though, such as Protobuf, but this requires that your fields are numbered or named. The following sections all use the following numbered and named data structure:

class MyProtobufObject {
public:
  int a = 949;
  std::string b = "Hello world!";
  std::vector<int> c {4, 5, 6};

  static leap::descriptor GetDescriptor(void) {
    return{
      "MyProtobufObject",
      {
        { 424, "a", &MyProtobufObject::a },
        { 425, "b", &MyProtobufObject::b },
        { 426, "c", &MyProtobufObject::c }
      }
    };
  }
};

Protobuf

Protobuf serialization can be done with OArchiveProtobuf:

#include <LeapSerial/IArchiveProtobuf.h>
#include <LeapSerial/OArchiveProtobuf.h>

void Foo() {
  MyProtobufObject myobj;
  std::stringstream ss;

  leap::Serialize<leap::OArchiveProtobuf>(ss, defaultPerson);
}

The resulting object can be parsed by Protobuf, if you have a schema for MyProtobufObject. You can also create the corresponding proto file by serializing the descriptor, like this:

std::stringstream ss;
leap::Serialize<leap::protobuf_v1>(
  ss,
  MyProtobufObject::GetDescriptor()
);

This is what you get:

message MyProtobufObject {
  required sint32 a = 424;
  optional string b = 425;
  repeated sint32 c = 426;
}

Right now, leap::protobuf_v1 and leap::protobuf_v2 are supported to ensure the generated proto file can be parsed by your chosen version of protoc.

JSON

JSON serialization is also supported. The protov3 specification's JSON mapping is used wherever possible. Currently, deserialization is not supported, but there is a leap::IArchiveJSON type which will provide deserialization once it's implemented.

#include <LeapSerial/ArchiveJSON.h>

void Foo() {
  MyProtobufObject obj;
  std::stringstream ss;
  leap::Serialize<leap::OArchiveJSON>(ss, obj);
}

Here's the resulting JSON:

{
  "a": 949,
  "b": "Hello world!",
  "c": [ 4, 5, 6 ]
}

Alternative Streams

You can also serialize to a memory buffer, if you don't like std::stringstream:

leap::MemoryStream ms;
MyClass b;
leap::Serialize(ms, b);

// In case you want to do something with the data
std::vector<uint8_t>& data = ms.GetDatadata();

// Or you can just round trip right from here
std::shared_ptr<MyClass> c = leap::Deserialize(ms);

Maybe you have a buffer already that you want to deserialize

char buf[1024];
FillMyBuffer(buf, 1024);

leap::BufferedStream bs{buf, sizeof(buf), sizeof(buf)};
std::shared_ptr<MyClass> mc = leap::Deserialize<MyClass>(bs);

Encryption and compression are supported, too. Encrypting and compressing streams are declared as a filter. Protobuf's zero copy streams are similarly implemented. Here's how you might encrypt a file directly to disk with an std::ofstream. Make sure you fill key with a cryptographic function or you might be vulnerable to a related key attack.

std::array<uint8_t, 32> myKey;

std::ofstream of("myfile.dat");
leap::OutputStreamAdapter osa(of);
leap::CompressionStream<Zlib> cs { osa };
leap::AESEncryptionStream ecs { cs, myKey };
MyObject obj;
leap::Serialize(ecs, obj);

If you use encryption and compression, make sure the compression step happens before the encryption step, otherwise you will wind up making the file size larger.

Owner
Leap Motion (Ultraleap)
Leap Motion, owned by Ultraleap, is a motion control hardware and software company.
Leap Motion (Ultraleap)
Comments
  • LeapSerial 0.3.1 causes firmware update issues

    LeapSerial 0.3.1 causes firmware update issues

    When I update the firmware on the Peripheral I get issues after the firmware update is finished.

    • Most of the time it succeeds and just stops the service after the update.
    • It worked one time and didn't stop the service but the images were corrupted.
    • One time it did not work and returns the following message after updating: [Info] Firmware is up to date. [Info] Leap Motion Controller detected: ID_MISSING [Info] ResourceHost added device with ID ID_MISSING
  • Fix protobuf compatibility and support

    Fix protobuf compatibility and support

    The protobuf build for LeapSerial isn't working any more. Fix it so we can run the tests. Also suppress warnings where possible--we can't fix the generated code and the warnings make it harder to see real issues.

    While we're at it, also fix SchemaWriterProtobuf2's access protections, and simplify the protobuf tests where possible.

  • Add benchmark project

    Add benchmark project

    Reports of slow performance in the AES encryption stream cipher have made it necessary to add a project to benchmark its performance and ensure that it approaches the performance that could be achieved with a plaintext read. IO should be the limiting factor with a block encryption system such as AES, not compute.

  • Do not force OSX arch flags, fix them for ARM

    Do not force OSX arch flags, fix them for ARM

    Do not use FORCE here, if the user truly wants to override these flags we should let them. Furthermore, the correct value for the CMAKE_OSX_ARCHITECTURE flag for ARM (ie, an IPhone build target) is actually arm, not x86_64;i386.

  • Change set_standard_output_variables to standard_project

    Change set_standard_output_variables to standard_project

    There's a bunch of stuff that set_standard_output_variables did that was beyond the scope of the name, and a number of them actually produced incorrect behavior if they weren't set prior to the first call to project. To ensure they're called in the correct order, change it into a wrapper around project(). Also changes from a macro to a function to avoid namespace leakage, and enforce the setting of the version property.

    Fixes problems introduced by me and reported by @gittyupagain in https://github.com/leapmotion/leapserial/pull/101

    • [x] Reviewed by @jdonald in lieu of @gittyupagain
    • [ ] Reviewed by @codemercenary
  • Fix vector-of-optionals defect

    Fix vector-of-optionals defect

    Currently the LeapSerial archiver does not write size values down for entries of an array or a map. This means that archivers which rely on exterior knowledge of the size--such as the descriptor archiver--do not know when the object is "finished", and therefore can only deserialize those parts of the object which are non-optional.

    Correct this while preserving backwards-compatibility by setting the high bit of the element count field on the vector when serializing mutable types. During deserialization, if this bit is set, a size integer is extracted from the stream before every deserialized array entry. If the bit is cleared, deserialization proceeds as normal. Also add a concept of "immutable" to identify those types for which a size field is redundant.

  • Add compression support to LeapSerial

    Add compression support to LeapSerial

    This adds a utility archive type that understands compression. This new feature uses a private version of zlib in order to minimize dependencies.

    Compression can be added by using the CompressionStream and DecompressionStream datatypes as so:

    std::ofstream ofs;
    leap::OutputStreamAdapter osa(ofs);
    leap::CompressionStream cs(osa);
    
    MySerializableType val;
    leap::Serialize(cs, val);
    
  • Add an optional header concept and corresponding unit test.

    Add an optional header concept and corresponding unit test.

    The STL will someday have something like this in it, then we can remove this implementation. In the meantime, however, we need to supply one of our own in order to be compatible with the Protobuf format without incurring a pretty big performance hit.

  • Disable LS_ENABLE_PROTOBUF_TESTS on Windows/ARM

    Disable LS_ENABLE_PROTOBUF_TESTS on Windows/ARM

    The CMake config was not properly tested for #105 and breaks much of leapserial when running int he ExternalBuilder job, so we can disable this for now. It works fine for Mac/Linux though.

  • Make toolchain filename consistent

    Make toolchain filename consistent

    The toolchain file toolchain-arm.cmake should be renamed to toolchain-arm32.cmake in order to make it consistent with the corresponding Android toolchain filenames.

  • Standardize LeapSerial source tree

    Standardize LeapSerial source tree

    Move all LeapSerial sources out of the funky LeapSerial directory, and fix target include paths and installation instructions to match the new configuration.

  • ODR and contrib libraries

    ODR and contrib libraries

    contrib libraries seem to have been simply copied into a directory. Will this not create issues with ODR (One Definition Rule) when I have additional libraries that try to link to their own copies ( of for instance ZLIB). There is no configuration option to point at my-own/external copies of zlib and other libraries

    Is there some kind of mechanism protecting users from this issue?

  • Get protobuf unit tests running on Mac

    Get protobuf unit tests running on Mac

    Couldn't get protobuf unit tests working on mac because of some problems with FindProtobuf.cmake on that platform. This command seems to install protobuf just fine with brew:

    brew install protobuf
    

    And FindProtobuf.cmake seems to find protoc just fine, but it doesn't seem to find the dylib properly nor does LeapSerialTest correctly link to it. Figure out why, fix it, and build protobuf on the mac platform.

    Commit a62f57b2774282414ac0c3fc6c361186c85e4739 shows as far as I got with this one. I rolled it back before #14 was merged so we could get some coverage and make some headway at least.

The OpenEXR project provides the specification and reference implementation of the EXR file format, the professional-grade image storage format of the motion picture industry.
The OpenEXR project provides the specification and reference implementation of the EXR file format, the professional-grade image storage format of the motion picture industry.

OpenEXR OpenEXR provides the specification and reference implementation of the EXR file format, the professional-grade image storage format of the mot

Jan 6, 2023
declarative polyamorous cross-system intermedia objects
declarative polyamorous cross-system intermedia objects

declarative polyamorous cross-system intermedia objects

Dec 27, 2022
A motion-activated LED light strip controller. Supports up to two motion detectors.

A simple standalone motion activated controller for WS2812b LED lights Version 0.30 adds the ability to change settings from a web interface without r

Mar 12, 2022
(Simple String Format) is an syntax of format and a library for parse this.

SSFMT (Simple String Format) is an syntax of format and a library for parse this. SSFMT != {fmt} SSFMT is NOT an API/library for parse {fmt} syntax !

Jan 30, 2022
mCube's ultra-low-power wake-on-motion 3-axis accelerometer
mCube's ultra-low-power wake-on-motion 3-axis accelerometer

MC3635 mCube's ultra-low-power wake-on-motion 3-axis accelerometer Based on mCube's Arduino demo driver, this sketch is specific for the MC3635 3-axis

Aug 26, 2022
Motion planner built upon Tesseract and Trajopt

motion_planner Motion planner built upon Tesseract and Trajopt The abb_example package is similar to the tesseract_ros_examples, but it contain more e

Jan 18, 2022
FluidNC - The next generation of motion control firmware
FluidNC - The next generation of motion control firmware

FluidNC (CNC Controller) For ESP32 Introduction FluidNC is the next generation of Grbl_ESP32. It has a lot of improvements over Grbl_ESP32 as listed b

Jan 3, 2023
DIY Zigbee CC2530 Motion sensor (AM312/ AM412/ BS312/ BS412), Temperature /Humidity /Pressure sensor (BME280), Ambient Light sensor (BH1750), 2.9inch e-Paper Module
DIY Zigbee CC2530 Motion sensor (AM312/ AM412/ BS312/ BS412), Temperature /Humidity /Pressure sensor (BME280), Ambient Light sensor (BH1750), 2.9inch e-Paper Module

How to join: If device in FN(factory new) state: Press and hold button (1) for 2-3 seconds, until device start flashing led Wait, in case of successfu

Feb 13, 2022
DIY Zigbee CC2530 Motion sensor (AM312/ AM412/ BS312/ BS412), Temperature /Humidity /Pressure sensor (BME280), Ambient Light sensor (BH1750), 2.9/2.13/1.54 inch e-Paper Module
DIY Zigbee CC2530 Motion sensor (AM312/ AM412/ BS312/ BS412), Temperature /Humidity /Pressure sensor (BME280), Ambient Light sensor (BH1750), 2.9/2.13/1.54 inch e-Paper Module

How to join: If device in FN(factory new) state: Press and hold button (1) for 2-3 seconds, until device start flashing led Wait, in case of successfu

Dec 9, 2022
Updated Vindriktning with Wifi Connectivity, Motion sensor, Temperature and Humidity
Updated Vindriktning with Wifi Connectivity, Motion sensor, Temperature and Humidity

Vindriktning-plus Updated Vindriktning with Wifi Connectivity, Motion sensor, Temperature and Humidity Inspired & parts of the code are used from: htt

Sep 20, 2022
Code accompanying our SIGGRAPH 2021 Technical Communications paper "Transition Motion Tensor: A Data-Driven Approach for Versatile and Controllable Agents in Physically Simulated Environments"
Code accompanying our SIGGRAPH 2021 Technical Communications paper

SIGGRAPH ASIA 2021 Technical Communications Transition Motion Tensor: A Data-Driven Framework for Versatile and Controllable Agents in Physically Simu

Apr 21, 2022
a Lightweight Motion Planning Package
a Lightweight Motion Planning Package

MPlib MPlib is a lightweight python package for motion planning, which is decoupled from ROS and is easy to set up. With a few lines of python code, o

Dec 30, 2022
Lidar-with-velocity - Lidar with Velocity: Motion Distortion Correction of Point Clouds from Oscillating Scanning Lidars
Lidar-with-velocity - Lidar with Velocity: Motion Distortion Correction of Point Clouds from Oscillating Scanning Lidars

Lidar with Velocity A robust camera and Lidar fusion based velocity estimator to undistort the pointcloud. This repository is a barebones implementati

Dec 15, 2022
The MoveIt motion planning framework
The MoveIt motion planning framework

The MoveIt Motion Planning Framework Easy-to-use open source robotics manipulation platform for developing commercial applications, prototyping design

Jan 2, 2023
Collection of Arduino sketches for TDK's combo accel/gyro motion sensor
Collection of Arduino sketches for TDK's combo accel/gyro motion sensor

ICM42688 Collection of Arduino sketches for TDK's combo accel/gyro motion sensor The basic sketch configures the sensors' data rates and full scale se

Dec 13, 2022
A cross platform shader language with multi-threaded offline compilation or platform shader source code generation
A cross platform shader language with multi-threaded offline compilation or platform shader source code generation

A cross platform shader language with multi-threaded offline compilation or platform shader source code generation. Output json reflection info and c++ header with your shaders structs, fx-like techniques and compile time branch evaluation via (uber-shader) "permutations".

Dec 14, 2022
A library to handle Apple Property List format in binary or XML

libplist A small portable C library to handle Apple Property List files in binary or XML format. Features The project provides an interface to read an

Dec 26, 2022
This is a helper library to abstract away interfacing with floppy disk drives in a cross-platform and open source library.
This is a helper library to abstract away interfacing with floppy disk drives in a cross-platform and open source library.

Adafruit Floppy This is a helper library to abstract away interfacing with floppy disk drives in a cross-platform and open source library. Adafruit Fl

Dec 19, 2022
Converts file formats supported by AdPlug to VGM format.

adlib2vgm Converts file formats supported by AdPlug to VGM format. Notice This tool is originally designed to work with the RetroWave OPL3 Sound Card.

Nov 25, 2022