MPack - A C encoder/decoder for the MessagePack serialization format / msgpack.org[C]

Introduction

MPack is a C implementation of an encoder and decoder for the MessagePack serialization format. It is:

The core of MPack contains a buffered reader and writer, and a tree-style parser that decodes into a tree of dynamically typed nodes. Helper functions can be enabled to read values of expected type, to work with files, to allocate strings automatically, to check UTF-8 encoding, and more.

The MPack code is small enough to be embedded directly into your codebase. Simply download the amalgamation package and add mpack.h and mpack.c to your project.

The MPack featureset can be customized at compile-time to set which features, components and debug checks are compiled, and what dependencies are available.

Build Status

Travis-CI AppVeyor Coveralls.io
Build Status Build Status Build Status

The Node API

The Node API parses a chunk of MessagePack data into an immutable tree of dynamically-typed nodes. A series of helper functions can be used to extract data of specific types from each node.

// parse a file into a node tree
mpack_tree_t tree;
mpack_tree_init_filename(&tree, "homepage-example.mp", 0);
mpack_tree_parse(&tree);
mpack_node_t root = mpack_tree_root(&tree);

// extract the example data on the msgpack homepage
bool compact = mpack_node_bool(mpack_node_map_cstr(root, "compact"));
int schema = mpack_node_i32(mpack_node_map_cstr(root, "schema"));

// clean up and check for errors
if (mpack_tree_destroy(&tree) != mpack_ok) {
    fprintf(stderr, "An error occurred decoding the data!\n");
    return;
}

Note that no additional error handling is needed in the above code. If the file is missing or corrupt, if map keys are missing or if nodes are not in the expected types, special "nil" nodes and false/zero values are returned and the tree is placed in an error state. An error check is only needed before using the data.

The above example decodes into allocated pages of nodes. A fixed node pool can be provided to the parser instead in memory-constrained environments. For maximum performance and minimal memory usage, the Expect API can be used to parse data of a predefined schema.

The Write API

The Write API encodes structured data to MessagePack.

// encode to memory buffer
char* data;
size_t size;
mpack_writer_t writer;
mpack_writer_init_growable(&writer, &data, &size);

// write the example on the msgpack homepage
mpack_start_map(&writer, 2);
mpack_write_cstr(&writer, "compact");
mpack_write_bool(&writer, true);
mpack_write_cstr(&writer, "schema");
mpack_write_uint(&writer, 0);
mpack_finish_map(&writer);

// finish writing
if (mpack_writer_destroy(&writer) != mpack_ok) {
    fprintf(stderr, "An error occurred encoding the data!\n");
    return;
}

// use the data
do_something_with_data(data, size);
free(data);

In the above example, we encode to a growable memory buffer. The writer can instead write to a pre-allocated or stack-allocated buffer, avoiding the need for memory allocation. The writer can also be provided with a flush function (such as a file or socket write function) to call when the buffer is full or when writing is done.

If any error occurs, the writer is placed in an error state. The writer will flag an error if too much data is written, if the wrong number of elements are written, if the data could not be flushed, etc. No additional error handling is needed in the above code; any subsequent writes are ignored when the writer is in an error state, so you don't need to check every write for errors.

Note in particular that in debug mode, the mpack_finish_map() call above ensures that two key/value pairs were actually written as claimed, something that other MessagePack C/C++ libraries may not do.

Comparison With Other Parsers

MPack is rich in features while maintaining very high performance and a small code footprint. Here's a short feature table comparing it to other C parsers:

MPack
(v1.1)
msgpack-c
(v3.2.0)
CMP
(v18)
CWPack
(v1.1)
No libc requirement
Growable memory writer ✓*
File I/O helpers ✓*
Propagating errors
Incremental parser
Tree stream parser
Compound size tracking
Automatic compound size

A larger feature comparison table is available here which includes descriptions of the various entries in the table.

This benchmarking suite compares the performance of MPack to other implementations of schemaless serialization formats. MPack outperforms all JSON and MessagePack libraries (except CWPack), and in some tests MPack is several times faster than RapidJSON for equivalent data.

Why Not Just Use JSON?

Conceptually, MessagePack stores data similarly to JSON: they are both composed of simple values such as numbers and strings, stored hierarchically in maps and arrays. So why not just use JSON instead? The main reason is that JSON is designed to be human-readable, so it is not as efficient as a binary serialization format:

  • Compound types such as strings, maps and arrays are delimited, so appropriate storage cannot be allocated upfront. The whole object must be parsed to determine its size.

  • Strings are not stored in their native encoding. Special characters such as quotes and backslashes must be escaped when written and converted back when read.

  • Numbers are particularly inefficient (especially when parsing back floats), making JSON inappropriate as a base format for structured data that contains lots of numbers.

  • Binary data is not supported by JSON at all. Small binary blobs such as icons and thumbnails need to be Base64 encoded or passed out-of-band.

The above issues greatly increase the complexity of the decoder. Full-featured JSON decoders are quite large, and minimal decoders tend to leave out such features as string unescaping and float parsing, instead leaving these up to the user or platform. This can lead to hard-to-find platform-specific and locale-specific bugs, as well as a greater potential for security vulnerabilites. This also significantly decreases performance, making JSON unattractive for use in applications such as mobile games.

While the space inefficiencies of JSON can be partially mitigated through minification and compression, the performance inefficiencies cannot. More importantly, if you are minifying and compressing the data, then why use a human-readable format in the first place?

Testing MPack

The MPack build process does not build MPack into a library; it is used to build and run the unit tests. You do not need to build MPack or the unit testing suite to use MPack.

See test/README.md for information on how to test MPack.

Owner
Comments
  • Initial work on generic mpack_write fixes #34

    Initial work on generic mpack_write fixes #34

    As discussed in #34 for now there is only the generic writer with C++ overloading. I added a c11=1 target to scons (for now). Before merging I can continue working on this pull request and we can discuss in #34. Let me know what you think.

  • Usage in Linux kernel modules

    Usage in Linux kernel modules

    Hi, we are using this nice library within Linux kernel modules. Since we only need the writer, a colleague took only the necessary code and changed/removed stuff to make it compile.

    This works but is of course not that nice. I haven't found any clean way of configuring the project accordingly.

    Did I miss something?

    best regards, Matthias

  • Possible encoding problems with differing double/int64 endianness

    Possible encoding problems with differing double/int64 endianness

    Hi Nicholas,

    I'm working on an ARM platform(little-endian) with your library. I was using doubles and floats, but this probably also counts for other types in mpack. The items in the the packet are endian swapped and not sent big-endian as described in the spec. https://github.com/msgpack/msgpack/blob/master/spec.md#formats-float

    When I send them to a server (i.e. x86), the values are also wrong. Using a union for mapping to uint64 and double is valid on big-endian systems, but invalid on little-endian. https://github.com/ludocode/mpack/blob/8c5bd668b515ded9e8e8e5a2f48033d2c3f6bd75/src/mpack/mpack-writer.c#L377-L384

    Perhaps it is a good idea to check for little-endian and swap the bytes when necessary?

  • MPACK_HAS_GENERIC autodetection is bugged with clang and/or C++ compiler

    MPACK_HAS_GENERIC autodetection is bugged with clang and/or C++ compiler

    With clang 3.7 the generic autodetection is bugged. Because of the following issue: See http://clang-developers.42468.n3.nabble.com/Generic-compiles-fine-in-c-source-td4037233.html

  • Undefined configuration and compiler definitions cause warnings

    Undefined configuration and compiler definitions cause warnings

    Hi Nicholas,

    I was compiling Mpack for a non-Linux platform today (non-GNU) and ran into a big list of warnings (46 to be exact). Since we want to compile our code with -Werror we really want to fix it. But we need your opinion on this.

    See some of our warnings below this. We are using the amalgamation package, so line-numbers are crazy, but you will get the point. The compiler moans about the check within #if while the variable is not defined.

    In the example of the first warning:

    #if MPACK_STDLIB
    

    should become:

    #if defined(MPACK_STDLIB) && MPACK_STDLIB
    

    but this creates a mess in the code. From my point of view, you want to use:

    #ifdef MPACK_STDLIB
    

    or

    #if defined(MPACK_STDLIB)
    

    but this changes the behavior when configuring Mpack, only setting config to 0 wouldn't suffice. You need to disable the config option completely.

    We are using GCC 4.9.3 Linaro arm-none-eabi cross-tools.

    I can fix this and make a PR for you. But (since it is your lib) I need your input on this.

    Thanks,

    Rik

    /3rdparty/mpack/src/mpack/mpack.h:78:5: warning: "MPACK_STDLIB" is not defined [-Wundef]
     #if MPACK_STDLIB
         ^
    /3rdparty/mpack/src/mpack/mpack.h:82:5: warning: "MPACK_STDIO" is not defined [-Wundef]
     #if MPACK_STDIO
         ^
    /3rdparty/mpack/src/mpack/mpack.h:85:5: warning: "MPACK_SETJMP" is not defined [-Wundef]
     #if MPACK_SETJMP
         ^
    /3rdparty/mpack/src/mpack/mpack.h:113:7: warning: "_MSC_VER" is not defined [-Wundef]
     #elif _MSC_VER
           ^
    /3rdparty/mpack/src/mpack/mpack.h:192:5: warning: "MPACK_STDLIB" is not defined [-Wundef]
     #if MPACK_STDLIB
         ^
    /3rdparty/mpack/src/mpack/mpack.h:224:5: warning: "MPACK_READ_TRACKING" is not defined [-Wundef]
     #if MPACK_READ_TRACKING && (!defined(MPACK_READER) || !MPACK_READER)
         ^
    /3rdparty/mpack/src/mpack/mpack.h:227:5: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
     #if MPACK_WRITE_TRACKING && (!defined(MPACK_WRITER) || !MPACK_WRITER)
         ^
    /3rdparty/mpack/src/mpack/mpack.h:231:9: warning: "MPACK_STDIO" is not defined [-Wundef]
         #if MPACK_STDIO
             ^
    /3rdparty/mpack/src/mpack/mpack.h:234:9: warning: "MPACK_READ_TRACKING" is not defined [-Wundef]
         #if MPACK_READ_TRACKING
             ^
    /3rdparty/mpack/src/mpack/mpack.h:237:9: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
         #if MPACK_WRITE_TRACKING
             ^
    /3rdparty/mpack/src/mpack/mpack.h:559:5: warning: "MPACK_READ_TRACKING" is not defined [-Wundef]
     #if MPACK_READ_TRACKING || MPACK_WRITE_TRACKING
         ^
    /3rdparty/mpack/src/mpack/mpack.h:559:28: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
     #if MPACK_READ_TRACKING || MPACK_WRITE_TRACKING
                                ^
    /3rdparty/mpack/src/mpack/mpack.h:780:5: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
     #if MPACK_WRITE_TRACKING
         ^
    /3rdparty/mpack/src/mpack/mpack.h:827:9: warning: "MPACK_SETJMP" is not defined [-Wundef]
         #if MPACK_SETJMP
             ^
    /3rdparty/mpack/src/mpack/mpack.h:833:9: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
         #if MPACK_WRITE_TRACKING
             ^
    /3rdparty/mpack/src/mpack/mpack.h:882:5: warning: "MPACK_STDIO" is not defined [-Wundef]
     #if MPACK_STDIO
         ^
    /3rdparty/mpack/src/mpack/mpack.h:906:5: warning: "MPACK_SETJMP" is not defined [-Wundef]
     #if MPACK_SETJMP
         ^
    /3rdparty/mpack/src/mpack/mpack.h:1203:5: warning: "MPACK_WRITE_TRACKING" is not defined [-Wundef]
     #if MPACK_WRITE_TRACKING
         ^
    /3rdparty/mpack/src/mpack/mpack.h:1298:5: warning: "MPACK_READ_TRACKING" is not defined [-Wundef]
     #if MPACK_READ_TRACKING
         ^
    /3rdparty/mpack/src/mpack/mpack.h:1351:9: warning: "MPACK_SETJMP" is not defined [-Wundef]
         #if MPACK_SETJMP
    
  • mpack_writer should not set flag to mpack_error_io when buffer to small

    mpack_writer should not set flag to mpack_error_io when buffer to small

    I think current behaviour is a bit weird when a static buffer is not big enough and more is tried to be written and the context is destroyed. The error flag is expected to be mpack_error_too_big but it is stated in the documentation (and emitted) currently as mpack_error_io.

  • Build for kernel module

    Build for kernel module

    This relates to: https://github.com/ludocode/mpack/issues/80

    The request contains everything I need to change to have it compiled withing a linux kernel module. Commit c92c2b3 looks not that nice to me. I think this one needs some refurbishment.

  • Support building with CMake or similar

    Support building with CMake or similar

    Hi,

    It would be really handy to be able to build this with CMake or some other standard build system, as Visual Studio .sln files aren't readily usable on Linux. I can't see any way to compile this on Linux currently, which is a shame, as I'd otherwise love to use this library.

  • Created a JUCE module for mpack.

    Created a JUCE module for mpack.

    This makes integrating mpack in C++ JUCE projects really swift and easy.

    Implementation Notes

    • JUCE Unity Build Module Format
    • The module code is explicitly designed to be CRLF to support MSVC. This is a consistent design choice for JUCE projects everywhere.

    Testing Question

    I'm open to ideas for how you would like to have these files be automatically tested as part of your CI.

    • You would definitely need a clone of JUCE from its master or develop branch.
    • You would need to build its Projucer app.
    • You will need to have a Projucer-based project ready with your module.
    • You can run the Projucer from the command-line:
      • Just resave the project.
        • pathToProjucer --resave path/to/MyProject.jucer

    I'm not sure what you would want to see run as part of this project. Although, JUCE provides a UnitTest class which could be integrated into the module, and selectively enabled via a module configuration macro.

    MSVC Warning Notes

    There are two types of warnings that come up. Seeing that I don't have the entire context for why the code is the way it is, I'm not totally sure how you would prefer tackling them so I'll leave that up to you!

    • C4127 is a common occurrence.
      • Example: if (sizeof(unsigned int) == 4)
    • There are a handful of truncation warnings (C4310).
      • mpack_write_byte_element(writer, (char)0xc0);
        • 0xc0 is 192 in decimal, causing overflow and therefore UB.

    Module Usage Example in JUCE's Projucer

    image

  • Improved c89 compatibility

    Improved c89 compatibility

    Heyo!

    Currently your code fails to compile when c_std=gnu89. However, the only C99 feature that you appear to use is the ability to declare variables inside for loops. This PR moves the declarations outside the for loops, and compiles when c_std=gnu89

    Cheers!

  • Overly used of inline keyword

    Overly used of inline keyword

    Hi,

    I'm integrating mpack into microcontroller/firmware and the use of inline is a no-go because it explodes the footprint of the firmware by crazy. The "magic" of inline is also not as you expect and the linux kernel coding style has some word about it:

    See Chapter 15: The inline disease https://www.kernel.org/doc/Documentation/CodingStyle

  • ESP32-C3 rebooting when executing functions from the MPack library

    ESP32-C3 rebooting when executing functions from the MPack library

    Hello, I'm testing the MPack library in an ESP32-C3 microcontroller. Below the code I'm using for testing (I added the MPack functions in the app_main function):

    #include <stdio.h>
    #include <stdint.h>
    #include <stddef.h>
    #include <string.h>
    
    #include "esp_wifi.h"
    #include "esp_system.h"
    #include "nvs_flash.h"
    #include "esp_event.h"
    #include "esp_netif.h"
    #include "protocol_examples_common.h"
    
    #include "freertos/FreeRTOS.h"
    #include "freertos/task.h"
    #include "freertos/semphr.h"
    #include "freertos/queue.h"
    
    #include "lwip/sockets.h"
    #include "lwip/dns.h"
    #include "lwip/netdb.h"
    
    #include "esp_log.h"
    #include "mqtt_client.h"
    
    #include "mpack.h"
    
    void print_data(char [], size_t);
    
    static const char *TAG = "MQTT_EXAMPLE";
    
    static void log_error_if_nonzero(const char *message, int error_code)
    {
        if (error_code != 0) {
            ESP_LOGE(TAG, "Last error %s: 0x%x", message, error_code);
        }
    }
    
    /*
     * @brief Event handler registered to receive MQTT events
     *
     *  This function is called by the MQTT client event loop.
     *
     * @param handler_args user data registered to the event.
     * @param base Event base for the handler(always MQTT Base in this example).
     * @param event_id The id for the received event.
     * @param event_data The data for the event, esp_mqtt_event_handle_t.
     */
    static void mqtt_event_handler(void *handler_args, esp_event_base_t base, int32_t event_id, void *event_data)
    {
        ESP_LOGD(TAG, "Event dispatched from event loop base=%s, event_id=%d", base, event_id);
        esp_mqtt_event_handle_t event = event_data;
        esp_mqtt_client_handle_t client = event->client;
        int msg_id;
        switch ((esp_mqtt_event_id_t)event_id) {
        case MQTT_EVENT_CONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_CONNECTED");
            msg_id = esp_mqtt_client_publish(client, "/topic/qos1", "data_3", 0, 1, 0);
            ESP_LOGI(TAG, "sent publish successful, msg_id=%d", msg_id);
    
            msg_id = esp_mqtt_client_subscribe(client, "/topic/qos0", 0);
            ESP_LOGI(TAG, "sent subscribe successful, msg_id=%d", msg_id);
    
            msg_id = esp_mqtt_client_subscribe(client, "/topic/qos1", 1);
            ESP_LOGI(TAG, "sent subscribe successful, msg_id=%d", msg_id);
    
            msg_id = esp_mqtt_client_unsubscribe(client, "/topic/qos1");
            ESP_LOGI(TAG, "sent unsubscribe successful, msg_id=%d", msg_id);
            break;
        case MQTT_EVENT_DISCONNECTED:
            ESP_LOGI(TAG, "MQTT_EVENT_DISCONNECTED");
            break;
    
        case MQTT_EVENT_SUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_SUBSCRIBED, msg_id=%d", event->msg_id);
            msg_id = esp_mqtt_client_publish(client, "/topic/qos0", "data", 0, 0, 0);
            ESP_LOGI(TAG, "sent publish successful, msg_id=%d", msg_id);
            break;
        case MQTT_EVENT_UNSUBSCRIBED:
            ESP_LOGI(TAG, "MQTT_EVENT_UNSUBSCRIBED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_PUBLISHED:
            ESP_LOGI(TAG, "MQTT_EVENT_PUBLISHED, msg_id=%d", event->msg_id);
            break;
        case MQTT_EVENT_DATA:
            ESP_LOGI(TAG, "MQTT_EVENT_DATA");
            printf("TOPIC=%.*s\r\n", event->topic_len, event->topic);
            printf("DATA=%.*s\r\n", event->data_len, event->data);
            break;
        case MQTT_EVENT_ERROR:
            ESP_LOGI(TAG, "MQTT_EVENT_ERROR");
            if (event->error_handle->error_type == MQTT_ERROR_TYPE_TCP_TRANSPORT) {
                log_error_if_nonzero("reported from esp-tls", event->error_handle->esp_tls_last_esp_err);
                log_error_if_nonzero("reported from tls stack", event->error_handle->esp_tls_stack_err);
                log_error_if_nonzero("captured as transport's socket errno",  event->error_handle->esp_transport_sock_errno);
                ESP_LOGI(TAG, "Last errno string (%s)", strerror(event->error_handle->esp_transport_sock_errno));
    
            }
            break;
        default:
            ESP_LOGI(TAG, "Other event id:%d", event->event_id);
            break;
        }
    }
    
    static void mqtt_app_start(void)
    {
        esp_mqtt_client_config_t mqtt_cfg = {
            .uri = CONFIG_BROKER_URL,
        };
    #if CONFIG_BROKER_URL_FROM_STDIN
        char line[128];
    
        if (strcmp(mqtt_cfg.uri, "FROM_STDIN") == 0) {
            int count = 0;
            printf("Please enter url of mqtt broker\n");
            while (count < 128) {
                int c = fgetc(stdin);
                if (c == '\n') {
                    line[count] = '\0';
                    break;
                } else if (c > 0 && c < 127) {
                    line[count] = c;
                    ++count;
                }
                vTaskDelay(10 / portTICK_PERIOD_MS);
            }
            mqtt_cfg.uri = line;
            printf("Broker url: %s\n", line);
        } else {
            ESP_LOGE(TAG, "Configuration mismatch: wrong broker url");
            abort();
        }
    #endif /* CONFIG_BROKER_URL_FROM_STDIN */
    
        esp_mqtt_client_handle_t client = esp_mqtt_client_init(&mqtt_cfg);
        /* The last argument may be used to pass data to the event handler, in this example mqtt_event_handler */
        esp_mqtt_client_register_event(client, ESP_EVENT_ANY_ID, mqtt_event_handler, NULL);
        esp_mqtt_client_start(client);
    }
    
    void app_main(void)
    {
        ESP_ERROR_CHECK(nvs_flash_init());
        ESP_ERROR_CHECK(esp_netif_init());
        ESP_ERROR_CHECK(esp_event_loop_create_default());
    
        /* This helper function configures Wi-Fi or Ethernet, as selected in menuconfig.
         * Read "Establishing Wi-Fi or Ethernet Connection" section in
         * examples/protocols/README.md for more information about this function.
         */
        //ESP_ERROR_CHECK(example_connect());
    
        mqtt_app_start();
    
    	// encode to memory buffer
    	char *data;
    	char data_r[256] = {0};
    	size_t size;
    	int len = 0;
    	mpack_writer_t writer;
    	mpack_writer_init_growable(&writer, &data, &size);
    
    	// write the example on the msgpack homepage
    	mpack_build_map(&writer);
    	mpack_write_cstr(&writer, "compact");
    	mpack_write_bool(&writer, true);
    	mpack_write_cstr(&writer, "schemaaaaa");
    	mpack_write_uint(&writer, 7);
    	mpack_complete_map(&writer);
    
            len = writer.position;
    
    	printf("Print 1: %u %u\n", len, (*writer.end));
    	printf("Print 2: %u %u\n", data[1], data[2]);
    
    	// finish writing
    	if (mpack_writer_destroy(&writer) != mpack_ok) {
    		  fprintf(stderr, "An error occurred encoding the data!\n");
    		  return;
    	}
    	else
    	{
    
    	}
    
    
    
    	// use the data
    
    	free(data);
    }
    
    
    void print_data(char data_in[], size_t length)
    {
    	for (size_t i = 0; i < length; i++)
    	{
    		printf("%X ", data_in[i]);
    	}
    }
    

    When I try to print the data in line printf("Print 2: %u %u\n", data[1], data[2]); I the microcontroller reboots:

    Print 1: 0 249
    Guru Meditation Error: Core  0 panic'ed (Load access fault). Exception was unhandled.
    
    Stack dump detected
    Core  0 register dump:
    MEPC    : 0x42007944  RA      : 0x42007942  SP      : 0x3fc91b90  GP      : 0x3fc8be00  
    0x42007944: app_main at /home/msurca/esp/IoTliftv1_0_3/tcp/build/../main/app_main.c:196 (discriminator 2)
    
    0x42007942: app_main at /home/msurca/esp/IoTliftv1_0_3/tcp/build/../main/app_main.c:196 (discriminator 2)
    
    TP      : 0x3fc7513c  T0      : 0x4005890e  T1      : 0x20000000  T2      : 0x3fc68a28  
    S0/FP   : 0x00000000  S1      : 0x00000000  A0      : 0x0000000f  A1      : 0x3fc917c8  
    A2      : 0x00000000  A3      : 0x00000001  A4      : 0x3fc8e000  A5      : 0x00000000  
    A6      : 0x42005cf4  A7      : 0x0000000a  S2      : 0x00000000  S3      : 0x00000000  
    0x42005cf4: console_write at /home/msurca/esp/esp-idf/components/vfs/vfs_console.c:71
    
    S4      : 0x00000000  S5      : 0x00000000  S6      : 0x00000000  S7      : 0x00000000  
    S8      : 0x00000000  S9      : 0x00000000  S10     : 0x00000000  S11     : 0x00000000  
    T3      : 0x00000000  T4      : 0x3ff00000  T5      : 0x00000000  T6      : 0x3fd34413  
    MSTATUS : 0x00001881  MTVEC   : 0x40380001  MCAUSE  : 0x00000005  MTVAL   : 0x00000002  
    0x40380001: _vector_table at ??:?
    
    MHARTID : 0x00000000  
    
    
    Backtrace:
    
    
    0x42007944 in app_main () at ../main/app_main.c:196
    196		printf("Print 2: %u %u\n", data[1], data[2]);
    #0  0x42007944 in app_main () at ../main/app_main.c:196
    #1  0x4205e806 in main_task (args=<optimized out>) at /home/msurca/esp/esp-idf/components/freertos/FreeRTOS-Kernel/portable/port_common.c:124
    #2  0x40387226 in prvTaskExitError () at /home/msurca/esp/esp-idf/components/freertos/FreeRTOS-Kernel/portable/riscv/port.c:132
    Backtrace stopped: frame did not save the PC
    ELF file SHA256: a7c731be59522570
    
    Rebooting...
    

    Also, if I change the content of "schema" by "schema12345" the end of the string is also 249. So, I am not sure if this is the function I should use to get the length of the data in the buffer. If I use the mpack_writer_buffer_used function the length I always get is 0.

    Finally, if I comment the line mpack_write_cstr(&writer, "compact");the program crashes and the microcontroller restarts. If the line mpack_build_map(&writer); is followed by the function mpack_write_cstr the program can be executed without crashing.

    Which could be the reason of these problems?

  • Warning about incompatible int types in mpack_snprintf calls

    Warning about incompatible int types in mpack_snprintf calls

    I copied the "amalgamation package" sources to my project and compiled it with a STM32 CubeIDE compiler. Got some warnings about incompatible types:

    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 427 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 477 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 489 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 492 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 544 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 547 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 556 C/C++ Problem
    format '%u' expects argument of type 'unsigned int', but argument 4 has type 'uint32_t' {aka 'long unsigned int'} [-Wformat=] mpack.c.line 559 C/C++ Problem
    

    Quickfix seems to be to replace all %u with %lu in the affected mpack_snprintf calls from above, where the compiler is complaining. Don't know if this could be fixed in the source repo here and still be platform independent?

  • comparison with libmpack

    comparison with libmpack

    There's another project (since 2016) with a similar name: https://github.com/libmpack/libmpack

    Would be useful to see it in your comparison table:

    https://github.com/ludocode/mpack#comparison-with-other-parsers

    I also wonder if the names are too similar? It looks like https://github.com/ludocode/mpack generates its library file as libmpack, which may conflict with https://github.com/libmpack/libmpack .

  • Stream reader with manual memory management?

    Stream reader with manual memory management?

    Hi, I really like how well this library is developed and might want to use it in a project. However, I'm not sure if the node reader API can be used the way I'd like. To my understanding, the API is able to:

    • let the user handle memory management: With mpack_tree_init_pool(), the library does not allocate any memory by itself. Both the input data and the nodes memory is allocated by the user.
    • parse a continuous stream: With mpack_tree_init_stream() I can use the library for processing a TCP stream of MessagePack data.

    However, it seems that these two features cannot be combined. The stream reader always allocates its own buffer for the input data, and for the nodes. So I cannot manage the memory by myself. Especially for environments where memory fragmentation needs to be minimized, that's a real drawback.

    Actually for me it would be good enough if I could simply allocate a large enough buffer for the data and for the nodes once, which shall then be used as-is without any reallocation (memory is available, but should not fragment too much during runtime). Probably this is (theoretically) possible even with mpack_tree_init_stream(), but there's still a problem: The documentation says that the reader cannot be reset if it is in error state. So in that case I have to destroy and create a new reader, which will also reallocate (and possibly fragment) the memory.

    Is my understanding about that correct? Or is there a way to recover from a stream reader error without any memory reallocation?

    By the way, somehow it is also a bit cumbersome that reading the data has to be implemented by a callback. I understand that for directly reading from a blocking TCP socket it works perfectly. But if the data is received by some other software component and then forwarded in blocks to some kind of "data processor", it is very cumbersome. It would be much simpler if I could allocate my own data buffer, and let the parser process it block by block.

    Example (simplified):

    char buffer[1024];
    size_t bufferSize = 0;
    mpack_tree_t tree;
    
    void init() {
        // let the stream parser know the start address of the data
        mpack_tree_init_stream(&tree, &buffer[0], ...);
    }
    
    void processTcpData(const char* data, size_t len) {
        // append new data to existing data
        memcpy(&buffer[bufferSize], data, len);
        bufferSize += len;
    
        // continue parsing (only the new data)
        mpack_tree_add_data(&tree, len);
    
        // process all new objects from the start of the buffer
        while (mpack_tree_parse(&tree)) {
            mpack_node_t node = mpack_tree_root(&tree);
            processNode(node);
        }
    
        // remove the processed data from the buffer, move any additional data to the buffer start
        size_t processedBytes = mpack_tree_remove_processed_data(&tree);
        bufferSize -= processedBytes;
    }
    

    Maybe the example is not perfectly correct, but I hope you get the idea :slightly_smiling_face: Such an API would be very useful for me. Do you know if something like that is possible, either with mpack or with any other C/C++ library?

    Thanks in advance.

  • builder page allocation leak and segfault

    builder page allocation leak and segfault

    hi, I ran into a segfault with the following code:

    mpack_writer_t writer;
    
    char *output_data = nullptr;
    size_t output_size = 0;
    mpack_writer_init_growable(&writer, &output_data, &output_size);
    
    
    const int N_OUTER = 10;
    const int N_INNER = 100;
    
    mpack_start_array(&writer, N_OUTER);
    for (int k=0; k<N_OUTER; k++) {
    	mpack_build_map(&writer);
    	mpack_write_str(&writer, "key", 3);
    	mpack_write_u64(&writer, k*k);
    	mpack_write_str(&writer, "value", 5);
    	mpack_start_array(&writer, N_INNER);
    	for (int i=0; i<N_INNER; i++) {
    		mpack_build_map(&writer);
    		mpack_write_str(&writer, "value0", 6);
    		mpack_write_i64(&writer, i);
    		mpack_write_str(&writer, "value1", 6);
    		mpack_write_i64(&writer, i*k);
    		mpack_complete_map(&writer);
    	}
    	mpack_finish_array(&writer);
    	mpack_complete_map(&writer); // <- segfault here
    }
    mpack_finish_array(&writer);
    
    
    mpack_writer_destroy(&writer);
    

    At the end of the 5th outer loop, the call to mpack_complete_map() will corrupt the internal memory of the builder, resulting in a segfault. Some mild debugging traced the issue back to the page allocation mechanism. According to the logs, we might be freeing one page and using the next without initializing it.

    The problem vanishes when writing different integer values. It seems to be triggered only by very specific alignment conditions.

    I would appreciate it ,if you could confirm the findings or give me a clue of any mismatching calls in my code. Thanks!

  • mpack crashes without a matching call to mpack_complete_*

    mpack crashes without a matching call to mpack_complete_*

    Taking the example from the README:

    #include <stdlib.h>
    #include <stdio.h>
    #include <mpack.h>
    
    int main()
    {
        char* data;
        size_t size;
        mpack_writer_t writer;
        mpack_writer_init_growable(&writer, &data, &size);
    
        // write the example on the msgpack homepage
        mpack_build_map(&writer);
        mpack_write_cstr(&writer, "compact");
        mpack_write_bool(&writer, true);
        mpack_write_cstr(&writer, "schema");
        mpack_write_uint(&writer, 0);
        //mpack_complete_map(&writer);
    
        // finish writing
        if (mpack_writer_destroy(&writer) != mpack_ok) {
            fprintf(stderr, "An error occurred encoding the data!\n");
            return 1;
        }
    
        // use the data
        printf("%.*s", (int)size, data);
        free(data);
        return 0;
    }
    
    glopes /tmp $ gcc -o t -g -I/tmp a.c mpack.c && valgrind -q ./t
    ==14417== Invalid free() / delete / delete[] / realloc()
    ==14417==    at 0x4C37DAD: realloc (vg_replace_malloc.c:1192)
    ==14417==    by 0x108D37: mpack_realloc (mpack.h:1842)
    ==14417==    by 0x10ABAB: mpack_growable_writer_teardown (mpack.c:1208)
    ==14417==    by 0x10B420: mpack_writer_destroy (mpack.c:1499)
    ==14417==    by 0x108C95: main (a.c:21)
    ==14417==  Address 0x52360b0 is 48 bytes inside a block of size 4,096 alloc'd
    ==14417==    at 0x4C32FB5: malloc (vg_replace_malloc.c:380)
    ==14417==    by 0x10DC1F: mpack_builder_begin (mpack.c:2470)
    ==14417==    by 0x10DCDF: mpack_builder_build (mpack.c:2495)
    ==14417==    by 0x10E1EC: mpack_build_map (mpack.c:2711)
    ==14417==    by 0x108C32: main (a.c:13)
    An error occurred encoding the data!
    

    This makes it impossible to use error chaining. In this case, I omitted the call to mpack_complete_map altogether, but it can also be skipped because of an earlier error. In fact, it's a problem even with:

    mpack_build_map(&writer);
    mpack_write_cstr(&writer, "foo");
    mpack_complete_map(&writer);
    mpack_writer_destroy(&writer);
    
MessagePack implementation for C and C++ / msgpack.org[C/C++]

msgpack for C/C++ It's like JSON but smaller and faster. Overview MessagePack is an efficient binary serialization format, which lets you exchange dat

Dec 5, 2022
Msgpack11 - A tiny MessagePack library for C++11 (msgpack.org[C++11])

What is msgpack11 ? msgpack11 is a tiny MsgPack library for C++11, providing MsgPack parsing and serialization. This library is inspired by json11. Th

Dec 1, 2022
Your binary serialization library

Bitsery Header only C++ binary serialization library. It is designed around the networking requirements for real-time data delivery, especially for ga

Dec 1, 2022
Nov 28, 2022
Cap'n Proto serialization/RPC system - core tools and C++ library
Cap'n Proto serialization/RPC system - core tools and C++ library

Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except

Dec 1, 2022
A C++11 library for serialization
A C++11 library for serialization

cereal - A C++11 library for serialization cereal is a header-only C++11 serialization library. cereal takes arbitrary data types and reversibly turns

Dec 2, 2022
Fast Binary Encoding is ultra fast and universal serialization solution for C++, C#, Go, Java, JavaScript, Kotlin, Python, Ruby, Swift

Fast Binary Encoding (FBE) Fast Binary Encoding allows to describe any domain models, business objects, complex data structures, client/server request

Dec 1, 2022
FlatBuffers: Memory Efficient Serialization Library

FlatBuffers FlatBuffers is a cross platform serialization library architected for maximum memory efficiency. It allows you to directly access serializ

Dec 8, 2022
Yet Another Serialization
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

Nov 30, 2022
Binary Serialization

Binn Binn is a binary data serialization format designed to be compact, fast and easy to use. Performance The elements are stored with their sizes to

Nov 24, 2022
Simple C++ 20 Serialization Library that works out of the box with aggregate types!

BinaryLove3 Simple C++ 20 Serialization Library that works out of the box with aggregate types! Requirements BinaryLove3 is a c++20 only library.

Sep 2, 2022
Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications

Zmeya Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serializ

Nov 2, 2022
CppSerdes is a serialization/deserialization library designed with embedded systems in mind
CppSerdes is a serialization/deserialization library designed with embedded systems in mind

A C++ serialization/deserialization library designed with embedded systems in mind

Nov 5, 2022
Serialization framework for Unreal Engine Property System that just works!

DataConfig Serialization framework for Unreal Engine Property System that just works! Unreal Engine features a powerful Property System which implemen

Nov 27, 2022
Header-only library for automatic (de)serialization of C++ types to/from JSON.

fuser 1-file header-only library for automatic (de)serialization of C++ types to/from JSON. how it works The library has a predefined set of (de)seria

Oct 20, 2022
Yet Another Serialization
Yet Another Serialization

YAS Yet Another Serialization - YAS is created as a replacement of boost.serialization because of its insufficient speed of serialization (benchmark 1

Sep 7, 2021
C++17 library for all your binary de-/serialization needs

blobify blobify is a header-only C++17 library to handle binary de-/serialization in your project. Given a user-defined C++ struct, blobify can encode

Oct 20, 2022
universal serialization engine

A Universal Serialization Engine Based on compile-time Reflection iguana is a modern, universal and easy-to-use serialization engine developed in c++1

Dec 2, 2022
Yet another JSON/YAML/BSON serialization library for C++.
Yet another JSON/YAML/BSON serialization library for C++.

ThorsSerializer Support for Json Yaml Bson NEW Benchmark Results Conformance mac linux Performance max linux For details see: JsonBenchmark Yet anothe

Oct 27, 2022