Protocol Buffers with small code size

Nanopb - Protocol Buffers for Embedded Systems

Build Status

Nanopb is a small code-size Protocol Buffers implementation in ansi C. It is especially suitable for use in microcontrollers, but fits any memory restricted system.

Using the nanopb library

To use the nanopb library, you need to do two things:

  1. Compile your .proto files for nanopb, using protoc.
  2. Include pb_encode.c, pb_decode.c and pb_common.c in your project.

The easiest way to get started is to study the project in "examples/simple". It contains a Makefile, which should work directly under most Linux systems. However, for any other kind of build system, see the manual steps in README.txt in that folder.

Generating the headers

Protocol Buffers messages are defined in a .proto file, which follows a standard format that is compatible with all Protocol Buffers libraries. To use it with nanopb, you need to generate .pb.c and .pb.h files from it:

python generator/nanopb_generator.py myprotocol.proto  # For source checkout
generator-bin/nanopb_generator myprotocol.proto        # For binary package

(Note: For instructions for nanopb-0.3.9.x and older, see the documentation of that particular version here)

The binary packages for Windows, Linux and Mac OS X should contain all necessary dependencies, including Python, python-protobuf library and protoc. If you are using a git checkout or a plain source distribution, you will need to install Python separately. Once you have Python, you can install the other dependencies with pip install protobuf grpcio-tools.

You can further customize the header generation by creating an .options file. See documentation for details.

Running the tests

If you want to perform further development of the nanopb core, or to verify its functionality using your compiler and platform, you'll want to run the test suite. The build rules for the test suite are implemented using Scons, so you need to have that installed (ex: sudo apt install scons or pip install scons). To run the tests:

cd tests
scons

This will show the progress of various test cases. If the output does not end in an error, the test cases were successful.

Note: Mac OS X by default aliases 'clang' as 'gcc', while not actually supporting the same command line options as gcc does. To run tests on Mac OS X, use: scons CC=clang CXX=clang. Same way can be used to run tests with different compilers on any platform.

For embedded platforms, there is currently support for running the tests on STM32 discovery board and simavr AVR simulator. Use scons PLATFORM=STM32 and scons PLATFORM=AVR to run these tests.

Build systems and integration

Nanopb C code itself is designed to be portable and easy to build on any platform. Often the bigger hurdle is running the generator which takes in the .proto files and outputs .pb.c definitions.

There exist build rules for several systems:

And also integration to platform interfaces:

Building nanopb - Using vcpkg

You can download and install nanopb using the vcpkg dependency manager:

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg install nanopb

The nanopb port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.

Owner
Comments
  • pb_decode does not restore tag field when using pb_callback_t in oneof

    pb_decode does not restore tag field when using pb_callback_t in oneof

    test.proto:

    syntax="proto3";
    
    message TestMessage {
        oneof type {
            string text = 1;
            uint32 number = 2;
        }
    }
    

    test.options:

    * anonymous_oneof:true
    

    I wrote myself a little Test-function that sets up a message. Then the message is encoded and decoded back:

    #define SEND_TEXT
    //#undef SEND_TEXT
    void test()
    {
        static char text[] = "someTextToTransport";
    
        /// prepare msg /////////////////////////////////////////////////////////////
        TestMessage messageToEncode = TestMessage_init_zero;
    #ifdef SEND_TEXT
        messageToEncode.which_type = TestMessage_text_tag;
        messageToEncode.text.arg = reinterpret_cast<void *>(text);
        messageToEncode.text.funcs.encode = [](pb_ostream_t *stream, const pb_field_iter_t *field,
                                               void *const *arg) {
            const char *text = reinterpret_cast<const char *>(*arg);
            if (!pb_encode_tag_for_field(stream, field)) {
                return false;
            }
            return pb_encode_string(stream, reinterpret_cast<const pb_byte_t *>(text),
                                    std::strlen(text));
        };
    #else
        static constexpr auto someNumber = 1337;
        messageToEncode.which_type = TestMessage_number_tag;
        messageToEncode.number = someNumber;
    #endif
    
        /// encode /////////////////////////////////////////////////////////////
        std::array<char, 1024> serializedMessage;
        auto outStream = pb_ostream_from_buffer(reinterpret_cast<pb_byte_t *>(serializedMessage.data()),
                                                serializedMessage.size());
        ASSERT_TRUE(pb_encode(&outStream, TestMessage_fields, &messageToEncode));
        const auto messageLength = outStream.bytes_written;
    
        /// decode /////////////////////////////////////////////////////////////
        std::array<char, 30> receivedText;
        auto inStream = pb_istream_from_buffer(
            reinterpret_cast<const pb_byte_t *>(serializedMessage.data()), messageLength);
    
        TestMessage messageToDecode = TestMessage_init_zero;
        messageToDecode.text.arg = reinterpret_cast<void *>(receivedText.data());
        messageToDecode.text.funcs.decode = [](pb_istream_t *stream, const pb_field_iter_t *field,
                                               void **arg) -> bool {
            char *receivedText = reinterpret_cast<char *>(*arg);
            if (stream->bytes_left > 30) {
                return false;
            }
    
            if (!pb_read(stream,
                         const_cast<pb_byte_t *>(reinterpret_cast<const pb_byte_t *>(receivedText)),
                         stream->bytes_left)) {
                return false;
            }
            return true;
        };
    
        /// decode /////////////////////////////////////////////////////////////
        ASSERT_TRUE(pb_decode(&inStream, TestMessage_fields, &messageToDecode));
    #ifdef SEND_TEXT
        ASSERT_EQ(messageToDecode.which_type, TestMessage_text_tag); // ! should be 1 but is 0
    #else
        ASSERT_EQ(messageToDecode.which_type, TestMessage_number_tag);
        ASSERT_EQ(messageToDecode.number, someNumber);
    #endif
    }
    

    However the messageToDecode.which_type is only correct when i am serializing an integer. Its always 0 when i use string. Am i doing something wrong? Thank you in advance

  • pb_release definition if PB_ENABLE_MALLOC disables causes issues with cmock

    pb_release definition if PB_ENABLE_MALLOC disables causes issues with cmock

    Hi, we are using cmock as test framework in our project and since I updated nanopb from 0.4.1 to 0.4.6 we encountered an issue with #define pb_release(fields, dest_struct) PB_UNUSED(fields); PB_UNUSED(dest_struct); in pb_decode.h:118. A known cmock issue https://github.com/ThrowTheSwitch/CMock/issues/176 results in an uncompilable test framework.

    The question is where is this pb_release defined for? I just removed it from our project, but since pb_release is also guarded in pb_decode.c, this line feels superfluous.

    #ifdef PB_ENABLE_MALLOC
    /* Release any allocated pointer fields. If you use dynamic allocation, you should
     * call this for any successfully decoded message when you are done with it. If
     * pb_decode() returns with an error, the message is already released.
     */
    void pb_release(const pb_msgdesc_t *fields, void *dest_struct);
    #else
    /* Allocation is not supported, so release is no-op */
    #define pb_release(fields, dest_struct) PB_UNUSED(fields); PB_UNUSED(dest_struct);
    #endif
    

    Best regards, Eike

  • Cannot install nanopb 0.4.6.4 on PlatformIO

    Cannot install nanopb 0.4.6.4 on PlatformIO

    The latest VSCode+PIO environment does not seem to like the version number of the latest versions: nanopb/[email protected]^0.4.6.4 Error message: ValueError: Invalid simple block '^0.4.6.4'

    I tried to upgrade from nanopb/[email protected]^0.4.6 but no newer version seems to work with PIO.

    Stacktrace:

    Library Manager: Removing Nanopb @ 0.4.6+4
    Library Manager: [email protected]+4 has been removed!
    Error: Traceback (most recent call last):
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\__main__.py", line 102, in main
        cli()  # pylint: disable=no-value-for-parameter
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\click\core.py", line 1130, in __call__
        return self.main(*args, **kwargs)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\click\core.py", line 1055, in main
        rv = self.invoke(ctx)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\cli.py", line 71, in invoke
        return super().invoke(ctx)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\click\core.py", line 1657, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\click\core.py", line 1404, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\click\core.py", line 760, in invoke
        return __callback(*args, **kwargs)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\click\decorators.py", line 26, in new_func
        return f(get_current_context(), *args, **kwargs)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\run\cli.py", line 144, in cli
        process_env(
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\run\cli.py", line 201, in process_env
        result = {"env": name, "duration": time(), "succeeded": ep.process()}
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\run\processor.py", line 83, in process
        install_project_env_dependencies(
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\package\commands\install.py", line 132, in install_project_env_dependencies
        _install_project_env_libraries(project_env, options),
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\package\commands\install.py", line 241, in _install_project_env_libraries
        spec = PackageSpec(library)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\package\meta.py", line 184, in __init__
        self._parse(self.raw)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\package\meta.py", line 291, in _parse
        raw = parser(raw)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\package\meta.py", line 316, in _parse_requirements
        self.requirements = tokens[1].strip()
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\platformio\package\meta.py", line 231, in requirements
        else semantic_version.SimpleSpec(str(value))
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\semantic_version\base.py", line 647, in __init__
        self.clause = self._parse_to_clause(expression)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\semantic_version\base.py", line 1043, in _parse_to_clause
        return cls.Parser.parse(expression)
      File "C:\Users\Norbi\.platformio\penv\lib\site-packages\semantic_version\base.py", line 1063, in parse
        raise ValueError("Invalid simple block %r" % block)
    ValueError: Invalid simple block '^0.4.6.4'
    
  • Documentation doesn't make it clear what difference nanopb makes/why we should use nanopb

    Documentation doesn't make it clear what difference nanopb makes/why we should use nanopb

    Hi,

    The homepage/readmes don't seem to make it clear what difference nanopb makes?

    From what I can tell it gives us a full protobuf library but it's considerably smaller than the normal protobuf library so we take up less ram? but how much smaller is it? and are there any trade offs for this size reduction? I would love if the documentation gave some indication what difference using nanopb will make.

    I ask because I've been asked to support a library which is using nanopb and I can't figure out why we decided to use it and if it's still useful for us.

    Thanks for all the work, Tom

  • Generate equality comparison function

    Generate equality comparison function

    Using memcmp does not work because the padding bytes will not necessarily be equal; I spent an hour until I figured this out.

    It would be really nice to simply auto-generate a function that compares two structs of same type.

    Ideally, it would tell you which field is different

    void compare_my_msg(my_msg_t * m1, my_msg_t * m2, my_msg_fields_eq_t * msg_fields_eq)

    but I would totally settle for a simple bool is_equal(my_msg_t * m1, my_msg_t * m2)

  • "mangle_names" option not parsed properly by CMake nanopb_generate_cpp() script

    Using NanoPb as in-source submodule from commit hash 529d26b (01-May-2022)

    CMakeLists.txt script

    ...
    find_package(Nanopb REQUIRED)
    ...
    set(NANOPB_OPTIONS "-s mangle_names:M_STRIP_PACKAGE")
    
    nanopb_generate_cpp(LIB_NANOPB_SRC_FILES LIB_NANOPB_HDR_FILES ${PROJECT_PROTO_SRC_FILES})
    ...
    

    results in the following exception stack from "nanopb_generator.py" during the build phase

    ...
    [build] Traceback (most recent call last):
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 2338, in <module>
    [build]     main_plugin()
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 2315, in main_plugin
    [build]     other_files[fdesc.name] = parse_file(fdesc.name, fdesc, options)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 2113, in parse_file
    [build]     f = ProtoFile(fdesc, file_options)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 1563, in __init__
    [build]     self.parse()
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 1648, in parse
    [build]     self.messages.append(Message(name, message, message_options, index, self.comment_locations))
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 1142, in __init__
    [build]     self.load_fields(desc, message_options)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 1179, in load_fields
    [build]     field = Field(self.name, f, field_options)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 583, in __init__
    [build]     self.ctype = names_from_type_name(desc.type_name)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 195, in names_from_type_name
    [build]     raise NotImplementedError("Lookup of non-absolute type names is not supported")
    [build] NotImplementedError: Lookup of non-absolute type names is not supported
    [build] Traceback (most recent call last):
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 2338, in <module>
    [build]     main_plugin()
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 2315, in main_plugin
    [build]     other_files[fdesc.name] = parse_file(fdesc.name, fdesc, options)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 2113, in parse_file
    [build]     f = ProtoFile(fdesc, file_options)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 1563, in __init__
    [build]     self.parse()
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 1648, in parse
    [build]     self.messages.append(Message(name, message, message_options, index, self.comment_locations))
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 1142, in __init__
    [build]     self.load_fields(desc, message_options)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 1179, in load_fields
    [build]     field = Field(self.name, f, field_options)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 583, in __init__
    [build]     self.ctype = names_from_type_name(desc.type_name)
    [build]   File "/home/richard/Professional/RPL/develop/firmware/ugv/PMB/UGV-power-management-board/build/proto-ipc/nanopb/generator/nanopb_generator.py", line 195, in names_from_type_name
    [build]     raise NotImplementedError("Lookup of non-absolute type names is not supported")
    [build] NotImplementedError: Lookup of non-absolute type names is not supported
    [build] --nanopb_out: protoc-gen-nanopb: Plugin failed with status code 1.
    ...
    

    for comparison, a set(NANOPB_OPTIONS "-s long_names:false") option is applied as expected

Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Aug 13, 2022
Protocol Buffers implementation in C

Overview This is protobuf-c, a C implementation of the Google Protocol Buffers data serialization format. It includes libprotobuf-c, a pure C library

Aug 13, 2022
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Aug 8, 2022
a small protobuf implementation in C

μpb - a small protobuf implementation in C Platform Build Status macOS ubuntu μpb (often written 'upb') is a small protobuf implementation written in

Aug 9, 2022
International obfuscated contest: Small C program to minify HTML sources and generate a minified HTML output.
International obfuscated contest: Small C program to minify HTML sources and generate a minified HTML output.

HTML Minifier C International obfuscated contest: Just a small C program to minify HTML sources and generate a minified HTML output. Using $ gcc html-

Jul 21, 2022
libcluon is a small and efficient, single-file and header-only library written in modern C++ to power microservices.

libcluon Linux & OSX Build (TravisCI) Win64 Build (AppVeyor) Test Coverage Coverity Analysis CII Best Practices libcluon is a small single-file, heade

Jul 14, 2022
Macesuted's Code

Macesuted's Code 这里存放了自 2021-5-1 以来我在学习 OI 过程中于各大 OJ 所做题目的 AC 代码。 仅供个人学习参考使用,请不要直接复制这些代码以 AC 对应题目,转载请注明出处。 我的 个人博客 中包含部分题目的题解。 我在这些 OJ 上的帐号: AtCoder:

Jun 9, 2022
A fast, byte-code interpreted language

Minima Minima is a small, portable, and fast programming language written in C. The Syntax Minima's syntax is optimized for a fast byte-code translati

May 10, 2022
Morse code decoding library
Morse code decoding library

ggmorse Morse code decoding library ggmorse2.mp4 ggmorse0.mp4 ggmorse1.mp4 Try it out You can easily test the library using the free GGMorse applicati

Jul 17, 2022
Google Protocol Buffers tools (C code generator).

About Google Protocol Buffers tools in Python 3.6+. C source code generator. Rust source code generator ( ?? ?? ?? under construction ?? ?? ?? ). prot

May 16, 2022
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Aug 13, 2022
Protocol Buffers implementation in C

Overview This is protobuf-c, a C implementation of the Google Protocol Buffers data serialization format. It includes libprotobuf-c, a pure C library

Aug 13, 2022
A protocol buffers library for C

PBC PBC is a google protocol buffers library for C without code generation. Quick Example package tutorial; message Person { required string name =

Aug 6, 2022
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Aug 8, 2022
Simple and fast C library implementing a thread-safe API to manage hash-tables, linked lists, lock-free ring buffers and queues

libhl C library implementing a set of APIs to efficiently manage some basic data structures such as : hashtables, linked lists, queues, trees, ringbuf

Jul 30, 2022
CUDA Custom Buffers and example blocks
CUDA Custom Buffers and example blocks

gr-cuda CUDA Support for GNU Radio using the custom buffer changes introduced in GR 3.10. Custom buffers for CUDA-enabled hardware are provided that c

Dec 9, 2021
CodeCompactor is an open source program designed for reducing the size of your code!

CodeCompacter An exciting, new and open source program for reducing the length of your code! Usage: ./CodeCompacter {ARGUMENTS} Arguments: -L {languag

Nov 28, 2021
A library of type safe sets over fixed size collections of types or values, including methods for accessing, modifying, visiting and iterating over those.

cpp_enum_set A library of type safe sets over fixed size collections of types or values, including methods for accessing, modifying, visiting and iter

Jun 16, 2022