Cap'n Proto serialization/RPC system - core tools and C++ library

Hello,

this patch adds support for observing availability of urgent/out-of-band data by adding a new promise to FdObserver. The patch itself is simple and without side effects: Unless OBSERVE_URGENT is passed when constructing the FdObserver object, no new code is run. The implementation mirrors what was already there for the POLLIN event.

Right, so... why do we need this? A little backstory: I gave Cap'n Proto a shot as RPC server for my home automation system, which controls various devices through a Raspberry Pi B+'s GPIO ports and a i2c GPIO multiplexer. Cap'n Proto did the job flawlessly! However, in order to receive interrupts from the i2c GPIO multiplexer, I have to test for the POLLPRI flag on the interrupt port's fd. Linux's GPIO abstraction layer sets POLLPRI when an edge change is detected on a GPIO port. Extending KJ's UnixEventPort implementation appeared to be the most simple way to achieve this goal and at the same time integrate with the RPC server's event loop. POLLPRI is also used by TCP and many drivers for specialized hardware (such as Bluetooth receivers.)

I would appreciate if this change could be merged so that I do not have to maintain a separate fork.

I have included a test which verifies that the new event works correctly with Linux's loopback TCP/IP out-of-band message implementation.

Cheers, Oliver

clang 8 build failed with error:

[ 15%] Building CXX object modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/encoding.c++:242:19: error: no matching conversion for functional-style cast from 'const char8_t [4]' to 'kj::StringPtr'
    result.addAll(StringPtr(u8"\ufffd"));
                  ^~~~~~~~~~~~~~~~~~~~
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const char8_t [4]' to 'const kj::StringPtr' for 1st argument
class StringPtr {
      ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const char8_t [4]' to 'kj::StringPtr' for 1st argument
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:73:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'decltype(nullptr)' (aka 'nullptr_t') for 1st argument
  inline StringPtr(decltype(nullptr)): content("", 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:74:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const char *' for 1st argument
  inline StringPtr(const char* value): content(value, strlen(value) + 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:141:20: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'ArrayPtr<const char>' for 1st argument
  inline constexpr StringPtr(ArrayPtr<const char> content): content(content) {}
                   ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:499:19: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const kj::String' for 1st argument
inline StringPtr::StringPtr(const String& value): content(value.cStr(), value.size() + 1) {}
                  ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:83:10: note: candidate template ignored: substitution failure [with T = char8_t [4]]: no matching function for call to 'instance'
  inline StringPtr(const T& t): StringPtr(t.c_str()) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:72:10: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
  inline StringPtr(): content("", 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:75:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
  inline StringPtr(const char* value, size_t size): content(value, size + 1) {
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:78:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
  inline StringPtr(const char* begin, const char* end): StringPtr(begin, end - begin) {}
         ^
1 error generated.
modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/build.make:257: recipe for target 'modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o' failed
make[2]: *** [modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o] Error 1

This is mostly copied from node-capnproto and async-unix. This commit enables integration with Cocoa's event loop (CFRunLoop).

Notes:

• I haven't committed any tests yet (however I have tried the AsyncIo::SimpleNetwork test with it and it works both when using wait, and running CFRunLoop normally)
• I haven't modified the makefiles (requires -framework CoreFoundation to be added to LDFLAGS for libkj-async)
• I've had to add a temporary AsyncCocoaIoContext struct as AsyncIoContext currently has a unixEventPort field

Is this something you would merge into kj? It would be useful for anyone that wants to integrate kj/capnproto into an OSX/iOS application, and for an Objective-C wrapper of capnproto.

Implements a $Cxx.omitSchemas annotation that strips schema information from the binary as discussed in https://github.com/capnproto/capnproto/issues/1498

👋 just found the latest release 0.9.0 does not compile on Mojave (macOS 10.14), error log is below:

In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/mutex.h:28:
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE'
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
                                                               ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                         ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                                                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN'
      uint column = KJ_CALLER_COLUMN())
                    ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN'
#define KJ_CALLER_COLUMN() __builtin_COLUMN()
                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
            ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor
  const char* fileName;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor
  const char* function;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor
  uint lineNumber;
       ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor
  uint columnNumber;
       ^
5 errors generated.
In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.c++:22:
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE'
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
                                                               ^
make[2]: *** [src/kj/CMakeFiles/kj.dir/mutex.c++.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                         ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                                                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN'
      uint column = KJ_CALLER_COLUMN())
                    ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN'
#define KJ_CALLER_COLUMN() __builtin_COLUMN()
                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
            ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor
  const char* fileName;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor
  const char* function;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor
  uint lineNumber;
       ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor
  uint columnNumber;
       ^
5 errors generated.
make[2]: *** [src/kj/CMakeFiles/kj.dir/source-location.c++.o] Error 1

The cmake command

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/capnp/0.9.0 -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_VERBOSE_MAKEFILE=ON -Wno-dev -DBUILD_TESTING=OFF -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk

relates to https://github.com/Homebrew/homebrew-core/pull/83289

This does the following:

• Adds a new general function ConnectionReceiver::run() to facilitate an async accept loop.
• Adds a queue and a TaskSet to the TlsConnectionReceiver to accept on its inner ConnectionReceiver without blocking on SSL_accept().
• Adds several new tests for the TlsConnectionReceiver.
• Converts several other TLS tests to use convenience helpers.

I wanted to get this up earlier than later to get feedback and also to figure out what to do about skip() not having an equivalent for async streams -- the omission seems deliberate so I didn't want to just add it in.

I'd still like to add some tests even though this is a straightforward copy of the sync stream implementation, since that one doesn't have tests aside from being used to implement TestPipe in serialize-packed-test.

I couldn't come up with a way to implement a buffered form of AsyncCapabilityStream, since there's no knowledge of message boundaries at the byte stream level, so handling the case where you complete a partial read of a control message range and then read the control messages of the following range isn't possible as there's no way to know the boundaries.

It is impossible to build capnproto in CAPNP_LITE mode and use library normally (through MallocMessageBuilder), even code which inits root and sets a field requires kj::systemPreciseMonotonicClock, which is not built if kj detects CAPNP_LITE.

Compiler&Linker: MSVC version 2022

error goes something along the lines:

undefined external symbol "class kj::MonotonicClock const & __cdecl kj::systemPreciseMonotonicClock(void)" ([email protected]@@[email protected]@XZ) in "public: void __cdecl kj::_::Mutex::wait(class kj::_::Mutex::Predicate &,class kj::Maybe<class kj::Quantity<__int64,class kj::_::NanosecondLabel> >,class kj::NoopSourceLocation)" ([email protected]@[email protected]@@[email protected]@[email protected][email protected][email protected][email protected]@@@[email protected]@@[email protected]@[email protected]@Z).

Today I was stumbling about cases where I could not print branded structs if the brand parameter was bound to a List({enum type}) via KJ_DBG. This happened due to BrandArgumentList receiving the enum binding with schema==nullptr and passing the binding to the constructor Type::Type(schema::Type::Which) (which would promptly reject the enum argument due to the missing type information).

After digging a bit into the schema generation facilities, I noticed something odd in generated_header_support.h:

When handling enums in BrandBindingFor_, the template specialization returns { 15, listDepth, nullptr }. However, I would expect the correct binding here to be {15, listDepth, schemas::EnumInfo<T>::schema->defaultBrand} (analogous to rawBrandedSchema<EnumType>()).

I suspect that this makes enum lists generally unreliable to use in the dynamic API when used through the C++ types. I also assume this likely never caused an issue for the following reasons:

• When using the dynamic API from a schema, SchemaLoader uses a different pathway (SchemaLoader::Impl::makeDep) to load the brand information
• C++ lists do not need the brand information, they directly derive all they need from the C++ types.

Is my suspicion correct, or am I chasing down the wrong avenue?

I'm not sure if we want this to be a default, or configurable behavior -- If your event loop was backed up enough for this to cause problems, I think that you'd be hurting from queueing writes regardless. And even if so, you could run into stack overflow if you queue enough messages, which this PR should also stop from happening.

First commit is same as (https://github.com/capnproto/capnproto/pull/1509) but I can't set the merge target to a branch in my own repo for a PR in this one.

This PR implements sending and receiving compressed messages. All permessage-deflate parameters are supported.

I've written an ew-test-bin for testing and it seems like things are working. I compared my compressed results with a few examples in the compression RFC and things matched up so that's a good sign. That said, I want to add to our existing KJ tests, though I suspect it might require modifying the fake server a bit.

There's still a couple KJ_DBGs sprinkled around for debugging reasons. The Context class is based on a similar class used by our CompressionStream implementation.

I want to create a Reader and pass it back to the caller so they can just get the values in the structure. Is this the right way to go about things?

get_reader(::MyNamespace::MyStruct::Reader& wanted_reader) {
    // I am trying to populate a reader with static values like for a unit test
    ::capnp::MallocMessageBuilder msg_builder;
    auto msg = msg_builder.initRoot<::MyNamespace::MyStruct::>();
    msg.setSomeField(888);

    // Serialize 
    size_t serialized_msg_size =
        ::capnp::computeSerializedSizeInWords(msg_builder) *
        sizeof(::capnp::word);
    ::capnp::byte out_array[serialized_msg_size] = {0};
    ::kj::ArrayPtr<::capnp::byte> out_array_ptr(out_array, serialized_msg_size);
    ::kj::ArrayOutputStream out_stream(out_array_ptr);
    ::capnp::writePackedMessage(out_stream, msg_builder);

    // Deserialize
    ::kj::ArrayPtr<capnp::byte> msg_byte_ptr(
        out_array, out_array + out_stream.getArray().size());
    ::kj::ArrayInputStream msg_stream(msg_byte_ptr);
    ::capnp::PackedMessageReader packed_msg(msg_stream);
    
    // !! Is this allowed? Does this do a copy?
    wanted_reader =
        packed_msg.getRoot<::MyNamespace::MyStruct>();
}


int main() {
    ::MyNamespace::MyStruct::Reader wanted_reader;

    // Get the reader
    get_reader(wanted_reader);

    // Use the reader now, did I mess up? Could someField be undefined?
    ::std::cout << wanted_reader.getSomeField() << ::std::endl;
}