Cap'n Proto serialization/RPC system - core tools and C++ library

Hello,

this patch adds support for observing availability of urgent/out-of-band data by adding a new promise to FdObserver. The patch itself is simple and without side effects: Unless OBSERVE_URGENT is passed when constructing the FdObserver object, no new code is run. The implementation mirrors what was already there for the POLLIN event.

Right, so... why do we need this? A little backstory: I gave Cap'n Proto a shot as RPC server for my home automation system, which controls various devices through a Raspberry Pi B+'s GPIO ports and a i2c GPIO multiplexer. Cap'n Proto did the job flawlessly! However, in order to receive interrupts from the i2c GPIO multiplexer, I have to test for the POLLPRI flag on the interrupt port's fd. Linux's GPIO abstraction layer sets POLLPRI when an edge change is detected on a GPIO port. Extending KJ's UnixEventPort implementation appeared to be the most simple way to achieve this goal and at the same time integrate with the RPC server's event loop. POLLPRI is also used by TCP and many drivers for specialized hardware (such as Bluetooth receivers.)

I would appreciate if this change could be merged so that I do not have to maintain a separate fork.

I have included a test which verifies that the new event works correctly with Linux's loopback TCP/IP out-of-band message implementation.

Cheers, Oliver

clang 8 build failed with error:

[ 15%] Building CXX object modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/encoding.c++:242:19: error: no matching conversion for functional-style cast from 'const char8_t [4]' to 'kj::StringPtr'
    result.addAll(StringPtr(u8"\ufffd"));
                  ^~~~~~~~~~~~~~~~~~~~
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'const char8_t [4]' to 'const kj::StringPtr' for 1st argument
class StringPtr {
      ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:70:7: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'const char8_t [4]' to 'kj::StringPtr' for 1st argument
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:73:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'decltype(nullptr)' (aka 'nullptr_t') for 1st argument
  inline StringPtr(decltype(nullptr)): content("", 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:74:10: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const char *' for 1st argument
  inline StringPtr(const char* value): content(value, strlen(value) + 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:141:20: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'ArrayPtr<const char>' for 1st argument
  inline constexpr StringPtr(ArrayPtr<const char> content): content(content) {}
                   ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:499:19: note: candidate constructor not viable: no known conversion from 'const char8_t [4]' to 'const kj::String' for 1st argument
inline StringPtr::StringPtr(const String& value): content(value.cStr(), value.size() + 1) {}
                  ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:83:10: note: candidate template ignored: substitution failure [with T = char8_t [4]]: no matching function for call to 'instance'
  inline StringPtr(const T& t): StringPtr(t.c_str()) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:72:10: note: candidate constructor not viable: requires 0 arguments, but 1 was provided
  inline StringPtr(): content("", 1) {}
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:75:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
  inline StringPtr(const char* value, size_t size): content(value, size + 1) {
         ^
/home/travis/build/chronoxor/CppSerialization/modules/capnproto/c++/src/kj/string.h:78:10: note: candidate constructor not viable: requires 2 arguments, but 1 was provided
  inline StringPtr(const char* begin, const char* end): StringPtr(begin, end - begin) {}
         ^
1 error generated.
modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/build.make:257: recipe for target 'modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o' failed
make[2]: *** [modules/capnproto/c++/src/kj/CMakeFiles/kj.dir/encoding.c++.o] Error 1

This is mostly copied from node-capnproto and async-unix. This commit enables integration with Cocoa's event loop (CFRunLoop).

Notes:

• I haven't committed any tests yet (however I have tried the AsyncIo::SimpleNetwork test with it and it works both when using wait, and running CFRunLoop normally)
• I haven't modified the makefiles (requires -framework CoreFoundation to be added to LDFLAGS for libkj-async)
• I've had to add a temporary AsyncCocoaIoContext struct as AsyncIoContext currently has a unixEventPort field

Is this something you would merge into kj? It would be useful for anyone that wants to integrate kj/capnproto into an OSX/iOS application, and for an Objective-C wrapper of capnproto.

Implements a $Cxx.omitSchemas annotation that strips schema information from the binary as discussed in https://github.com/capnproto/capnproto/issues/1498

👋 just found the latest release 0.9.0 does not compile on Mojave (macOS 10.14), error log is below:

In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/mutex.h:28:
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE'
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
                                                               ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                         ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                                                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN'
      uint column = KJ_CALLER_COLUMN())
                    ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN'
#define KJ_CALLER_COLUMN() __builtin_COLUMN()
                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
            ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor
  const char* fileName;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor
  const char* function;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor
  uint lineNumber;
       ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor
  uint columnNumber;
       ^
5 errors generated.
In file included from /tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.c++:22:
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:64: error: use of undeclared identifier '__builtin_FILE'
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
                                                               ^
make[2]: *** [src/kj/CMakeFiles/kj.dir/mutex.c++.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:26: error: use of undeclared identifier '__builtin_FUNCTION'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                         ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:60:60: error: use of undeclared identifier '__builtin_LINE'
      const char* func = __builtin_FUNCTION(), uint line = __builtin_LINE(),
                                                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:61:21: error: use of undeclared identifier '__builtin_COLUMN'
      uint column = KJ_CALLER_COLUMN())
                    ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:27:28: note: expanded from macro 'KJ_CALLER_COLUMN'
#define KJ_CALLER_COLUMN() __builtin_COLUMN()
                           ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:59:13: error: constexpr constructor must initialize all members
  constexpr SourceLocation(Badge = Badge{}, const char* file = __builtin_FILE(),
            ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:77:15: note: member not initialized by constructor
  const char* fileName;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:78:15: note: member not initialized by constructor
  const char* function;
              ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:79:8: note: member not initialized by constructor
  uint lineNumber;
       ^
/tmp/capnp-20210815-80151-159a7vv/capnproto-c++-0.9.0/src/kj/source-location.h:80:8: note: member not initialized by constructor
  uint columnNumber;
       ^
5 errors generated.
make[2]: *** [src/kj/CMakeFiles/kj.dir/source-location.c++.o] Error 1

The cmake command

cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/capnp/0.9.0 -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_VERBOSE_MAKEFILE=ON -Wno-dev -DBUILD_TESTING=OFF -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk

relates to https://github.com/Homebrew/homebrew-core/pull/83289

This saves some bookkeeping when the caller doesn't plan to wait for the RPC response. We can even avoid sending a Return message.

Unfortunately it does not seem possible to do this automatically, for a few reasons:

1. To detect that the caller doesn't care about the response, we'd have to detect that they drop the promise shortly after calling send(). However, send() is required to enqueue the message to the stream immediately, to maintain e-order. In order to set the hint bit based on the promise being dropped, we'd need some mechanism to reserve a place in the queue and block the stream from continuing until we've decided whether the promise is going to be dropped. This bookkeeping would probably be costly.
2. Pipelined capabilities are promise capabilities, i.e. their whenMoreResolved() is expected to return a promise that resolves when the RPC returns. If we decide to elide the Return message, we cannot implement this correctly. With sendForPipeline(), we can say that the app is opting out of whenMoreResolved().

Following https://capnproto.org/install.html build capnproto library with -O2 compiler option. Is there any recommended way of installing capnpproto library by building it with -O3 option instead of -O2? Also, is there any reason for using -O2 istead of -O3 option in the installer?

This is still a WIP:

• [ ] Needs https://github.com/capnproto/capnproto/pull/1579
• [ ] Needs some cleanup, specifically TestConnectResponseImpl is duplicated
  • [x] also need to figure out what to do about the DelayedEof usage in ConnectResponseImpl. In workerd (https://github.com/cloudflare/workerd/pull/162) the suggestion was to define a new class, but is that really the best? (there is a lot of duplication and it seems like we could pass a bool through to the class to customise whether DelayedEof is used)

Test Plan

In edgeworker repo for ASAN:

$ bazel test @capnp-cpp//src/capnp/compat:http-over-capnp-test --config=asan

When bootstrapping a new Cap'n Proto implementation in another language, it can be useful to make the first code-generator in the same language as the language you're planning to write the code generator in (of course). But until the schema.capnp file has been bootstrapped at least once for the library being developed, this is difficult.

You may bootstrap a simple compiler in another language like C++, but then you won't be able to reuse any of the code and be forced to rewrite from scratch in the target language once up and running.

It'd be useful if another included output format existed like capnpc-capnp, to dump the input code generator request in another format that's easier to bootstrap in the target language. JSON is decently popular enough that it's well supported basically everywhere.

This PR adds capnpc-json, an included output plugin that simply dumps the entire code generator request in JSON format to stdout.

Implements a variation on connect() that allows for optimistic piplining of outbound writes while the CONNECT tunnel is still being established. Useful for reducing latency. Best implemented as a separate method since not all uses will want to, or be capable of, making use of pipelining.

Work in progress: Needs tests