I tried compiling the core library on VC 2015 RC. It seems that the only remaining build issue is the following struct containing a variadic template.

I can't parse what this is doing, but I'm wondering if there is a simple way to replace it with something similar - even if that's just a few expanded out templates....

template <class What, typename With, class... IfStmt> struct replace_type { static constexpr bool do_replace = disjunctionIfStmt::value...::value; using type = typename replace_type_impl<do_replace, What, With>::type; };

That's something I'm thinking of implementing for a while now and I would like to get some feedback on the idea.

The only benefit the "old" sb_actor did provide was saving 1-2 lines of code: behavior init_behavior = ... instead of behavior make_behavior() override { ... }. It would make more sense to focus on viability of non-class based actors instead, which I personally almost always prefer anyways. The new stateful_actor could have an interface like this:

using inc_and_fetch_atom = atom_constant<atom("IncFetch")>;

struct counter_state {
  int counter = 0;
};

behavior counter(stateful_actor<counter_state>* self) {
  [](inc_and_fetch_atom) {
    return *self->counter += 1;
  }
}

int main() {
  auto cntr = spawn(counter);
}

Note the *self-> syntax. This is how a stateful actor could access its state. The more verbose version would be self->state.counter += 1. Since accessing the state is expected to be a very common operation, some syntactic sugar might be in order (even though it's a bit hacky).

A huge benefit of this design is the following:

struct slave_state {
  actor master;
};

struct master_state {
  actor slave;
};

behavior master(stateful_actor<master_state>* self) {
  *self->slave = spawn<monitored>(slave, self);
  // ... more stuff ...
}

When modeling the same with event_based_actors, killing the master is a memory leak. The slave keeps a reference to the master and vice versa. Actors have to actively break the cycle by setting references to invalid_actor in on_exit. Decoupling the state in this way allows the runtime to safely run the destructor for the state once the actor has finished execution!

Opinions?

1. Fix issue that promise doesn't get invalidated after delivering in non-chained cases
2. Fix issue that typed promise delivers error response message to non-request senders
3. Add move semantics to typed promise
4. Add typed_response_promise::valid()
5. Add local_actor::make_ready_response_promise()
6. Improve unit test

IMHO, pending() would be a better name than valid(). I also thought about the name make_fulfilled_response_promise(). But then, I realised that the standard library has been using the term ready (see here), and we should follow existing practice. There is std::future::valid(), but no similar things for std::promise. The name valid() seems a little bit odd and counter-intuitive on response promise. A fulfilled promise should be a valid promise. It's just closed an no longer pending. Users are even tempted to expect a valid promise to be a fulfilled promise, given the semantics of std::future::valid().

Relates to #353

The broker is currently limited to TCP-like backends. A new datagram backend for connectionless networking would significantly broaden its usefulness.

The alias config_value_map has a very misleading name. One would expect std::map<config_value> or something similar. However, it's defined as dictionary<dictionary<config_value>>. CAF's dictionary<T> is similar to std::map<string, T> but with a string_view-based interface for more efficient access.

The reason config_value_map is defined in this way is that this type is supposed to store the content of an .ini file. In other words, this type associates categories to a map of settings.

Here are my current ideas for new aliases:

using settings_map = dictionary<config_value>;
using configuration_map = dictionary<settings_map>;

This allows for intuitively named variables:

void fill_settings(settings_map& settings, ...);
void parse_config(configuration_map& configuration, ...);

I'm not sure if configuration_map signals "this is a map over settings" clearly enough. Thoughts?

It turns out that the CAF unit testing framework does not support compound Boolean expressions like x && y and x || y. The tricky part is that no diagnostic was ever issued for such use. So, you are doing it wrong, and you don't know it. I've seen such use in several unit test suits. So I fixed them and re-ran the tests. Guess what? The once-OK io_dynamic_remote_actor test suite failed. The mis-use did cover up problems! To prevent future mis-use, I modified the testing framework a bit, so that code with compound Boolean expressions is rejected.

This pull request adds the ability to use the thread to core affinity in CAF. Pinning threads to specific cores can, in general, improve performances and can also permit the isolation of different threads in order to avoid performance degradation (e.g., worker threads and blocking actors)

The patch adds the Affinity Manager which is called by the Actor System every time a new thread is created and manages the affinity setting for each of them. There is the possibility to set the affinity for the different types of threads with the following configurations:

• affinity.worker-cores for the Run-time threads
• affinity.blocking-cores for the blocking actors
• affinity.detached-cores for the detached actors
• affinity.other-cores for other threads

Each configuration supports a string that specifies a set of groups of cores in which each thread must be pinned. For instance, the string "<0,3><4-7>" specifies two groups of cores, the first is composed by the cores 0, 3 and the second by the cores 4, 5, 6, 7. Thus the first thread spawned will be restricted to the threads 0 and 3 and the second thread to the cores 4, 5, 6, and 7. Then the third thread will be pinned on the first group again and so on.

The patch supports both Linux and Windows, macOS, instead, does not support thread to core affinity. Right now I only test it with g++ on Linux and with the Microsoft C++ compiler on Windows 10.

I am available for any clarification.

Almost all event-based actors have an others >> handler to discard unexpected messages, usually printing the message.

This is quite clumsy and repetitive. We could either provide a default handler (e.g. others >> report_unexpected) or allow actors to specify a handler for unexpected messages separately. If defined, this handler would be used to consume all uncaught messages, essentially overriding the default of leaving the message in the mailbox.

I haven't really changed my code other than updating libcppa to recent master. Now I am getting the following segfault upon program start. The program dies in the constructor of cppa::memory_managed for some to me unclear reason.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe8ff1700 (LWP 27069)]
0x00007ffff6b68511 in cppa::memory_managed::memory_managed() () at /home/matthias/src/libcppa/./cppa/memory_managed.hpp:36
36      class memory_managed {
(gdb) bt
#0  0x00007ffff6b68511 in cppa::memory_managed::memory_managed() () at /home/matthias/src/libcppa/./cppa/memory_managed.hpp:36
#1  0x00007ffff6b8bbb8 in cppa::ref_counted::ref_counted() () at /home/matthias/src/libcppa/src/ref_counted.cpp:36
#2  0x00007ffff6b01532 in cppa::channel::channel() () at /home/matthias/src/libcppa/./cppa/channel.hpp:70
#3  0x00007ffff6b01313 in cppa::actor::actor() () at /home/matthias/src/libcppa/src/actor.cpp:59
#4  0x00007ffff6b676a2 in cppa::local_actor::local_actor(bool) () at /home/matthias/src/libcppa/src/local_actor.cpp:74
#5  0x00007ffff6b8e887 in cppa::scheduled_actor::scheduled_actor(bool) () at /home/matthias/src/libcppa/src/scheduled_actor.cpp:37
#6  0x00007ffff6b35c5c in _ZN4cppa6detail19abstract_actor_baseINS_15scheduled_actorELb1EEC2IJbEEEDpOT_ ()
    at /home/matthias/src/libcppa/./cppa/detail/abstract_actor.hpp:78
#7  0x00007ffff6b34cb7 in _ZN4cppa6detail14abstract_actorINS_15scheduled_actorEEC2IJbEEEDpOT_ ()
    at /home/matthias/src/libcppa/./cppa/detail/abstract_actor.hpp:198
#8  0x00007ffff6b340a0 in cppa::detail::abstract_scheduled_actor::abstract_scheduled_actor(int) ()
    at /home/matthias/src/libcppa/./cppa/detail/abstract_scheduled_actor.hpp:120
#9  0x00007ffff6b3315e in cppa::event_based_actor::event_based_actor() () at /home/matthias/src/libcppa/src/event_based_actor.cpp:39
#10 0x00007ffff79cc234 in cppa::sb_actor<vast::event_source>::sb_actor() () at /home/matthias/opt/gcc/include/cppa/sb_actor.hpp:45
#11 0x00007ffff79ba6fc in vast::event_source::event_source(cppa::intrusive_ptr<cppa::actor>, cppa::intrusive_ptr<cppa::actor>) ()
    at /home/matthias/vast/src/vast/event_source.cc:108

Summary

This set of changes addresses performance issues when distributing CAF applications.

The BASP broker is currently bottlenecked by deserializing incoming messages. For evaluating the performance, I've used the simple_streaming benchmark, which runs a trivial streaming source that produces short strings endlessly and a sink that drops everything it receives.

Baseline

The baseline deploys both actors in the same process:

$ simple_streaming -m both
975586 messages/s
924538 messages/s
1088718 messages/s
976075 messages/s
994920 messages/s
1095617 messages/s
789944 messages/s
782430 messages/s
825262 messages/s
848951 messages/s

Distributed

Deploying each actor in its own process (connected via localhost) drops the performance to roughly half on current master (with the source already running in the background):

$ simple_streaming -m sink -p 4242
538069 messages/s
557683 messages/s
566474 messages/s
556036 messages/s
537674 messages/s
568108 messages/s
559711 messages/s
547152 messages/s
550078 messages/s
542460 messages/s

Looking at the bandwidth via iftop -i lo reveals that CAF only utilizes 46 mbits.

Distributed after the Patch

With this patch, performance doubles:

$ ./build/release/bin/simple_streaming --middleman.workers=2 -m sink -p 4242
1125932 messages/s
1254350 messages/s
1258735 messages/s
1285260 messages/s
1289829 messages/s
1215599 messages/s
1326506 messages/s
1274993 messages/s
1239853 messages/s
1388835 messages/s

Looking at the bandwidth via iftop -i lo again shows that CAF utilizes about 100 mbits now.

And yeah, this is now faster than the baseline. I don't know how that's possible. 🤷‍♂️

The single-process setup is a topic for another day, though.

Here's the epoll patch. Note that it avoids the linear channels scanning in the event handling section by using map(fd)->function. The patch seems to work (have seen a bit of testing in production environment).

Most unit tests for the new flow API currently sit between 75-90% code coverage for their tested components. There are also some components that fall below 75%. For a key API like this, we should strive for >= 95% (line) test coverage for everything under caf::flow.

At least these things are currently lacking:

• Response handles lack an as_single and as_observable overload when working with typed actors
• Typed actor views have no make_observable member function

If we start digging, we may find more.

Actors that read from an SPSC buffer with a non-stop writer at the other end can end up in a loop where each pull action immediately causes another pull action.

Here's a potential patch:

diff --git a/libcaf_core/caf/scheduled_actor.hpp b/libcaf_core/caf/scheduled_actor.hpp
index dcea886fd..7a735a938 100644
--- a/libcaf_core/caf/scheduled_actor.hpp
+++ b/libcaf_core/caf/scheduled_actor.hpp
@@ -807,6 +807,10 @@ private:
   /// message.
   std::vector<action> actions_;
 
+  /// Flag that tells delay() to push actions to the mailbox if we are already
+  /// in run_actions.
+  bool running_actions_ = false;
+
   /// Stores resources that block the actor from terminating.
   std::vector<disposable> watched_disposables_;
 };
diff --git a/libcaf_core/src/scheduled_actor.cpp b/libcaf_core/src/scheduled_actor.cpp
index 6bc9ecd64..96b5576df 100644
--- a/libcaf_core/src/scheduled_actor.cpp
+++ b/libcaf_core/src/scheduled_actor.cpp
@@ -606,7 +606,12 @@ void scheduled_actor::schedule(action what) {
 }
 
 void scheduled_actor::delay(action what) {
-  actions_.emplace_back(std::move(what));
+  // If we are already in run_actions, we force the action through the mailbox
+  // in order to break hot loops that would otherwise starve any other activity.
+  if (!running_actions_)
+    actions_.emplace_back(std::move(what));
+  else
+    schedule(std::move(what));
 }
 
 disposable scheduled_actor::delay_until(steady_time_point abs_time,
@@ -1242,6 +1247,8 @@ void scheduled_actor::watch(disposable obj) {
 }
 
 void scheduled_actor::run_actions() {
+  running_actions_ = true;
+  auto guard = detail::make_scope_guard([this] { running_actions_ = false; });
   if (!actions_.empty()) {
     // Note: can't use iterators here since actions may add to the vector.
     for (auto index = size_t{0}; index < actions_.size(); ++index) {

Currently, there's no way to cancel a delayed or scheduled message. However, both internally use the actor clock, which already supports that. So it should be straightforward to re-implement these function on top of actor_clock::schedule and then simply return the disposable from that call.

The CAF homepage (https://www.actor-framework.org/) links to the CAF doxygen documentation page (https://codedocs.xyz/actor-framework/actor-framework/).

On the doxygen documentation page, it appears that all inheritance graphs and collaboration graphs are broken links.

See for example:

• https://codedocs.xyz/actor-framework/actor-framework/classcaf_1_1actor.html
• https://codedocs.xyz/actor-framework/actor-framework/classcaf_1_1callback.html

When designing a responsive application with an actor system (or any other concurrency framework), it is usually important for message handlers to yield control frequently, e.g., by cutting large amounts of work into smaller batches.

One of the main challenges during development for me in this regard is to actually identify the handlers that are currently running for an actor, how long they took, and whether they sometimes take longer than expected. Now I could, in theory, add logic to the beginning and end of every actor handler for debugging purposes, which would allow for runtime extrapolation from the log, but this would be much easier to implement in CAF directly. The log approach doesn't scale well, and it doesn't work on production systems depending on the minimum log level enabled.

I think there's an easy way to make this easier to debug that isn't even a breaking change to CAF: Introducing optionally named message handlers.

// before
using my_actor_handle = caf::typed_actor<
  caf::result<R1>(T...),
  caf::result<R2>(U...)
>;

// after
using my_actor_handle = caf::typed_actor<
  caf::named_mpi<"foo", caf::result<R1>(T...)>,
  caf::named_mpi<"bar", caf::result<R2>(U...)>
>; 

This name could be used in log messages and metrics in various places:

• Actor metrics that include how much time is spent in actor handlers compared to the total, or on average for a given handler, or even show percentiles for runtimes for individual handlers.
• Exception handlers that show which handler they originated in, because they can retrieve the last executed handler's name.
• Compile failures for typed actor interface mismatches that show exactly which behavior is mismatching by name.
• Fancy debug consoles for all CAF applications similar to tokio console.

String literal NTTPs should be possible with C++17 with some template trickery, so I think this should be implementable for current CAF.

This is somewhere between a feature request and an unstructured brainstorming session I'd like to start; please move this over into a discussion if that format is better suited than an issue. I just wanted to get this idea written down as I think it'd make developing and maintaining CAF applications much easier.