Intricacies of C++ Callbacks

I. Overview

Recently I have been working on a C++ library which wraps a non HFT third party financial asynchronous API from a broker. Most of the API's adhere to the following pattern:

Each id must be unique and there can be many outstanding requests for the same data with different configuration parameters and id's which will complete in a random order due to the asynchronous nature. The id is easily guaranteed unique with a global atomic int.

This lends itself well to encapsulation into a class to handle the verification of the configuration parameters, the storage of the data, and the various synchronizations. An example of this type of pattern is as follows:

This article is not about the configuration of this class but about the issues that can arise from C++ callbacks many of which are the same as pub/sub type issues.

II. First Iteration

As a first attempt let's look at how you might call a bunch of callbacks associated with an event. From now on we will ignore the id type for simplification. As a first attempt something like this may be produced:

While this looks perfectly safe there is one large issue which is that the locks on the callbacks is held during all the time consuming invocations which could represent the majority of the lock hold time and block threads trying to add or remove callbacks. Another issue here is that the mutex prevents the invocation of the callbacks for a different set of configuration parameters at the same time limiting the asynchronous nature.

III. Improve Callbacks Lock Held

In an attempt to improve on the lock hold duration and allow multiple simultaneous invocations the following might be attempted:

So we make a copy of the callbacks under the lock before invoking them which should be much faster than the actual invocations and then invoke them out of the lock. We also use a shared mutex which allows multiple invocations with different parameters simultaneously. While this greatly improves the lock hold time, it introduces a subtle race condition where the callback may no longer be valid when invoked due to it having been removed.

IV. Improve On Race Condition

We can improve on the callback removal race condition by changing the std::function to std::shared_ptr<std::function> and using std::weak_ptr. An example is as follows:

If the callback is removed before being called then the weak pointer cannot be converted to a shared pointer and thus not be called. However, there is an even more subtle race condition here where the functor itself or the object containing the functor is deleted during the callback which will result in undefined behavior most likely a crash.

V. Callback Wrapper

We can overcome the problem with the shared pointer mentioned above by creating a callback wrapper and storing a shared pointer to the callback wrapper instead of to the functor directly.

The interesting part of this is the atomic bool which is acquired before invoking the callback and blocks the destroy if in a callback so that it cannot be unregistered, i.e. removed from the callback list.

With this new helper class and already discussed optimizations, the invocation of callbacks now becomes as follows:

And the corresponding register and unregister become the as follows:

These functions can be easily modified to pass known fixed parameters which in the case described above would be for variants with an int reqId.

VI. Summary

Hope you have enjoyed this overview of C++ callbacks and some of the gotchas along the way. Any comments or feedback is appreciated.