Dive into SObjectizer 5.5. Seventh part: Message Limits

Dive into SObjectizer-5.5
SObjectizer Team, Jan 2016
Seventh Part: Message Limits
(at v.5.5.15)

This is the next part of the series of presentations with deep
introduction into features of SObjectizer-5.5.
This part is dedicated to message limits feature.
This feature is intended to use in protection of agents from
overloading.

Introduction

The main way of interaction between agents in SObjectizer
is asynchronous messages.
It makes very easy to use “fire-and-forget” scheme:
● an agent A finishes its own part of work, sends the result as a
message to agent B and starts new processing;
● agent B receives a message, processes it and sends a new
message to agent C;
● agent C receives a message, processes it and sends…
This is very simple scheme but...

...what if some agent in this chain processes messages
much slower than the other agents which send new
messages to it?
This problem is known as overloading of agent.

Overloading leads to growth of event queues.
Event queues eat more and more memory.
Growing memory usage slows down the application.
Lower speed leads to faster growth of event queues…
As a result of overloading the application is degraded and
should be killed in most cases.

There is no simple solution for overloading problem because
a good overload control scheme for a particular application-
specific agent will be very domain-specific and, probably,
can’t be reused in another project.
So, there is no ready-to-use just out-of-box tool or
mechanism in SObjectizer which could be seen as ‘one size
fits all’ solution for overloading problem.

But there is a mechanism which can protect agents from
overloading in simple situations.
This mechanism is known as message limits.

Message limits is an optional feature which can be turned on
for individual agent.
If message limits is turned on for an agent SObjectizer will
count instances of messages in agent’s event queue.
When number of messages of particular type exceeds limit
for that type SObjectizer performs an overlimit reaction
which is specified by user.

Types Of Overlimit Reactions

There are several overlimit reactions which can be
performed by SObjectizer:
● dropping new messages;
● killing the whole application by std::abort();
● redirecting new messages to another mbox;
● transforming new messages to messages of different
types (with possibility to redirect them to different
mboxes).

Message dropping is the simplest form of overlimit reactions.
It means that a new message of type T which exceeds limit
for messages of type T will be ignored and will not be
pushed to event queue of the subscriber.
This type of reaction can be useful if there is an intensive
flow of messages and losing of several messages is not a
problem. For example: ignored message will be repeated by
sender after some timeout.

Another simple overlimit reaction is killing the whole
application by std::abort().
This overlimit reaction can be used in scenarios where the
application must guarantee some throughput but can’t do
that.
For example: an agent should process messages quite
quickly. But there is a bug in the agent’s implementation and
sometimes it fell into infinite loop. The only solution there is
killing the app and restart it.

The next overlimit reaction is redirection of a new message
to another mbox.
This type of reaction could be used in different schemes:
● a very simple load balancing with agents A1, A2, ..., An. Agent A1
redirects messages to A2. A2 redirects to A3 and so on;
● different kind of processing by different subscribers. There could
be agent N which does a normal processing. On overload it
redirects messages to agent P which just sends back a reply
“system is busy, try later”.

The last and most complex reaction is transformation with
optional redirection.
This type of reaction could be used in implementing complex
domain-specific schemes of overload control.
For example, an overloaded agent could transform new messages
into signals about overloaded state and sends it to an overload
controller. The controller can perform some actions like changing
processing params, disconnecting some clients, switching off some
logging and so on...

Maximum Overlimit Reaction Deep

There is a possibility of redirection loops for overlimit
reactions of ‘redirect’ and ‘transform’ types.
For example: agent A1 redirects message to A2, A2
redirects to A3, …, An redirects back to A1.
Without a control of recursion of overlimit reactions infinite
loops inside message delivery procedure could arise.

Because of that SObjectizer limits the number of overlimit
reactions which are performed during the delivery of a
message instance.
This is called ‘overlimit_deep’ and the current maximum
value of overlimit_deep is 16.
This value is hardcoded in the current SObjectizer
implementation and can’t be changed in run-time.

When maximum value of overlimit_deep is reached then a
delivery of message is aborted by raising so_5::exception_t
with the appropriate error code.
Please note that if it is a delivery via multi-consumer mbox
then some subscribers will not receive a message. Even if
they do not use message limits at all.
It is because the whole delivery procedure is canceled.

How To Specify Message Limits?

This part of presentation contains some examples of
message limits definition.
But before going deep into code samples some important
rules first...

If an agent defines message limit for some message type T it
must also define message limits for all other message types
it processes.
If an agent defines message limit for type T and tries to
make a subscription for message of type M which hasn’t
defined limit there will be an error.

Message limits are defined at the moment of agent’s
construction and can’t be changed or removed later.
Definition of message limits are passed to the constructor of
so_5::agent_t or to coop_t::define_agent() method via so_5::
context_t object.

Definition of message limits with ‘drop message’ overlimit
reaction could look like:
// For ordinary agents:
class request_processor : public so_5::agent_t {
public :
request_processor( context_t ctx )
: so_5::agent_t( ctx
// No more than 20 messages with dropping of extra messages.
+ limit_then_drop< request >( 20 )
// No more than 1 message with dropping of extra messages.
+ limit_then_drop< get_status >( 1 ) )
...
};
// For ad-hoc agents:
coop.define_agent( coop.make_agent_context()
+ so_5::agent_t::limit_then_drop< request >( 20 )
+ so_5::agent_t::limit_then_drop< get_status >( 1 ) )...

Definition of message limits with ‘abort the app’ overlimit
reaction could look like:
// For ordinary agents:
class hardware_interface : public so_5::agent_t {
public :
hardware_interface( context_t ctx )
// Usually there must be no more than 10 waiting commands.
// But if something goes wrong and hardware part doesn't respond
// it is better to terminate the whole application.
+ limit_then_abort< outgoing_apdu_command >( 500 ) )
...
};
// For ad-hoc agents:
coop.define_agent( coop.make_agent_context()
+ so_5::agent_t::limit_then_abort< outgoing_apdu_command >( 500 ) );

Definition of message limits with ‘redirect’ overlimit reaction
could look like:
class normal_request_processor : public so_5::agent_t
{
public :
normal_request_processor(
context_t ctx,
// Message box of agent for handling requests in overload mode.
// In this mode requests are not processed but negative response with the appropriate
// result code is sent back (and this fact is stored in the application log).
so_5::mbox_t overload_mode_processor )
// We can hold no more that 10 requests in queue.
// All extra messages must be redirected to overload-mode processor
// for generation of negative replies.
+ limit_then_redirect< request >( 10,
[overload_mode_processor] { return overload_mode_processor; } )
...
};

Definition of message limits with ‘transform’ overlimit
reaction could look like (for the case of a message):
class request_processor : public so_5::agent_t
{
public :
normal_request_processor( context_t ctx )
// We can hold no more that 10 requests in queue.
// For all extra messages negative replies must be generated.
+ limit_then_transform( 10, [](const request & evt) {
return make_transformed< negative_reply >(
// Mbox for sending reply is got from original request.
evt.reply_to(),
// All other arguments are passed to negative_reply constructor.
error_code::processor_is_busy );
} ) )
{}
...
};

Definition of message limits with ‘transform’ overlimit
reaction could look like (for the case of a signal):
class long_operation_performer : public so_5::agent_t
{
public :
long_operation_performer( context_t ctx, so_5::mbox_t mbox_for_notifications )
// If we cannot process previous get_status signal
// then we are busy and can tell about this.
+ limit_then_transform< get_status >( 1,
[this, mbox_for_notifications] {
// The result of the transformation is another signal and because of that
// there is only one argument for make_transform (the target mbox).
return make_transformed< status_busy >( notification_mbox );
} ) )
{}
...
};

Some Final Words

This presentation is just a short introduction in the topic of
message limits.
For more detailed information please see the corresponding
section of Project’s Wiki.
But it is necessary to remember that message limits is not a
fullfledged solution for overload problem...

Message limits could be seen as a basic building blocks for
more complex application- and domain-specific overload
control schemes.
Some examples of such schemes which are based on collector-
performer agents pairs are represented in several examples in the
SObjectizer’s distribution (they are also described in Project’s Wiki):
● work_generation;
● collector_performer_pair and collector_many_performers;
● simple_message_deadline;
● prio_work_stealing.

Additional Information:
Project’s home: http://sourceforge.net/projects/sobjectizer
Documentation: http://sourceforge.net/p/sobjectizer/wiki/
Forum: http://sourceforge.net/p/sobjectizer/discussion/
Google-group: https://groups.google.com/forum/#!forum/sobjectizer
GitHub mirror: https://github.com/masterspline/SObjectizer

Dive into SObjectizer 5.5. Seventh part: Message Limits

More Related Content

What's hot (20)

Viewers also liked (11)

Similar to Dive into SObjectizer 5.5. Seventh part: Message Limits (20)

More from Yauheni Akhotnikau (12)

Recently uploaded (20)

Dive into SObjectizer 5.5. Seventh part: Message Limits