Event-Driven Systems With MongoDB

EVENT-DRIVEN SYSTEMS
BACKED BY MONGODB
bit.ly/2DUYXNF

ANDRII LITVINOV
Platform engineer at Synergy
Sports Technology

CHALLENGES
• Duplicate records matching
• Game statistics calculations
• Reports read model calculations

Orders Payments
M
e
s
s
a
g
e
b
r
o
k
e
r
Place
Order
Order
Order
Placed
Order
Placed
Payment
Accepted
Payment
Payment
Accepted

M
e
s
s
a
g
e
b
r
o
k
e
r
Orders
Crash
Order
Placed
✕
Order
Placed
?Payments
✓
Order
Place
Order

KEY TAKEAWAY:
PERSIST STATE AND
PUBLISH EVENT
ATOMICALLY

EVENT SOURCING
Service 1
Event 1
Event 1
Event 1
Event 1
Event
Service 2
Event 1
Event 1
Event 1
Event 1
Event

APPLICATION EVENTS
Service
M
e
s
s
a
g
e
b
r
o
k
e
r
Entity EventLocal
transaction
Event
Publisher

TRANSACTION LOG TAILING
Service
M
e
s
s
a
g
e
b
r
o
k
e
r
Event
Publisher
Transaction Log
Entity

OUR CONSIDERATIONS
• Go with Oplog tailing
• Event sourcing requires major redesign
• MongoDB 3.6 Change Streams just released
• MongoDB 3.4
• No ACID transactions support so far

OPLOG TAILING
• More efforts to implement { w: "majority" }
• Proven, reliable, widely used in production
• Emit event from Primary { w: 1 }
Pros
Cons
• Private undocumented API
• Advised against by MongoDB

CHANGE STREAMS 3.6
• Single collection per connection
• Not yet widely used in production
• { w: "majority" } out of the box
• Stable, well-documented API
Pros
Cons
• No way to emit from Primary { w: 1}
• Recommended by MongoDB

Event-Driven Systems With MongoDB

TAKEAWAY:
FOR PRIMARY-SECONDARY-
ARBITER (PSA) REPLICA SET
USE OPLOG TAILING

CHANGE STREAMS 4.0
• { w: "majority" } out of the box
• Option to resume from a given Timestamp
• Subscribe to database or entire deployment
Pros
Cons
• Only supports { w: "majority" }, no { w: 1}
• Recommended by MongoDB

TAKEAWAY:
FOR MONGODB 4.0 REPLICA
SET WITH 3+ DATA-
BEARING NODES USE
CHANGE STREAMS

var @object = (BsonDocument) operation["o"];
switch ((string) operation["op"])
{
case "i":
{
EmitDomainEvents(@object);
break;
}
case "u":
{
if (@object.TryGetValue("$set", out var set))
EmitDomainEvents((BsonDocument) set);
else
EmitDomainEvents(@object);
break;
}
}

var handler = container.GetInstance<IEventHandler<OrderPlaced>>();
Lifestyle.Singleton.CreateRegistration(() =>
new EventHandlersConsumer("payments", database,
new Dictionary<string, Func<DomainEvent, Task>>
{
{
nameof(OrderPlaced),
@event => handler.Handle((OrderPlaced) @event)
}
}, logger), container));

await events
.Find(filter.And(
filter.In("event._t", handlers.Keys),
filter.Gt(envelope => envelope.Timestamp, checkpoint.Position)))
.Sort(Builders<EventEnvelope>.Sort.Ascending(envelope => envelope.Timestamp))
.ForEachAsync(async envelope =>
{
if (handlers.TryGetValue(envelope.Event.GetType().Name, out var handler))
{
if (!string.IsNullOrEmpty(envelope.CorrelationId))
TraceContext.Set(envelope.CorrelationId, envelope.EventId);
await handler.Invoke(envelope.Event);
}
await SaveCheckpoint(checkpoint, envelope.Timestamp);
}, cancellationToken);

public async Task Handle(OrderPlaced @event)
{
if (@event.Amount >= 200)
{
var delay = new Random().Next(3000, 30000);
await Task.Delay(delay);
}
var payment = new Payment(ObjectId.GenerateNewId(), @event.SourceId);
payment.Process(@event.Amount);
await payments.Create(payment);
}

public void Process(decimal amount)
{
Amount = amount;
if (Amount >= 300)
{
Status = PaymentStatus.Rejected;
RecordEvent(new PaymentRejected(Id, OrderId, Amount));
}
else
{
Status = PaymentStatus.Accepted;
RecordEvent(new PaymentAccepted(Id, OrderId, Amount));
}
}

var futureEvent = observable.FirstOfType<OrderFulfilled, OrderDiscarded>(orderId);
var command = await request.Body.ReadAs<PlaceOrder>();
await handler.Handle(command);
try
{
await futureEvent;
response.StatusCode = (int) HttpStatusCode.Created;
}
catch (TimeoutException)
{
response.StatusCode = (int) HttpStatusCode.Accepted;
}
var order = await orders.Find(o => o.Id == orderId).FirstAsync();
await response.Body.Write(order);

SUMMARY
• Use change streams with MongoDB 4.0
and 3+ data-bearing nodes
• Write state and publish event atomically
• Use oplog tailing prior MongoDB 4.0
or with 2 data-bearing nodes

RESOURCES
Replication
https://www.slideshare.net/Stripe_talks/mongo-db-oplog
https://www.compose.com/articles/the-mongodb-oplog-and-node-js/
https://stackoverflow.com/questions/17878835/what-does-the-h-in-oplog-rs-document-stand-for
https://stackoverflow.com/questions/26935419/what-do-the-oplog-fields-actually-mean
Event-Driven-Architecture
https://www.nginx.com/blog/event-driven-data-management-microservices/
https://microservices.io/
https://skillsmatter.com/skillscasts/12115-keynote-not-just-events-developing-asynchronous-microservices
Domain model
http://dddcommunity.org/library/vernon_2011/

THANK YOU! QUESTIONS?
https://twitter.com/andriilitvinov
https://github.com/andrii-litvinov
https://www.upwork.com/fl/andriilitvinov

Event-Driven Systems With MongoDB

More Related Content

What's hot (20)

Similar to Event-Driven Systems With MongoDB (20)

Recently uploaded (20)

Event-Driven Systems With MongoDB

Editor's Notes