The Dual write problem

The dual write problem
Jeppe Cramon - @jeppec
Cloud Create ApS

@jeppec
Essential complexity of 2 way integration
Component
Warehouse
Component
Order
Component
Billing
UI
Send-
Invoice
Save-Order
Order:Save-Order()è
call Warehouse:Reserve-Items()
call Billing:Send-Invoice()
commit()
Reserve-
Items
Local transaction between
the 3 Components

@jeppec
Let’s (micro)service’ify this

@jeppec
Accidental complexity from distributed service integration
Warehouse
Service
Order
Service
Billing
Service
UI
Send-
Invoice
Save-Order
Reserve-
Items
Local transaction between 2
local “Services”
DUAL WRITE
using Remote call

@jeppec
Microservices turns a
local functional call
into a distributed problem

@jeppec
With cross SERVICE integration
we’re bound by the laws of
distributed computing

@jeppec
The 11 Fallacies of Distributed Computing
These fallacies are assumptions architects, designers and developers
of distributed systems are likely to make. The fallacies will be proven
wrong in the long run - resulting in all sorts of troubles and pains for
the solution and architects who made the assumptions.
1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.
9. The System is atomic/monolithic
10. The System is Finished
11. Business logic can and should be centralized
See https://arnon.me/wp-content/uploads/Files/fallacies.pdf

@jeppec
Order:Save-Order()è
call Warehouse:Reserve-Items()
if (Billing:Call-Failed:Too-Busy?)
Wait-A-While()
if (Billing:Call-Failed:Too-Busy?)
Wait-A-Little-While-Longer()
if (Billing:Call-Failed:IO-Error?)
Save-We-Need-Check-If-Call-Billing-Succeded-After-All
AND We-Need-To-Retry call Order:Save-Order and call Warehouse:Reserve-Items
AND Tell-Customer-That-This-Operation-Perhaps-Went-Well
if (Billing:Call-Went-Well?)
commit()
Accidental complexity from distributed service integration
Warehouse
Service
Order
Service
Billing
Service
UI
Send-
Invoice
Save-Order
Reserve-
Items
Local transaction between 2
Components

@jeppec
Remote Call
Aka. RPC/REST/SOAP/GraphQL…

@jeppec
What’s the challenge using
state mutating
Request/Response calls
between
distributed components?

@jeppec
Synchronous Request/Response
lowers our tolerance for faults
• When you get an IO error
• When servers crash or restarts
• When databases are down
• When deadlocks occurs in our databases
• Do you retry?
With synchronous Request/Response we can loose business data if there’s no automatic retry mechanism.
Also if the operation we retry isn’t idempotent* we risk having the side effect multiple times!
Client Server
Processing
The same message can be
processed more than once
*Idempotence describes the quality of an
operation in which result and state does
not change if the operation is performed
more than 1 time
Request
Processing
Duplicated Request
Duplicated Response
Response

@jeppec
Idempotence
Don’t press the button twice!

@jeppec
Ensuring
transactional
consistency
becomes much harder

@jeppec
Sales system
Sales
Update customer status (e.g. Gold customer)
Bookkeeping
Deliver goods
Delivery
system
Deliveries
Customer/
CRM system
Customers
SAP
Bookkeeping
Complete
Purchase
Transaction
Coordinator
Transactional
Resource
Request-to-Prepare
Commit
Prepared
Done
Prepare
Phase
Commit
Phase
2 Phase Commit

@jeppec
What’s wrong with distributed transactions?
• Transactions lock resources while active
• Services are autonomous
• Can’t be expected to finish within a certain time interval
• Locking keeps other transactions from completing their
job
• Locking doesn’t scale
• X Phase Commit is fragile by design

How to handle the dual
write problem?

@jeppec
Dual write problem
Order
Service
Billing
Service
Client
Send-
Invoice
Save-Order
Mutating
Remote call
Expected
transaction
boundary No simple solution (except a classic monolith)
• Local-commit-then-publish
• If the app crashes after local-
commit then the publish
operation isn’t performed
• Publish-then-local-commit
• If the local-commit fails then
we’ve already formed the publish
operation
• Billing Service may receive
published message BEFORE the
Order Service has committed
its transaction

@jeppec
Dual write problem
Orchestration / Process Manager / Saga*
Order
Service
Billing
Service
Client
Send-
Invoice
Transaction
boundary
Transaction
boundary
Save-Order
• Requires only local transactions
• Single coordinator
• Increases eventual consistency
• Requires idempotent operations
due to retries
• Requires compensations /
rollback
• Tighter coupling as the
orchestrator instructs other
services about what to do
* This isn’t the real Saga pattern

@jeppec
Dual write problem
Choreography
public class OrderService {
public void handle(AcceptOrder cmd) {
orderRepository.add(new Order(cmd.orderId,
cmd.orderLines));
eventTopic.publish(new OrderAccepted(cmd.orderId,
cmd.orderLines));
}
}
public class BillingService {
public void handle(OrderAccepted cmd) {
invoiceRepository.add(new Invoice(cmd.orderId,
cmd.orderLines));
emailGateway.send(new OrderAcceptedEmail(cmd.orderId,
cmd.orderLines));
}
}

@jeppec
Dual write problem
Choreography
Order
Service
Billing
Service
Client
Send-
Invoice
Save-Order
Transaction
boundary
Transaction
boundary
• No Single coordinator, but global
system state and coordination
logic is scattered across all
participants
• Increases eventual consistency
due to retries
• Looser coupling (if events are
used)
• Suffers from a Dual Write problem
it self
Events/Commands
Queue

@jeppec
Outbox pattern
For Choreography

@jeppec
Dual write problem
Choreography with Outbox pattern
Order
Service
Billing
Service
Client
Send-
Invoice
Save-Order
Transaction
boundary
Transaction
boundary
• You can either use a Change
Data Capture (CDC) approach
e.g. using a database Write
Ahead Log (WAL)
• Risk of CRUD events!
• Or a manual OutBox pattern
where Save-Order writes data
and the event/command into
the OutBox within the same
database. Typically requires
polling and coordination +
recovery options in a multi cluster
setup
Events/Commands
Change
Data
Capture
/
Polling
Async Push or Poll

@jeppec
Saga pattern
A failure management pattern

@jeppec
Trip reservation as a Saga
Start
Rent Car Book Hotel Reserve Flight
Cancel Hotel
Cancel Car
Book
Car Rental Hotel Rental
WorkQueue:
- Queue: “Rent Car”
- Queue: “Book Hotel”
- Queue: “Reserve Flight”
Routing slip
Flight Booking

@jeppec
Start
Cancel Hotel
Cancel Car
Book Book
WorkQueue:
- Queue: “Book Hotel”
- Queue : “Reserve Flight”
WorkStack:
- ReservationId: ABC123
Cancellation: “Cancel Car”
Routing slip Flight Booking

@jeppec
Start
Cancel Hotel
Cancel Car
Book Book
WorkQueue:
- Queue : “Reserve Flight”
WorkStack:
- Hotel ReservationId: XYZ
Cancellation: “Cancel Hotel”
- Car ReservationId: ABC123
Routing slip
Book
Flight Booking

@jeppec
Start
Cancel Hotel
Cancel Car
Book Book Book
WorkStack:
Routing slip
Flight Booking

@jeppec
Start
Cancel Hotel
Cancel Car
Book Book Book
Cancel
WorkStack:
Routing slip
Flight Booking

@jeppec
Start
Cancel Hotel
Cancel Car
Book Book Book
Cancel
Cancel
WorkStack:
Routing slip
Flight Booking

@jeppec
Start
Cancel Hotel
Cancel Car
Book Book Book
Cancel
Cancel
Cancelled
Flight Booking

@jeppec
Dual write problem
Choreography with Event Sourcing
Order
Service
Billing
Service
Client
Send-
Invoice
Save-Order
Transaction
boundary
Transaction
boundary
Event Store
Async
Push
or Pull
• Natively event driven: The Service
consumes and produces events
• Proven Audit Trail
• Flexible and adaptable
• Captures business intentions as
events

@jeppec
Event Sourcing
Business-Objects/Aggregates track their own Domain Events
and derive their state from said Events
OrderCreated ProductAdded ProductAdded ProductRemoved ProductAdded OrderAccepted
Time
07:39
Time
07:40
Time
07:41
Time
07:45
Time
07:46
Time
07:50

@jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
EventType SerializedEvent
Order 14237 100 0 2014-01-06 7:39 {Guid-1} OrderCreated <serialized event>…
Order 14237 101 1 2014-01-06 7:40 {Guid-2} ProductAdded <serialized event>…
Order 14237 103 3 2014-01-06 7:45 {Guid-4} ProductRemoved <serialized event>…
Order 14237 105 5 2014-01-06 7:50 {Guid-6} OrderAccepted <serialized event>…
Order

@jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
Order
Orderline

@jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
Order
Orderline
Orderline

@jeppec
Event Replaying
Type Aggregate
Identifier
GlobalOrder Event
Order
Timestamp Event
Identifier
Order
Accepted: true
Orderline
Orderline

@jeppec
Client handled subscriptions
RSocketEvent
StreamSubscription
Local storage
EventStore
RSocketEvent
StreamSubscription
Local storage
EventStreamSubscription
Message
EventStreamSubscription
Message
EventStoreEventStreamPublisher
EventStoreEventStreamPublisher
Event
Event
Event Event
Supports
Single Instance
Subscriber,
which ensures
that only one
instance of
Subscriber B
has an active
subscription.
Other
instances of
the same
subscriber are
hot-standby
<<Topic Subscriber>>
Customer_Service:Some_Ac:OrderEvents
<<Topic Publisher>>
Sales_Service:OrderEvents
RSocketServer
tcp://subscribe-event-stream
A
B Subscriber B
RSocket Request/Stream
Event-Stream
Subscriber A
RSocket Request/Stream
Event-Stream
Flux<PersistedEvent> eventStream(long fromInclusiveGlobalOrder,
Option<String> subscriptionId)

@jeppec
ReactiveBus Pub/Sub
JdbcEventStores eventStores = ….
var customerAggregateType = AggregateType.from("Customer", CustomerId.class);
var customerEventStore = eventStores.getEventStoreFor(customerAggregateType);
var streamName = EventStreamName.from("CustomersStream");
reactiveBus.addEventStreamPublisher(new EventStoreEventStreamPublisher(streamName, customerEventStore, unitOfWorkFactory));
Publisher:
reactiveBus.subscribeToEventStream(
SubscriberId.from(”SalesService->CustomerEvents"),
EventStreamName.from("CustomersStream"),
EventStreamSubscriptionParameters.parameters(GlobalOrderSubscribeFromToken.fromStartOfStream()))
.doOnNext(payload -> {
System.out.println(”Received: " + e.payload.getClass() + "@" + e.globalOrder);
}).subscribe();
Subscriber:

@jeppec
Pub/Sub – Subscriber event handling
reactiveBus.subscribeToEventStream(SubscriberId.from(”SalesService->OrderEvents"),
EventStreamName.from(”OrdersStream"),
EventStreamSubscriptionParameters.parameters(GlobalOrderSubscribeFromToken.fromStartOfStream()))
.doOnNext(payload -> {
if (event instanceof OrderAdded e) {
.. = e.orderNumber;
} else if (event instanceof ProductAddedToOrder e) {
… = e.productId;
} else if (event instanceof OrderAccepted) {
…accepted = true;
}
).subscribe();

@jeppec
Pub/Sub – Subscriber event handling
@EventHandler
private void handle(OrderAdded e, EventMessage message) {
….
}
@EventHandler
private void handle(ProductAddedToOrder e) {
.…
}
@EventHandler
private void handle(OrderAccepted e) {
….
}

@jeppec
For more
see
Blog: https://cramonblog.wordpress.com/
Homepage: http://cloudcreate.dk/
Twitter: @jeppec

The Dual write problem

More Related Content

What's hot (20)

Similar to The Dual write problem (20)

Recently uploaded (20)

The Dual write problem