Stream-based Data Synchronization

Stream Based
Data Synchronization
Klemen Verdnik

1.1 Who am I, What I Do?
• low-level programming enthusiast 
(audio and video DSP routines, tight loop
optimizations)
• embedded systems (graphic EQ with DSP,
ﬂeet management, mobile payment)
• familiar with iOS SDK since 2008
• vox.io (web / mobile sip, xmpp)
• layer.com (messaging)
• obsession with synchronization protocols

2.1 What is Data Synchronization?
• Having data
consistency across two
or more networked
entities

2.1.1 Example
Toggle Switch App

2.1.2 How to Design the System?
• Simple server
Toggle Switch App

• Simple server
• Simple client
Toggle Switch App

• Simple server
• Simple client
• Simple data
structure
{
lightsOn: true
}
Toggle Switch App

2.2 Other Use Cases
• E-mail (IMAP, POP)
• Messaging (iMessages, Hangouts)
• Photo sharing (Photo Stream, Google Photos)
• File sharing (Dropbox, iCloud Drive)
• Online text editors / spreadsheet editors (Google Docs)
• Multiplayer Games (Minecraft)

2.3 Types of Data Synchronization
• File
synchronization

• File
synchronization
• Text / document
synchronization

• File
synchronization
• Text / document
synchronization
• Data model
synchronization

2.4 Approaches to Data
Synchronization

2.4.1 Absolute Synchronization (copying)
• Copying (wholesale transfer) is ok
when dealing with small data-sets
(e.g. refreshing weather
forecast, RSVP list ...)

• Figuring out differences between previously
fetched data-sets costs CPU and memory
O(n ⋁ m)

O(n ⋁ m)
Dan
Alex
Blake
Emily
George
Caroline
You

O(n ⋁ m)
Dan
Alex
Blake
Emily
George
Caroline
You
Dan
Alex
Emily
George
Caroline
You
≠

2.4.2 Relative Synchronization (changes)
• Getting data up-to-date with changes 
instead of full data sets.
(a.k.a. deltas)

2.4 What are Deltas?
Delta encoding is a way to describe differences
between two datasets.

2.5.1 How to Encode Deltas?
insert ― adds new values to dataset
update ― updates existing values in dataset
delete ― deletes existing values from dataset
+
✔
-
• Three primitive operations

0000000: 4749 4654 2B31 0d00 0d00 9100 00b6 6257 GIFT+1........bW
0000010: 0804 0456 2c27 e5aa 7f21 f904 0000 0000 ...V,'...!......
0000020: 002c 0000 0000 0d00 0d00 0002 318c 8f29 .,..........1..)
0000030: 3000 7986 944f 8823 260d 0feb b620 0b03 0.y..O.#&.... ..
0000040: 2e97 e1a4 0f79 920c 60a5 28e5 c452 abc6 .....y..`.(..R..
[ {
type: "update",
offset: 0x03,
values: [ 0x38, 0x39, 0x61 ]
}, {
type: "insert",
offset: 0x50,
values: [ 0xCE, 0xE1, 0x50, 0x96, 0x89, 0x48, 0x9D, 0x02, 0x43, 0x62,
0x8D, 0x98, 0x28, 0x00, 0x00, 0x3B ]
} ]
• Example on how to encode binary data changes

0000000: 4749 4638 3961 0d00 0d00 9100 00b6 6257 GIF89a........bW
0000010: 0804 0456 2c27 e5aa 7f21 f904 0000 0000 ...V,'...!......
0000020: 002c 0000 0000 0d00 0d00 0002 318c 8f29 .,..........1..)
0000030: 3000 7986 944f 8823 260d 0feb b620 0b03 0.y..O.#&.... ..
0000040: 2e97 e1a4 0f79 920c 60a5 28e5 c452 abc6 .....y..`.(..R..
[ {
type: "update",
offset: 0x03,
values: [ 0x38, 0x39, 0x61 ]
}, {
type: "insert",
offset: 0x50,
0x8D, 0x98, 0x28, 0x00, 0x00, 0x3B ]
} ]

0000000: 4749 4638 3961 0d00 0d00 9100 00b6 6257 GIF89a........bW
0000010: 0804 0456 2c27 e5aa 7f21 f904 0000 0000 ...V,'...!......
0000020: 002c 0000 0000 0d00 0d00 0002 318c 8f29 .,..........1..)
0000030: 3000 7986 944f 8823 260d 0feb b620 0b03 0.y..O.#&.... ..
0000040: 2e97 e1a4 0f79 920c 60a5 28e5 c452 abc6 .....y..`.(..R..
0000050: cee1 5096 8948 9d02 4362 8d98 2800 003b ..P..H..Cb..(..;
[ {
type: "update",
offset: 0x03,
values: [ 0x38, 0x39, 0x61 ]
}, {
type: "insert",
offset: 0x50,
0x8D, 0x98, 0x28, 0x00, 0x00, 0x3B ]
} ]

• Example on how to encode text changes
83: //
84: // Toggles the private ivar `_lightSwitchState` boolean, updates the
85: // background image, plays a sound and transmits the change over network.
86: //
87: func toggleAndSendLightSwitchState() {
88: self.lightSwitchState = !self.lightSwitchState
> 89: self.lightSwitchClient.sendLightSwitchState(self.lightSwitchState)
90: }

• Example on how to encode text changes (diﬀ patch)
83: //
86: //
> 89: self.lightSwitchClient?.sendLightSwitchState(self.lightSwitchState)
90: }
--- 89: self.lightSwitchClient.sendLightSwitchState(self.lightSwitchState)
+++ 89: self.lightSwitchClient?.sendLightSwitchState(self.lightSwitchState)

• Example on how to encode text changes (insert operation)
83: //
86: //
> 89: self.lightSwitchClient?.sendLightSwitchState(self.lightSwitchState)
90: }
{
type: "insert",
offset: 2781,
values: [ "?" ]
}

• Example on how to encode custom data model changes
{
guests: [ "Alex", "Blake", "Caroline", "Dan", "Emily", "George" ]
}

• Example on how to encode custom data model changes
{
guests: [ "Alex", "Blake", "Caroline", "Dan", "Emily", "George" ]
}
{
type: "delete",
guest: [ "Blake" ]
}
{
guests: [ "Alex", "Caroline", "Dan", "Emily", "George" ]
}

3. Stream Based Synchronization

3.1 The Motivation
• Minimum data redundancy

3.1 The Motivation
• Speed / minimum bandwidth

3.1 The Motivation
• Fast writes = good concurrency characteristics

3.1 The Motivation
• Distributability and scalability

3.1 The Motivation
• Ofﬂine support

3.2 Stream of Mutations
• Clients with an open connection receive a live
stream of events from the server

3.2 Stream of Mutations
• Example "To Do" app

3.2.1 Example (To-do List App Data-model)
• Live synchronized list of
to-do tasks
public struct Todo {
public class List: NSObject {
private let tasks: Array<Task> = []
}
}

to-do tasks
• Task element consists of:
checkbox, label and
color public struct Todo {
}
}
public class Task: NSObject {
public private(set) var identifier: NSUUID
public private(set) var completed: Bool
public private(set) var title: String
public private(set) var label: ColorLabel
public enum ColorLabel: UInt8 {
case None = 0, Red, Orange, Yellow, Green,
Turquoise, Blue, Purple, Pink
}
}
}

to-do tasks
checkbox, label and
color
• Tasks can be added,
edited and removed
public func create(title: String, label: Task.ColorL
public func update(identifier: NSUUID, completed: Bo
public func remove(identifier: NSUUID)
}
}
public class Task: NSObject {
public private(set) var completed: Bool
public private(set) var title: String
public private(set) var label: ColorLabel
public enum ColorLabel: UInt8 {
case None = 0, Red, Orange, Yellow, Green,
Turquoise, Blue, Purple, Pink
}
}
}

to-do tasks
checkbox, label and
color
• Tasks can be added,
edited and removed

3.2.2 Example (To-do List Sync Data-model)
• Todo.List user actions
turn into events (𝚫)

• Simple concrete objects
describing changes
public struct Sync {
public class Event: NSObject {
public enum Type: UInt8 {
case Insert = 0, Update, Delete
}
public private(set) var type: Type
public private(set) var completed: Bool?
public private(set) var title: String?
public private(set) var label: Int?
}
}

describing changes
• Serializable
public class Event: NSObject, Serializable {
public enum Type: UInt8 {
case Insert = 0, Update, Delete
}
}
}
public protocol Serializable: class {
init(fromDictionary dictionary: Dictionary<String, AnyObject>)
func toDictionary() -> Dictionary<String, AnyObject>
}

describing changes
• Serializable

• Creating new task
{ // serialized event structure
type: 0, // 0 = Insert
identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", // client generated id
completed: false, // an incomplete task
title: "Buy Milk", // task description
label: 0 // color tag
}

• Editing an existing task
{ // event structure
type: 1, // 1 = Update
identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", // reference to task
completed: true // new state
}

3.2.3 Example (To-do List Sync and Transport)
• Receive live serialized
Events from the
server. public protocol TransportDelegate: class {
func transport(transport: Transport, didReceiveObject object: Serializable)
func transportDidConnect(transport: Transport)
func transportDidDisconnect(transport: Transport)
}
public class Client: NSObject, TransportDelegate {
public private(set) var stream: Stream = Stream()
public private(set) var transport: Transport
public private(set) var todoList: Todo.List
public private(set) var publishedEvents: Array<Event> = []
private func publish(event: Event) -> Bool
public func transport(transport: Transport, didReceiveObject object: Serializab
}
}

Events from the
server.
• Send serialized
Events to server.
public protocol TransportDelegate: class {
func transport(transport: Transport, didReceiveObject object: Serializable)
func transportDidConnect(transport: Transport)
func transportDidDisconnect(transport: Transport)
}
public class Client: NSObject, TransportDelegate {
public private(set) var stream: Stream = Stream()
public private(set) var transport: Transport
public private(set) var todoList: Todo.List
public private(set) var publishedEvents: Array<Event> = []
private func publish(event: Event) -> Bool
public func transport(transport: Transport, didReceiveObject object: Serializab
}
}

Events from the
server.
• Send serialized
Events to server.

3.3 Let the Streaming Begin
• Data consistent, as long as clients remain connected

• Missing out on events puts the client out-of-sync

{ // event structure
type: 1,
identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd",
completed: true
}

• Data consistent, as long as clients remain
connected
• Clients can recover from out-of-sync state
• Server's responsibility beside broadcasting should
also be preserving the events

3.4 Persistent Stream
• Think of it as a linear magnetic tape, or as a storage
with a WORM behavior
• Append only
• Immutable events
• Journal of all the events that have happened

3.4 Persistent Stream
• Always copy all the events? (too expensive)
• Integrity check by hashing events? (only detects
mismatches)
How does a client know if it's got all the events?

3.5 Event Discovery
• Sequencing Events on server

3.5 Event Discovery
public private(set) var seq: Int?
public private(set) var identifier: NSUUID?
}
}

3.5 Event Discovery
• Sequence is a linear function f(x)=x reproducible on client

3.5 Event Discovery
• Client only needs to know the seq value of 
the last event f(x<12)=x

3.5 Event Discovery
• Figuring out missing events by subtracting the set of seqs

3.5 Event Discovery
// Seq values pulled from all events the client has.
// [ 0, 1, 2, 10, 11, 12 ]
let seqsOfEvents: Set = events.map({ $0.seq })
// Calculated sequence ranging from 0 to 12.
// [ 0, 1, 2, 3 ... 12 ]
let seqsOfAllEvents: Set = [Int](0...12)
// Diffed set of seq values.
// [ 3, 4, 5, 6, 7, 8, 9 ]
let seqsOfMissingEvents: Set = seqOfAllEvents.subtract(seqOfEvents)
• Figuring out missing events by subtracting the set of seqs

4. Event and Model Reconciliation
Outbound and inbound reconciliation

4.1 Outbound Reconciliation
• Turning user actions (model changes) into Events

public func create(title: String, label: Task.ColorLabel) -> Sync.Event
public func update(identifier: NSUUID, completed: Bool?, title: String?, label: Task.ColorLabel?) -> Sync.Event?
public func remove(identifier: NSUUID) -> Sync.Event?
}

let todoList = List()
let event = todoList.create("Buy Milk", label: Task.ColorLabel.None)
print("event: '(event)", event)
// event: {
// type: 0, // 0 = Insert
// identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", // client generated id
// completed: false, // an incomplete task
// title: "Buy milk", // task description
// label: 0 // task without a label
// }

• Publishing events
let todoList = List()
let event = todoList.create("Buy Milk", label: Task.ColorLabel.None)
print("event: '(event)", event)
// event: {
// type: 0, // 0 = Insert
// identifier: "cb55ceec-b9ae-4bd9-8783-7dbf3e9cb2cd", // client generated id
// completed: false, // an incomplete task
// title: "Buy milk", // task description
// label: 0 // task without a label
// }
// Sends the event to the stream over the network.
self.syncClient.publish(event)

4.2 Inbound Reconciliation
• Apply Events onto the model

4.2 Inbound Reconciliation
• Apply Events onto the model
private func apply(event: Sync.Event) -> Bool {
switch event.type {
case .Insert: // Task creation
let task = Task(identifier: event.identifier, completed: event.completed!, title: event.title!, label: Task.ColorL
self.tasks.append(task)
case .Update: // Task updates
let task = self.task(event.identifier)
if task == nil {
return false
}
task!.update(event.completed!, title: event.title!, label: Task.ColorLabel(rawValue: event.label!)!)
case .Delete: // Task removal
if !self.removeTask(event.identifier) {
return false
}
}
return true
}
}

4.3 Ofﬂine Support
• Events generated ofﬂine have to be published eventually

• Queue generated events; drain queue for publication

✔
• Generating redundant events while ofﬂine

4.4 Reducing the Edit Distance
• Events describing the same mutation

• Causes stream pollution
• Increases the edit distance

1. Insert Event merges with 
Update Events → single Insert Event
2. Update Event merge with 
the rest of Update Events → single Update Event
3. Last Update Event deﬁnes ﬁnal state.
4. Delete Event clobbers other Event types
Simple set of rules when queueing:

var mergedEvents = Array<Event>()
for oldEvent in events.reverse() {
if oldEvent.identifier != self.identifier {
// Event not mergeable, due to the identifier mismatch.
mergedEvents.append(oldEvent)
continue
} else if self.type == Type.Delete {
// Rule #4
self.reset()
self.type = Type.Delete
} else if self.type == Type.Update && (oldEvent.type == Type.Insert || oldEvent.type == Type.Update) {
// Rule #1, #2, #3
self.completed = self.completed ?? oldEvent.completed
self.title = self.title ?? oldEvent.title
self.label = self.label ?? oldEvent.label
}
}
mergedEvents.append(self)
return mergedEvents
}
}

4.5 Conﬂict Resolution
• Concurrent systems experience conﬂicts when two
or more nodes (clients) work on the same resource
at the same time.

• Concurrent systems experience conﬂicts when two
or more nodes (clients) work on the same resource
at the same time.
• Example: a client deletes a Todo Task before
another client tries to mutate it.

Possible conﬂict resolutions:
• Bring the deleted task back (last writer wins)
• Deleted task stays deleted (ﬁrst writer wins)
• Ask the User what to do? (requires user interaction)

5. Order of Events
Event sequence dictates the order they were written to 
stream this puts the Events in total order
• Task objects will be in the exact same order, deﬁned by 
the Event.seq
• Task mutations will be applied in the same manner on all
clients

5. Order of Events (total order)
• Queued events must be published in batches

5.1 Total Order (sequential writes)
• Synchronized sequential writes block other clients from
writing

5.1 Total Order (sequential writes)
• Synchronized sequential writes block other clients from
writing - violates our fast concurrent writes requirement
serial
writes
concurrent
writes

5.1 Total Order (oﬄine support)
• Both clients online

• Left client loses connection

• Ofﬂine client adds more To-do tasks to the list

• Online client also adds a Todo task to the list

• Left client comes back online ― events generated ofﬂine
get published and fall at the end (higher seq values)

5.2 Causal Order
Causes must precede their effects - effects come after causes,
and never before

5.2 Causal Order
Causes must precede their effects - effects come after causes,
and never before
cause
eﬀect

5.2 Causal Order
• Generated Event is an eﬀect caused by user taking action /
responding to the UI.
• Events should be reconciled in the same order as they were
generated by clients.
• Events should be applied onto the app model under the same
conditions as it was when author generated the events.
• Total order cannot guarantee Events will be written to stream in
the same order they were generated.

5.2.1 Order Based on Timestamps
Client B's events are
written before Client A's,
even though Client A
generated them ﬁrst.

Encoding local time
with events.

Sorting events based
on the embedded
timestamp.

• No guarantee
time will be the
same on all
devices
• Clock skew
• Manual override

5.2.2 Version Vectors
• Reconstructing Events' order as it was perceived by the author
based on happened-before information.
• Provides causality-tracking basic principle in some optimistic
(lazy) replication algorithms.
• Allows the client to operate independently from the server.
• When all clients eventually publish their events, it brings other online
clients into a consistent state eventual consistency.

How to encode happened-before information?
}
}

1. Information of what's
the last seen event -
event.seq
public private(set) var precedingSeq: Int
}
}

event.seq
2. Keep unpublished
events in order -
event.clientSeq
public private(set) var clientSeq: Int
}
}

event.seq
2. Keep unpublished
events in order -
event.clientSeq
3. Order
/// Event sorting closure
static public let causalOrder = { (e1: Event, e2: Event)
if e1.precedingSeq == e2.precedingSeq {
return e1.clientSeq < e2.clientSeq
}
return e1.precedingSeq < e2.precedingSeq
}
public private(set) var clientSeq: Int
}
}

var mergedEvents = Array<Event>()
for oldEvent in events.sort(Event.causalOrder) {
if oldEvent.identifier != self.identifier {
// etc...
public class List: NSObject, ModelReconciler {
public func apply(events: Array<Sync.Event>) -> Bool {
for event in events.sort(Sync.Event.causalOrder) {
let success = self.apply(event)
// etc...
Minor adjustment in outbound / inbound reconciliation:

• Newly published events generated ofﬂine are ordered by
their causality.

• Concurrent writes - no need for batched writes anymore, 
due to clientSeq.

• Concurrent writes - events can be written with undetermined
order; order can be reconstructed on clients
serial
writes
concurrent
writes

6. Advantages
• Shared source - minimal redundancy

6. Advantages
• Lightweight data structure - fast delivery

6. Advantages
• Minimal server logic (low CPU)

6. Advantages
• Short writes - high concurrency

6. Advantages
• Scalable / distributable

6. Advantages
• Scalable / distributable
• Ofﬂine support

7. Disadvantages
• Server simplicity = client complexity

7. Disadvantages
• Rogue clients = stream pollution

7. Disadvantages
• Clients must read full stream

7. Disadvantages
• Clients must read full stream
• Partial sync difﬁcult to implement

END_OF_STREAM
questions?
klemen.verdnik@gmail.comgithub.com/chipxsd
@chipxsd

Stream-based Data Synchronization

More Related Content

What's hot (13)

Viewers also liked (7)

Similar to Stream-based Data Synchronization (20)

Recently uploaded (20)

Stream-based Data Synchronization