SlideShare a Scribd company logo

Server side data sync for mobile apps with silex

Michele Orselli, profile picture
Michele Orselli

The document discusses the implementation of a data synchronization API for mobile applications using Silex, focusing on various design choices, conflict resolution strategies, and alternative approaches. It elaborates on the architecture required to handle multiple mobile platforms and user-generated data, including MongoDB integration and sync logic for diverse resources. Furthermore, the document presents specific API endpoints and code snippets to illustrate the API's functionality in managing changes and resolving conflicts effectively.

Software
Related topics:
Mobile Apps Overview
1 of 99
Downloaded 72 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
Implementing data synchronization API for mobile apps with Silex
Michele Orselli CTO@Ideato _orso_ micheleorselli / ideatosrl mo@ideato.it
Agenda scenario design choices implementation alternative approaches
Server side data sync for mobile apps with silex
Server side data sync for mobile apps with silex
Sync scenario A B C
Sync scenario ABC ABC ABC
Dealing with conflicts A1 A2 ?
Brownfield project Scenario several mobile apps for tracking user generated data (calendar, notes, bio data) iOS & Android ~10 K users steadily growing at 1.2 K/month
MongoDB Scenario Legacy App based on Codeigniter Existing RPC-wannabe-REST API for data sync
For every resource get updates: Scenario POST /m/:app/get/:user_id/:res/:updated_from send updates: POST /m/:app/update/:user_id/:res_id/:dev_id/:res
api
Scenario ~6 different resources, ~12 calls per sync apps sync by polling every 30 sec every call sync little data
Challenge Rebuild sync API for old apps + 2 incoming Enable image synchronization More efficient than previous API
Server side data sync for mobile apps with silex
Existing Solutions Tstamps, Vector clocks, CRDTs syncML, syncano Algorithms Protocols/API Azure Data sync Platform couchDB, riak Storage
Not Invented Here? Don't Reinvent The Wheel, Unless You Plan on Learning More About Wheels J. Atwood
Architecture 2 different mobile platforms Several teams with different skill level Changing storage wasn’t an option Forcing a particular technology client side wasn’t an option
Architecture c1 server c2 c3 sync logic conflicts resolution thin clients
Implementation In the sync domain all resources are managed in the same way
Implementation For every app: one endpoint for getting new data one endpoint for pushing changes one endpoint for uploading images
The new APIs GET /apps/:app/users/:user_id/changes[?from=:from] POST /apps/:app/users/:user_id/merge POST /upload/:res_id/images
Silex Implementation
Silex Implementation Col 1 Col 2 Col 3
Silex Implementation Col 1 Col 2 Col 3 Sync Service
Silex Implementation Col 1 Col 2 Col 3 Sync Service
Silex Implementation Col 1 Col 2 Col 3 Sync Service
Silex Implementation Col 1 Col 2 Col 3 Sync Service
Silex Implementation Col 1 Col 2 Col 3 Sync Service
Silex Implementation Col 1 Col 2 Col 3 Sync Service
Silex Implementation Col 1 Col 2 Col 3 Sync Service
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/changes”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $syncService = $app[‘syncService’]; $syncService->sync($lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/changes”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $syncService = $app[‘syncService’]; $syncService->sync($lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/changes”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $syncService = $app[‘syncService’]; $syncService->sync($lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/changes”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $syncService = $app[‘syncService’]; $syncService->sync($lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
Silex Implementation $app['mongodb'] = new MongoDb(
); $app[‘changesRepo’] = new ChangesRepository( $app[‘mongodb’] ); $app[‘syncService’] ? new SyncService( $app[‘changesRepo’] );
Get changes GET /apps/:app/users/:user_id/changes?from=:from timestamp?
Server suggest the sync time timestamp are inaccurate server suggests the “from” parameter to be used in the next request
Server suggest the sync time GET /changes { ‘next’ : 12345, ‘data’: [
] } c1 server
Server suggest the sync time GET /changes { ‘next’ : 12345, ‘data’: [
] } c1 server GET /changes?from=12345 { ‘next’ : 45678, ‘data’: [
] }
operations: {‘op’: ’add’, id: ‘1’, ’data’:[
]} {‘op’: ’update’, id: ‘1’, ’data’:[
]} {‘op’: ’delete’, id: ‘1’} {‘op’: ’add’, id: ‘2’, ’data’:[
]} states: {id: ‘1’, ’data’:[
]} {id: 2’, ’data’:[
]} {id: ‘3’, ’data’:[
]} what to transfer
what to transfer we choose to transfer states {id: ‘1’, ’type’: ‘measure’, ‘_deleted’: true} {id: 2’, ‘type’: ‘note’} {id: ‘3’, ‘type’: ‘note’} ps: soft delete all the things!
unique identifiers How do we generate an unique id in a distributed system?
unique identifiers How do we generate an unique id in a distributed system? UUID (RFC 4122): several implementations in PHP (https://github.com/ramsey/uuid)
unique identifiers How do we generate an unique id in a distributed system? Local/Global Id: only the server generates GUIDs clients use local ids to manage their records
unique identifiers POST /merge { ‘data’: [ {’lid’: ‘1’, 
}, {‘lid’: ‘2’, 
} ] } c1 server { ‘data’: [ {‘guid’: ‘58f0bdd7-1400’, ’lid’: ‘1’, 
}, {‘guid’: ‘6f9f3ec9-1400’, ‘lid’: ‘2’, 
} ] }
conflict resolution algorithm (plain data) mobile generated data are “temporary” until sync to server server handles conflicts resolution
conflict resolution algorithm (plain data) conflict resolution: domain indipendent: e.g. last-write wins domain dipendent: use domain knowledge to resolve
function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data)
function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data)
function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data) no conflict
function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data) remote wins
function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data) server wins
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ } c1 server
conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge c1 server {‘ok’ : { ’guid’: ‘af54d’ }} {‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
conflict resolution algorithm (hierarchical data) How to manage hierarchical data? { ‘lid’ : ‘123456’, ‘type’ : ‘baby’, 
 } { ‘lid’ : ‘123456’, ‘type’ : ‘temperature’, ‘baby_id : ‘123456’ }
conflict resolution algorithm (hierarchical data) How to manage hierarchical data? 1) sync root record 2) update ids 3) sync child records { ‘lid’ : ‘123456’, ‘type’ : ‘baby’, 
 } { ‘lid’ : ‘123456’, ‘type’ : ‘temperature’, ‘baby_id : ‘123456’ }
conflict resolution algorithm (hierarchical data) function syncHierarchical($data) { sortByHierarchy($data); foreach ($data as $newRootRecord) { $s = findByGuid($newRootRecord->getGuid()); if($newRecord->isRoot()) { if (!$s) { add($newRootRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } 

conflict resolution algorithm (hierarchical data) function syncHierarchical($data) { sortByHierarchy($data); foreach ($data as $newRootRecord) { $s = findByGuid($newRootRecord->getGuid()); if($newRecord->isRoot()) { if (!$s) { add($newRootRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } 
 parent records first
conflict resolution algorithm (hierarchical data) function syncHierarchical($data) { sortByHierarchy($data); foreach ($data as $newRootRecord) { $s = findByGuid($newRootRecord->getGuid()); if($newRecord->isRoot()) { if (!$s) { add($newRootRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } 

conflict resolution algorithm (hierarchical data) function syncHierarchical($data) { sortByHierarchy($data); foreach ($data as $newRootRecord) { $s = findByGuid($newRootRecord->getGuid()); if($newRecord->isRoot()) { if (!$s) { add($newRootRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } 
 no conflict

 if ($newRootRecord->updated > $s->updated) { update($s, $newRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } else { updateRecordIds($s, $data); updateRemote($newRecord, $s); } } else { sync($data); } } conflict resolution algorithm (hierarchical data) remote wins

 if ($newRootRecord->updated > $s->updated) { update($s, $newRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } else { updateRecordIds($s, $data); updateRemote($newRecord, $s); } } else { sync($data); } } conflict resolution algorithm (hierarchical data) server wins
conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge c1 server
conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } {‘update’ : { ‘lid’: ‘1’, ’guid’: ‘af54d’ }} {‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
enforcing domain constraints e.g. “only one temperature can be registered in a given day” how to we enforce domain constraints on data?
enforcing domain constraints e.g. “only one temperature can be registered in a given day” how to we enforce domain constraints on data? 1) relax constraints
enforcing domain constraints e.g. “only one temperature can be registered in a given day” how to we enforce domain constraints on data? 1) relax constraints 2) integrate constraints in sync algorithm
enforcing domain constraints from findByGuid to findSimilar first lookup by GUID then by domain rules “two measures are similar if are referred to the same date”
enforcing domain constraints c1 server
enforcing domain constraints { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } c1 server
enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } c1 server
enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server
enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server
enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server { ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
dealing with binary data Binary data uploaded via custom endpoint Sync data remains small Uploads can be resumed
dealing with binary data Two steps* 1) data are synchronized 2) related images are uploaded * this means record without file for a given time
dealing with binary data POST /merge { ‘lid’ : 1, ‘type’ : ‘baby’, ‘image’ : ‘myimage.jpg’ } { ‘lid’ : 1, ‘guid’ : ‘ac435-f8345’ } c1 server POST /upload/ac435-f8345/image
What we learned Implementing this stuff is tricky Explore existing solution if you can Understanding the domain is important
vector clocks
vector clocks
CRDT Conflict-free Replicated Data Types (CRDTs) Constraining the types of operations in order to: - ensure convergence of changes to shared data by uncoordinated, concurrent actors - eliminate network failure modes as a source of error
Couchbase Mobile Gateways handles sync Data flows through channels - partition data set - authorization - limit the data Use revision trees
Riak Distributed DB Eventually/Strong Consistency Data Types Configurable conflict resolution - db level for built-in data types - application level for custom data
That’s all folks! Questions? Please leave feedback! https://joind.in/12959
http://www.objc.io/issue-10/sync-case-study.html http://www.objc.io/issue-10/data-synchronization.html https://dev.evernote.com/media/pdf/edam-sync.pdf http://blog.helftone.com/clear-in-the-icloud/ http://strongloop.com/strongblog/node-js-replication-mobile-offline-sync-loopback/ http://blog.denivip.ru/index.php/2014/04/data-syncing-in-core-data-based-ios-apps/?lang=en http://inessential.com/2014/02/15/vesper_sync_diary_8_the_problem_of_un http://culturedcode.com/things/blog/2010/12/state-of-sync-part-1.html http://programmers.stackexchange.com/questions/206310/data-synchronization-in-mobile-apps-multiple- devices-multiple-users http://bricklin.com/offline.htm http://blog.couchbase.com/why-mobile-sync Links
Links Vector Clocks http://basho.com/why-vector-clocks-are-easy/ http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks http://basho.com/why-vector-clocks-are-hard/ http://blog.8thlight.com/rylan-dirksen/2013/10/04/synchronization-in-a-distributed-system.html CRDTs http://christophermeiklejohn.com/distributed/systems/2013/07/12/readings-in-distributed-systems.html http://www.infoq.com/presentations/problems-distributed-systems https://www.youtube.com/watch?v=qyVNG7fnubQ Riak http://docs.basho.com/riak/latest/dev/using/conflict-resolution/ Couchbase Sync Gateway http://docs.couchbase.com/sync-gateway/ http://www.infoq.com/presentations/sync-mobile-data API http://developers.amiando.com/index.php/REST_API_DataSync https://login.syncano.com/docs/rest/index.html
Credits phones https://www.flickr.com/photos/15216811@N06/14504964841 wat http://uturncrossfit.com/wp-content/uploads/2014/04/wait-what.jpg darth http://www.listal.com/viewimage/3825918h blueprint: http://upload.wikimedia.org/wikipedia/commons/5/5e/Joy_Oil_gas_station_blueprints.jpg building: http://s0.geograph.org.uk/geophotos/02/42/74/2427436_96c4cd84.jpg brownfield: http://s0.geograph.org.uk/geophotos/02/04/54/2045448_03a2fb36.jpg no connection: https://www.flickr.com/photos/77018488@N03/9004800239 no internet con https://www.flickr.com/photos/roland/9681237793 vector clocks: http://en.wikipedia.org/wiki/Vector_clock crdts: http://www.infoq.com/presentations/problems-distributed-systems

More Related Content

PDF
Implementing data sync apis for mibile apps @cloudconf
Michele Orselli
 
PDF
Implementing Server Side Data Synchronization for Mobile Apps
Michele Orselli
 
PPTX
Firebase ng2 zurich
Christoffer Noring
 
PDF
Angular.js Primer in Aalto University
SC5.io
 
PDF
Me and my importers
Donny Wals
 
PPT
Android wearpp
Paul Trebilcox-Ruiz
 
PPTX
Powering Heap With PostgreSQL And CitusDB (PGConf Silicon Valley 2015)
Dan Robinson
 
PPTX
Parse Advanced
Tushar Acharya
 
Implementing data sync apis for mibile apps @cloudconf
Michele Orselli
 
Implementing Server Side Data Synchronization for Mobile Apps
Michele Orselli
 
Firebase ng2 zurich
Christoffer Noring
 
Angular.js Primer in Aalto University
SC5.io
 
Me and my importers
Donny Wals
 
Android wearpp
Paul Trebilcox-Ruiz
 
Powering Heap With PostgreSQL And CitusDB (PGConf Silicon Valley 2015)
Dan Robinson
 
Parse Advanced
Tushar Acharya
 

What's hot (20)

PPT
Developing A Real World Logistic Application With Oracle Application - UKOUG ...
Roel Hartman
 
PDF
Bringing Transactional Guarantees to MongoDB
Paul Robinson
 
PDF
React lecture
Christoffer Noring
 
PDF
JS Fest 2019. Anjana Vakil. Serverless Bebop
JSFestUA
 
PDF
My Top 5 APEX JavaScript API's
Roel Hartman
 
PPTX
Creating sub zero dashboard plugin for apex with google
Roel Hartman
 
PDF
Use Kotlin scripts and Clova SDK to build your Clova extension
LINE Corporation
 
PDF
Akka: Actor Design & Communication Technics
Alex Fruzenshtein
 
PPTX
Building multi lingual and empatic bots - Sander van den Hoven - Codemotion A...
Codemotion
 
PPTX
Rxjs swetugg
Christoffer Noring
 
PPTX
Rxjs ngvikings
Christoffer Noring
 
PDF
Inversion Of Control
Chad Hietala
 
PDF
Designing The Right Schema To Power Heap (PGConf Silicon Valley 2016)
Dan Robinson
 
PDF
Integrating React.js with PHP projects
Ignacio MartĂ­n
 
PDF
Controller specs
Alexander Miller
 
PPTX
201410 2 fiware-orion-contextbroker
FIWARE
 
PPTX
Using Change Streams to Keep Up with Your Data
MongoDB
 
PPTX
Bare-knuckle web development
Johannes Brodwall
 
PDF
Tweaking the interactive grid
Roel Hartman
 
PDF
Goal Based Data Production with Sim Simeonov
Databricks
 
Developing A Real World Logistic Application With Oracle Application - UKOUG ...
Roel Hartman
 
Bringing Transactional Guarantees to MongoDB
Paul Robinson
 
React lecture
Christoffer Noring
 
JS Fest 2019. Anjana Vakil. Serverless Bebop
JSFestUA
 
My Top 5 APEX JavaScript API's
Roel Hartman
 
Creating sub zero dashboard plugin for apex with google
Roel Hartman
 
Use Kotlin scripts and Clova SDK to build your Clova extension
LINE Corporation
 
Akka: Actor Design & Communication Technics
Alex Fruzenshtein
 
Building multi lingual and empatic bots - Sander van den Hoven - Codemotion A...
Codemotion
 
Rxjs swetugg
Christoffer Noring
 
Rxjs ngvikings
Christoffer Noring
 
Inversion Of Control
Chad Hietala
 
Designing The Right Schema To Power Heap (PGConf Silicon Valley 2016)
Dan Robinson
 
Integrating React.js with PHP projects
Ignacio MartĂ­n
 
Controller specs
Alexander Miller
 
201410 2 fiware-orion-contextbroker
FIWARE
 
Using Change Streams to Keep Up with Your Data
MongoDB
 
Bare-knuckle web development
Johannes Brodwall
 
Tweaking the interactive grid
Roel Hartman
 
Goal Based Data Production with Sim Simeonov
Databricks
 
Ad

Viewers also liked (6)

PDF
PHP Experience 2016 - [Palestra] Scaling with Microservice
iMasters
 
KEY
Building a-self-sufficient-team
Filippo De Santis
 
PDF
Symfony e micro (non così tanto) services
Michele Orselli
 
PDF
Exposing M2M to the REST of us
Matteo Collina
 
PDF
Vagrant for real (codemotion rome 2016)
Michele Orselli
 
PDF
Dependency Management with Composer
Jordi Boggiano
 
PHP Experience 2016 - [Palestra] Scaling with Microservice
iMasters
 
Building a-self-sufficient-team
Filippo De Santis
 
Symfony e micro (non così tanto) services
Michele Orselli
 
Exposing M2M to the REST of us
Matteo Collina
 
Vagrant for real (codemotion rome 2016)
Michele Orselli
 
Dependency Management with Composer
Jordi Boggiano
 
Ad

Similar to Server side data sync for mobile apps with silex (20)

PDF
WordPress Realtime - WordCamp SĂŁo Paulo 2015
Fernando Daciuk
 
PDF
Advanced redux
Boris Dinkevich
 
PDF
Reduxing like a pro
Boris Dinkevich
 
PPTX
Nodejs do teste de unidade ao de integração
VinĂ­cius Pretto da Silva
 
PPTX
SharePoint Conference 2018 - APIs, APIs everywhere!
SĂ©bastien Levert
 
KEY
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Tatsuhiko Miyagawa
 
PDF
Bootstrat REST APIs with Laravel 5
Elena Kolevska
 
PDF
Doctrine For Beginners
Jonathan Wage
 
PPTX
Reactive programming every day
Vadym Khondar
 
PDF
Burn down the silos! Helping dev and ops gel on high availability websites
Lindsay Holmwood
 
PDF
Complex Sites with Silex
Chris Tankersley
 
PDF
Writing RESTful web services using Node.js
FDConf
 
PDF
Bag Of Tricks From Iusethis
Marcus Ramberg
 
PPTX
REST API for your WP7 App
Agnius Paradnikas
 
PDF
Silex meets SOAP & REST
Hugo Hamon
 
PPTX
SharePoint Saturday Belgium 2018 - APIs, APIs everywhere!
SĂ©bastien Levert
 
PPTX
APIs, APIs Everywhere!
BIWUG
 
PPTX
Express JS
Alok Guha
 
PDF
Make WordPress realtime.
Josh Hillier
 
PPTX
Managing State in React Apps with RxJS by James Wright at FrontCon 2019
DevClub_lv
 
WordPress Realtime - WordCamp SĂŁo Paulo 2015
Fernando Daciuk
 
Advanced redux
Boris Dinkevich
 
Reduxing like a pro
Boris Dinkevich
 
Nodejs do teste de unidade ao de integração
VinĂ­cius Pretto da Silva
 
SharePoint Conference 2018 - APIs, APIs everywhere!
SĂ©bastien Levert
 
Remedie: Building a desktop app with HTTP::Engine, SQLite and jQuery
Tatsuhiko Miyagawa
 
Bootstrat REST APIs with Laravel 5
Elena Kolevska
 
Doctrine For Beginners
Jonathan Wage
 
Reactive programming every day
Vadym Khondar
 
Burn down the silos! Helping dev and ops gel on high availability websites
Lindsay Holmwood
 
Complex Sites with Silex
Chris Tankersley
 
Writing RESTful web services using Node.js
FDConf
 
Bag Of Tricks From Iusethis
Marcus Ramberg
 
REST API for your WP7 App
Agnius Paradnikas
 
Silex meets SOAP & REST
Hugo Hamon
 
SharePoint Saturday Belgium 2018 - APIs, APIs everywhere!
SĂ©bastien Levert
 
APIs, APIs Everywhere!
BIWUG
 
Express JS
Alok Guha
 
Make WordPress realtime.
Josh Hillier
 
Managing State in React Apps with RxJS by James Wright at FrontCon 2019
DevClub_lv
 

More from Michele Orselli (20)

PDF
Tackling Tech Debt with Rector
Michele Orselli
 
PDF
Comunicare, condividere e mantenere decisioni architetturali nei team di svil...
Michele Orselli
 
PDF
A dive into Symfony 4
Michele Orselli
 
PDF
A recommendation engine for your applications codemotion ams
Michele Orselli
 
PDF
A recommendation engine for your applications phpday
Michele Orselli
 
PDF
Hopping in clouds - phpuk 17
Michele Orselli
 
PDF
A recommendation engine for your php application
Michele Orselli
 
PDF
Hopping in clouds: a tale of migration from one cloud provider to another
Michele Orselli
 
PDF
Vagrant for real codemotion (moar tips! ;-))
Michele Orselli
 
PDF
Migrare a Symfony 3
Michele Orselli
 
PDF
Vagrant for real
Michele Orselli
 
PDF
Continuous, continuous, continuous
Michele Orselli
 
PDF
Deploy a PHP App on Google App Engine
Michele Orselli
 
PDF
Deploy a php app on Google App Engine
Michele Orselli
 
PDF
Sf2 wtf
Michele Orselli
 
PDF
Manage a project portfolio
Michele Orselli
 
PDF
Developing sustainable php projects
Michele Orselli
 
PDF
Zend Framework 2 per chi viene da Symfony2
Michele Orselli
 
PDF
BDD - Buzzword Driven Development - Build the next cool app for fun and for.....
Michele Orselli
 
PDF
Extreme automation
Michele Orselli
 
Tackling Tech Debt with Rector
Michele Orselli
 
Comunicare, condividere e mantenere decisioni architetturali nei team di svil...
Michele Orselli
 
A dive into Symfony 4
Michele Orselli
 
A recommendation engine for your applications codemotion ams
Michele Orselli
 
A recommendation engine for your applications phpday
Michele Orselli
 
Hopping in clouds - phpuk 17
Michele Orselli
 
A recommendation engine for your php application
Michele Orselli
 
Hopping in clouds: a tale of migration from one cloud provider to another
Michele Orselli
 
Vagrant for real codemotion (moar tips! ;-))
Michele Orselli
 
Migrare a Symfony 3
Michele Orselli
 
Vagrant for real
Michele Orselli
 
Continuous, continuous, continuous
Michele Orselli
 
Deploy a PHP App on Google App Engine
Michele Orselli
 
Deploy a php app on Google App Engine
Michele Orselli
 
Sf2 wtf
Michele Orselli
 
Manage a project portfolio
Michele Orselli
 
Developing sustainable php projects
Michele Orselli
 
Zend Framework 2 per chi viene da Symfony2
Michele Orselli
 
BDD - Buzzword Driven Development - Build the next cool app for fun and for.....
Michele Orselli
 
Extreme automation
Michele Orselli
 

Recently uploaded (20)

PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PDF
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
PPTX
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
PDF
medical staffing services at VALiNTRY
Tiyan Grayson
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
System and Network Administraation Chapter 3
jaramharo
 
PDF
AI in Product Development-omnex systems
omnex systems
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PPTX
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
PDF
Navsoft: AI-Powered Business Solutions & Custom Software Development
Navsoft
 
PDF
Nekopoi APK 2025 free lastest update
umarali35kp
 
PDF
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
PDF
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
PPTX
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
PPTX
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
ISO 45001 Occupational Health and Safety Management System
EHA Soft Solutions
 
medical staffing services at VALiNTRY
Tiyan Grayson
 
history of c programming in notes for students .pptx
Vijayakumari829030
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
System and Network Administraation Chapter 3
jaramharo
 
AI in Product Development-omnex systems
omnex systems
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
Navsoft: AI-Powered Business Solutions & Custom Software Development
Navsoft
 
Nekopoi APK 2025 free lastest update
umarali35kp
 
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
Transform Your Business with a Software ERP System
Canada Software Company
 

Server side data sync for mobile apps with silex

  • 1. Implementing data synchronization API for mobile apps with Silex
  • 2. Michele Orselli CTO@Ideato _orso_ micheleorselli / ideatosrl mo@ideato.it
  • 3. Agenda scenario design choices implementation alternative approaches
  • 9. Brownfield project Scenario several mobile apps for tracking user generated data (calendar, notes, bio data) iOS & Android ~10 K users steadily growing at 1.2 K/month
  • 10. MongoDB Scenario Legacy App based on Codeigniter Existing RPC-wannabe-REST API for data sync
  • 11. For every resource get updates: Scenario POST /m/:app/get/:user_id/:res/:updated_from send updates: POST /m/:app/update/:user_id/:res_id/:dev_id/:res
  • 12. api
  • 13. Scenario ~6 different resources, ~12 calls per sync apps sync by polling every 30 sec every call sync little data
  • 14. Challenge Rebuild sync API for old apps + 2 incoming Enable image synchronization More efficient than previous API
  • 16. Existing Solutions Tstamps, Vector clocks, CRDTs syncML, syncano Algorithms Protocols/API Azure Data sync Platform couchDB, riak Storage
  • 17. Not Invented Here? Don't Reinvent The Wheel, Unless You Plan on Learning More About Wheels J. Atwood
  • 18. Architecture 2 different mobile platforms Several teams with different skill level Changing storage wasn’t an option Forcing a particular technology client side wasn’t an option
  • 19. Architecture c1 server c2 c3 sync logic conflicts resolution thin clients
  • 20. Implementation In the sync domain all resources are managed in the same way
  • 21. Implementation For every app: one endpoint for getting new data one endpoint for pushing changes one endpoint for uploading images
  • 22. The new APIs GET /apps/:app/users/:user_id/changes[?from=:from] POST /apps/:app/users/:user_id/merge POST /upload/:res_id/images
  • 24. Silex Implementation Col 1 Col 2 Col 3
  • 25. Silex Implementation Col 1 Col 2 Col 3 Sync Service
  • 26. Silex Implementation Col 1 Col 2 Col 3 Sync Service
  • 27. Silex Implementation Col 1 Col 2 Col 3 Sync Service
  • 28. Silex Implementation Col 1 Col 2 Col 3 Sync Service
  • 29. Silex Implementation Col 1 Col 2 Col 3 Sync Service
  • 30. Silex Implementation Col 1 Col 2 Col 3 Sync Service
  • 31. Silex Implementation Col 1 Col 2 Col 3 Sync Service
  • 32. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/changes”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $syncService = $app[‘syncService’]; $syncService->sync($lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 33. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/changes”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $syncService = $app[‘syncService’]; $syncService->sync($lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 34. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/changes”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $syncService = $app[‘syncService’]; $syncService->sync($lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 35. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/changes”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $syncService = $app[‘syncService’]; $syncService->sync($lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 36. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 37. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 38. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 39. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 40. Silex Implementation $app->get(“/apps/{mApp}/users/{userId}/merge”, function ($mApp, $userId, $app, $request) { $lastSync = $request->get('from', null); $data = $request->get(‘data’, false); $syncService = $app[‘syncService’]; $syncService->merge($data, $lastSync, $userId); $response = new JsonResponse( $syncService->getResult() ); return $response; }
  • 41. Silex Implementation $app['mongodb'] = new MongoDb(
); $app[‘changesRepo’] = new ChangesRepository( $app[‘mongodb’] ); $app[‘syncService’] ? new SyncService( $app[‘changesRepo’] );
  • 42. Get changes GET /apps/:app/users/:user_id/changes?from=:from timestamp?
  • 43. Server suggest the sync time timestamp are inaccurate server suggests the “from” parameter to be used in the next request
  • 44. Server suggest the sync time GET /changes { ‘next’ : 12345, ‘data’: [
] } c1 server
  • 45. Server suggest the sync time GET /changes { ‘next’ : 12345, ‘data’: [
] } c1 server GET /changes?from=12345 { ‘next’ : 45678, ‘data’: [
] }
  • 46. operations: {‘op’: ’add’, id: ‘1’, ’data’:[
]} {‘op’: ’update’, id: ‘1’, ’data’:[
]} {‘op’: ’delete’, id: ‘1’} {‘op’: ’add’, id: ‘2’, ’data’:[
]} states: {id: ‘1’, ’data’:[
]} {id: 2’, ’data’:[
]} {id: ‘3’, ’data’:[
]} what to transfer
  • 47. what to transfer we choose to transfer states {id: ‘1’, ’type’: ‘measure’, ‘_deleted’: true} {id: 2’, ‘type’: ‘note’} {id: ‘3’, ‘type’: ‘note’} ps: soft delete all the things!
  • 48. unique identifiers How do we generate an unique id in a distributed system?
  • 49. unique identifiers How do we generate an unique id in a distributed system? UUID (RFC 4122): several implementations in PHP (https://github.com/ramsey/uuid)
  • 50. unique identifiers How do we generate an unique id in a distributed system? Local/Global Id: only the server generates GUIDs clients use local ids to manage their records
  • 51. unique identifiers POST /merge { ‘data’: [ {’lid’: ‘1’, 
}, {‘lid’: ‘2’, 
} ] } c1 server { ‘data’: [ {‘guid’: ‘58f0bdd7-1400’, ’lid’: ‘1’, 
}, {‘guid’: ‘6f9f3ec9-1400’, ‘lid’: ‘2’, 
} ] }
  • 52. conflict resolution algorithm (plain data) mobile generated data are “temporary” until sync to server server handles conflicts resolution
  • 53. conflict resolution algorithm (plain data) conflict resolution: domain indipendent: e.g. last-write wins domain dipendent: use domain knowledge to resolve
  • 54. function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data)
  • 55. function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data)
  • 56. function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data) no conflict
  • 57. function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data) remote wins
  • 58. function sync($data) { foreach ($data as $newRecord) { $s = findByGuid($newRecord->getGuid()); if (!$s) { add($newRecord); send($newRecord); continue; } if ($newRecord->updated > $s->updated) { update($s, $newRecord); send($newRecord); continue; } updateRemote($newRecord, $s); } conflict resolution algorithm (plain data) server wins
  • 59. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } server
  • 60. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
  • 61. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
  • 62. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ } c1 server
  • 63. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ } c1 server
  • 64. conflict resolution algorithm (plain data) { ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } { ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ } { ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge c1 server {‘ok’ : { ’guid’: ‘af54d’ }} {‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
  • 65. conflict resolution algorithm (hierarchical data) How to manage hierarchical data? { ‘lid’ : ‘123456’, ‘type’ : ‘baby’, 
 } { ‘lid’ : ‘123456’, ‘type’ : ‘temperature’, ‘baby_id : ‘123456’ }
  • 66. conflict resolution algorithm (hierarchical data) How to manage hierarchical data? 1) sync root record 2) update ids 3) sync child records { ‘lid’ : ‘123456’, ‘type’ : ‘baby’, 
 } { ‘lid’ : ‘123456’, ‘type’ : ‘temperature’, ‘baby_id : ‘123456’ }
  • 67. conflict resolution algorithm (hierarchical data) function syncHierarchical($data) { sortByHierarchy($data); foreach ($data as $newRootRecord) { $s = findByGuid($newRootRecord->getGuid()); if($newRecord->isRoot()) { if (!$s) { add($newRootRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } 

  • 68. conflict resolution algorithm (hierarchical data) function syncHierarchical($data) { sortByHierarchy($data); foreach ($data as $newRootRecord) { $s = findByGuid($newRootRecord->getGuid()); if($newRecord->isRoot()) { if (!$s) { add($newRootRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } 
 parent records first
  • 69. conflict resolution algorithm (hierarchical data) function syncHierarchical($data) { sortByHierarchy($data); foreach ($data as $newRootRecord) { $s = findByGuid($newRootRecord->getGuid()); if($newRecord->isRoot()) { if (!$s) { add($newRootRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } 

  • 70. conflict resolution algorithm (hierarchical data) function syncHierarchical($data) { sortByHierarchy($data); foreach ($data as $newRootRecord) { $s = findByGuid($newRootRecord->getGuid()); if($newRecord->isRoot()) { if (!$s) { add($newRootRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } 
 no conflict
  • 71. 
 if ($newRootRecord->updated > $s->updated) { update($s, $newRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } else { updateRecordIds($s, $data); updateRemote($newRecord, $s); } } else { sync($data); } } conflict resolution algorithm (hierarchical data) remote wins
  • 72. 
 if ($newRootRecord->updated > $s->updated) { update($s, $newRecord); updateRecordIds($newRootRecord, $data); send($newRootRecord); continue; } else { updateRecordIds($s, $data); updateRemote($newRecord, $s); } } else { sync($data); } } conflict resolution algorithm (hierarchical data) server wins
  • 73. conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge c1 server
  • 74. conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
  • 75. conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
  • 76. conflict resolution algorithm (hierarchical data) { ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } c1 server POST /merge { ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ } { ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } {‘update’ : { ‘lid’: ‘1’, ’guid’: ‘af54d’ }} {‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
  • 77. enforcing domain constraints e.g. “only one temperature can be registered in a given day” how to we enforce domain constraints on data?
  • 78. enforcing domain constraints e.g. “only one temperature can be registered in a given day” how to we enforce domain constraints on data? 1) relax constraints
  • 79. enforcing domain constraints e.g. “only one temperature can be registered in a given day” how to we enforce domain constraints on data? 1) relax constraints 2) integrate constraints in sync algorithm
  • 80. enforcing domain constraints from findByGuid to findSimilar first lookup by GUID then by domain rules “two measures are similar if are referred to the same date”
  • 82. enforcing domain constraints { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } c1 server
  • 83. enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } c1 server
  • 84. enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server
  • 85. enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server
  • 86. enforcing domain constraints { ‘lid’: ‘1’, ‘when’: ‘20141005’ } { ’guid’: ‘af54d’, ‘when’: ‘20141005’ } POST /merge c1 server { ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
  • 87. dealing with binary data Binary data uploaded via custom endpoint Sync data remains small Uploads can be resumed
  • 88. dealing with binary data Two steps* 1) data are synchronized 2) related images are uploaded * this means record without file for a given time
  • 89. dealing with binary data POST /merge { ‘lid’ : 1, ‘type’ : ‘baby’, ‘image’ : ‘myimage.jpg’ } { ‘lid’ : 1, ‘guid’ : ‘ac435-f8345’ } c1 server POST /upload/ac435-f8345/image
  • 90. What we learned Implementing this stuff is tricky Explore existing solution if you can Understanding the domain is important
  • 93. CRDT Conflict-free Replicated Data Types (CRDTs) Constraining the types of operations in order to: - ensure convergence of changes to shared data by uncoordinated, concurrent actors - eliminate network failure modes as a source of error
  • 94. Couchbase Mobile Gateways handles sync Data flows through channels - partition data set - authorization - limit the data Use revision trees
  • 95. Riak Distributed DB Eventually/Strong Consistency Data Types Configurable conflict resolution - db level for built-in data types - application level for custom data
  • 96. That’s all folks! Questions? Please leave feedback! https://joind.in/12959
  • 97. http://www.objc.io/issue-10/sync-case-study.html http://www.objc.io/issue-10/data-synchronization.html https://dev.evernote.com/media/pdf/edam-sync.pdf http://blog.helftone.com/clear-in-the-icloud/ http://strongloop.com/strongblog/node-js-replication-mobile-offline-sync-loopback/ http://blog.denivip.ru/index.php/2014/04/data-syncing-in-core-data-based-ios-apps/?lang=en http://inessential.com/2014/02/15/vesper_sync_diary_8_the_problem_of_un http://culturedcode.com/things/blog/2010/12/state-of-sync-part-1.html http://programmers.stackexchange.com/questions/206310/data-synchronization-in-mobile-apps-multiple- devices-multiple-users http://bricklin.com/offline.htm http://blog.couchbase.com/why-mobile-sync Links
  • 98. Links Vector Clocks http://basho.com/why-vector-clocks-are-easy/ http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clocks http://basho.com/why-vector-clocks-are-hard/ http://blog.8thlight.com/rylan-dirksen/2013/10/04/synchronization-in-a-distributed-system.html CRDTs http://christophermeiklejohn.com/distributed/systems/2013/07/12/readings-in-distributed-systems.html http://www.infoq.com/presentations/problems-distributed-systems https://www.youtube.com/watch?v=qyVNG7fnubQ Riak http://docs.basho.com/riak/latest/dev/using/conflict-resolution/ Couchbase Sync Gateway http://docs.couchbase.com/sync-gateway/ http://www.infoq.com/presentations/sync-mobile-data API http://developers.amiando.com/index.php/REST_API_DataSync https://login.syncano.com/docs/rest/index.html
  • 99. Credits phones https://www.flickr.com/photos/15216811@N06/14504964841 wat http://uturncrossfit.com/wp-content/uploads/2014/04/wait-what.jpg darth http://www.listal.com/viewimage/3825918h blueprint: http://upload.wikimedia.org/wikipedia/commons/5/5e/Joy_Oil_gas_station_blueprints.jpg building: http://s0.geograph.org.uk/geophotos/02/42/74/2427436_96c4cd84.jpg brownfield: http://s0.geograph.org.uk/geophotos/02/04/54/2045448_03a2fb36.jpg no connection: https://www.flickr.com/photos/77018488@N03/9004800239 no internet con https://www.flickr.com/photos/roland/9681237793 vector clocks: http://en.wikipedia.org/wiki/Vector_clock crdts: http://www.infoq.com/presentations/problems-distributed-systems