SlideShare a Scribd company logo

Handling Changes to Your Server-Side Data Model

Gilt Tech Talks, profile picture
Gilt Tech Talks

The document discusses challenges in handling server-side data model changes for mobile applications, particularly comparing web and iOS platforms. It uses the example of a hypothetical social network, Chirper, to illustrate how inefficient data handling can arise from re-sending unchanged messages even with newly applied tags. The introduction of mockingbird expressions is proposed as a solution to decouple the client and server data structures, allowing more efficient queries without requiring advanced knowledge of the server's data model.

Technology
1 of 47
Downloaded 15 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
Evan Coyne Maloney Principal Engineer Mobile Gracefully handling changes to your server-side data model
Mobile Things change. …sometimes more often than we’d like.
Mobile Handling change on the Web • No app review process • “Go live” with the push of a button • Immediate deployment to production • All users get changes right away • No need to test “on device” • Won’t spend hours in provisioning hell • Easy to perform automated testing on the hardware you’ll be using in production Handling change Web vs. iOS
Mobile Handling change on iOS • Mandatory app review • Can take a week or more • Must test “on device” • If you only test in the simulator, you don’t really know what your code is doing • How does it perform? • It is exhibiting device-only crashing? Handling change Web vs. iOS
Mobile Web vs. iOS Handling change Web vs. iOS Web iOS Fully automated testing Partially manual testing Push a button and go live Going live requires Apple’s permission, which in turn requires waiting Can fix stuff right away Being able to fix stuff right away requires careful planning and lots of work (…and even then you can’t fix everything)
Mobile These differences have consequences Your iOS apps can become an anchor on your server-side environment Handling change Web vs. iOS
Mobile An Example
Mobile Chirper Handling change An Example A hypothetical social network • Unidirectional relationships between users (following) • Encourages very short messages; optimized for mobile input • Allows followers to tag a message to categorize it
Mobile Chirper Handling change An Example A hypothetical social network • Launched by an early-stage startup with limited resources • Built a simple system to launch quickly • Having trouble scaling • Need to bring hosting and bandwidth costs down
Mobile Chirper’s Message Feed Handling change An Example • The feed contains all messages that: • Are new to the user • Were delivered previously, but have newly-applied tags [{ "message_id": 10271972, "message": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert"] }] • When the Chirper app launches, it retrieves the updated message feed from the server • Messages in the feed arrive within a JSON array:
Mobile Not the most efficient Handling change An Example • Any time anyone tags a message, the entire message will be re-sent in the feed update, even though previously-received messages are already stored in the client app • Tags themselves can add a lot of redundant data, especially in heavily active feeds with long tag names • Chirper has decided to refactor their feeds to be more efficient
Mobile An example scenario Handling change An Example 1. Someone adds a new tag to an old message 2. Two new messages arrive
Mobile How it looks in the current feed Handling change An Example [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] 503 bytes (compact JSON)
Mobile A refactored feed Handling change An Example { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so much better than the album version.", "author_id": 6022 }, "10272012": { "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216 } } } 344 bytes (compact JSON)
Mobile Success? Handling change An Example • The response size is now about ⅔ of what it was • Savings will be even better under many scenarios Great, but… These savings won’t lower Chirper’s bandwidth costs until: 1. They build and submit a new app 2. Apple approves it 3. A critical mass of users update to the latest version
Mobile The root of the problem Handling change Designing for change • The client app and the server must always agree on the structure of the data • On the client side, this knowledge has to be compiled in to the application binary • Once the app has been submitted, it can’t be modified • But, we can design a way around this!
Mobile How we build server-driven apps today Handling change Designing for change 1. The server maintains knowledge of the data necessary to drive the app 2. The app requests data, and receives a data structure in response 3. The app will “ask questions of the data” to do its work 4. This requires having knowledge of the data structure
Mobile How we build server-driven apps today Handling change Designing for change 1. The server maintains knowledge of the data necessary to drive the app 2. The app requests data, and receives a data structure in response 3. The app will “ask questions of the data” to do its work 4. This requires having knowledge of the data structure
Mobile How we could build server-driven apps Handling change Designing for change 1. The server maintains knowledge of the data necessary to drive the app 2. The app requests data, and receives in response: • The data itself • A set of instructions for how to extract values from the data structure 3. The app will still “ask questions of the data” to do its work, but it will use the server-provided instructions to do it — the client no longer needs advance knowledge of the server’s data model!
Mobile 1. The exact structure of the data 2. What questions it needs answered from the data 3. How to translate those questions into procedures for extracting answers from the data The Client needs to know: The Server needs to know: 1. The exact structure of the data Rethink where we require knowledge Handling change Designing for change
Mobile Rethink where we require knowledge Handling change Designing for change 1. What questions it needs answered from the data 1. The exact structure of the data 2. The set of questions each app version needs answered from the data 3. How to translate those questions into procedures for extracting answers from the data The Client needs to know: The Server needs to know:
Mobile How can we do this? Handling change Designing for change
Mobile Introducing Mockingbird Expressions
Mobile • Text strings that specify values within an iOS application’s runtime environment • Objective-C and Swift variables • Server-side data models • Device information • Screen size, system clock • OS-level information • iOS version, system permissions, user settings • Provided by the Mockingbird Data Environment library Mockingbird Expressions An Introduction Mockingbird Expressions What are they?
Mobile Mockingbird Expressions An Introduction Mockingbird Expressions What do they look like? Variables Functions The expression: The expression: $foo ^currentTime() Returns the object bound to the Mockingbird variable name “foo”. Returns an NSDate object containing the current time. $dict[key].propVal ^formatLongDate(^currentTime()) Selects the object associated with key from the NSDictionary bound to the name “dict”, and returns the value of its propVal property. Returns an NSString containing the current date in the format “August 11, 2014”. $UIDevice.model ^filter($users|$item.firstName == Jill) Equivalent to the Objective-C code: [[UIDevice currentDevice] model] Returns a subset of the collection $users containing only the items whose firstName property equals “Jill”.
Mobile We can use Mockingbird Expressions to describe how to extract answers from data Mockingbird Expressions An Introduction
Mobile Chirper + Mockingbird
Mobile 1. What are the IDs of the new messages? 2. What are the IDs of the messages with newly- applied tags? 3. What is the content of a given message? 4. Who is the author of a given message? 5. Which tags are associated with a given message? Mockingbird Expressions A Use Case Questions Chirper asks its data
Mobile [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case 1. What are the IDs of the new messages? ^list(^filter($feed|!$item.is_tag_update)|$item.message_id)Old feed New feed^list($feed.messages|$key)
Mobile 2. What are the IDs of the messages with newly-applied tags? [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case ^list(^filter($feed|$item.is_tag_update)|$item.message_id) ^filter(^list($feed.message_tags|$key)|!$feed.messages[$item]) Old feed New feed
Mobile 3. What is the content of a given message? [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case ^associate($feed|$item.message_id|$item.content)[$id] $feed.messages[$id].content Old feed New feed
Mobile 4. Who is the author of a given message? [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case ^associate($feed|$item.message_id|$item.author_id)[$id] $feed.messages[$id].author_id Old feed New feed
Mobile 5. Which tags are associated with a given message? [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case ^associate($feed|$item.message_id|$item.tags)[$id] ^list($feed.message_tags[$id]|$feed.tags[$item]) Old feed New feed
Mobile { "new_message_ids": "^list(^filter($feed|!$item.is_tag_update)|$item.message_id)", "newly_tagged_message_ids": "^list(^filter($feed|$item.is_tag_update)|$item.message_id)", "content_for_message_with_id": "^associate($feed|$item.message_id|$item.content)[$id]", "author_id_for_message_with_id": "^associate($feed|$item.message_id|$item.author_id)[$id]", "tags_for_message_with_id": "^associate($feed|$item.message_id|$item.tags)[$id]" } Mockingbird Expressions A Use Case Old feed We can now separate out the expressions needed to query the data { "new_message_ids": "^list($feed.messages|$key)", "newly_tagged_message_ids": "^filter(^list($feed.message_tags|$key)|!$feed.messages[$item])", "content_for_message_with_id": "$feed.messages[$id].content", "author_id_for_message_with_id": "$feed.messages[$id].author_id", "tags_for_message_with_id": "^list($feed.message_tags[$id]|$feed.tags[$item])" } New feed
Mobile This architecture now relies on “model/query decoupling” wherein Mockingbird Expressions A Use Case The ability to query a set of data is separated from knowledge of that data’s structure
Mobile How would this work in practice?
Mobile 1. The server maintains a schema version for the data model it uses • Whenever a non-backwards-compatible schema change is introduced, the schema version increments 2. Any time the schema version changes, the server publishes a corresponding set of query expressions for that schema version 3. When an app fetches new data, it checks the schema version returned by the server. If the app doesn’t have the query expressions for that version, it can get them from the server • Apps will cache query expressions • Apps can include a compiled-in set of query expressions to avoid having to download them the first time an app launches Mockingbird Expressions Code Samples Model/Query Decoupling using Mockingbird Expressions
Mobile 1. When we retrieve feed data from the server, we expose it to the variable space: Mockingbird Expressions Code Samples Populating the Mockingbird Data Environment NSData* json = ... // Unicode text containing JSON structure from server id feedData = [NSJSONSerialization JSONObjectWithData:json options:0 error:nil]; [[MBVariableSpace instance] setVariable:@"feed" value:feedData]; The expression $feed now refers to the value of feedData 2. We load the query expressions associated with the current schema into a dictionary and expose it to the variable space: NSDictionary* queries = ... // query names -> expressions [[MBVariableSpace instance] setVariable:@"queries" value:queries]; The expression $queries now refers to the NSDictionary queries
Mobile 3. Query for new messages: Mockingbird Expressions Code Samples Querying the data NSString* query = [MBExpression asString:@"$queries.new_message_ids"]; NSArray* newMessageIDs = [MBExpression asObject:query]; The array newMessageIDs contains two elements: ( 10272003, 10272012 ) 4. Query for messages with updated tags: NSString* query = [MBExpression asString:@"$queries.newly_tagged_message_ids"]; NSArray* newlyTaggedMessageIDs = [MBExpression asObject:query]; The array newlyTaggedMessageIDs contains one element: ( 10271972 ) 5. Get the content of a message given a message ID: [[MBVariableSpace instance] setVariable:@"id" value:@(10272003)]; NSString* query = [MBExpression asString:@"$queries.content_for_message_with_id"]; NSString* content = [MBExpression asNumber:query]; The string content is “I like the live Penelope so much better than the album version.”
Mobile Mockingbird Expressions Code Samples Querying the data 7. Get the tags associated with a given message: [[MBVariableSpace instance] setVariable:@"id" value:@(10272012)]; NSString* query = [MBExpression asString:@"$queries.tags_for_message_with_id"]; NSArray* tags = [MBExpression asObject:query]; The array tags contains three elements: ( "Pinback", "concert", "live music" ) 6. Get the author ID of a message given a message ID: [[MBVariableSpace instance] setVariable:@"id" value:@(10271972)]; NSString* query = [MBExpression asString:@"$queries.author_id_for_message_with_id"]; NSNumber* authorID = [MBExpression asNumber:query]; The NSNumber authorID contains the integer value 5375.
Mobile What can you use this for?
Mobile 1. No App Store update required to introduce support for new schemas • New sets of query expressions can be downloaded to add immediate support for new schemas 2. Maintain legacy support while evolving your services • You won’t need to run multiple versions of your backend just to maintain legacy support 3. Simplify your backend codebase • Avoid spaghetti code — no need for version-check conditionals littered throughout your server-side codebase 4. A/B testing made easy • Want to support multiple schemas simultaneously? No problem! Just switch between sets of query expressions. This makes server-driven A/B testing easy. Mockingbird Expressions Use Cases Benefits
Mobile When can you use this?
Mobile Mockingbird Availability Mockingbird Data Environment availability The Mockingbird Data Environment is part of the Mockingbird Library open-source project from Gilt Groupe. Coming Fall 2014
Mobile Mockingbird Data Environment availability Available today! Coming Fall 2014 Mockingbird Availability The Mockingbird Data Environment is part of the Mockingbird Library open-source project from Gilt Groupe.
Mobile Mockingbird Toolbox open source project https://github.com/gilt/mockingbird-toolbox Mockingbird Availability available at:
Evan Coyne Maloney Principal Engineer Mobile Gracefully handling changes to your server-side data model

More Related Content

PDF
iOS Testing With Appium at Gilt
Gilt Tech Talks
 
PDF
Mobile Testing at Gilt
Gilt Tech Talks
 
PDF
Screenplay - Next generation automated acceptance testing
John Ferguson Smart Limited
 
PDF
Automating Mobile Testing at Gilt with Appium
Sauce Labs
 
PPTX
Yelp Tech Talks: Mobile Testing 1, 2, 3
Yelp Engineering
 
PDF
Interaction Design L04 - Materialise and Coupling
Mohammad Shaker
 
PDF
All the world's a stage – the next step in automated testing practices
John Ferguson Smart Limited
 
PPTX
Humans by the hundred
Yelp Engineering
 
iOS Testing With Appium at Gilt
Gilt Tech Talks
 
Mobile Testing at Gilt
Gilt Tech Talks
 
Screenplay - Next generation automated acceptance testing
John Ferguson Smart Limited
 
Automating Mobile Testing at Gilt with Appium
Sauce Labs
 
Yelp Tech Talks: Mobile Testing 1, 2, 3
Yelp Engineering
 
Interaction Design L04 - Materialise and Coupling
Mohammad Shaker
 
All the world's a stage – the next step in automated testing practices
John Ferguson Smart Limited
 
Humans by the hundred
Yelp Engineering
 

What's hot (18)

PDF
What TDD Can Teach Us About API Design
Jon Phenow
 
PPTX
Microservices Summit - The Human Side of Services
Yelp Engineering
 
PDF
Shawn Wallace - Test automation in brownfield applications
QA or the Highway
 
PDF
How to Build a 5-star App
Nudge Software Inc.
 
PDF
Using Prototypes to Validate Product Strategy - Product Camp DC Presentation ...
3Pillar Global
 
PPTX
Continuous delivery in the cloud - DevConFu 2014
Kristoffer Deinoff
 
PDF
Goodtests
fxsperling
 
PPTX
An Overview of Selenium
adamcarmi
 
PDF
Shifting is more than shifting left
PractiTest
 
PPTX
DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney
Andreas Grabner
 
PPTX
Automated Visual Testing in NSW.Gov.AU
Applitools
 
PDF
How Optimizely Scaled its REST API with asyncio
Optimizely
 
PDF
How We Became World Leaders in Agile Data Warehousing - Teradata Partners - O...
Em Campbell-Pretty
 
PPTX
State of the Union - Mobile Performance
SOASTA
 
PPTX
BDD Selenium for Agile Teams - User Stories
Sauce Labs
 
PDF
Translating Tester-Speak Into Plain English: Simple Explanations for 8 Testin...
Neotys
 
PDF
8 Blind Spots Often Overlooked When Testing on Mobile
Neotys
 
PPTX
ONE-SIZE DOESN'T FIT ALL - EFFECTIVELY (RE)EVALUATE A DATA SOLUTION FOR YOUR ...
DevOpsDays Tel Aviv
 
What TDD Can Teach Us About API Design
Jon Phenow
 
Microservices Summit - The Human Side of Services
Yelp Engineering
 
Shawn Wallace - Test automation in brownfield applications
QA or the Highway
 
How to Build a 5-star App
Nudge Software Inc.
 
Using Prototypes to Validate Product Strategy - Product Camp DC Presentation ...
3Pillar Global
 
Continuous delivery in the cloud - DevConFu 2014
Kristoffer Deinoff
 
Goodtests
fxsperling
 
An Overview of Selenium
adamcarmi
 
Shifting is more than shifting left
PractiTest
 
DevOps and Performance - Why, How and Best Practices - DevOps Meetup Sydney
Andreas Grabner
 
Automated Visual Testing in NSW.Gov.AU
Applitools
 
How Optimizely Scaled its REST API with asyncio
Optimizely
 
How We Became World Leaders in Agile Data Warehousing - Teradata Partners - O...
Em Campbell-Pretty
 
State of the Union - Mobile Performance
SOASTA
 
BDD Selenium for Agile Teams - User Stories
Sauce Labs
 
Translating Tester-Speak Into Plain English: Simple Explanations for 8 Testin...
Neotys
 
8 Blind Spots Often Overlooked When Testing on Mobile
Neotys
 
ONE-SIZE DOESN'T FIT ALL - EFFECTIVELY (RE)EVALUATE A DATA SOLUTION FOR YOUR ...
DevOpsDays Tel Aviv
 
Ad

Similar to Handling Changes to Your Server-Side Data Model (20)

PDF
The Developers World
Ronald Northrip
 
PDF
Nt1310 Unit 3 Language Analysis
Nicole Gomez
 
PDF
Web application intro
Tobias Pfeiffer
 
PDF
08 ajax
Ynon Perek
 
PDF
iPhone Python love affair
Anna Callahan
 
PDF
Web application intro + a bit of ruby (revised)
Tobias Pfeiffer
 
PDF
Emerging Languages: A Tour of the Horizon
Alex Payne
 
PDF
Why Our Code Smells
TiNguyn863920
 
PDF
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...
jnewland
 
PDF
Building native mobile apps using web technologies
Hjörtur Hilmarsson
 
PDF
Mobile Development Architecture Ppt with Slides, Book Notes on using Web Silv...
Bala Subra
 
PDF
Web Service and Mobile Integrated Day I
Anuchit Chalothorn
 
PDF
CI/CD non-breaking changes exercise - Cork Software Crafters - February 2020
Paulo Clavijo
 
PDF
Going Mobile !
Christopher Duffy
 
PDF
Software development - the java perspective
Alin Pandichi
 
PDF
CV_Serhiy_Medvedyev_2015
Serhiy Medvedyev
 
PDF
Phonegap deep-dive
alunny
 
PDF
Iphone lecture imp
Pragati Singh
 
PDF
Angular.js - JS Camp UKraine 2013
Max Klymyshyn
 
PDF
DDD/CQRS - I must learn to repeat myself
Douglas Reith
 
The Developers World
Ronald Northrip
 
Nt1310 Unit 3 Language Analysis
Nicole Gomez
 
Web application intro
Tobias Pfeiffer
 
08 ajax
Ynon Perek
 
iPhone Python love affair
Anna Callahan
 
Web application intro + a bit of ruby (revised)
Tobias Pfeiffer
 
Emerging Languages: A Tour of the Horizon
Alex Payne
 
Why Our Code Smells
TiNguyn863920
 
Continuous (Production) Integration: Ruby on Rails Application Monitoring wit...
jnewland
 
Building native mobile apps using web technologies
Hjörtur Hilmarsson
 
Mobile Development Architecture Ppt with Slides, Book Notes on using Web Silv...
Bala Subra
 
Web Service and Mobile Integrated Day I
Anuchit Chalothorn
 
CI/CD non-breaking changes exercise - Cork Software Crafters - February 2020
Paulo Clavijo
 
Going Mobile !
Christopher Duffy
 
Software development - the java perspective
Alin Pandichi
 
CV_Serhiy_Medvedyev_2015
Serhiy Medvedyev
 
Phonegap deep-dive
alunny
 
Iphone lecture imp
Pragati Singh
 
Angular.js - JS Camp UKraine 2013
Max Klymyshyn
 
DDD/CQRS - I must learn to repeat myself
Douglas Reith
 
Ad

More from Gilt Tech Talks (11)

PDF
“Get Stuff Done Faster: Why Engineers Should Work with the ‘Dark Side’ of Tech”
Gilt Tech Talks
 
PPTX
Optimizely at Gilt--November 2013 presentation
Gilt Tech Talks
 
PDF
The Agile PMO (fall 2014 version)
Gilt Tech Talks
 
PDF
An Intro to Swift
Gilt Tech Talks
 
PDF
Beyond the Crystal Ball: The Agile PMO
Gilt Tech Talks
 
PDF
Exploring Docker at Gilt
Gilt Tech Talks
 
PDF
Scaling Gilt: from monolith ruby app to micro service scala service architecture
Gilt Tech Talks
 
PDF
PostgreSQL Setup Using Docker
Gilt Tech Talks
 
PDF
Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...
Gilt Tech Talks
 
PDF
Scala Self Types by Gregor Heine, Principal Software Engineer at Gilt
Gilt Tech Talks
 
PDF
Virtualization at Gilt
Gilt Tech Talks
 
“Get Stuff Done Faster: Why Engineers Should Work with the ‘Dark Side’ of Tech”
Gilt Tech Talks
 
Optimizely at Gilt--November 2013 presentation
Gilt Tech Talks
 
The Agile PMO (fall 2014 version)
Gilt Tech Talks
 
An Intro to Swift
Gilt Tech Talks
 
Beyond the Crystal Ball: The Agile PMO
Gilt Tech Talks
 
Exploring Docker at Gilt
Gilt Tech Talks
 
Scaling Gilt: from monolith ruby app to micro service scala service architecture
Gilt Tech Talks
 
PostgreSQL Setup Using Docker
Gilt Tech Talks
 
Roadmap for the Trillion Sensor Universe -- a Gilt-hosted, Internet of Things...
Gilt Tech Talks
 
Scala Self Types by Gregor Heine, Principal Software Engineer at Gilt
Gilt Tech Talks
 
Virtualization at Gilt
Gilt Tech Talks
 

Recently uploaded (20)

PPT
What is a Computer? Input Devices /output devices
GulJan8
 
PDF
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
PDF
WOOl fibre morphology and structure.pdf for textiles
Rajendrakumar868651
 
PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PPTX
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
PDF
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
PDF
project resource management chapter-09.pdf
ssuser00e6e21
 
PDF
Developing a website for English-speaking practice to English as a foreign la...
IAESIJAI
 
PDF
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
PDF
2021 HotChips TSMC Packaging Technologies for Chiplets and 3D_0819 publish_pu...
KeXue5
 
PPTX
OMC Textile Division Presentation 2021.pptx
JahangirAlam816069
 
PDF
TrustArc Webinar - Click, Consent, Trust: Winning the Privacy Game
TrustArc
 
PDF
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
PDF
Getting Started with Data Integration: FME Form 101
Safe Software
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PDF
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
PDF
Microsoft Solutions Partner Drive Digital Transformation with D365.pdf
Microsoft Dynamics
 
PDF
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
PDF
Web App vs Mobile App What Should You Build First.pdf
KodekX
 
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
What is a Computer? Input Devices /output devices
GulJan8
 
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
WOOl fibre morphology and structure.pdf for textiles
Rajendrakumar868651
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
MicrosoftCybserSecurityReferenceArchitecture-April-2025.pptx
SilvioHayashi
 
A novel scalable deep ensemble learning framework for big data classification...
IAESIJAI
 
project resource management chapter-09.pdf
ssuser00e6e21
 
Developing a website for English-speaking practice to English as a foreign la...
IAESIJAI
 
Univ-Connecticut-ChatGPT-Presentaion.pdf
ry7r52mzb4
 
2021 HotChips TSMC Packaging Technologies for Chiplets and 3D_0819 publish_pu...
KeXue5
 
OMC Textile Division Presentation 2021.pptx
JahangirAlam816069
 
TrustArc Webinar - Click, Consent, Trust: Winning the Privacy Game
TrustArc
 
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
Getting Started with Data Integration: FME Form 101
Safe Software
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
How ambidextrous entrepreneurial leaders react to the artificial intelligence...
IAESIJAI
 
Microsoft Solutions Partner Drive Digital Transformation with D365.pdf
Microsoft Dynamics
 
Architecture types and enterprise applications.pdf
ChristopherTHyatt
 
Web App vs Mobile App What Should You Build First.pdf
KodekX
 
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 

Handling Changes to Your Server-Side Data Model

  • 1. Evan Coyne Maloney Principal Engineer Mobile Gracefully handling changes to your server-side data model
  • 2. Mobile Things change. …sometimes more often than we’d like.
  • 3. Mobile Handling change on the Web • No app review process • “Go live” with the push of a button • Immediate deployment to production • All users get changes right away • No need to test “on device” • Won’t spend hours in provisioning hell • Easy to perform automated testing on the hardware you’ll be using in production Handling change Web vs. iOS
  • 4. Mobile Handling change on iOS • Mandatory app review • Can take a week or more • Must test “on device” • If you only test in the simulator, you don’t really know what your code is doing • How does it perform? • It is exhibiting device-only crashing? Handling change Web vs. iOS
  • 5. Mobile Web vs. iOS Handling change Web vs. iOS Web iOS Fully automated testing Partially manual testing Push a button and go live Going live requires Apple’s permission, which in turn requires waiting Can fix stuff right away Being able to fix stuff right away requires careful planning and lots of work (…and even then you can’t fix everything)
  • 6. Mobile These differences have consequences Your iOS apps can become an anchor on your server-side environment Handling change Web vs. iOS
  • 8. Mobile Chirper Handling change An Example A hypothetical social network • Unidirectional relationships between users (following) • Encourages very short messages; optimized for mobile input • Allows followers to tag a message to categorize it
  • 9. Mobile Chirper Handling change An Example A hypothetical social network • Launched by an early-stage startup with limited resources • Built a simple system to launch quickly • Having trouble scaling • Need to bring hosting and bandwidth costs down
  • 10. Mobile Chirper’s Message Feed Handling change An Example • The feed contains all messages that: • Are new to the user • Were delivered previously, but have newly-applied tags [{ "message_id": 10271972, "message": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert"] }] • When the Chirper app launches, it retrieves the updated message feed from the server • Messages in the feed arrive within a JSON array:
  • 11. Mobile Not the most efficient Handling change An Example • Any time anyone tags a message, the entire message will be re-sent in the feed update, even though previously-received messages are already stored in the client app • Tags themselves can add a lot of redundant data, especially in heavily active feeds with long tag names • Chirper has decided to refactor their feeds to be more efficient
  • 12. Mobile An example scenario Handling change An Example 1. Someone adds a new tag to an old message 2. Two new messages arrive
  • 13. Mobile How it looks in the current feed Handling change An Example [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] 503 bytes (compact JSON)
  • 14. Mobile A refactored feed Handling change An Example { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so much better than the album version.", "author_id": 6022 }, "10272012": { "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216 } } } 344 bytes (compact JSON)
  • 15. Mobile Success? Handling change An Example • The response size is now about ⅔ of what it was • Savings will be even better under many scenarios Great, but… These savings won’t lower Chirper’s bandwidth costs until: 1. They build and submit a new app 2. Apple approves it 3. A critical mass of users update to the latest version
  • 16. Mobile The root of the problem Handling change Designing for change • The client app and the server must always agree on the structure of the data • On the client side, this knowledge has to be compiled in to the application binary • Once the app has been submitted, it can’t be modified • But, we can design a way around this!
  • 17. Mobile How we build server-driven apps today Handling change Designing for change 1. The server maintains knowledge of the data necessary to drive the app 2. The app requests data, and receives a data structure in response 3. The app will “ask questions of the data” to do its work 4. This requires having knowledge of the data structure
  • 18. Mobile How we build server-driven apps today Handling change Designing for change 1. The server maintains knowledge of the data necessary to drive the app 2. The app requests data, and receives a data structure in response 3. The app will “ask questions of the data” to do its work 4. This requires having knowledge of the data structure
  • 19. Mobile How we could build server-driven apps Handling change Designing for change 1. The server maintains knowledge of the data necessary to drive the app 2. The app requests data, and receives in response: • The data itself • A set of instructions for how to extract values from the data structure 3. The app will still “ask questions of the data” to do its work, but it will use the server-provided instructions to do it — the client no longer needs advance knowledge of the server’s data model!
  • 20. Mobile 1. The exact structure of the data 2. What questions it needs answered from the data 3. How to translate those questions into procedures for extracting answers from the data The Client needs to know: The Server needs to know: 1. The exact structure of the data Rethink where we require knowledge Handling change Designing for change
  • 21. Mobile Rethink where we require knowledge Handling change Designing for change 1. What questions it needs answered from the data 1. The exact structure of the data 2. The set of questions each app version needs answered from the data 3. How to translate those questions into procedures for extracting answers from the data The Client needs to know: The Server needs to know:
  • 22. Mobile How can we do this? Handling change Designing for change
  • 24. Mobile • Text strings that specify values within an iOS application’s runtime environment • Objective-C and Swift variables • Server-side data models • Device information • Screen size, system clock • OS-level information • iOS version, system permissions, user settings • Provided by the Mockingbird Data Environment library Mockingbird Expressions An Introduction Mockingbird Expressions What are they?
  • 25. Mobile Mockingbird Expressions An Introduction Mockingbird Expressions What do they look like? Variables Functions The expression: The expression: $foo ^currentTime() Returns the object bound to the Mockingbird variable name “foo”. Returns an NSDate object containing the current time. $dict[key].propVal ^formatLongDate(^currentTime()) Selects the object associated with key from the NSDictionary bound to the name “dict”, and returns the value of its propVal property. Returns an NSString containing the current date in the format “August 11, 2014”. $UIDevice.model ^filter($users|$item.firstName == Jill) Equivalent to the Objective-C code: [[UIDevice currentDevice] model] Returns a subset of the collection $users containing only the items whose firstName property equals “Jill”.
  • 26. Mobile We can use Mockingbird Expressions to describe how to extract answers from data Mockingbird Expressions An Introduction
  • 28. Mobile 1. What are the IDs of the new messages? 2. What are the IDs of the messages with newly- applied tags? 3. What is the content of a given message? 4. Who is the author of a given message? 5. Which tags are associated with a given message? Mockingbird Expressions A Use Case Questions Chirper asks its data
  • 29. Mobile [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case 1. What are the IDs of the new messages? ^list(^filter($feed|!$item.is_tag_update)|$item.message_id)Old feed New feed^list($feed.messages|$key)
  • 30. Mobile 2. What are the IDs of the messages with newly-applied tags? [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case ^list(^filter($feed|$item.is_tag_update)|$item.message_id) ^filter(^list($feed.message_tags|$key)|!$feed.messages[$item]) Old feed New feed
  • 31. Mobile 3. What is the content of a given message? [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case ^associate($feed|$item.message_id|$item.content)[$id] $feed.messages[$id].content Old feed New feed
  • 32. Mobile 4. Who is the author of a given message? [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case ^associate($feed|$item.message_id|$item.author_id)[$id] $feed.messages[$id].author_id Old feed New feed
  • 33. Mobile 5. Which tags are associated with a given message? [{ "message_id": 10271972, "content": "Is there going to be an encore? I really hope they play Penelope!", "author_id": 5375, "tags": ["Pinback", "concert", "live music"], "is_tag_update": true }, { "message_id": 10272003, "content": "I like the live Penelope so much better than the album version.", "author_id": 6022, "tags": ["Pinback", "concert", "live music"] }, { "message_id": 10272012, "content": "They played that last night. Expect June or Grey Machine tonight.", "author_id": 4216, "tags": ["Pinback", "concert", "live music"] }] { "tags": ["Pinback", "concert", "live music"], "message_tags": { "10271972": [ 0, 1, 2 ], "10272003": [ 0, 1, 2 ], "10272012": [ 0, 1, 2 ] }, "messages": { "10272003": { "content": "I like the live Penelope so mu "author_id": 6022 }, "10272012": { "content": "They played that last night. E "author_id": 4216 } } } Mockingbird Expressions A Use Case ^associate($feed|$item.message_id|$item.tags)[$id] ^list($feed.message_tags[$id]|$feed.tags[$item]) Old feed New feed
  • 34. Mobile { "new_message_ids": "^list(^filter($feed|!$item.is_tag_update)|$item.message_id)", "newly_tagged_message_ids": "^list(^filter($feed|$item.is_tag_update)|$item.message_id)", "content_for_message_with_id": "^associate($feed|$item.message_id|$item.content)[$id]", "author_id_for_message_with_id": "^associate($feed|$item.message_id|$item.author_id)[$id]", "tags_for_message_with_id": "^associate($feed|$item.message_id|$item.tags)[$id]" } Mockingbird Expressions A Use Case Old feed We can now separate out the expressions needed to query the data { "new_message_ids": "^list($feed.messages|$key)", "newly_tagged_message_ids": "^filter(^list($feed.message_tags|$key)|!$feed.messages[$item])", "content_for_message_with_id": "$feed.messages[$id].content", "author_id_for_message_with_id": "$feed.messages[$id].author_id", "tags_for_message_with_id": "^list($feed.message_tags[$id]|$feed.tags[$item])" } New feed
  • 35. Mobile This architecture now relies on “model/query decoupling” wherein Mockingbird Expressions A Use Case The ability to query a set of data is separated from knowledge of that data’s structure
  • 36. Mobile How would this work in practice?
  • 37. Mobile 1. The server maintains a schema version for the data model it uses • Whenever a non-backwards-compatible schema change is introduced, the schema version increments 2. Any time the schema version changes, the server publishes a corresponding set of query expressions for that schema version 3. When an app fetches new data, it checks the schema version returned by the server. If the app doesn’t have the query expressions for that version, it can get them from the server • Apps will cache query expressions • Apps can include a compiled-in set of query expressions to avoid having to download them the first time an app launches Mockingbird Expressions Code Samples Model/Query Decoupling using Mockingbird Expressions
  • 38. Mobile 1. When we retrieve feed data from the server, we expose it to the variable space: Mockingbird Expressions Code Samples Populating the Mockingbird Data Environment NSData* json = ... // Unicode text containing JSON structure from server id feedData = [NSJSONSerialization JSONObjectWithData:json options:0 error:nil]; [[MBVariableSpace instance] setVariable:@"feed" value:feedData]; The expression $feed now refers to the value of feedData 2. We load the query expressions associated with the current schema into a dictionary and expose it to the variable space: NSDictionary* queries = ... // query names -> expressions [[MBVariableSpace instance] setVariable:@"queries" value:queries]; The expression $queries now refers to the NSDictionary queries
  • 39. Mobile 3. Query for new messages: Mockingbird Expressions Code Samples Querying the data NSString* query = [MBExpression asString:@"$queries.new_message_ids"]; NSArray* newMessageIDs = [MBExpression asObject:query]; The array newMessageIDs contains two elements: ( 10272003, 10272012 ) 4. Query for messages with updated tags: NSString* query = [MBExpression asString:@"$queries.newly_tagged_message_ids"]; NSArray* newlyTaggedMessageIDs = [MBExpression asObject:query]; The array newlyTaggedMessageIDs contains one element: ( 10271972 ) 5. Get the content of a message given a message ID: [[MBVariableSpace instance] setVariable:@"id" value:@(10272003)]; NSString* query = [MBExpression asString:@"$queries.content_for_message_with_id"]; NSString* content = [MBExpression asNumber:query]; The string content is “I like the live Penelope so much better than the album version.”
  • 40. Mobile Mockingbird Expressions Code Samples Querying the data 7. Get the tags associated with a given message: [[MBVariableSpace instance] setVariable:@"id" value:@(10272012)]; NSString* query = [MBExpression asString:@"$queries.tags_for_message_with_id"]; NSArray* tags = [MBExpression asObject:query]; The array tags contains three elements: ( "Pinback", "concert", "live music" ) 6. Get the author ID of a message given a message ID: [[MBVariableSpace instance] setVariable:@"id" value:@(10271972)]; NSString* query = [MBExpression asString:@"$queries.author_id_for_message_with_id"]; NSNumber* authorID = [MBExpression asNumber:query]; The NSNumber authorID contains the integer value 5375.
  • 41. Mobile What can you use this for?
  • 42. Mobile 1. No App Store update required to introduce support for new schemas • New sets of query expressions can be downloaded to add immediate support for new schemas 2. Maintain legacy support while evolving your services • You won’t need to run multiple versions of your backend just to maintain legacy support 3. Simplify your backend codebase • Avoid spaghetti code — no need for version-check conditionals littered throughout your server-side codebase 4. A/B testing made easy • Want to support multiple schemas simultaneously? No problem! Just switch between sets of query expressions. This makes server-driven A/B testing easy. Mockingbird Expressions Use Cases Benefits
  • 43. Mobile When can you use this?
  • 44. Mobile Mockingbird Availability Mockingbird Data Environment availability The Mockingbird Data Environment is part of the Mockingbird Library open-source project from Gilt Groupe. Coming Fall 2014
  • 45. Mobile Mockingbird Data Environment availability Available today! Coming Fall 2014 Mockingbird Availability The Mockingbird Data Environment is part of the Mockingbird Library open-source project from Gilt Groupe.
  • 46. Mobile Mockingbird Toolbox open source project https://github.com/gilt/mockingbird-toolbox Mockingbird Availability available at:
  • 47. Evan Coyne Maloney Principal Engineer Mobile Gracefully handling changes to your server-side data model