Patterns of the Lambda Architecture -- 2015 April - Hadoop Summit, Europe
3 likes982 views
The document outlines the Lambda architecture, emphasizing its layers: batch, speed, and serving. It discusses architectural trade-offs related to data truth, processing capabilities, and system resilience while exploring patterns for training recommendation systems and handling data updates. Additionally, it presents various real-world application scenarios and common objections to implementing this architecture, underscoring the importance of moving truth to the edge of the architecture.
![Article Title
Article Body Lorem ipsum dolor sit
amet, consectetur adipiscing elit, sed do
eiusmod tempor incididunt ut labore et
dolore magna aliqua. Ut enim ad minim
veniam, quis nostrud exercitation ullamco
laboris nisi ut aliquip ex ea commodo
consequat. Duis aute irure dolor in
reprehenderit in voluptate velit esse cillum
dolore eu fugiat nulla pariatur. Excepteur
sint occaecat cupidatat non proident.
Author Name
Author
Photo
Author Bio
Lorem ipsum
dolor sit amet,
consectetur
adipiscing elit,
sed do eiusmod
tempor
Comments
"First Post"
(8/8/2014 by Commenter 1)
"lol"
(8/8/2014 by Commenter 2)
"No comment"
(8/8/2014 by Commenter 3)
article show rendered
{
"title":"Article Title",
"body":"Article Body Lorem [...]",
"author":{ ... },
"comments: [
{"comment_id":1, "body":"First Post",...},
{"comment_id":2, "body":"lol",...},
...
]}](https://image.slidesharecdn.com/201410-hadoopsummiteurope-patternsofthelambdaarchitecturev6-150415074209-conversion-gate01/85/Patterns-of-the-Lambda-Architecture-2015-April-Hadoop-Summit-Europe-53-320.jpg)
Patterns of the Lambda Architecture -- 2015 April - Hadoop Summit, Europe
- 1. Patterns of the Lambda Architecture Truth and Lies at the Edge of Scale
- 2. Pattern Set
- 5. Search w/ Update Build Indexes A Ton of Text Historical Index Live Indexer More Text Recent Index API
- 6. Search w/ Update Build Indexes A Ton of Text Historical Index Live Indexer More Text Recent Index API Main Index
- 7. Search w/ Update Build Indexes A Ton of Text Historical Index Live Indexer More Text Recent Index API Updates Index
- 8. Search w/ Update Build Indexes A Ton of Text Historical Index Live Indexer More Text Recent Index API Serve Result
- 9. Lambda Architecture Build Indexes A Ton of Text Historical Index Live Indexer More Text Recent Index API Batch Speed Serving
- 15. Lambda Arch Layers • Batch layer Deep Global Truth throughput • Speed layer Relevant Local Truth throughput • Serving layer Rapid Retrieval latency
- 16. Lambda Arch: Technology • Batch layer Hadoop, Spark, Batch DB Reports • Speed layer Storm+Trident, Spark Str., Samza,AMQP, … • Serving layer Web APIs, Static Assets, RPC, …
- 19. Lambda Architecture λ(v) • Pure Function on immutable data
- 20. Two Big Ideas • Fine-grained control over architectural tradeoffs • Truth lives at the edge, not the middle
- 22. Ideal Data System • Capacity -- Can process arbitrarily large amounts of data • Affordability -- Cheap to run • Simplicity -- Easy to build, maintain, debug • Resilience -- Jobs/Processes fail&restart gracefully • Justification -- Incorporates all relevant data into result • Comprehensiveness-- Answer questions about any subject • Accuracy -- Few approximations or avoidable errors • Responsiveness -- Low latency for delivering results • Recency -- Promptly incorporates changes in world
- 23. Patterns
- 25. Pattern: Train / React • Model of the world lets you make immediate decisions • World changes slowly, so we can re-build model at leisure • Relax: Recency • Batch layer: Train a machine learning model • Speed layer: Apply that model • Examples: most Machine Learning thingies
- 26. Search w/ Update Build Indexes A Ton of Text Historical Index Live Indexer More Text Recent Index API
- 27. Pattern: Baseline / Delta • Understanding the world takes a long time • World changes much faster than that, and you care • Relax: Simplicity, Accuracy • Batch layer: Process the entire world • Speed layer: Handle any changes since last big run • Examples: Real-time Search index; Count Distinct; other Approximate Stream Algorithms
- 29. 48 24 42 12 12 6 24 24 42 6 6 6 6 6 6 6 9 4 - 5 - Guess Using Neighbors ?
- 30. 48 24 42 12 12 6 24 24 42 6 6 6 6 6 6 6 9 4 - 5 - Guess Using Neighbors 12÷3 = 4 24÷5 ≈ 5 9
- 31. 48 24 42 12 12 6 24 24 42 6 6 6 6 6 6 6 9 4 - 5 - …But Don’t Fix Neighbors meh
- 32. Batch Updates Graph 42 30 36 11 10 6 21 21 36 4 6 6 6 5 5 4 9 3 9 6
- 33. Pagerank Converge Pagerank Friend Relations User Pagerank Retrieve Bob’s Facebook Ntwk Bob Bob’s Friends’ Pageranks Estimate Bob’s Pagerank But don’t bother updating Bob’s Friends (or friends friends or …) API
- 34. Pattern: Guess Beats Blank • You can’t calculate a good answer quickly • But Comprehensiveness is a must • Relaxing: Accuracy, Justification • Batch layer: finds the correct answer • Speed layer: makes a reasonable guess • Examples:Any time the sparse-data case is also the most valuable
- 35. Marine Corp’s 80% Rule “Any decision made with more than 80% of the necessary information is hesitation” — “The Marine Corps Way” Santamaria & Martino
- 36. A Guess Beats a Blank • You can’t calculate a good answer quickly • But Comprehensiveness is a must • Relaxing: Accuracy, Justification • Batch layer: finds the correct answer • Speed layer: makes a reasonable guess • Examples:Any time the sparse-data case is also the most valuable
- 37. Pattern: World/Local • Understanding the world needs full graph • You can tell a little white lie reading immediate graph only • Relaxing: Accuracy, Justification • Batch layer: uses global graph information • Speed layer: just reads immediate neighborhood • Examples:“Whom to follow”, Clustering, anything at 2nd- degree (friend-of-a-friend)
- 38. Network Security Find Potential Evilness Connection Counts Agents of Interest Store Interaction Net Connect ions Detected Evilnesses Approximate Streaming Agg Agent of Interest? Dashboard
- 39. Pattern: Slow Boil/Flash Fire • Two tempos of data: months vs milliseconds • Short-term data too much to store • Long-term data too much to handle immediately • Often accompanies Baseline / Deltas • Examples: • Trending Topics • Insider Security
- 40. Banking, Oversimplified Reconcile Accounts Account Balances Event Store Transaction Update Records
- 41. Banking, Oversimplified Reconcile Accounts Account Balances Event Store Transaction Update Records nice-to-haveessential This wins over fast layer
- 42. Pattern: C-A-P Tradeoffs • C-A-P tradeoffs: • Can’t depend on when data will roll in (Justification) • Can’t live in ignorance (Comprehensiveness) • Batch layer: The final answer • Speed layer: Actionable views • Examples: Security (Authorization vs Auditing), lots of counting problems
- 43. Common Theme The System Asymptotes to Truth over time
- 44. Two Big Ideas • Fine-grained control over those architectural tradeoffs • Truth lives at the edge, not the middle
- 45. Lambda Architecture for a Dinky Little Blog
- 46. Blog: Traditional Approach • Familiar with the ORM Rails-style blog: • Models: User, Article, Comment • Views: • /user/:id (user info, links to their articles and comments); • /articles (list of articles); • /articles/:id (article content, comments, author info)
- 47. User id 3 name joeman homepage http://… photo http://… bio “…” Article id 7 title The Crisis body These are… author_id 3 created_at 2014-08-08 Comment id 12 body “lol” article_id 7 author_id 3
- 48. Author Name Author Bio Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Author Photo Joe has written 2 Articles: “A Post about My Cat” Donec nec justo eget felis facilisis fermentum.Aliquam porttitor mauris sit amet orci.Aenean dignissim pellentesque (… read more) “Welcome to my blog” Donec nec justo eget felis facilisis fermentum.Aliquam porttitor mauris sit amet orci.Aenean dignissim pellentesque (… read more) Article Title Article Body Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident. Author Name Author Photo Author Bio Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor Comments "First Post" (8/8/2014 by Commenter 1) "lol" (8/8/2014 by Commenter 2) "No comment" (8/8/2014 by Commenter 3) article show user show
- 51. Models Trigger Reporters update article update user update comments Δ article Δ user Δ com’nt models user com’nt article history compact article user’s # articles expanded user user’s # comments sidebar user compact comment expanded article exp’d article compact article user’s # articles exp’d user sidebar user user’s # comments compact comment micro user micro user
- 52. Serve Report Fragments exp’d article compact article user’s # articles exp’d user sidebar user user’s # comments compact comment micro user show article Article Title Article Body Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident. Author Name Author Photo Author Bio Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor Comments "First Post" (8/8/2014 by Commenter 1) "lol" (8/8/2014 by Commenter 2) "No comment" (8/8/2014 by Commenter 3)
- 53. Article Title Article Body Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident. Author Name Author Photo Author Bio Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor Comments "First Post" (8/8/2014 by Commenter 1) "lol" (8/8/2014 by Commenter 2) "No comment" (8/8/2014 by Commenter 3) article show rendered { "title":"Article Title", "body":"Article Body Lorem [...]", "author":{ ... }, "comments: [ {"comment_id":1, "body":"First Post",...}, {"comment_id":2, "body":"lol",...}, ... ]}
- 54. Serve Report Fragments Article Title Article Body Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident. Author Name Author Photo Author Bio Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor Comments "First Post" (8/8/2014 by Commenter 1) "lol" (8/8/2014 by Commenter 2) "No comment" (8/8/2014 by Commenter 3) exp’d article compact article user’s # articles exp’d user sidebar user user’s # comments compact comment micro user show user
- 55. Reports are Cheap update article update user update comments Δ article Δ user Δ com’nt models user com’nt article history compact article user’s # articles expanded user user’s # comments sidebar user compact comment expanded article exp’d article compact article user’s # articles exp’d user sidebar user user’s # comments compact comment micro user micro user list articles show article list user’s articles show user
- 56. • (…hack hack hack…) /articles/v2/show.json /articles/v1/show.json • (…hack hack hack…) What data model would you like to receive? {“title”:”…”, “body”:”…”,…} j/k lol can I have Data Engineer Web Engineer {“title”:”…”, “body”:”…”, “snippet”:…}
- 57. Syndicated Data • Reports are cheap, single-concern, and faithful to the view. • You start thinking like the customer, not the database • All pages render in O(1): • Your imagination doesn’t have to fit inside a TCP timeout • Data is immutable, flows are idempotent: • Interface change is safe • Data is always _there_, • Asynchrony doesn’t affect consumers • Everything is decoupled: • Way harder to break everything
- 58. Syndicated Data • The Data is always _there_ • …but sometimes it’s more perfect than other times.
- 59. • Lambda architecture isn’t about speed layer / batch layer. • It's about • moving truth to the edge, not the center; • enabling fine-grained tradeoffs against fundamental limits; • decoupling consumer from infrastructure • decoupling consumer from asynchrony • …with profound implications for how you build your teams λ Arch: Truth, not Plumbing
- 60. …
- 61. …
- 68. Cannot have Consistency Product Resolver key words mfr & model ASIN VendorListing Fetch Products Unified Products Resolve & Update Listings Unify Products
- 69. Objections
- 70. Objections • Three objections 1.Why hasn't it been done before 2.Architecture Astronaut 3.I'm not at high scale • Response 1.Chef/Puppet/Docker/etc 2.Chef/Puppet/Docker/etc 3.Shut Up
- 72. Objections • Two APIs? Really? • Yes. Guilty.That’s dumb and must be fixed. • Spark or Samza, if you’re willing to only drink one flavor of Kool-Aid • EZbake.io, a CSC / 42six project to attack this • …but we shouldn’t be living at the low level anyhow
- 73. Objections • Orchestration: “logical plan” (dataflow graph) • Optimization/Allocation: “physical plan” (what goes where) • Resource Projector: instantiates infrastructure • HTTP listeners,Trident streams, Oozie scheduling, ETL flows, cron jobs, etc • Transport Machineries: • move data around, fulfilling locality/ordering/etc guarantees • Data Processing: UDFs and operators