The Other Way of Doing Big Data
Download as KEY, PDF3 likes1,756 views
The document outlines a new approach to managing big data through a decoupled, federated architecture that emphasizes resilience, simplicity, and optimization for people. It discusses various components like HBase, MySQL, and Elasticsearch, highlighting the importance of managing system complexity and failure. The author advocates for a lightweight and declarative system design to improve scalability and reduce costs.
The Other Way of Doing Big Data
- 1. How to Win at Scale and its Influence on People Philip (flip) Kromer CTO, Infochimps.com
- 2. Big Data is Inevitable It Demands a New Approach
- 4. There’s Another Way You’re Going to Have to follow It
- 5. There’s Another Way You’re Going to Have to follow It It Might be a Better Way
- 8. Federated Truth email MySQL HBase s3 spreadsheets elasticsearch elasticsearch HDFS hipchat redis mongo MongoDB log files salesforce zabbix hubspot ADP Chargify BC/BS ZenDesk google docs
- 9. Low Coupling
- 10. Reliable Resilient
- 11. • Manage 100s of machines: architecture as code • Contain system complexity: relentlessly decouple • Maintain coherency: federated truth • Manage true costs: optimize for people not machines • Manage failure & change:resiliency engineering
- 12. The Other Way Declarative, not Homogenous Decoupled, not Standardized Federated, not Centralized Simple, not Performant Resilient, not Reliable
- 13. Declarative
- 14. Architecture as Code Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh flume ElasticSearch ElasticSearch Operations Application Ironfan + ops ics.com Hadoop On-Demand Hadoop Chef
- 15. Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh flume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop HM NN ZK RS RS RS RS RS RS
- 16. provision machine run state settings standard components cluster-specific facet groups
- 18. Lightweight Dashboard Lightweight Dashboard HBase HBase HM NN ZK API Data Transport ESh flume ElasticSearch RS RS ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop RS RS RS RS regionserver ssh nfs datanode zbx stargate log tasktracker fw zookeeper
- 19. Wins from Declarative Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh flume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop
- 20. Recapitulatable
- 21. Portable
- 22. Decoupled
- 23. Our Stack Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh flume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop
- 24. Our Stack
- 25. Our Stack
- 26. Engineer : System = 1:10 • >60 distinct components • 50-150 machines • 1 ops + 5 hackers + 1 analyst
- 27. Self-similar Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh flume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop HM NN ZK RS RS ssh ssh hb 2d mstr hb master nfs nfs RS RS namenode zbx 2d nn zbx log jobtracker log zookeeper RS RS fw zookeeper fw alpha beta regionserver ssh regionserver ssh nfs nfs datanode datanode zbx zbx stargate log stargate log tasktracker fw tasktracker fw zookeeper gamma delta
- 28. Example: Scraper Scraper disk tail’er decorator sink Jobs database
- 29. Scraper flume Scraper disk tail’er decorator sink Jobs database while true: get_job fetch_url dump_to_disk
- 30. Scraper flume Scraper disk tail’er decorator sink Jobs database while true: ensures get_job reliable fetch_url delivery dump_to_disk
- 31. Scraper flume Scraper disk tail’er decorator sink Jobs database while true: ensures parse get_job reliable raw fetch_url delivery => dump_to_disk objects
- 32. Scraper flume Scraper disk tail’er decorator sink Jobs database while true: ensures parse store get_job reliable raw object fetch_url delivery => => dump_to_disk objects database
- 34. Simple
- 36. • Immediately Understandable • Clear Interface • Few Moving Parts
- 37. Federated
- 38. Data Stores in Production • HBase • MySQL • ElasticSearch • Redis • Cassandra • sqlite • TokyoTyrant • whisper (graphite) • SimpleDB • file system • MongoDB • S3
- 39. Programs Used for This Talk • Emacs • Skitch • Keynote • finder • Preview • flickr.com • Chrome • google image search • ruby (pry) • ssh
- 40. How’s my Batch Job Going? • 1 x Job Status • 1 x Counters & App Metrics • N x Task Status • M x Machine System Stats • 1 x Cloud Status • 1 x Chef Server
- 41. Dataflow is All
- 42. Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh flume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop System Diagram Dataflow Workflow
- 43. Lightweight Lightweight Dashboard Dashboard HBase HBase API Data Transport ESh flume ElasticSearch ElasticSearch Operations Application ops ics.com Hadoop On-Demand Hadoop System Diagram Dataflow Workflow Org Chart
- 44. Robots are Cheap People are Important
- 45. Expensive / Not Expensive 1 trillion 10 kb objects: • 100 % in RAM: $ 212,000 /mo • 10% in Ram: $ 21,000 /mo • On Disk: $ 3,000 /mo • On S3: $ 1,200 /mo
- 46. Expensive / Not Expensive 1 trillion 10 kb objects: • 100 % in RAM: $ 212,000 /mo • 10% in Ram: $ 21,000 /mo • On Disk: $ 3,000 /mo • On S3: $ 1,200 /mo 1 Intern, part-time: $ 1,500 /mo
- 47. Scalability is People
- 49. Monolithic Software means Meetings
- 52. n^2 law of coupling 100 things 5 + 3 + 2 things + 2 (tax)
- 53. n^2 law of coupling 2500 + 900 + 400 + 400 = 10,000 things 4200 things to go wrong to go wrong
- 55. Infochimps.com 2011 text search Planet of the API acct'g APIs infochimps.com models A/B testing cloud services
- 56. Infochimps.com 2012 datasets catalog API API docs text search content dashboards Planet of the API acct'g APIs auth & payment layout console models A/B testing blog press cloud services collateral
- 57. Infochimps.com 2012 (infochimps) icsexpl catalog API (saas) capuchin elasticsrch kanzi beergoggls Planet of the MongoDB APIs george george alphamale MySQL redis WPEngine totem cloud services hubspot
- 58. this drawing fits in my head datasets catalog API this app fits in my head, and my laptop
- 59. Infochimps.com 2012 (infochimps) icsexpl catalog API (saas) capuchin elasticsrch kanzi beergoggls Planet of the MongoDB APIs george george alphamale MySQL redis WPEngine totem cloud services hubspot
- 60. fin. http://infochimps.com http://github.com/infochimps-labs
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: This is on a 15-person organization\nFederated, meaning the data is semantically disparate\n
- #10: \n
- #11: \n
- #12: people are walking around as if we used to have one kind of database and now we have two\nThe important fact isn’t that one of them is sharded \nThe important fact is that they’re proliferating -- and that’s a good thing.\n
- #13: Google, Facebook, Amazon had to solve the scalability problem\n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: Now I know this sounds like the lunacy of a ritalin-addled architecture astronaut spending too much time on StackOverflow. \n
- #40: Now I know this sounds like the lunacy of a ritalin-addled architecture astronaut spending too much time on StackOverflow. \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: $200k on 146 Amazon EC2 m2.4xlarge\n$20k 10 TB Data, 10% Ram: $ / month, on 57 Amazon EC2 m2.xlarge\n$3k 10 TB Data, Disk: $ / month, on 6 Amazon EC2 c1.xlarge\n$1.2k 10 TB s3\n\n10 TB Ram: $ / month, on 146 Amazon EC2 m2.4xlarge \n 10_000 * 2.00 * 24 * 30.25 / 68.4 = \n $212,280\n10 TB Data, 10% Ram: $ / month, on 57 Amazon EC2 m2.xlarge \n 0.1 * 10_000 * 0.50 * 24 * 30.25 / 17.5 = \n $20,743\n10 TB Data, Disk: $ / month, on 6 Amazon EC2 c1.xlarge\n machines, price, disk, ram = [6, 0.68, 1_690, 7] ; [(tot_disk = disk * machines), (machine_dollars_mo = (machines * price * 24 * 30.25).round)] $2,962\n10 TB Data, S3: $1,250 / month\n1 intern, $10/hr, 25 hrs/wk, not incl. overhead: $1,100 / month\n\n
- #47: $200k on 146 Amazon EC2 m2.4xlarge\n$20k 10 TB Data, 10% Ram: $ / month, on 57 Amazon EC2 m2.xlarge\n$3k 10 TB Data, Disk: $ / month, on 6 Amazon EC2 c1.xlarge\n$1.2k 10 TB s3\n\n10 TB Ram: $ / month, on 146 Amazon EC2 m2.4xlarge \n 10_000 * 2.00 * 24 * 30.25 / 68.4 = \n $212,280\n10 TB Data, 10% Ram: $ / month, on 57 Amazon EC2 m2.xlarge \n 0.1 * 10_000 * 0.50 * 24 * 30.25 / 17.5 = \n $20,743\n10 TB Data, Disk: $ / month, on 6 Amazon EC2 c1.xlarge\n machines, price, disk, ram = [6, 0.68, 1_690, 7] ; [(tot_disk = disk * machines), (machine_dollars_mo = (machines * price * 24 * 30.25).round)] $2,962\n10 TB Data, S3: $1,250 / month\n1 intern, $10/hr, 25 hrs/wk, not incl. overhead: $1,100 / month\n\n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: \n
- #61: \n