Bhupeshbansal bigdata
Download as PPT, PDF0 likes682 views
The document discusses big data challenges and solutions. It describes how specialized systems like Hadoop are more efficient than relational databases for large-scale data. It provides examples of open source projects that can be used for tasks like storage, search, streaming data, and batch processing. The document also summarizes the design of the Voldemort distributed key-value store and how it was inspired by Dynamo and Memcached.
![Vector Clock Vector Clock [Lamport] provides way to order events in a distributed system. A vector clock is a tuple {t1 , t2 , ..., tn } of counters. Each value update has a master node When data is written with master node i, it increments ti. All the replicas will receive the same version Helps resolving consistency between writes on multiple replicas If you get network partitions You can have a case where two vector clocks are not comparable. In this case Voldemort returns both values to clients for conflict resolution Proprietary & Confidential 02/06/12](https://image.slidesharecdn.com/bhupeshbansalbigdatafinal-120206143007-phpapp02/85/Bhupeshbansal-bigdata-35-320.jpg)
Bhupeshbansal bigdata
- 1. Big Data and Me Bhupesh Bansal Feb 3, 2012
- 2. Relational Model Architecture Reference : http:// www.slideshare.net / adorepump / voldemort-nosql
- 3. Linkedin 2006 Reference : http://www.slideshare.net/linkedin/linked-in-javaone-2008-tech-session-comm
- 4. Relational model The relational model is a triumph of computer science: General Concise Well understood But then again: SQL is a pain Hard to build re-usable data structures Hides performance issues/details
- 5. Specialized Systems Architecture Reference : http:// www.slideshare.net / adorepump / voldemort-nosql
- 6. Linkedin 2007 Reference : http://www.slideshare.net/linkedin/linked-in-javaone-2008-tech-session-comm
- 7. Specialized systems Specialized systems are efficient (10-100x) Search: Inverted index Offline: Hadoop, Terradata, Oracle DWH Memcached In memory systems (social graph) Specialized system are scalable New data and problems Graphs, sequences, and text
- 8. Batch Driven Architecture Reference : http:// www.slideshare.net / bhupeshbansal /hadoop-user-group-jan2010
- 9. Motivation I : Big Data 02/06/12 Reference : algo2.iti.kit.edu/.../fopraext/index.html
- 10. Motivation II: Data Driven Features
- 11. Motivation III: Makes Money 02/06/12 Proprietary & Confidential
- 12. Motivation IV: Big Data is cool 02/06/12
- 13. Reference : http:// www.slideshare.net / BenSiscovick /the-business-of-big-data-ia-ventures-8577588
- 14. Big Data Challenges Large scale data processing Use all available signals eg. Weblogs, Social signals (twitter/facebook/linkedin) Data Driven Applications Refine data push back to user for consumption Near real time feedback loop Keep continuously improving
- 15. Why is this hard ? Large scale data processing TB/PB of data Traditional storage systems cannot handle the scale Data Driven Applications Need to run complex machine learning algorithms on this data scale Near real time analysis improves application performance and usage.
- 16. Some good news !! Hadoop Biggest single driver for large scale data economy Scales, works, easy to use Memcached Works, scales and is fast Open source world Lot of awesome people working on awesome systems eg. hBase, memcached, Voldemort, kafka, mahout etc. Sharing across companies Common practices/knowledge sharing across companies.
- 17. What works !! Simplicity Go with the simplest design possible. Near real time Async/Batch processing Put computation to background as much as possible Duplicate data everywhere Build customized solution for each problem Duplicate data as needed Data river Publish events and let all systems consume at their own pace Monitoring/Alerting Keep a close eye on things and build a strong dev-ops team
- 18. What doesn’t works !! Magic systems Auto configure, Auto tuning Very hard to get it right instead have easy configuration and better monitoring Open source If Not supported by strong engineering team internally Be ready to have folks spend 30-40% time on understanding, helping open source components Silver bullets One system to solve all scaling problems eg. Hbase Build separate systems for separate problems Central data source Don’ t lock your data let it flow Use (Kafka, Scribe or any publish/subscribe system)
- 19. Open source Very very important for any company today Do not reinvent the wheel Do not write a line of code if not needed 90/10 % rule Pick up open source solutions, fix what is broken Big plus for hiring Stand on shoulder of crowd
- 20. Open source: Storage Problem: You want to store TB of data for user consumption in real time Latency < 50 ms Scale 10,000 QPS + Solutions Big table design eg. Hbase Amazon Dynamo design eg. Voldemort Cache with persistence eg. Membase Document based storage eg. MongoDB
- 21. Open source: Publish/Subscribe Problem: Data River for all other systems to get their feed Solutions Strong data guarantees eg. ActiveMQ, RabbitMQ, HornetQ Log feeds eg. Scribe, flume Kafka A great mix of both the world
- 22. Open source: Real time analysis Problem: Analyze a stream of data and do simple analysis/reporting Solutions Splunk General purpose but high maintenance expansive analysis tool OpenTSDB Simple but scalable metrics reporting Yahoo S4/Twitter Storm Online map-reduce ish New systems will need lots of love and care
- 23. Open source: Search Problem: unstructured queries on data Solutions Lucene Most tested common search (but just a) library Solr Old system with lot of users but bad design Elastic Search Very well designed but new system Linkedin search open source systems sensieDB, zoie
- 24. Open source: Batch computation Problem: You want to process TB of data Solutions is simple: Use Hadoop Hadoop workflow manager Azkaban Oozie Query Native Java code Cascading Hive Pig
- 25. Open source: Other Serialization Avro, Thrift, protocol buffers Compression Snappy, LZO Monitoring Ganglia
- 26. My personal picks !! Storage: Pure key-value lookup : Voldemort Range queries, Hadoop job support: Hbase Batch generated Read only data serving: Voldemort Publish/Subscribe HornetQ OR Kafka Search ElasticSearch Hadoop Azkaban Hive and Native Java code
- 27. Jeff Dean’s Thoughts Very practical advice on building good reliable distributed systems. Highlights Back of the envelope calculations Understand your base numbers well Scale for 10X not 100X Embrace chaos/failure and design around it Monitor/status hooks at all levels Important not to try to be all things for everybody Reference : http:// www.slideshare.net / xlight /google-designs-lessons-and-advice-from-building-large-distributed-systems
- 28. How Voldemort was born ? Reference : 1) http://www.slideshare.net/bhupeshbansal/hadoop-user-group-jan2010 2) http://www.slideshare.net/adorepump/voldemort-nosql
- 29. Why NoSQL ? TBs of data Sharding the only way to scale No joins possible (Data is split across machines) Specialized systems eg search, network feed breaks relational model No constraints, triggers, etc disappear Lots of denormalization Latency is key Relational DB depend on caching layer to achieve high throughput and low latency
- 30. Inspired By Amazon Dynamo & Memcached Amazon ’s Dynamo storage system Works across data centers Eventual consistency Commodity hardware Memcached Actually works Really fast Really simple
- 31. ACID Vs CAP ACID Great for single centralized server. CAP Theorem Consistency (Strict), Availability , Partition Tolerance Impossible to achieve all three at same time in distributed platform Can choose 2 out of 3 Dynamo chooses High Availability and Partition Tolerance by sacrificing Strict Consistency to Eventual consistency Proprietary & Confidential 02/06/12
- 32. Consistent Hashing Key space is Partitioned Many small partitions Partitions never change Partitions ownership can change Replication Each partition is stored by ‘N’ nodes Proprietary & Confidential 02/06/12
- 33. R+W > N N - The replication factor R - The number of blocking reads W - The number of blocking writes If R+W > N then we have a quorum-like algorithm Guarantees that we will read latest writes OR fail R, W, N can be tuned for different use cases W = 1, Highly available writes R = 1, Read intensive workloads Knobs to tune performance, durability and availability Proprietary & Confidential 02/06/12
- 34. Versioning & Conflict Resolution Eventual Consistency allows multiple versions of value Need a way to understand which value is latest Need a way to say values are not comparable Solutions Timestamp Vector clocks Provides global ordering. No locking or blocking necessary
- 35. Vector Clock Vector Clock [Lamport] provides way to order events in a distributed system. A vector clock is a tuple {t1 , t2 , ..., tn } of counters. Each value update has a master node When data is written with master node i, it increments ti. All the replicas will receive the same version Helps resolving consistency between writes on multiple replicas If you get network partitions You can have a case where two vector clocks are not comparable. In this case Voldemort returns both values to clients for conflict resolution Proprietary & Confidential 02/06/12
- 36. Client API Data is organized into “stores”, i.e. tables Key-value only But values can be arbitrarily rich or complex Maps, lists, nested combinations … Four operations PUT (Key K, Value V) GET (Key K) MULTI-GET (Iterator<Key> K), DELETE (Key K) / (Key K , Version ver) No Range Scans
- 39. Read-only storage engine Throughput vs. Latency Index building done in Hadoop Fully parallel transfer Very efficient on-disk structure Heavy reliance on OS pagecache Rollback! Reference : http:// www.slideshare.net / bhupeshbansal /hadoop-user-group-jan2010
- 40. What do we use Hadoop/Voldemort for ? Proprietary & Confidential 02/06/12
- 41. Batch Driven Architecture Reference : http:// www.slideshare.net / bhupeshbansal /hadoop-user-group-jan2010
- 42. Data Flow Driven Architecture Reference : http:// sna-projects.com /blog/2011/08/ kafka /
- 43. Questions
Editor's Notes
- #4: Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- #7: Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- #11: Statistical learning as the ultimate agile development tool (Peter Norvig), “business logic” through data rather than code
- #30: No Joins Across data domains due to APIs Within data domains due to performance Natural operation: getAll(id…) Latency: if you want to call 30 services on your main pages, they better be quick (30 * 20ms = 600ms)
- #32: - Strong Consistency: all clients see the same view, even in presence of updates - High Availability: all clients can find some replica of the data, even in the presence of failures Partition-tolerance: the system properties hold even when the system is partitioned high availability : Mantra for websites Better to deal with inconsistencies, because their primary need is to scale well to allow for a smooth user experience.
- #33: Hashing .. Why do we need it ?? Basic problem : Clients need to know which data is where ?? Many ways of solving it Central configuration Hashing Linear hashing works : issue is when cluster is dynamic ?? KeyHash –node IDmapping change for a lot of entries When you add new slots Consistent hashing : preserves key –Node mapping for most of the keys and only change the minimal amount needed How to do it ?? Number of partitions ---------------------------- Arbitrary , each node is allocated many partitions (better load balancing and fault tolerance) Few hundreds to few thousands .. Key partition mapping is fixed and only ownership of partitions can change
- #35: Give example of read and writes with vector clocks Pros and cons vs paxos and 2pc User can supply strategy for handling cases where v1 and v2 are not comparable.
- #36: Fancy way of doing Optimistic locking
- #37: Very simple APIS NO Range Scans .. . No iterator on KeySet / Entry SET : Very hard to fix performance Have plans to provide such an iterator
- #38: Explain about partitions Make things fast by removing slow things, not by tuning HTTP client not performant Separate caching layer
- #40: Transfer time: 30 minutes Can max out a gb network, so be careful
- #43: Example: member data--does not make sense to repeatedly join positions, emails, groups, etc. Explain about joins How to better model in java? Json like data model
- #44: Questions, comments, etc