Taming Big Data with NoSQL
0 likes841 views
The presentation discusses the advantages of NoSQL databases for handling big data in distributed systems, emphasizing their scalability and flexibility. It highlights important concepts such as consistency levels, conflict resolution, and partitioning strategies while referencing the CAP theorem. The document also briefly mentions related technologies like Hadoop, Spark, and Kafka, and stresses the importance of reliable data analysis.
Taming Big Data with NoSQL
- 1. This presentation includes information that is confidential and proprietary to Basho Technologies and should not be forwarded or distributed without Basho's prior written consent. © 2014. Basho Technologies, Inc. All Rights Reserved. This presentation includes information that is confidential and proprietary to Basho Technologies and should not be forwarded or distributed without Basho's prior written consent. © 2014. Basho Technologies, Inc. All Rights Reserved. Matt Brender Developer Advocate Taming Big Data with NoSQL
- 2. Relational databases are not bad Data Scientist Big Data
- 3. Basho Confidential 3 Data Scientist Big Data
- 6. 6 And it’s a distributed systems problem
- 7. Basho Confidential 7 Beyond the Scope of one tool Beyond the Scope of one file system Beyond the Scope of one database
- 9. 9 NoSQL is
- 10. 10
- 15. Eventually Consistent C = Consistency A = Availability P = Partition Tolerance Client Client DBDBDB Network Partition Cap theorem states that a distributed system can at most support 2 out of these 3 properties
- 17. 17 Conflict Resolution Last Write Wins vs. Causal Context
- 19. 19 Partition Strategy Master Slave Slave Slave OR Node 1 Node 2 Node 3
- 21. 21 “data is powerful when stored and analyzed”
- 22. Relational databases are not bad
- 23. 23 Storing
- 25. Basho Confidential 25 Report on this
- 26. 26 vs
- 27. 27 A single scalable system
- 28. 28 A single scalable system
- 29. 29 Analyzing
- 30. What kind of questions do you need to ask?
- 32. 32 Error Analysis? As simple as NoSQL + Solr
- 33. 33 Patterns? Machine Learning? Multi-client writes to NoSQL & HDFS
- 34. OR
- 35. 35 NoSQL + ETL process => other datastore + Spark and/or Hadoop M/R
- 36. OR
- 37. 37 NoSQL & Apache Storm to Kafka to HDFS
- 38. 38
- 39. LAMDA 39
- 40. So we agree. NoSQL is helpful.
- 41. Side Note: NoSQL is a terrible term
- 42. 42 In Review
- 43. 43 You can’t analyze what you don’t have. And you don’t want an analysis system to be unreliable.
- 44. THE COST OF DOWNTIME 44
- 46. 46 everything works at small scale
- 49. • Hadoop A framework that allows for the distributed processing of large data sets across clusters • Spark A fast, general engine for large-scale data processing • Storm A distributed real-time computation system • NoSQL A collection of highly scalable, highly available systems that fall within CAP theorem • Solr Apache project for indexing text for search • Kafka Distributed scalable pub/sub messaging queue Summary