Choosing which big data, nosql or database technology to use
8 likes3,675 views
The document addresses the complexities of selecting appropriate database technologies for big data, emphasizing that 'big data' is not just about sheer volume but also involves understanding workload parameters like query complexity and access patterns. It highlights the importance of analyzing specific use cases to determine the appropriate technology—whether relational or NoSQL—while considering factors such as read-write intensity, data consistency, and latency. Mark Madsen, an expert in the field, underscores the significance of building proof of concepts to match workload requirements with suitable database solutions.
Choosing which big data, nosql or database technology to use
- 1. One Size Doesn’t Fit All Choosing which big data, NoSQL or database technology to use March 14, 2012 Mark R. Madsen http://ThirdNature.net
- 2. The problem of “big” is three problems of volume Computations! Number Amount of users! of data!
- 3. Big data? Unstructured data isn’t really unstructured. The problem is that this data is unmodeled. The real challenge is complexity.
- 6. You must understand your workload ‐ throughput and response time requirements aren’t enough. ▪ 100 simple queries accessing month‐to‐date data ▪ 90 simple queries accessing month‐to‐date data plus 10 complex queries using two years of history ▪ Hazard calculation for the entire customer master ▪ Performance problems are rarely due to a single factor.
- 7. Workload: One big query or many small queries? Retrieval: small return set or large? Selectivity: large volume of data scanned or small?
- 10. Important workload parameters to know • Read‐intensive vs. write‐intensive • Mutable vs. immutable data • Immediate vs. eventual consistency
- 11. Important workload parameters to know • Read‐intensive vs. write‐intensive • Mutable vs. immutable data • Immediate vs. eventual consistency • Short vs. long access latency
- 12. Important workload parameters to know • Read‐intensive vs. write‐intensive • Mutable vs. immutable data • Immediate vs. eventual consistency • Short vs. long access latency • Predictable vs. unpredictable data access patterns
- 13. Types of workloads Write‐biased: Read‐biased: ▪ OLTP ▪ Query ▪ OLTP, batch ▪ Query, simple retrieval ▪ OLTP, lite ▪ Query, complex ▪ Object persistence ▪ Query‐hierarchical / ▪ Data ingest, batch object / network ▪ Data ingest, real‐time ▪ Analytic Mixed? Inline analytic execution, operational BI
- 14. Matching to parameters, at assumption of data scale Workload Write‐ Read‐ Updateable Eventual Un‐ Compute parameters biased biased data consistency predictable intensive ok query path Standard RDBMS Parallel RDBMS NoSQL (kv, dht, obj) Hadoop* Streaming database You see the problem: it’s an intersection of multiple parameters, and this chart only includes the first tier of parameters. Plus, workload factors can completely invert these general rules of thumb.
- 15. Matching to parameters, at assumption of data scale Workload Complex Selective Low latency High High ingest parameters queries queries queries concurrency rate Standard RDBMS Parallel RDBMS NoSQL (kv, dht, obj) Hadoop Streaming database You have to look at the combination of workload factors: data scale, concurrency, latency & response time, then chart the parameters.
- 17. Image Attributions Thanks to the people who supplied the images used in this presentation: Holy Grail – © Monty Python Ltd. Cupcakes – <lost attribution on Flickr> rock‐fall‐roadblock.jpg ‐ http://www.flickr.com/photos/wsdot/4679360979/ roadblock‐sheep.jpg ‐ http://www.flickr.com/photos/brizo_the_scot/4013939756/ Slide 17
- 18. About the Presenter Mark Madsen is president of Third Nature, a technology research and consulting firm focused on business intelligence, analytics and information management. Mark is an award-winning author, architect and former CTO whose work has been featured in numerous industry publications. During his career Mark received awards from the American Productivity & Quality Center, TDWI, Computerworld and the Smithsonian Institute. He is an international speaker, contributing editor at Intelligent Enterprise, and manages the open source channel at the Business Intelligence Network. For more information or to contact Mark, visit http://ThirdNature.net.
- 19. About Third Nature Third Nature is a research and consulting firm focused on new and emerging technology and practices in analytics, business intelligence, and performance management. If your question is related to data, analytics, information strategy and technology infrastructure then you‘re at the right place. Our goal is to help companies take advantage of information-driven management practices and applications. We offer education, consulting and research services to support business and IT organizations as well as technology vendors. We fill the gap between what the industry analyst firms cover and what IT needs. We specialize in product and technology analysis, so we look at emerging technologies and markets, evaluating technology and hw it is applied rather than vendor market positions.