Anything data (revisited)
0 likes110 views
This document provides guidelines for working with big data technologies and systems. It emphasizes the importance of understanding business needs and value over chasing new technologies. It recommends starting simply rather than over-engineering, and learning key concepts like data structures, query languages, processing models and costs before implementing solutions. The document also discusses architectural complexity, operational challenges and evolving approaches to data processing.
Anything data (revisited)
- 1. Anything data revisited Big, Streaming, NoSQL, Cloud, Science … a sloppy travel guide
- 2. whoami ( linkedin. com/in/ahmetakyol )
- 3. whoami ( linkedin. com/in/ahmetakyol )
- 4. whoami - Dilbert already did it
- 5. Who are these people or who are you ?
- 6. Who are these people or who are you ?
- 7. Why a travel guide ? “... Martin is an excellent map reader even in the most hectic Italian traffic … And after Martin and Cindy left us, we did better because we had learned from what they had showed us … When there’s no guide available, it helps to have someone who understands how to read the maps, tracks, signs, and indications. When we’re on our own, it helps to learn how to do those things ourselves“ “Software projects are always traveling in areas they don’t know “ Ron Jeffries (from his foreword for PoAPA book)
- 8. Why a ‘sloppy’ travel guide - (Big Data Landscape 2012 )
- 9. Why a ‘sloppy’ travel guide - (Big Data Landscape 2017 )
- 10. Why a ‘sloppy’ travel guide - (many others: ai,iot ...)
- 11. Why a ‘sloppy’ travel guide - ( the ‘n’ V’s of Big Data )
- 12. Chasing Cool Technologies - Big Data Envy “We continue to see organizations chasing ‘cool’ technologies, taking on unnecessary complexity and risk when a simpler choice would be better.” “ While we've long understood the value of Big Data to better understand how people interact with us, we've noticed an alarming trend of Big Data envy: organizations using complex tools to handle ‘not-really-that-big’ Data.” “ The Apache Cassandra database promises massive scalability on commodity hardware, but we have seen teams overwhelmed by its architectural and operational complexity. Unless you have data volumes that require a 100+ node cluster, we recommend against using Cassandra. ” https://www.thoughtworks.com/radar/techniques/big-data-envy
- 13. Big Data Envy - architectural complexity (expectation) from ‘10000 foot view’ big data systems may seem like ‘good old n-tier’s
- 14. Big Data Envy - architectural complexity (example) A dataflow diagram from a good (but still a) reference application. Real life examples are usually more complex !
- 16. Big Data Envy - architectural complexity (aws example) Big Data Architectural Patterns and Best Practices on AWS : https://www.youtube.com/watch?v=RNrsIlweCno
- 17. Big Data Envy - architectural complexity (blueprints)
- 18. Big Data Envy - operational complexity (devops)
- 19. Big Data Envy - operational complexity (devops) http://www.slideshare.net/jcmia1/apache-spark-20-tuning-guide ● Tuning JVM, OS and each (big) data system ● Choosing right hardware for each ‘right solution’ ● Orchestrating / monitoring / debugging many small applications running on and/or interacting with such distributed systems OOM Troubleshooting example for Apache Spark
- 20. Know thyself - reaching the cliff of confusion https://www.vikingcodeschool.com/posts/why-learning-to-code-is-so-damn-hard
- 21. What is your learning style ? “ What’s a better learning strategy: covering a subject in full detail from top-to- bottom, or progressively sharpening a quick overview? “
- 22. How about an expanding/evolving learning style ? Lifelong learning is the "ongoing, voluntary, and self-motivated" pursuit of knowledge for either personal or professional reasons. Therefore, it not only enhances social inclusion, active citizenship, and personal development, but also self-sustainability, as well as competitiveness and employability.
- 23. The Unknown Unknowns - the iceberg of ignorance In his acclaimed study “The Iceberg of Ignorance”, consultant Sidney Yoshida concluded: “Only 4% of an organization’s front line problems are known by top management, 9% are known by middle management, 74% by supervisors and 100% by employees…”
- 24. Guidelines - the very first principle (business value) “DDD isn’t first and foremost about technology. In its most central principles, DDD is about discussion, listening, understanding, discovery, and business value, all in an effort to centralize knowledge. If you are capable of understanding the business in which your company works, you can at a minimum participate in the software model discovery process to produce a Ubiquitous Language.” “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software” the very first principle of the agile manifesto
- 25. Guidelines - science before technology (business value)
- 26. Guidelines - garbage dump or compulsive hoarding (business value)
- 27. Guidelines - making simple but not simpler ● “ Make things as simple as possible, but not simpler.” (Albert Einstein) ● As simple as possible: no over-engineering search for the simplest feasible solution possible ○ feasible ‘ready’ solution ○ fully managed solutions ○ manageable packed solutions with support ○ solutions known for stability, manageability ● Not simpler: no under-engineering ○ right task, right tool ○ right usage: design patterns, best practices
- 28. Guidelines - right task right tool isn’t enough
- 29. Guidelines - right task right tool right usage DynamoDB Design Patterns and Best Practices : https://www.youtube.com/watch?v=PDQ3jbDyTQ4
- 30. Guidelines - don’t let API fool you (cassandra) CQL Under The Hood : https://www.youtube.com/watch?v=CY5-bWpqAVA
- 31. Guidelines - don’t let API fool you (cassandra) CQL Under The Hood : https://www.youtube.com/watch?v=CY5-bWpqAVA
- 32. Guidelines - learn data paths and structures ( C* ) learning “write path”, “read path” and main internal data structures gives critical hints about “do’s and don’ts”; especially anti-patterns: ● Queue-like designs ● Intensive updates ● Deletes http://www.slideshare.net/doanduyhai/cassandra-nice-use-cases-and-worst-anti-patterns
- 33. Guidelines - loading data, layouts and file formats (hdfs) ● Data distribution , small files problem ● Row v.s. columnar formats ● I/O advantage, read only what you need: ○ Vertical: projection ○ Horizontal: predicate pushdown
- 34. Guidelines -SQL or not (Spark as a Compiler) https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html
- 35. Guidelines -SQL or not (Beam Combine vs GroupBy) https://issues.apache.org/jira/browse/BEAM-2477
- 36. Guidelines -SQL or not ( Spark RDD vs Spark DF and SQL) https://databricks.com/session/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets
- 37. Guidelines - learning from costs (google)
- 38. Guidelines - learning from costs (bigquery)
- 39. Guidelines - learning from costs (kinesis) “ Pricing is based on volume of data ingested into Amazon Kinesis Firehose, which is calculated as the number of data records you send to the service, times the size of each record rounded up to the nearest 5KB. For example, if your data records are 42KB each, Amazon Kinesis Firehose will count each record as 45 KB of data ingested. ” “ A record is the data that your data producer adds to your Amazon Kinesis Stream. A PUT Payload Unit is counted in 25KB payload “chunks” that comprise a record. For example, a 5KB record contains one PUT Payload Unit, a 45KB record contains two PUT Payload Units, and a 1MB record contains 40 PUT Payload Units. PUT Payload Unit is charged with a per million PUT Payload Units rate. ”
- 40. Cloud computing - simple example “ a system, which tracks price changes for my desirable products in online stores (which I trust to buy from) and notifies me over the email when price drops. “ http://www.bebetterdeveloper.com/coding/architecture/serverless-system-architecture-using-aws.html
- 41. Cloud computing - simple “serverless” example http://www.bebetterdeveloper.com/coding/architecture/serverless-system-architecture-using-aws.html “ a system, which tracks price changes for my desirable products in online stores (which I trust to buy from) and notifies me over the email when price drops. “
- 42. Cloud computing - serverless real world example
- 43. Guidelines - learn windows of opportunity (streaming) SELECT sensorid, Count(*) AS count FROM sensorreadings TIMESTAMP by time GROUP BY sensorid, tumblingwindow(second, 10)
- 44. Guidelines - learn windows of opportunity (streaming) SELECT sensorid, Count(*) AS count FROM sensorreadings TIMESTAMP by time GROUP BY sensorid, hoppingwindow(second, 10, 5)
- 45. Guidelines - learn windows of opportunity (streaming) The Evolution of Massive-Scale Data Processing : https://goo.gl/f31iXP
- 46. Guidelines - data processing evolution (history) The Evolution of Massive-Scale Data Processing : https://goo.gl/f31iXP
- 47. Guidelines - data processing evolution (unified/continuous) https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html
- 48. Guidelines (bonus) - know thy theorem ( CAP )
- 49. Guidelines (bonus) - know thy theorem ( PACELC )