Concepts, use cases and principles to build big data systems (1)
6 likes3,504 views
The document provides a comprehensive overview of big data concepts, principles, and practical use cases, highlighting the importance of big data in various applications. It outlines the architectural design using lambda architecture, details the data engineering process, and includes a case study on a chat bot with a video recommendation engine. Additionally, it offers guidance for students on learning big data and resources for further education.
Concepts, use cases and principles to build big data systems (1)
- 1. Concepts, use cases and principles to build big data systems http://www.bigdatavietnam.org https://www.facebook.com/bigdatavn Compiled by Nguyễn Tấn Triều
- 2. Key Contents 1. Introduction to the key Big Data concepts ○ The Origins of Big Data ○ What is Big Data ? ○ Why is Big Data So Important ? ○ How Is Big Data Used In Practice ? 2. Introduction to the key principles of Big Data Systems ○ How to design Data Pipeline in 6 steps ○ Using Lambda Architecture for big data processing 3. Practical case study ○ Chat bot with Video Recommendation Engine 4. FAQ for student
- 3. Introduction to the key Big Data concepts ○ The Origins of Big Data ○ What is Big Data ? ○ Why is Big Data so important ? ○ How Is Big Data used in practice ?
- 4. The Origins of Big Data
- 6. What is Big Data ?
- 7. What is Big Data ?
- 8. What is Big Data ?
- 9. Why is Big Data So Important ?
- 10. Why is Big Data So Important ?
- 12. Source: https://internetofthingsagenda.techtarget.com/definition/Internet-of-Things-IoT How Is Big Data Used In Practice ?
- 13. How Is Big Data Used In Practice ?
- 14. Why is Big Data So Important ?
- 16. How Is Big Data Used In Practice ? Device Analytics Which device is most popular used ?
- 17. How Is Big Data Used In Practice ? Time-series Analytics The peak hours of system
- 18. How Is Big Data Used In Practice ? GeoLocation Heatmap Analytics
- 19. Introduction to the key principles of Big Data Systems ○ How to design Data Pipeline in 6 steps ○ Using Lambda Architecture for big data processing
- 20. How to design Data Pipeline Systems Collecting → Storing → Processing → Analyzing → Learning → Visualizing Data engineering process: 3 tasks 1. Collecting a. Concepts b. Technology 2. Storing a. Big Data Storage Concepts b. Big Data Storage Technology 3. Processing a. Big Data Processing Concepts b. Big Data Processing Technology Data Science/Machine Learning process: 3 tasks 4) Analyzing → 5) Learning → 5) Visualizing
- 21. Data Engineer Tasks Data Analyst Tasks Big Data Analytics Lifecycle Collecting Storing Processing Analyzing Learning Visualizing
- 22. (Collecting) → Storing → Processing → Analyzing → Learning → Reacting
- 23. Collecting
- 24. Collecting tools Batch collecting: Apache Sqoop ( from DBMS to Apache Hadoop) Real-time collecting: Log Collector with Apache Kafka
- 25. Collecting → (Storing) → Processing → Analyzing → Learning → Reacting
- 26. Storing Concepts ● Clusters ● Scale-Up vs Scale-Out ● File Systems and Distributed File Systems ● NoSQL ● Sharding ● Replication ● Sharding and Replication ● CAP Theorem
- 27. Clusters
- 29. Database in Big Data
- 30. NoSQL
- 31. NoSQL
- 32. Sharding
- 36. CAP Theorem
- 37. Collecting → Storing → (Processing) → Analyzing → Learning → Reacting
- 38. Processing concepts ● Parallel Data Processing ● Distributed Data Processing ● Hadoop ● Processing Workloads ● Cluster ● Processing in Batch Mode ● Processing in Realtime Mode
- 41. Hadoop Hadoop is a versatile framework that provides both processing and storage capabilities
- 42. Batch processing (offline processing)
- 44. Cluster
- 45. Map and Reduce Tasks
- 46. Processing in Realtime Mode
- 47. When standard relational database (Oracle,MySQL, ...) is not good enough the “analytic system” MySQL database from a startup, tracking all actions in mobile games: iOS, Android, ...
- 48. 3 common problems in Big Data System 1. Size: the volume of the datasets is a critical factor. 2. Complexity: the structure, behaviour and permutations of the datasets is a critical factor. 3. Technologies: the tools and techniques which are used to process a sizable or complex dataset is a critical factor.
- 49. Key ideas of Lambda Architecture in Big Data System
- 50. Practical case study Chat bot with Video Recommendation Engine
- 51. Problem ● A company want to develop a chat bot for news recommendation ● They want to classify data into standard categories (26 categories) for user-friendly query ● The engineering team have develop a data pipeline for system
- 52. Solution Diagram Big Data is here Author @tantrieuf31
- 53. Problem: Topic Classification for News
- 54. Solution Diagram
- 55. FAQ for students How to learn Big Data ? Job Opportunity Ref resources
- 56. How to learn Big Data ? 1. Have lots of passion, curiosity with data 2. Knowledge about data structure, statistics and basic maths 3. Love to solve complex problems with data-driven mindset 4. Database knowledge: when to use NoSQL vs RDBMS 5. Knowledge about distributed computing 6. Linux / Open Source Tools 7. Programming language: Python / Java / SQL / JavaScript 8. English skills
- 57. Big Data Job Market is really hot https://www.class-central.com/subject/big-data
- 58. Some good books for self-learning ● http://sachvui.com/ebook/du-lieu-lon-big-data.281.html ● https://drive.google.com/open?id=0B3dHGVpTXDOhQXJCR01PVkpQMGM ● https://drive.google.com/file/d/1rPvfio6EkaUvGtgfQoq9p9Fa2ljOMIn1/view?usp=sharing ● https://drive.google.com/open?id=0B3dHGVpTXDOhVTBKX09NUnlLcm8