Data Science Toolchain 101
0 likes651 views
This document provides an overview of key concepts in data science and related technologies. It defines data science as extracting knowledge from data using various techniques. It then discusses concepts like the data-information-knowledge hierarchy, Apache Spark for large-scale data processing, YARN for resource management, RDDs for fault-tolerant databases, Apache Hive for data warehousing, HDFS for file storage, HBase for non-relational databases, Parquet for efficient data encoding, columnar databases for analytics, and the differences between OLTP for transactions and OLAP for analysis.
Data Science Toolchain 101
- 2. “Data Science is the extraction of knowledge from data using mathematics, statistics, computer science, machine learning, pattern recognition, predictive analysis, etc.” - Wikipedia “Information is not knowledge” - Einstein
- 3. - DIKW - Apache Spark - YARN - RDDs - Apache Hive - HDFS - Parquet - Niometrics - Columnar DB - DBMS - HBase - OLAP - OLTP
- 5. - signals, symbols, raw facts - first-line products of observation - unorganized Data 0101 1101 1011 1001 0101 1101 1011 1001 0101 1101 1011 1001 0110 0001 1010 1111 0101 1101 1011 1001 0101 1101 1011 1001
- 6. Information - inferred from data - answers interrogative questions - data that is now useful through organization and structuring
- 7. Knowledge - knowledge is subjective - many consider it as applied information - synthesis of multiple information - contextualized consolidated information
- 8. Wisdom - an appreciation of the why - knowing the right things to do - very immaterial
- 10. Apache Spark - fast engine for large-scale data processing - support for Python, Scala, Java - SparkSQL, MLlib, GraphX, Streaming
- 12. YARN - Yet Another Resource Negotiator - resource management for computing resources in a cluster - can be seen as a distributed operating system - separates resource management from Hadoop data processing layer
- 13. Apache Hive - data warehouse infrastructure - provides data querying, analysis, and aggregation - developed initially by Facebook
- 14. Resilient Distributed Databases (RDDs) - fault-tolerant database management system used for cluster computing - done by chunking data across multiple nodes and racks for redundancy - a common feature in cluster computing
- 15. HDFS - Hadoop Distributed File System - Runs with RDDs in managing a fault-tolerant file system - Java based and spans clusters of commodity servers
- 17. HBase - Distributed non-relational database - modelled after Google’s BigTable - runs on top of HDFS - fault-tolerance on sparse and large data - supports compression, in-memory filtering and operations
- 18. Parquet - Columnar file format - stores data in columns instead of rows as in traditional relational databases - for efficient compression and encoding - more aligned for OLAP
- 19. Columnar Database - Stores tables as sections of columns - advantageous for data warehousing - more efficient for computations over large numbers of rows with similar column items - more aligned for OLAP
- 20. Online Transaction Processing (OLTP) - Information processes that facilitate transactions - data entry and retrieval - provide data for data warehousing - emphasis on fast simple single querying, - ACID, and multi-access - involve operational business processes
- 21. Online Analytical Processing (OLAP) - low transactional volume - complex queries with aggregation - OLAP uses data from OLTP systems - queries involve traversing massive quantities of data - involves business intelligence/data science activities
- 24. Thank You