Available platforms for Big Data 2.0
- 1. Available platforms for Big Data 2.0 Based on presentations during Brno Data Week 2018 by prof Sherif Sakr Created by: Tichý, T. & Luhan, J. (Feb. 2019) https://www.chedteb.eu/
- 2. Dealing with data diversity in Big Data 2.0 Due to the fact that data is very diverse, we have to find the right platform which suits our needs. Big Data 2.0
- 3. Spark • Apache Spark is a fast, general engine for large scale data processing on a computing cluster (new engine for Hadoop) • Developed initially at UC Berkeley, in 2009, in Scala, it is currently supported by Databricks • One of the most active and fastest growing Apache projects • Committers from Cloudera, Yahoo, Databricks, UC Berkeley, Intel, Groupon and others Big Data 2.0: Platforms
- 4. Spark vs Hadoop Big Data 2.0: Platforms
- 5. Spark vs Hadoop Big Data 2.0: Platforms
- 6. Flink • Apache Flink is a distributed in-memory data processing framework which represents a flexible alternative for the MapReduce framework that supports both batch and real-time processing. • Flink has originated from the Stratosphere research project that was started at the Technical University of Berlin in 2009 before joining the Apache‘s Incubator in 2014. • Instead of the map and reduce abstractions, Flink uses a directed graph approach that leverages in-memory storage for improving the performance of the runtime execution. • Has a fast growing community. Big Data 2.0: Platforms
- 7. Hadoop vs Spark vs Flink Big Data 2.0: Platforms
- 8. Big Data 2.0 Processing Systems: SQL-On-Hadoop
- 9. Apache Hive • The first system that was introduced to support SQL-on-Hadoop with familiar relational database concepts such as tables and columns. • Hive has been widely used in many organizations to manage and process large volumes of data, such as Facebook, eBay, LinkedIn and Yahoo! • It supports an SQL-like declarative language, HiveQL, which represents a subset of SQL92 and therefore can be easily understood by anyone who is familiar with SQL. • Hive queries automatically compile into MapReduce jobs that are run by using Hadoop. Big Data 2.0: SQL based platforms
- 10. Impala • Open source project, built by Cloudera, to provide a massively parallel processing SQL query engine that runs natively in Apache Hadoop. • By using Impala, the user can query data which is stored in Hadoop Distributed File System (HDFS). • It uses the same metadata, SQL syntax (HiveQL) of Apache Hive. Big Data 2.0: SQL based platforms
- 11. IBM Big SQL • The SQL interface for the IBM big data processing platform, InfoSphere BigInsights • Big SQL relies on a built-in query optimizer that rewrites the input query as a local job to help minimize latencies by using Hadoop dynamic scheduling mechanisms. • It uses a massively parallel processing SQL engine that is deployed directly on the physical Hadoop Distributed File System (HDFS). Big Data 2.0: SQL based platforms
- 12. Presto • Open source distributed SQL query engine, built by Facebook, for running interactive analytic queries against large scale structured data sources of sizes of gigabytes up to petabytes. • Presto allows querying data where it lives, including Hive, NoSQL databases (e.g., Cassandra, HBase), relational databases or even proprietary data stores • A single Presto query can combine data from multiple sources • Presto has been recently adopted by big companies and applications such as Netflix and Airbnb Big Data 2.0: SQL based platforms
- 13. To be continued In the next episode we will focus on big streams (data that is constantly generated), how to analyze and vizualize them.