seminar presentation on apache-spark
Download as PPTX, PDF3 likes918 views
Apache Spark is a fast, general engine for large-scale data processing. It was created at UC Berkeley and is now an Apache open-source project. Spark provides high-level APIs in Scala, Java, Python and R that support distributed computing using the MapReduce paradigm. It can run on Hadoop, Mesos and standalone, and supports real-time streaming, machine learning and graph processing.
seminar presentation on apache-spark
- 1. SUBMITTED TO: SUBMITTED BY: Mrs. Suman singh Nikita Vijay (HOD of CSE Dept.) B. Tech –VIII sem(CSE) A SEMINAR PRESENTATION ON “Introduction To Apache Spark”
- 2. ● Need of new generation distributed system ● Hardware/software evolution in last decade ● Apache Spark • Components of Apache Spark ● Why Spark? ● Who are using Spark? Agenda
- 3. ● Lot has been changed from 2000 ● Both hardware and software gone through changes ● Big data has become necessity now ● Let’s look at what changed over decade Why we need new generation?
- 4. ● Disk was cheap so disk was primary source of data ● Network was costly so data locality ● RAM was very costly ● Single core machines were dominant RAM is the king • RAM is primary source of data and we use disk for fallback ● Network is speedier ● Multi core machines are commonplace State of hardware in 2000 Now
- 5. ● Object orientation was the king ● Software optimized for single core ● No open frameworks for creating ○ Distributed storage ○ Distributed processing ● SQL was the only dominant way for data analysis Now •Functional programming is on rise ● Software needs to exploit multiple cores on single node There are good frameworks to create distributed systems ○ HDFS for storage ● NoSQL is real alternative now Software in 2000
- 6. ● Very few companies had big data issue ● Batch processing system ruled the world ● Volume was big concern compare to velocity ● Mostly used for ○ Search ○ Log analysis ● All companies use big data ● Velocity is as much concern as volume Needs of real time are as much important as batch processing Big Data processing needs in 2000 NOW
- 7. • A fast and general engine for large scale data processing • Created by AMPLab • Written in Scala •Licensed under Apache Apache Spark
- 10. Benefits of a Unified Platform • No copying of data between systems •Combine processing types in one program • Code reuse • One system to learn • One system to maintain
- 11. Mesos, a distributed system framework as class project in UC Berkeley in 2009. ● Spark to test how mesos works ● Focused on ○ Iterative programs (ML) ○ Unifying real time and batch processing ● Open sourced in 2010 History of Apache Spark
- 12. ● You can spark on top any distributed system ● It can run on ○ Yarn ○ Apache Mesos ○ It’s own cluster Runs everywhere
- 13. ● Apache Spark is highly modular The original version contained only 1600 lines of scala code ● Apache Spark API is extremely simple compared Java API of M/R ● API is concise and consistent Small and Simple
- 15. • In Spark, you can cache hdfs data in main memory of worker nodes • Spark analysis can be executed directly on in memory data ● Shuffling also can be done from in memory ● Fault tolerant In-memory aka Speed
- 16. ● No separate storage layer ● Integrates well with HDFS ● Can run on Hadoop 1.0 and Hadoop 2.0 YARN ● Excellent integration with ecosystem projects like Apache Hive, HBase etc Integration with Hadoop
- 17. ● Written in Scala but API is not limited to it ● Offers API in ○ Scala ○ Java ○ Python ● You can also do SQL using SparkSQL Multi language API
- 18. Who are using Spark