Apache hama @ Samsung SW Academy
Download as PPT, PDF4 likes908 views
Apache Hama is an open source Bulk Synchronous Parallel computing framework written in Java. It provides a general BSP engine and supports iterative algorithms, graph processing, machine learning and continuous event processing on big data. Hama uses a message passing model and guarantees deadlock freedom. It is suitable for applications with complex relationships like social networks.
Apache hama @ Samsung SW Academy
- 1. Apache Hama a Bulk Synchronous Parallel Computing Edward J. Yoon <edwardyoon@apache.org>
- 2. Who Am I • Edward J. Yoon – @eddieyoon • Founder of Apache Hama • PMC member of Apache BigTop • Oracle Employee
- 3. What’s Hama? • Open Source – Under Apache 2.0 License • Written In Java • Apache Top Level Project
- 5. Characteristics • a General BSP computing engine – M/R like Input/Output Formatter • SequenceFile, Text, Accumulo, Hbase, …, etc. – Job Manager – Checkpoint Recovery • Streaming and Pipes – Python, C++, …, etc. • Graph and Machine Learning Packages – K-means, Gradient Descent, Collaborative Filtering
- 6. Bulk Synchronous Parallel? • Originally introduced by Valiant • a Sequence of supersteps
- 7. Compare to M/R and MPI • Supports message-passing paradigm style of application development • Provides a flexible, simple, and easy-to-use small APIs • Enables to perform better than MPI for communication-intensive applications • Guarantees impossibility of deadlocks or collisions in the communication mechanisms
- 8. So, fit for what? • Processing Big Data w/ complicated relationships – e.g., graph or network. • Iterative or Recursive scientific applications • Continuous Event Processing
- 9. Which is the Big Data?
- 10. Could be applied to • Analyze user actions and patterns • Social Target Marketing • Observe evolution of Social networks • Detect anomaly rapidly in Real-time • Business Intelligence
- 11. Internals • Pluggable RPC Architecture for message transfer – e.g., Hadoop RPC, Avro RPC, …, etc. • Message Collector, Bundler, and Compressor to reduce network overheads and contentions – e.g., Snappy, Bzip2, …, etc.
- 12. BSP API public abstract void bsp(BSPPeer<K1, V1, K2, V2, M> peer) throws IOException, SyncException;
- 13. BSP Examples • Pi Calculation • Sparse Matrix-Vector Multiplication • K-means Clustering • Gradient Descent
- 14. Graph API public void compute(Iterator<M> messages) throws IOException;
- 15. Graph Examples • In-link Count • Single Source Shortest Path • Pagerank • Bipartitie Matching • Semi-Clustering
- 17. SSSP Performance • a SSSP for random graph of 1 billion edges is computed in 400 seconds on 1 Oracle BDA