Indic threads pune12-apache-crunch
Download as PPT, PDF0 likes831 views
The document discusses issues with MapReduce pipelines and presents Apache Crunch as a solution to these challenges, highlighting its features such as lazy evaluation and support for various data formats. It explains the data model, operations, and system workflow within Apache Crunch, along with examples like word count and sorting. Additionally, it provides contact information and resources for further exploration of Apache Crunch.
Indic threads pune12-apache-crunch
- 2. Agenda : Issues with MapReduce pipelines Solving with Apache Crunch Data Model & Operations System Workflow Examples Question & Answers 2
- 3. Issues with MapReduce Pipelines Unit Testing pipeline ?? You must be joking !! Can someone tell me where is the business logic ?? Chain performance?? Learn Latin(pig) first!! 3
- 4. Apache Crunch Is a Java library Contains Collections which can excute Parallel operations Lazy evaluation of Collections at runtime Operations merged at runtime to have efficient chains. Available @ http://incubator.apache.org/crunch/ Based on Google FlumeJava paper 4
- 5. Apache Crunch Supports Hadoop version 1 and 2-alpha Supports HBase, jdbc etc Works with Writables, Avro, Thrift and proto-buffers Scala varient also exists Integration with R and Clojure in process Archetype exists for creating sample maven project 5
- 6. Apache Crunch : Data Model Pipeline MRPipeline MemPipeline PCollection<T> PTable<K,V> PGroupTable<K,V> Source<T> Target<T> Emitter<T> 6 PType<K,V>
- 7. Apache Crunch : Operations DoFn<S,T> CombineFn<S,T> FilterFn<T> Joins Cartesian Sort SecondarySort PObject<T> BloomFilters 7
- 8. Apache Crunch : System Workflow Construct a pipeline Pipeline.done() Map Map Map GBK GBK Reduce Reduce 8 Output
- 9. Apache Crunch : Examples WordCount example Avro example Sorting example SecondarySort Join Example BloomFilters 9
- 10. Write to me : rsharma@apache.org Example src : http://github.com/rahul0208 10 Blog : devlearnings.wordpress.com