Hadoop gets Groovy
Download as PPTX, PDF9 likes2,969 views
The document discusses using Groovy to interact with Hadoop. Some key points: - Groovy allows mappers and reducers to be written more concisely using closures and lists compared to Java. - Groovy scripts can be run from Hadoop jobs by compiling scripts into classes at runtime. - Performance of Groovy has improved with Java 7 but Pig should be used over Groovy if possible for production jobs. - Groovy is best used for testing and extending Hadoop classes, not entire production pipelines.
![& Configuration gets better
conf['mapscript'] = new File(src).text
String scriptText = conf['mapscript']
conf.add([
window:60000,
'redscript':reduceScript
])
Extending to Job class trickier –subclassing better
Page 12
© Hortonworks Inc. 2012](https://image.slidesharecdn.com/2012-06-hadoopgetsgroovy-120604053240-phpapp01/85/Hadoop-gets-Groovy-12-320.jpg)
![New today! script driven MR jobs!
protected void setup(Mapper.Context ctx) {
this.ctx = ctx
this.conf = ctx.configuration
ScriptCompiler comp = new ScriptCompiler(conf)
String scriptText = conf['mapscript']
map = comp.parse(scriptText, this, ctx)
}
protected void map(Writable key, Writable value,
Mapper.Context ctx) {
map.setProperty('key',key)
map.setProperty('value',value)
map.run()
}
Page 13
© Hortonworks Inc. 2012](https://image.slidesharecdn.com/2012-06-hadoopgetsgroovy-120604053240-phpapp01/85/Hadoop-gets-Groovy-13-320.jpg)
Hadoop gets Groovy
- 1. Hadoop gets Groovy Steve Loughran– Hortonworks stevel at hortonworks.com @steveloughran Berlin, June 2012 © Hortonworks Inc. 2012
- 2. Where are you in this diagram? Hadoop Skills Doug,Owen Arun, Jakob Groovy Skills @steveloughran James Strachan Guillamue Laforge Page 2 © Hortonworks Inc. 2012
- 3. Grumpy : Groovy Hadoop Library • Something lightweight for testing • Wanted to play in the M/R layer • Already using Groovy • Liked: JVM integration, tooling, libraries, IntelliJ IDEA, Books… git@github.com:steveloughran/grumpy.git Page 3 © Hortonworks Inc. 2012
- 4. What is Groovy? A dynamic language within the JVM • Java++ –Maps, lists, tuples, Closures • Flavours of Ruby and Python –'Duck' typing, Grails, (Scripting) A way to do things in the JVM that Sun didn't imagine Page 4 © Hortonworks Inc. 2012
- 5. Can use & subclass java classes: class LineCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> { static final def emitKey = new Text("lines") static final def one = new IntWritable(1) void map(LongWritable key, Text value, Mapper.Context context) { context.write(emitKey, one) } } Page 5 © Hortonworks Inc. 2012
- 6. Closures & lists class CountReducer2 extends Reducer { def reduce(Text k, Iterable values, Reducer.Context ctx) { def sum = values.collect() {it.get() }.sum() ctx.write(k, new IntWritable(sum)); } } Page 6 © Hortonworks Inc. 2012
- 7. Closures & lists values.collect() { it.get() }.sum() List<values> -> List<int> -> int Page 7 © Hortonworks Inc. 2012
- 8. Result: MR jobs in Groovy In: gate1,b46cca4d3f5f313176e50a0e38e7fde3,,2006-10-30,16:06:17,Fleurball gate1,f1191b79236083ce59981e049d863604,,2006-10-30,16:06:20,vklaptop gate1,b45c7795f5be038dda8615ab44676872,,2006-10-30,16:06:21,Franky Panky gate1,02e73779c77fcd4e9f90a193c4f3e7ff,,2006-10-30,16:06:23, gate1,eef1836efddf8dbfe5e2a3cd5c13745f,,2006-10-30,16:06:24,Vas gate1,b46cca4d3f5f313176e50a0e38e7fde3,,2006-10-30,16:06:32,Fleurball gate1,f1191b79236083ce59981e049d863604,,2006-10-30,16:06:36,vklaptop gate1,b45c7795f5be038dda8615ab44676872,,2006-10-30,16:06:37,Franky Panky gate1,eef1836efddf8dbfe5e2a3cd5c13745f,,2006-10-30,16:06:38,Vas gate1,02e73779c77fcd4e9f90a193c4f3e7ff,,2006-10-30,16:06:43, gate1,2afaf990ce75f0a7208f7f012c8d12ad,,2006-10-30,16:06:54,Smiley Out: 163,198,223 device sightings! Page 8 © Hortonworks Inc. 2012
- 9. why no Pig? Sliding Window Debounce void map(LongWritable key, BlueEvent event, Mapper.Context context) { BlueEvent ev2 = window.insert(event) List<BlueEvent> expired = window.purgeExpired(event) expired.each { evt -> emit(context, evt) } } void cleanup(Mapper.Context context) { window.each { evt -> emit(context, evt) } } Page 9 © Hortonworks Inc. 2012
- 10. Device sightings by day for 2007 1600000 1400000 1200000 1000000 800000 600000 400000 200000 0 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141 146 151 156 161 166 171 176 181 186 191 196 201 206 211 216 221 226 231 236 241 246 251 256 261 266 271 276 281 286 291 296 301 306 311 316 321 326 331 336 341 346 351 356 361 366 Page 10 © Hortonworks Inc. 2012
- 11. Improving Hadoop APIs Configuration.metaClass.setAt = { key, val -> set(key.toString(), val.toString()) } Configuration.metaClass.getAt = { key -> get(key) } Configuration.metaClass.add = {map -> map.each {elt -> set((elt.key).toString(), (elt.value).toString() ) } Page 11 © Hortonworks Inc. 2012
- 12. & Configuration gets better conf['mapscript'] = new File(src).text String scriptText = conf['mapscript'] conf.add([ window:60000, 'redscript':reduceScript ]) Extending to Job class trickier –subclassing better Page 12 © Hortonworks Inc. 2012
- 13. New today! script driven MR jobs! protected void setup(Mapper.Context ctx) { this.ctx = ctx this.conf = ctx.configuration ScriptCompiler comp = new ScriptCompiler(conf) String scriptText = conf['mapscript'] map = comp.parse(scriptText, this, ctx) } protected void map(Writable key, Writable value, Mapper.Context ctx) { map.setProperty('key',key) map.setProperty('value',value) map.run() } Page 13 © Hortonworks Inc. 2012
- 14. Things to consider • Performance: Groovy 2 on Java7 • 'False friends' -Types, if(), exceptions • If you can use Pig, use it. • Use Groovy for testing, extending Hadoop classes (output formatter, etc) • Play with YARN and Giraph with it Page 14 © Hortonworks Inc. 2012
- 15. Questions? hortonworks.com Page 15 © Hortonworks Inc. 2012
- 16. hortonworks.com Page 16 © Hortonworks Inc. 2012
- 17. Performance? • Groovy 1 over-introspects • HLL hides a lot of overhead • If your work is I/O bound, less important • Speed of development vs execution • Need to benchmark on Java 7 Page 17 © Hortonworks Inc. 2012
Editor's Notes
- #3: What is the knowledge/skill level of the audience?I’m taking about Groovy, not time to cover Hadoop as wellDisclaimer: I am a Groovy user, not an Expert.
- #5: How to describe Groovy? It depends on what you are doing with it? You can look at it and come to different conclusions based on your useIt's like Java with the datatypes they forgotIt's got ruby concepts (closures), keywords from python, and dynamic 'duck' typingIt lets you do things in the JVM that the original Java authors didn't expect
- #8: It's a map and reduce -in the reduce. Turtles all the way down. Or at least elephants.