Introduction to Big data tdd and pig unit
0 likes1,175 views
The document discusses test-driven development (TDD) and how to apply it to Pig programs using the PigUnit testing framework. TDD involves writing tests before code to ensure new code passes all tests. PigUnit allows running Pig scripts from JUnit tests, overriding variables, inspecting relation values, and preventing side effects from STORE statements. The document provides an example of using TDD with PigUnit to test a Pig program.
Introduction to Big data tdd and pig unit
- 1. What is Big Data And Why learn Hadoop View Big Data and Hadoop Course at www.edureka.co/my-course/big-data-and-hadoop www.edureka.co/big-data-and-hadoop
- 2. How it Works? LIVE Online Class Class Recording in LMS 24/7 Post Class Support Module Wise Quiz Project Work Verifiable Certificate Twitter @edurekaIN, Facebook /edurekaIN, Slide 2 use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 3. Objectives What is TDD ? I Can’t follow TDD because… Traditional Development Cycle Vs TDD Why Unit Test Pig? What is PigUnit? TDD Using PigUnit- Demo Slide 3 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 4. What is TDD? TDD stands for Test Driven Development Test Driven Development aims to shorten the development cycles It aims to “get something now and perfect it later” approach The typical process involves “RED-GREEN-REFACTOR” cycle It’s a part of larger software design paradigm- “Extreme Programming” Test Driven Development requires tests to be written before code itself! It leads to a better code which is just enough to pass the tests 100% code coverage is ensured for TDD based code Slide 4 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 5. I Can’t follow TDD Because… “It’s working! Let’s freeze it for now” The release date is quite aggressive! It slows down our development cycle We are already short staffed What are Testers supposed to do? All (or possibly more) reasons above lead the teams for “Technical Debt” Slide 5 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 6. -Albert Einstein Slide 6 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 7. Time Taken to Fix Bugs 1000 750 500 250 0 Design Implementation QA Post-release Slide 7 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 8. Traditional Development Test Design Implement Slide 8 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 9. TDD Implement Design Test Test Slide 9 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 10. TDD Design Test Test Implement Slide 10 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 11. TDD Design Test Test Implement Slide 11 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 12. Why Unit Test Pig? Pig is NOT a programming language Pig is a Data Flow Language It just converts the Pig Latin data flows to Map-Reduce jobs The best use-case for Pig in Big Data projects is for “Data Factory” operations Since we are not talking about a “programming language”, does testing make sense? Pig already comes with the diagnostic operators, so extra testing will be overhead! All of the above reasons lead to even bigger problems, as the testing in Big Data world is data driven in nature Slide 12 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 13. What is PigUnit? PigUnit is the Unit testing framework for Pig scripts It is not really a *Unit framework It’s a library which can be used within JUnit tests to » Run Pig scripts from within JUnit tests » Override variables in Pig scripts to provide data from tests rather than from external sources such as HDFS » Inspect the values of your Pig script relations » Make your STORE statements into no-ops so that your Pig scripts run without side effects. Slide 13 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop
- 14. TDD Using PigUnit - Demo Slide 14 Twitter @edurekaIN, Facebook /edurekaIN, use #askEdureka for Questions www.edureka.co/big-data-and-hadoop