Ataas2016 - Big data hadoop and map reduce - new age tools for aid to testing and qa by aditya garg
0 likes394 views
The document discusses the use of big data tools like Hadoop and MapReduce in testing and quality assurance. It highlights the challenges posed by big data applications across various industries and details how QA teams can utilize these tools for efficient testing processes. Key components such as HDFS, Flume, and HBase are introduced as solutions to manage and analyze large data sets.
Ataas2016 - Big data hadoop and map reduce - new age tools for aid to testing and qa by aditya garg
- 1. Big Data - Hadoop and MapReduce - new age tools for aid to testing and QA by Aditya Garg Confidential | Copyright © QAAgility Technologies
- 2. Aditya Garg @Adigindia Co-Founder and Director QAAgility.com Co-founder & Steering Committee Member of Agile Testing Alliance – run meetup groups across multiple cities Co-creator and licensed trainer of Agile Testing Alliance’s certifications CP-BAT, CP-MAT, CP-AAT, CP-SAT Co-Author of a book on Selenium Co-Author of a book on Selenium Love Cooking Indian Dishes Tasting (Testing) World food Travelling and meeting testers (Get inspired and may be inspire a few) @adigindia https://www.linkedin.com/in/adigarg
- 3. Big Data - Hadoop and MapReduce - new age tools for aid to testing and QA Topic for the presentation for aid to testing and QA
- 4. What is this Confidential | Copyright © QA Agility Technologies
- 5. 1. How to test Big Data applications ? 2. How can QA and Testing What are we going to discuss ? 2. How can QA and Testing team use Big Data tools for their testing needs ?
- 6. 1. How to test Big Data applications ? 2. How can QA and Testing What are we going to discuss ? 2. How can QA and Testing team use Big Data tools for their testing needs ?
- 7. What is Big Data ? Is it just too much Hype or Confidential | Copyright © QA Agility Technologies Is it just too much Hype or reality ?
- 8. Let us start with what exactly is BigData Confidential | Copyright © QA Agility Technologies
- 9. Which Search Engine do you use ? http://searchstorage.techtarget.com/definition all-that How much data does Google store ? https://www.cirrusinsight.com/blog/how-much-data-does-google-store http://searchstorage.techtarget.com/definition /Kilo-mega-giga-tera-peta-and-all
- 10. On Search Engines – Anyone using DuckDuckGo?
- 12. Key Points in Big Data 1.Volume – Data Explosion 2.Velocity 3.Variety 4.Veracity
- 13. Key Points in Big Data Ref: IBM.com
- 14. Definition Big datais the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional Ref: goo.gl/iWZhjJ management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. http://www.forbes.com/sites/gilpress/2014/09/03/12-big-data-definitions-whats- yours/#379879e621a9
- 15. Big Data Application 1. Finance 2. Insurance 3. Health Care 4. Agriculture 5. Defense5. Defense 6. Manufacturing 7. Aero Space 8. Oil and Gas 9. Advertisement and Marketing 10.Election Campaigns 11. List goes on --- applicability across industries
- 16. Lets go back to definition Big datais the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processingtools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization.
- 17. Tools solving Big Data Challenge Confidential | Copyright © QA Agility Technologies
- 18. Tool solving the Big Data Challenge
- 19. Hadoop – Key components HDFS and MR *Source Udacity
- 20. 1. Sqoop takes data from regular RDBMS and puts it into HDFS 2. Flume ingests data into HDFS as it is generated by external systems 3. HBASE is real time Hadoop Ecosystem *Source Udacity 3. HBASE is real time database on top of HDFS 4. Hue is a graphical front end to the cluster 5. Oozie is workflow management tool 6. Mahout is Machine Learning library
- 21. HDFS • HDFS stands for Hadoop Distributed File System, which is the storage system used by Hadoop. The following is a high-level architecture that explains how HDFSarchitecture that explains how HDFS works.
- 22. Map Reduce Ref: Emanuele Della Valle @manudellavalle
- 23. Understanding MapReduce Demo – Word Count Confidential | Copyright © QA Agility Technologies Demo – Word Count Given an input file, count unique words
- 24. WordCount – Map Reduce Reference : http://wearecloud.cz/media/files/prezentace-biz/Big%20Data%20v%20Cloudu.ppt
- 25. How can QA and Testing team use Big Data tools Confidential | Copyright © QA Agility Technologies team use Big Data tools for their testing needs ?
- 26. Problem Statement and Solution using Hadoop and MapReduce Confidential | Copyright © QA Agility Technologies and MapReduce
- 27. MTBT – Multicast Tick by Tick Adapter Input was exchange feed – Output given to HFT Engine Legacy Adaptor (3rd Party) connects to the TAP – and converts to a format which can be used by HFT MTBT - Adaptor Exchange TAP – Co-location servers listen to it at high speed can be used by HFT Platforms (Algorithmic Trading Platforms) New Adaptor – being made Inhouse – to increase the speed by 10 Times HFT Engine
- 28. MTBT – Multicast Tick by Tick Adapter •Client was trying to build a brand new MTBT Exchange Adaptor •The adaptor was being developed in C and Unix and was to run in a co-location with NSE (National Stockwas to run in a co-location with NSE (National Stock Exchange) •The new adaptor was supposed to increase the overall speed by more than 10 times from the existing adaptor •The Goal was to test the new adaptor
- 29. MTBT - Adaptor Challenges -------------------------------------------------- 1. Manually next to impossible 2. Even few seconds samples were running into large MegaBytes (MB) files 3. Manually impossible to compare MTBT – Challenges Input Output Output over time 3. Manually impossible to compare the legacy records with the New code processed records 4. Daily processed data ran into 150 Giga Bytes (GB) plus files
- 30. MTBT – SOLUTION 1 Reduce LEGACY MTBT - Output file into a standard format 2 Reduce NEW INHOUSE MTBT output file into a standard format 3 Compare the two files 4 Generate Report
- 31. QA team can use the tools in multiple scenarios 1. Beta Testing 2. Repeated execution effectiveness – applying analytics ( R) 3. Capturing Customer feedback and Other scenarios – Big Data Tool implementation Confidential | Copyright © QA Agility Technologies 3. Capturing Customer feedback and channeling the same for smarter test execution 4. Extracting relevant information from repeated regression cycles from QC 5. Adding intelligence on the data generated by the testing team
- 32. Thank you and Jai Hind Questions ? @adigIndia@adigIndia @AgileTA #GTR2016
- 33. Contact Please contact us at info@QAAgility.com Confidential | Copyright © QAAgility Technologies MUMBAI 711, Rupa Solitaire MBP, Mahape Navi Mumbai-400701 DENMARK 1 Lindebo 7 Lej - 42, 2630 Tasstrup, Copenhagen +45.7164.0278 denmark@qaagility.com USA 200 E Campus View Blvd. Suite 200, Columbus, OH