Hadoop etl
- 2. What is ETL?
- 3. ETL… • Extract: Extract data from source system and make it accessible for further processing. • Clean Ensures quality of data in warehouse. • Transform: Applies a set of rules on data flowing from source to data. • Load Load in destination database. • Staging
- 4. Benefits & Limitations of Traditional ETL • Benefit • Helps to address issues like Change Management, Slowly changing dimensions, inserts, updates, etc. • Limitations • Early in process someone has to decide what data is important, who can access it, what should be updated. • Original raw data is not stored and can’t be retrieved.
- 5. ETL Bottleneck in Big Data Analytics • Business benefits of analyzing of Big Data can be significant.
- 8. ETL Tools • For Traditional ETL tools like Teradata Warehouse Builder, DataStage are used. • For ETL with Hadoop, Apache Sqoop, Apache Flume are used .
- 9. Apache SQOOP • Sqoop is designed to transfer data between Hadoop and Relational Databases. • You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.
- 11. What is Sqoop ? • A different paradigm Data Application Data
- 12. • A very scalable different paradigm Data Data Application Data Application Data
- 13. • Where did the transform go ? Application Data TTT TTT TTT TTT
- 14. Any Questions ?
- 15. References • https://www.ibm.com/developerworks/library/bd-hivetool/index.html • https://www.csgsolutions.com/blog/what-is-etl/ • https://software.intel.com/sites/default/files/article/402274/etl-big-data- with-hadoop.pdf