OWF12/Java Michael hirt
Download as PPTX, PDF1 like1,506 views
This document discusses big data and Talend's goal of democratizing big data through its open source integration platform. It begins by defining big data and explaining the challenges it poses related to volume, velocity, variety and other factors. It then outlines Talend's goal of providing intuitive graphical tools to design and run big data jobs within Hadoop, abstracting away the underlying code generation. The document stresses that data quality is especially important for big data and how Talend supports implementing data quality checks either as part of loading data into Hadoop or as a separate job after loading. Finally it provides an overview of Talend's roadmap to add support for additional Hadoop technologies over time such as HCatalog, Oozie and more
OWF12/Java Michael hirt
- 1. Tackling Big Data with Hadoop and Graphical Open Source Integration Michaël Hirt Data Integration Product Manager
- 2. Agenda 1. What is Big Data ? 2. Talend’s Goal 3. What’s next ? Big Data Quality and Big Data management 4. Talend Open Studio for Big Data in action © Talend 2011 2
- 3. What is Big Data?
- 4. What Is BIG Data? 2,300 tweets per second "Big data" (June 2011) is information of extreme size, 50 gigabytes of data diversity, complexity per person on Earth 50,000,000,000 and need for rapid 300 exabytes total processing. 200 billion Ted Friedman - Information intelligent devices Infrastructure and Big Data Projects Key Initiative Overview - July 2011 200,000,000,000 2015 275 exabytes of data flowing over the Internet each day 275,000,000,000,000,000,000 © Talend 2011 2020 4
- 5. How to define Big data is…. Hans Rosling – uses big data to analyze world health trends Key Takeaway #1 volume, variety, velocity © Talend 2011 – Stri2y Private & Confidential © Talend 2011 5
- 6. The 6 Dimensions of BIG Data Primary challenges Volume Velocity Variety Complexity And also Validation Lineage © Talend 2011 6
- 7. Key Takeaway #2 Forces us to think © Talend 2011 differently © Talend 2011 – Stri2y Private & Confidential 7
- 8. Traditional Data Flows CRM ETL Normalized Traditional Data ERP Data Data Warehouse Quality Finance • Scheduled–daily or weekly, sometimes more frequently. Business Business Analyst User • Volumes rarely exceed terabytes Warehouse Administrator Executives © Talend 2011 – Stri2y Private & Confidential © Talend 2011 8
- 9. The new world of big data Social Networking CRM ERP Big Data Finance © Talend 2011 – Stri2y Private & Confidential © Talend 2011 9
- 10. The new world of big data Social Networking CRM Mobile Devices ERP Big Data Finance © Talend 2011 – Stri2y Private & Confidential © Talend 2011 10
- 11. The new world of big data Social Networking CRM Mobile Devices ERP Transactions Finance Network Devices Big Data Sensors © Talend 2011 – Stri2y Private & Confidential © Talend 2011 11
- 12. Data driven business enables data governance supports information decisions drives Information provides value to the business If you can't rely on your information then Your the result can be missed opportunities, or business higher costs. Matthew West and Julian Fowler (1999). Developing High Quality Data Models. The European Process Industries STEP Technical Liaison Executive (EPISTLE). © Talend 2011 – Stri2y Private & Confidential © Talend 2011 12
- 13. BIG Data Management Big Data Big Data Management Production Big Data Big Data Big Data RDBMS Integration Quality Consumption Analytical DB NoSQL DB ERP/CRM Mining SaaS Social Media Analytics Web Analytics Log Files Storage Processing Search RFID Filtering Call Data Records Sensors Enrichment Machine-Generated Turn Big Data into actionable information © Talend 2011 13
- 14. BIG data driven business enables BIG data governance supports BIG BIG information decisions drives Information provides value to the business If you can't rely on your information then the result can be missed opportunities, or BIG higher costs. business Matthew West and Julian Fowler (1999). Developing High Quality Data Models. The European Process Industries STEP Technical Liaison Executive (EPISTLE). © Talend 2011 – Stri2y Private & Confidential © Talend 2011 14
- 15. Our goal
- 16. Talend – The Market Leading Unified Integration Platform Talend Enterprise Data Data MDM ESB BPM Quality Integration Commercial license Subscription model Studio Repository Deployment Execution Monitoring Open source license Talend Open Studio for Free of charge Optional support Data Data Quality Integration MDM ESB Recognized as the open source leader in each of its market category by all industry analysts © Talend 2011 16
- 17. Trying to get from this… © Talend 2011 – Stri2y Private & Confidential © Talend 2011 17
- 18. to this… Why Talend… ONLY Talend generates code that is executed within map reduce. This open approach removes the limitation of a proprietary “engine” to provide a truly unique and powerful set of tools for big data. © Talend 2011 – Stri2y Private & Confidential © Talend 2011 18
- 19. “Big Data for the Masses”
- 20. Goal: Democratize Big Data Talend Open Studio for Big Data “Big Data for the Masses” Improves efficiency of big data job design with graphic interface Abstracts and generates code Run transforms inside Hadoop Native support for HDFS, Pig, HBase, Pig Sqoop and Hive Apache License 2.0 Embedded in Hortonworks Data Platform …an open source Certifed with Cloudera, MapR and ecosystem Grenplum © Talend 2011 – Stri2y Private & Confidential © Talend 2011 20
- 21. Big Data – How about Data Quality? © Talend 2012
- 22. Poor Data Quality + Big Data = Big Problems Poor Data Quality * Big Data = Big Problems^2 Key Takeaway #3 In big data… poor data quality can be magnified at huge scale © Talend 2011 23
- 23. Two methods for inserting data quality into a big data job 1. Pipelining: as part of the load process 2. Load the cluster then implement and execute a data quality map reduce job © Talend 2011 24
- 24. E-T-L Extract – Transform - Load © Talend 2011 – Stri2y Private & Confidential © Talend 2011 25
- 25. E- DQ -L Extract – Improve/Cleanse - Load © Talend 2011 – Stri2y Private & Confidential © Talend 2011 26
- 26. Pipelining: data quality with big data CRM DQ ERP DQ Finance Big Data Social Networking • Use traditional data quality tools • No new programming, no PHDs • Once and done Mobile Devices © Talend 2011 – Stri2y Private & Confidential © Talend 2011 27
- 27. Big data alternative: Load and improve within the cluster CRM DQ ERP DQ Finance Big Data Social Networking • Load first, improve later • Really complex to build, limited tools • Constant on, increments Mobile Devices • Insane performance © Talend 2011 – Stri2y Private & Confidential © Talend 2011 28
- 28. Let us show you… © Talend 2012
- 29. What’s next for Talend Big Data? © Talend 2012
- 30. Talend Open Studio for Big Data 4.1: Hive & 5.1:HCatalog 4.0: HDFS 4.2: Pig 5.0: Hbase Sqoop & Oozie © Talend 2011 31
- 31. big 2012 data now Q4 2013 Talend Open Studio for Big Data Packaged within Hortonworks Data Platform …Eclipse tools for HIVE, HDFS, PIG, SCOOP …supports Oozie, Hcatalog, Kerberos Free to download and use under the Apache license …democratizing big data through intuitive tools © Talend 2011 – Stri2y Private & Confidential © Talend 2011 32
- 32. Questions / Thanks for attending mhirt_at_talend.com
Editor's Notes
- #15: This is a classic diagram that maps how business and data are related. Nothing is new. This never changes. In fact in becomes even more important today.
- #17: We accomplish this innovation by offering two editions of our products. The Talend Open Studio, at the bottom of this diagram, is a set of free open source products for Data Quality, Data Integration, Master Data Management, Enterprise Service Bus and Business Process Management. And when you are ready to deploy, you can purchase a Talend Enterprise commercial license, which includes the features found in world-class integration solutions such as extreme scalability, high availability and 24x7 mission-critical support – all backed by a large services and partner ecosystem. Unlike competitors “non-integrated” integration products, Talend’s uniqueness is in the unification of our products – they are built from the same unified platform, maximizing your productivity and providing greater software reuse and repeatability. An analogy would be the user experience you see with the integration of the iPod, iPad and iPhone. As shown in this picture, our products leverage the same studio, repository, and deployment, execution and monitoring tools to maximize your productivity. As modular products, you can buy what you need when you need it, or easily combine them to solve more comprehensive integration problems.
- #18: For instance, this is a SIMPLE drawing of how the map reduce features work. This is abstract and does not reflect the complexity of code. Still pretty complex.
- #19: Big data has an OPERATION DI challenge. This is the core of what talend was built on and part of our DNA. We simplify the process of implementation to speed projects and increase adoption.Note: I am trying to get a recording that can be embedded in the slide that will build a HDFS load as you speak. It is so simple that it was completed in the time it took for me to present this slide!
- #21: Finally, the entire big data world has been built as an open source ecosystem. This all makes sense… talend is the open source leader.To this end we will introduce the first compelte set of tools that will democratize big data. Talend Open Studio for Big Data
- #24: However, with big data comes significant challenges. For example, poor data quality can be magnified at huge scale. Consider a small company with 100 customers. Assume they had a bad address for three customers and sent a mailer out to their list. Three mailers would be returned and they would have wasted about 5 dollars or so. Now imagine the world of big data where this number of customers expands across business lines and companies and partners to millions. The costs are big. Even more interesting is the ability to not only use the data but to analyze. Across your customer base, how could you monitor and analyze every interaction they ever had with you (social media, web, stores, etc). This is large amounts of data. A small problem with the data can lead to very LARGE issues with analysis, invalidating the entire reason for big data. Data quality is KEY for big data – it is a core tenant of our strategy.
- #30: demo