Empowering you with Democratized Data Access, Data Science and Machine Learning
0 likes477 views
The document discusses IBM's solutions for democratizing data access and operationalizing machine learning through tools like IBM Big SQL, which enables integration and analysis of disparate data sources. It highlights case studies of various companies leveraging Big SQL for enhancing productivity, optimizing data workflows, and gaining actionable insights, all while addressing common challenges faced during data analysis. The content emphasizes the performance and capabilities of Big SQL, as well as its role in supporting enterprises to modernize their data architectures.
![28
Reduce data center footprint with Spectrum Scale
HDFS
Raw
data
ext4
ext4
write move copy
Traditional
applicationsCopies in both HDFS and ext4
Spectrum
Scale
Application
writes direct
to Hadoop
Path with
NFS/SMB/
Object/POSIX
direct-read with
NFS/SMB/
Object/POSIX
Raw
data
Traditional
applications
Multiple copies with HDFS based workflow
Spectrum Scale in-place analytics (No copies required)
Hadoop
analysis Jobs Hadoop
analysis Jobs
Direct read,one version
Data Scientists waste daysjust copying data to HDFS No copies required with Spectrum Scale
Costly data protection - Default uses 3-wayreplication with HDFS
IBM ESS Software RAID eliminatesneed for 3 way replication. Just 30%
extra storage requirement.
[Erasure coding in HDFS has limitations and is good only for cold data]
Traditional
applications Traditional
applications
HDFS
APIs
HDFS
APIs
Example: For 5PB of data, HDFS requires 15PB of storage Example: For 5PB of data, ESS requires 6.5PB of storage
Copy process can take hours/days & eventually results are based on stale data.](https://image.slidesharecdn.com/panditprasadempoweringyouwithdemocratizeddataaccessdatascienceandmachinelearning-170921020146/85/Empowering-you-with-Democratized-Data-Access-Data-Science-and-Machine-Learning-28-320.jpg)
Empowering you with Democratized Data Access, Data Science and Machine Learning
- 1. Empowering YOU with Democratized Data Access, Data Science and Machine Learning
- 2. 2 Acknowledgements and Disclaimers Availability. References in this presentationto IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. The workshops, sessions and materials have beenprepared by IBM or the session speakers and reflecttheir own views. They are provided for informational purposes only, and are neither intended to,nor shall have the effect of being,legal or other guidanceor advice to any participant. While efforts were made to verify the completeness and accuracy of theinformationcontained in this presentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the useof, or otherwise relatedto, this presentation or any other materials. Nothingcontainedin this presentationis intendedto, nor shall havethe effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicablelicense agreementgoverning theuse of IBM software. All customer examples described are presentedas illustrations of how thosecustomers have usedIBM products and theresults they may have achieved. Actual environmental costs andperformancecharacteristics may vary by customer. Nothing containedin these materials is intended to, nor shall have the effect of, statingor implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. © Copyright IBM Corporation 2017. All rights reserved. — U.S. Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM, the IBM logo, ibm.com, Big SQL are trademarks or registered trademarks of International Business Machines Corporationin the United States, other countries, or both. If these andother IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or TM), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at §“Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml §TPC Benchmark, TPC-DS, and QphDS are trademarks of Transaction Processing Performance Council §Cloudera, the Cloudera logo, Cloudera Impala are trademarks of Cloudera. §Hortonworks, the Hortonworks logo and other Hortonworks trademarks are trademarks of Hortonworks Inc. in the UnitedStates andother countries. §Other company, product,or service names may be trademarks or service marks of others.
- 3. 3 Operationalizing machine learning and getting actionable insights from disparate sources of data has been a huge challenge Data Integration/Data Engineering Team Data Science Team and Data Engineering Team Application Development Team Line of Business Data still lives in Silos IBM Db2 Operationalize Machine Learning Organization needs to act fast ACT NOW!
- 4. 4 Amy Chris Data Engineer Ryan Data Scientist Chris Data Engineer Nick Application Developer Amy needs to work with different teams who perform specific tasks to execute the campaign 4 Product details Customer details Sales campaign Operationalize Machine Learning
- 5. 5 With Big SQL, Amy’s team can save time on execution and enhance Productivity Federation Spark Integration Application Integration Ryan Data Scientist Nick Application Developer ONE TIME DEVELOPMENT 5 Product Details Sales Campaign Customer Details IBM Big SQL Chris Data Engineer
- 6. 6 Democratize Data Science and Machine Learning Data Ingestion Data Transformation/ Data Science/ Machine Learning Data Visualization Virtualize disparate data sources like Hadoop, RDBMS, and Object Stores (S3) to join data in a single query Manipulate data and operationalize data science models written in various languages Perform data discovery, analyze, and visualize business results in notebooks or other BI tools
- 7. 7 How are other companies benefiting by using Big SQL
- 8. 8 Want to modernize your EDW without long and costly migration efforts Offloading historical data from Oracle, Db2, Netezza because reaching capacity Operationalize machine learning Need to query, optimize and integrate multiple data sources from one single endpoint Slow query performance for SQL workloads Require skill set to migrate data from RDBMS to Hadoop / Hive Do you have any of these challenges?
- 9. 9 Big Fish Games – Uses Big SQL to combine disparate data to drive product innovation through customer feedback with the use of analytics “The ability to answer complicated questions with data from disparate sources will allow our analysts to focus on answering business questions without having to worry about where the data lives or waiting on a project to perform the data integration for them.” -- David Darden, BI Engineering Manager, Big Fish Games Business need: • Understand which product features resonate the best with the gaming community • Increase cross sell and up-sell opportunities Solution: • Combines structured (customer) data from PureData System for Analytics with semi-structured (game log) data in Hadoop Beta Experience and Outcomes: Puts users in charge of data analysis - Access to data without technology getting in the way Faster insights – fast data movement to Hadoop, 3X faster than Sqoop Leverage existing skills - IBM Big SQL enables and leverages existing SQL skill set
- 10. 10 Southwest Power Pool – Uses Big SQL to federate and reuse applications while taking first steps in establishing an enterprise data lake Business need: • Near term requirements: Offload less frequently used data • Long term vision: Build a data lake infrastructure Solution: • Offloaded cold data to Hadoop and reuse applications • Combine cold data on Hadoop with hot data on Netezza to derive insights Business Benefits: Application portability – SQL compatibility enables reuse of application with minimal business query modifications - Support for Netezza functions Federation – query capabilities between Netezza and Hadoop Leverage existing skills - IBM Big SQL enables and leverages existing SQL skill set Big SQL provides SQL query, federation, transactional, performance, and security capabilities, which will combine with streaming data, governance, and BI analytics in later phases.
- 11. 11 EY – Query data from different sources using Big SQL to help prevent fraud Business need: • Analyze data to quickly detect fraud threats • Forensic data analytics is seen as a key capability to invest in Solution: • EY is now able to detect potential threats before they escalate Business Benefits: High performance helps queries to run in minutes, not hours, helping clients rapidly identify and eliminate threats Gathers data from multiple sources and applies real- time analytics to identify hidden patterns and anomalies Ability to process a wide variety of data types, from journal entries and payment streams to email, news feeds and social media. EY is a global leader in assurance,tax, transaction and advisory services. The insights and quality services EY delivers help build trust and confidence in the capital markets and in economies the world over, and help to build a better working world for EY’s people,clients and communities. Transformation EY provides its clients with comprehensive protection against fraud and security risks. IBM Analytics helps EY rapidly detect potential threats before they escalate.
- 12. 12 Vestas – Leverages complex queries and performance capabilities to turn climate into capital with Big data using IBM Big SQL “In our development strategy, we see growing our library in the range of 18 to 24 petabytes of data. And while it’s fairly easy to build that library, we needed to make sure that we could gain knowledge from that data.” — Lars Christian Christensen, vice president, Vestas Wind Systems The transformation: Successful analysis resulted in 97% decrease in response times for wind forecasting information to pinpoint optimal turbine placement, maximizes power generation and reduces energy costs. Business Benefits: High performance helps queries to run in minutes, not hours, helping clients rapidly identify and eliminate threats Complex query processing helps manage and analyze weather and location data for calculating the right location for turbines Business need: • Need to process complex queries on large volume of data • Analyze wind forecasting information for ideal placement of turbines Solution: • Analyzed petabytes of data using complex queries on weather and location data for turbine placements • Successful placements of turbines lead to increased customer’s ROI
- 13. 13 "In a half-day workshop, we were able to show the chemical company how big data analytics work and were able to identify four new customers." —Dr. Michael Kowolenko, Senior Research Scholar, Poole College of Management, NC State University The transformation: The Poole College of Management at NC State University is developing the next generation of data- driven decision makers, utilizing an IBM big data solution based on PowerLinux technology. This system allows its students to effectively manage and analyze large volumes of structured and unstructured data from a variety of sources. NC State University Poole College of Management – Helping businesses uncover new opportunities with IBM on PowerLinux Business Benefits: Application portability – SQL compatibility enables reuse of application with minimal business query modifications - Support for Netezza functions Federation – query capabilities between Netezza and Hadoop Leverage existing skills - IBM Big SQL enables and leverages existing SQL skill set Business need: • Make data driven decision on what new courses to add • Enable students to get real-world experience on Big Data Solution: • Created a curriculum that enables students to apply Big data analytics to real-world problems • Help businesses identify new opportunities
- 14. 14 Major North American Food Retailer, implements HDP on IBM POWER Business need: • Gain a competitive advantage by retaining and analyzing their store level loyalty program data • Bring outsourced analytics back in-house Solution: • Consolidation of client transaction data into a Hortonworks Data Platform on Linux on IBM Power Systems. • SAP Customer Activity Repository (CAR) application, powered by SAP HANA, connected to the data lake to enable real-time insights. Business Benefits: • More efficient and flexible in-store experiences for their clients to increase client loyalty and purchases. Time to Value HDP 2.6 running on a cluster of 9 IBM Power System servers Full solution deployed by IBM lab services and an IBM Business Partner in < 2 weeks Trial to production in 2 months
- 15. 15 Big SQL is the only SQL-on-Hadoop solution to understand SQL syntax from other vendors and products, including: Oracle, IBM DB2 and Netezza. For this reason, Big SQL is the ultimate hybrid engine to optimize EDW workloads on an open Hadoop platform What is IBM Big SQL?
- 16. 16 Federation and Spark Performance Enterprise and Security SQL Compatibility Relational Databases Leads performance metrics on high volumes of data and concurrent streams Automatic memory management Role and Column level Security Ranger Integration NoSQL Object Stores Core Themes of Big SQL
- 17. 17 Big SQL queries heterogeneous systems in a single query - only SQL-on-Hadoop that virtualizes more than 10 different data sources: RDBMS, NoSQL, HDFS or Object Store Big SQL Fluid Query (federation) Oracle SQL Server Teradata DB2 Netezza (PDA) Informix Microsoft SQL Server Hive HBase HDFS Object Store (S3) WebHDFS Big SQL allows query federation by virtualizing data sources and processing where data resides Hortonworks Data Platform (HDP) Data Virtualization
- 18. 18 § Easy porting of enterprise applications § Ability to work seamlessly with Business Intelligence tools like Cognos to gain insights § Big SQL integrates with Information Governance Catalog by enabling easy shared imports to InfoSphere Metadata Asset Manager, which allows: -Analyze assets -Utilize assets in jobs -Designate stewards for the assets Oracle SQL DB2 SQL Netezza SQL Big SQL SQL syntax tolerance (ANSI SQL Compliant) Cognos Analytics InfoSphere Metadata Asset Manager Big SQL is a synergetic SQL engine that offers SQL compatibility, portability and collaborative ability to get composite analysis on data Data Offloading and Analytics
- 19. 19 BRANCH_A FINANCE (security admin)BRANCH_B Role Based Access Control enables separation of Duties / Audit Row Level Security Row and Column Level Security Big SQL offers row and column level access control (RBAC) among other security settings Data Security
- 20. 20 PERFORMANCE Big SQL 5.0 is 3.2x faster than Spark SQL 2.1 (4 ConcurrentStreams)SNAPSHOT OF 100TB HADOOP-DS I/O (vs Spark) Big SQL reads 12x less data Big SQL writes 30x less data COMPRESSION 60% SPACE SAVED WITH PARQUET AVERAGE CPU USAGE 76.4% MAX I/O THROUGHPUT READ 4.4 GB/SEC WRITE 2.8 GB/SEC WORKING QUERIES Big SQL’s Performance at a Glance Leads performance metrics on high volumes of data and concurrent streams
- 21. 21 Right Tool for the Right Job Not Mutually Exclusive. Hive, Big SQL & Spark SQL can co-exist and complement each other in a cluster Big SQL Federation Complex Queries High Concurrency Enterprise ready Application portability All open source file formats Spark SQL Machine learning Data exploration Simpler SQL Hive In-memory cache Geospatial analytics ACID capabilities Fast ingest Ideal tool for Data Scientists and discovery Ideal tool for BI Data Analysts and production workloads Ideal tool for simple BI Data Analysts and production workloads
- 22. 22 Summary – Get more for Less with Big SQL §Big SQL is really a powerful runtime that makes access to Hive Tables fast and secure. §Big SQL supports ANSI SQL 2003, 2008, and even parts of SQL 2011! §Though SQL for HBase can be achieved with projects like Apache Phoenix, Big SQL provides seamless access to both HBase and Hive tables with the ability to join them too! §Big SQL can start working with existing Hive and HBase tables in Hadoop §With Big SQL Nicknames, you can provide seamless access to remote data sources to allow users to see and experiment with data – without the time and cost associated with building ingest processes. §All of this capability, provided with a single driver, one connection, a single robust and consistent ANSI compliant SQL dialect – with unified security managed for all object types. Big SQL will save you Time and Money
- 23. 23 Act Now - Available until Dec 31, 2017!
- 24. 24 Innovation Pervasive in the Design Power Systems S822LC for Big Data Not Just Another Intel Server NVIDIA: Tesla K80 GPU Accelerator Linux by Redhat: Redhat7.2 Linux OS Mellanox:InfiniBand/Ethernet Connectivity in and out of server HGST:OptionalNVMe Adapters Alpha Data with Xilinx FPGA: OptionalCAPIAccelerator Broadcom:OptionalPCIe Adapters QLogic:OptionalFiberChannel PCIe Samsung:SSDs & NVMe Hynix,Samsung,Micron:DDR4 IBM: POWER8 CPU
- 25. 25 IBM Power S822LC for Big Data and Hortonworks Combine to Deliver Leadership in Hadoop Environments • Hive/Tez Performance results are basedonIBM Internal Testing of 10 queries (simple, medium, andcomplex) withvaryingruntimes runningagainst a 10TB database. Thetests were run on10x IBM Power System S822LC for Big Data 20 cores / 40 threads, 2 X POWER8 2.92GHz, 256 GB memory, RHEL 7.2,, HDP 2.5.3 comparedto the published x86/Hortonworks results running on10x AWS d2.8xlarge EC2 nodes running HDP 2.5; details canbe foundat https://hortonworks.com/blog/apache-hive-going-memory-computing/ . Data as of February 28, 2017) • BigSQL Performance results are basedonIBM Internal Testing of all 50TPC-DS queries selected by Hortonworks. There is a majordifferencewith the 10 longest running queries basedonthe 10TB result Hortonworks teamachieved with 10 x AWS d2.8xlarge EC2 dataNodes runningwith HDP 2.5.2 (details can befound at https://hortonworks.com/blog/apache-hive-going-memory-computing/). 11 x S822LC for BigData Power servers wereusedas dataNodes running BigSQL and IOP 4.2.5. Data as of July 12, 2017. • Conducted under laboratory condition,individual result can vary based on workloadsize, use of storagesubsystems & other conditions. • POWER8 and Hortonworksdeliver1.70X the throughput comparedto Hortonworks Hive/Tez running on x86 – 70% More QpH based on the averageresponsetime – complete the same amountofwork with less system resources – 41% Reduction on averagein query response time – reduced response time enablesmaking business decisionsfaster. • IBM BigSQLon IBM PowerSystems can deliver3.5X faster query times on average for the mostcomplex queries 70% More Throughput
- 26. 26 Data lake with IBM Spectrum Scale Unleash new storage economies on a global scale. Block iSCSI Client workstations Users and applications Compute farm Traditional applications GLOBAL Namespace Analytics Transparent HDFS OpenStack Cinder Glance Manilla Object Swift S3 Transparent Cloud Powered by IBM Spectrum Scale Automated data placement and data migration Disk Tape Shared Nothing Cluster Flash New Gen applications Transparent Cloud Tier Worldwide Data Distribution (R/W) Site B Site A Site C SMBNFS POSIX File Consolidate all your unstructured data storage on Spectrum Scale with unlimited and painless scaling of capacity and performance Encryption DR Site AFM-DR JBOD/JBOF Spectrum Scale RAID Compression 4000+ clients
- 27. 27 Why Spectrum Scale for Big Data & Analytics Extreme scalability with parallel file system architecture Data + Metadata Node Data + Metadata Node Data + Metadata Node Data + Metadata Node No centralized metadata node bottleneck. Every node can serve as data and metadata in the cluster. Global namespace that can span geographies Active – Active replicas of data for real time global collaboration Reduce datacenter footprint with industry’s best in-place analytics True software defined storage that can be purchased as software only OR pre-integrated system Data NFS SMB POSIX Object HDFS API Access to the data using any of the industry standard protocols IBM Elastic Storage Server (ESS) (Pre-integrated system) IBM Spectrum Scale (SW only)
- 28. 28 Reduce data center footprint with Spectrum Scale HDFS Raw data ext4 ext4 write move copy Traditional applicationsCopies in both HDFS and ext4 Spectrum Scale Application writes direct to Hadoop Path with NFS/SMB/ Object/POSIX direct-read with NFS/SMB/ Object/POSIX Raw data Traditional applications Multiple copies with HDFS based workflow Spectrum Scale in-place analytics (No copies required) Hadoop analysis Jobs Hadoop analysis Jobs Direct read,one version Data Scientists waste daysjust copying data to HDFS No copies required with Spectrum Scale Costly data protection - Default uses 3-wayreplication with HDFS IBM ESS Software RAID eliminatesneed for 3 way replication. Just 30% extra storage requirement. [Erasure coding in HDFS has limitations and is good only for cold data] Traditional applications Traditional applications HDFS APIs HDFS APIs Example: For 5PB of data, HDFS requires 15PB of storage Example: For 5PB of data, ESS requires 6.5PB of storage Copy process can take hours/days & eventually results are based on stale data.
- 29. 29 Page 29 HDP and IBM Systems – Better Together 5 Mission Critical Support Ø Stable, trusted Hadoop platform on proven Power System with outstanding client support Performance and Price/Performance – Leading performance for SQL and Spark workloads Ø 1.70X the throughput compared to Hortonworks running on x86 and a 3X price performance guarantee 2 TCO at Scale with HDP on Power Systems : Ø Host mixed application workloads on a single global filesystem with IBM Spectrum Scale Ø Up to 3X reduction of storage and compute infrastructure moving to Power Systems and IBM Elastic Storage Server vs commodity scale out x86 4 Flexibility – Richest family of Linux servers to match your workload’s scale and reliability needs1 Designed for Cognitive + AI – Obtain your ML/DL results faster with AI on Power servers Ø PowerAI is the only commercial offering containing all key deep learning frameworks § Caffe, TensorFlow, Torch, Theano, OpenBLAS, NCCL, NVIDIA DIGITS* 3
- 30. 30 Scaling Data Science on Big Data Date: Wed, 9/20 @ 11:00 AM Room: C2.3 1 Ingesting Data at Blazing Speed using Apache ORC Data: Wed, 9/20 @ 4:20 PM Room: C4.7 2 Open metadata and governance with Apache Atlas Date: Wed, 9/20 @ 5:10 PM Room: C4.6 Empowering YOU with Democratized Data Access, Data Science and Machine Learning Date: Wednesday, 9/20 @ 6:00 PM Room: C4.5 Breaching the 100TB mark with SQL over Hadoop Date: Thurs, 9/21 @ 2:20 PM Room: C2.3 Apache Spark, Apache Zeppelin and Data Science Date: Thurs, 9/21 @ 6:00 PM Room: C4.5 Visit IBM Booth for More Information! Check out the Breakout Sessions