How Glidewell Moves Data to Amazon Redshift
Download as PPTX, PDF2 likes1,030 views
Glidewell Laboratories moved data to Amazon Redshift using Attunity CloudBeam to enable analytics and business intelligence. Attunity CloudBeam extracts data from Glidewell's on-premise databases, applies transformations, and loads the data into Amazon Redshift. This provides Glidewell's employees access to timely data in Redshift to support analytics using Tableau and Dundas Dashboards. The migration addressed Glidewell's challenges around managing growing data from multiple sources and providing a robust analytics platform to support their global expansion.
How Glidewell Moves Data to Amazon Redshift
- 1. Glidewell Laboratories Moves Data to Amazon Redshift with Attunity CloudBeam October 20, 2015
- 2. Agenda • Overview of Amazon Redshift by Mike Ruiz, Solution Architect, AWS Emerging Partner Team • Overview of Attunity CloudBeam by Rodan Zadeh, Director of Product Management, Attunity • Glidewell Customer Story by Mike Selberis, CIO, Glidewell Dental Laboratories
- 3. Agenda • Overview of Amazon Redshift by Mike Ruiz, Solution Architect, AWS Emerging Partner Team
- 4. Amazon Redshift Fast, simple, petabyte-scale data warehousing for less than $1,000/TB/Year
- 5. Data warehousing done the AWS way • Easy to provision • Pay as you go, no up front costs • Fast, cheap, easy to use • SQL
- 6. Constant Innovation: 95+ new features since launch… • Regions – N. Virginia, Oregon, Dublin, Tokyo, Singapore, Sydney, Frankfurt • Certifications – PCI, SOC 1/2/3, HIPAA and FedRamp • Security – Load/unload encrypted files, Resource-level IAM, Temporary credentials, HSM/CloudHSM, Audit Logging • Manageability – Snapshot sharing, backup/restore progress indicators, SNS Alerts, faster cluster creation, cross-region backups, faster resize, WLM resource management • Query – Regex, Cursors, MD5, SHA1, Time zone, workload queue timeout, approximate count distinct, distributed tables, concurrency increased to 50 from 15, user defined functions • Ingestion – S3 Manifest, LZOP/LZO, JSON built-ins, UTF-8 4byte, invalid character substitution, CSV, auto datetime format detection, epoch, load from EMR/HDFS/SSH… Full list: http://docs.aws.amazon.com/redshift/latest/dg/doc-history.html
- 7. Amazon Redshift architecture • Leader Node – SQL endpoint – Stores metadata – Coordinates query execution • Compute Nodes – Local, columnar storage – Isolated from end-user – Execute queries in parallel – Load, backup, restore via Amazon S3 – Parallel load from Amazon DynamoDB – Load from EMR/HDFS/SSH *New* • Single node version available 10 GigE (HPC) Ingestion Backup Restore JDBC/ODBC
- 8. Amazon Redshift lets you start small and grow big Single Node (2 TB) Cluster 2-32 Nodes (4 TB – 64 TB) Cluster 2-100 Nodes (32 TB – 1.6 PB) Dense Storage Node (dw1.xlarge) 2 TB, 16 GB RAM, 2 cores Dense Compute Node (dw2.large) 0.16 TB, 16 GB RAM, 2 cores 8XL Dense Storage Node (dw1.8xlarge) 16 TB, 128 GB RAM, 16 cores, 10 GigE 8XL Dense Compute Node (dw2.8xlarge) 2.56 TB, 128 GB RAM, 16 cores, 10 GigE
- 9. Amazon Redshift integrates with multiple data sources Dashboarding Reporting & BI Ad Hoc Analysis Amazon Redshift Amazon EMRAmazon DynamoDBAmazon RDS Amazon S3 On-Premise database Flat files On-Premise Warehouse
- 10. Amazon Redshift works with your existing analysis tools JDBC/ODBC Amazon Redshift
- 11. Common customer use cases • Reduce costs by extending DW rather than adding HW • Respond faster to business drivers by avoiding procurement loop. • Improve performance by an order of magnitude • Make more data available for analysis • Access business data via standard reporting tools • Add analytic functionality to applications • Scale DW capacity as demand grows • Reduce HW and SW costs by an order of magnitude Extend On-Premise DW Companies with Big Data SaaS Companies
- 12. Redshift powers Clickstream Analytics for Amazon.com • Performance – Scan 2.25 trillion rows of data: 14 minutes – Load 5 billion rows data: 10 minutes – Backfill 150 billion rows of data: 9.75 hours – Pig Amazon Redshift: 2 days to 1 hr • 10B row join with 700 M rows – Oracle Amazon Redshift: 90 hours to 8 hrs • Reduced number of SQLs by a factor of 3 • Cost – 1.6 PB cluster – 100 node dw1.8xl (3-yr RI) – $180/hr • Complexity – 20% time of one DBA • Backup • Restore • Resizing
- 13. Resources • Detail Pages – http://aws.amazon.com/redshift – https://aws.amazon.com/marketplace/redshift/ • Presentations & Webinars: – http://www.youtube.com/watch?v=JxLpj_TnisM (SF Summit Presentation) – http://www.youtube.com/watch?v=R1m-fwzXMow (Best Practices 1 of 2) – http://www.youtube.com/watch?v=7ySzRTOyK6o (Best Practices 2 of 2)
- 14. Agenda • Overview of Attunity CloudBeam by Rodan Zadeh, Director of Product Management, Attunity
- 15. Right Data. Right Place. Right Time. DW HD To Analytic Platform To Many Locations To One Location Enable Analytics Share & Distribute Scale Reporting Consolidate Away from OLTP
- 16. Attunity CloudBeam – Highlights • Simplicity – Automated, “Click-to-Load” Solution • Performance – Optimized data transfer • Real-time Data – Incremental/Continuous data loads • Free Trial – “Try before you Buy” • One-Stop Shop – Integrated into the AWS Redshift 16
- 17. Attunity CloudBeam – Solutions for AWS • Amazon Redshift: – Load data for BI / Analytics – One time and incrementally • Amazon RDS / DBs on EC2: – Migrate data from on-premise data centers – Enable disaster recovery (on-premise to cloud) • Amazon S3: – Transfer data for archiving, storage, content availability – Enable disaster recovery
- 18. Attunity CloudBeam for Amazon Redshift & S3 Simplicity: Automated, Click-2-Load Solution Performance: Optimized data transfer Real-time: Continuous data loads Free Trial: Try before you Buy 1-Stop-Shop: Integrated with AWS Marketplace Pricing Options: Express (hourly), Premium (hourly) and BYOL
- 19. How Attunity CloudBeam Works 1. Extract data from source database 2. Apply filters and transformations 3. Generate data files for transfer 4. Make optimized transfer to Amazon S3 5. Orchestrate COPY and MERGE for Amazon Redshift 6. Provide CDC (change data capture) for incremental loading AWS Region EC2 Machine M3.Large Attunity AMI RedshiftS3 On-Premises Source DB(s)
- 20. Agenda • Glidewell Customer Story by Mike Selberis, CIO, Glidewell Dental Laboratories
- 21. Glidewell Laboratories • Founded in 1970 and based in California • Largest dental lab in the United States with a market presence in Europe and Latin America • Provides high-quality dental lab products, services, and materials to dental professionals • Over 4K employees worldwide • Over 3M custom units processed and manufactured annually
- 22. Technology • Hybrid cloud environment with multiple on-premise datacenters and AWS as public cloud provider • Use Attunity CloudBeam to synchronize data between on- premises databases and Amazon Redshift • Provide end users with timely access to data on Amazon Redshift to support business analytics and business intelligence • Tableau Software and Dundas Dashboards are used for analytics and data visualization
- 23. CAD/CAM Digital Manufacturing MillScan Design
- 24. Glidewell Data Explosion • Lab management system (ERP/CRM) • Case tracking, logistics, technician performance • Manufacturing data (CAD/CAM) • Manufacturing Automation • Robotics • Cloud-based manufacturing system (CloudPoint) • IoT (Future) Introduction of Digital Manufacturing
- 25. Our Challenge • Managing an explosion of data and disparate data sources • Bridging on-premise data with cloud-generated data • Providing a robust data analytics platform for our employees, customers, and engineers • Development resource constraints • Global expansion • Speed of action
- 26. Our Solution Dundas Dashboards Tableau Analytics Ad Hoc reporting and analysis Amazon Redshift Transactional Data (ERP/CRM) CloudPoint Customer DataScan, Design, & Manufacturing Data Scan, Design, & Manufacturing Data Legacy Manufacturing Data
- 27. Glidewell Laboratories Moves Data to Amazon Redshift with Attunity CloudBeam Click here to watch the recorded webinar