[よくわかるAmazon Redshift]Amazon Redshift最新情報と導入事例のご紹介
2 likes2,086 views
- Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service in the cloud. It uses massively parallel processing and columnar storage to enable fast queries on large data sets for a fraction of the cost of traditional data warehousing. - Some key features include automatic scaling, continuous backups, integrated security and access controls, integration with other AWS services like S3 and DynamoDB, and simple point-and-click management. - Customers are seeing significant improvements in performance, often 50-100x faster than alternatives like Hive, as well as large cost reductions of up to 80% compared to on-premises data warehousing.
![Customers on Performance
“Redshift is twenty times faster than Hive” (5x – 20x reduction in query times) link
…[Redshift] performance has blown away everyone here (we generally see 50-100x speedup
over Hive). link
“We saw…2x improvement in query times and a 50% reduction in costs”
We regularly process multibillion row datasets and we do that in a matter of hours. link
“Queries that used to take hours came back in seconds. Our analysts are orders of magnitude
more productive.” (20x – 40x reduction in query times) link
“Did I mention it's ridiculously fast? We'll be using it immediately to provide our analysts
an alternative to Hadoop.”](https://image.slidesharecdn.com/20140219redshiftupdatesv1tokyo-140224010117-phpapp02/85/Amazon-Redshift-Amazon-Redshift-36-320.jpg)
![Customers on Cost
“We found that Amazon Redshift offers the performance we needed while freeing us from
the licensing costs of our previous solution” link
“[Redshift] cost saving is even more impressive…Our analysts like [Redshift] so much they
don’t want to go back.” (4x reduction in cost over HIVE) link
“We saw 50% reduction in costs”
“Not only did we avoid 3 months of development work [we] saved approximately $80,000 in
labor…Competitive Advantage realized with just a few clicks.”
“[Amazon Redshift] took an industry famous for its opaque pricing, high TCO and unreliable
results and completely turned it on its head.” link
“[Redshift] has reduced our storage and processing costs significantly, helping us to realize
another 60-70 percent savings.” link](https://image.slidesharecdn.com/20140219redshiftupdatesv1tokyo-140224010117-phpapp02/85/Amazon-Redshift-Amazon-Redshift-37-320.jpg)
![Customer on Ease of Use
“With Amazon Redshift and Tableau, anyone in the company can set up any queries they
like…It’s very flexible.” link
“Compared to Hadoop [Redshift] is much easier for analysts to use. What may have been a
Hadoop project can become just a query in Redshift.” link
“We can spin up an Amazon Redshift cluster, take a snapshot, and scale servers in minutes
instead of days.” link
“…our team was able to provision Redshift in a matter hours vs. weeks with on-premises
servers.”
“Amazon Redshift is simple to use and reliable. With one click, we can rapidly scale up or down
in real time in alignment with business requirements.” link
“Customers can get consistent, accurate, and useful data fast - in weeks not months or years.”
link](https://image.slidesharecdn.com/20140219redshiftupdatesv1tokyo-140224010117-phpapp02/85/Amazon-Redshift-Amazon-Redshift-38-320.jpg)
[よくわかるAmazon Redshift]Amazon Redshift最新情報と導入事例のご紹介
- 1. AWSプロダクトシリーズ よくわかるAmazon Redshift 2014/02/19 アマゾン データ サービス ジャパン株式会社
- 2. Amazon Redshift Fast, simple, petabyte-scale data warehousing for less than $1,000/TB/Year Rahul Pathak |Senior Product Manager
- 3. a lot faster a lot cheaper a whole lot simpler Petabyte scale Massively parallel Amazon Redshift Relational data warehouse Fully managed; zero admin
- 4. Amazon Redshift Quick Overview Amazon Redshift 概要のおさらい
- 5. Amazon Redshift architecture • Leader Node – – – • SQL endpoint Stores metadata Coordinates query execution JDBC/ODBC Compute Nodes – – – – Local, columnar storage Execute queries in parallel Load, backup, restore via Amazon S3 Parallel load from Amazon DynamoDB • Hardware optimized for data processing • Two hardware platforms 10 GigE (HPC) – – DW1: HDD; scale from 2TB to 1.6PB DW2: SSD; scale from 160GB to 256TB Ingestion Backup Restore
- 6. Amazon Redshift has security built-in • • Customer VPC SSL to secure data in transit Encryption to secure data at rest – – – JDBC/ODBC AES-256; hardware accelerated All blocks on disks and in Amazon S3 encrypted HSM Support • 10 GigE (HPC) No direct access to compute nodes • Internal VPC Audit logging & AWS CloudTrail integration • Amazon VPC support Ingestion Backup Restore
- 7. Amazon Redshift is easy to use • Provision in minutes • Monitor query performance • Point and click resize • Built in security • Automatic backups
- 8. Provision a data warehouse in minutes
- 10. Point and click resize • Resize while remaining online via AWS Console or API • Provision a new cluster in the background and copy data in parallel from node to node • Only charged for source cluster until SQL endpoint has automatically been switched over via DNS
- 11. Amazon Redshift continuously backs up your data and recovers from failures • Replication within the cluster and backup to Amazon S3 to maintain multiple copies of data at all times • Backups to Amazon S3 are continuous, automatic, and incremental – Designed for eleven nines of durability • Continuous monitoring and automated recovery from failures of drives and nodes • Able to restore snapshots to any Availability Zone within a region • Easily enable backups to a second region for disaster recovery
- 12. Amazon Redshift integrates with multiple data sources Corporate Datacenter DynamoDB Amazon Redshift Amazon S3 Amazon RDS Amazon EMR
- 13. New Features That Introduced After re:Invent 2013 re:Invent 2013以降の主なアップデート
- 14. Feature Delivery in 2013 Unload logs (7/5) Temp Credentials (4/11) Sharing snapshots (7/18) DUB (4/25) Resource Level IAM (8/9) SHA1 Builtin (7/15) SOC1/2/3 (5/8) Statement Timeout (7/22) WLM Timeout/Wildcards (8/1) UTF-8 Substitution (8/29) JDBC Fetch Size (6/27) Kinesis EMR/HDFS/SSH copy, Distributed Tables, Audit Logging/CloudTrail, Concurrency, Resize Perf., Approximate Count Distinct, SNS Alerts (11/13) Service Launch (2/14) Split_part, Audit tables (10/3) EIP Support for VPC Clusters (12/28) PCI (8/22) SIN/SYD (10/8) PDX (4/2) Distributed Tables, Single Node Cursor Support, Maximum Connections to 500 (12/13) JSON, Regex, Cursors (9/10) NRT (6/5) CRC32 Builtin, CSV, Restore Progress (8/9) Timezone, Epoch, Autoformat (7/25) 4 byte UTF-8 (7/18) Unload Encrypted Files HSM Support (11/11)
- 15. Summary of Updates after re:Invent • Amazon Redshift - New Features Galore (2013/11/11) – – – – – – – – • • • Distributed Tables - You now have more control over the distribution of a table's rows across compute nodes. Remote Loading - You can now load data into Redshift from remote hosts across an SSH connection. Approximate Count Distinct - You can now use a variant of the COUNT function to approximate the number of matching rows. Workload Queue Memory Management - You can now apportion available memory across work queues. Key Rotation - You can now direct Redshift to rotate keys for an encrypted cluster. HSM Support - You can now direct Redshift to use an on-premises Hardware Security Module (HSM) or AWS CloudHSM to manage the encryption master and cluster encryption keys. Database Auditing and Logging - You can log connections and user activity to Amazon S3. SNS Notification - Redshift can now issue notifications to an Amazon SNS topic when certain events occur. Automated Cross-Region Snapshot Copy for Amazon Redshift (2013/11/14) Faster & More Cost-Effective SSD-Based Nodes for Amazon Redshift(2014/01/24) AWS CloudFormation Adds Support for Redshift and More (2014/02/10)
- 16. Amazon Redshift Node Types DW1.XL: 16 GB RAM, 2 Cores 3 Spindles, 2 TB compressed storage • Optimized for I/O intensive workloads • High disk density DW1.8XL: 128 GB RAM, 16 Cores, 24 Spindles 16 TB compressed, 2 GB/sec scan rate • On demand at $0.85/hour • As low as $1,000/TB/Year • Scale from 2TB to 1.6PB DW2.L *New*: 16 GB RAM, 2 Cores, 160 GB compressed SSD storage • High performance at smaller storage size • High compute and memory density • On demand at $0.25/hour • As low as $5,500/TB/Year • Scale from 160GB to 256TB DW2.8XL *New*: 256 GB RAM, 32 Cores, 2.56 TB of compressed SSD storage
- 17. Amazon Redshift is priced to let you analyze all your data Price Per Hour for DW1.XL Single Node Effective Annual Price per TB On-Demand $ 1.250 $ 5,475 1 Year Reservation $ 0.750 $ 3,283 3 Year Reservation $ 0.452 $ 1,981 DW1 (HDD) Effective Annual Price per TB On-Demand $ 0.330 $ 18,068 1 Year Reservation $ 0.211 $ 11,570 3 Year Reservation $ 0.130 $ 7,127 No charge for leader node • Price Per Hour for DW2.L Single Node Number of nodes x cost per hour • DW2 (SSD) • No upfront costs • Pay as you go
- 18. Security, visibility and control • Audit logging Redshift • SNS Alerts
- 19. Visibility and control AWS CloudTrail System Activity Creates, Changes, Deletes, Resizes • Audit logging • SNS Alerts Amazon Redshift Database Activity Logins, Login failures, Queries, Loads Amazon S3
- 20. Visibility and control • • Audit logging Monitoring Security Maintenance Errors SNS Alerts Amazon Redshift SNS Topic
- 21. Batch operations • Cluster Creation • Faster Resize Amazon Corporate Amazon EC2 Data Center EMR Amazon Redshift Amazon S3
- 22. Batch operations • Cluster Creation • Faster Resize Amazon Corporate Amazon EC2 Data Center EMR Amazon Redshift Amazon S3
- 23. Batch operations • Cluster Creation • Faster Resize 15-20 min 3 min
- 24. Batch operations • Cluster Creation • Faster Resize 29 hours 7 hours
- 29. How Customers Leverage Amazon Redshift Amazon Redshift 活用事例
- 30. Common Customer Use Cases Traditional Enterprise DW SaaS Companies • Improve performance by an order of magnitude • Add analytic functionality to applications Make more data available for analysis • Scale DW capacity as demand grows • • • • • Reduce costs by extending DW rather than adding HW Companies with Big Data Access business data via standard reporting tools • Reduce HW & SW costs by an order of magnitude Migrate completely from existing DW systems Respond faster to business; provision in minutes
- 32. Japanese Redshift Customer – ALBERT • Business Challenge – • Why AWS? – – • Given their data volumes, RDBMS tuning and archiving was causing them a lot of operational pain and costing them money Amazon Redshift’s performance and ability to handle large data sets allowed them to make it the core engine of their analytics, enabling them to provide a private DMP (Data Management Platform) for their customers on the Cloud PostgreSQL is their primary RDBMS, and connectivity by PostgreSQL drivers is big technical advantage to choose Redshift. Benefits for their business – – Ability to start small and scale as needed Scalability and flexibility dramatically lowered the cost of ownership
- 33. Japanese Redshift Customer – Sansan • Business Challenge – Since “Eight” is business card management solution for consumers, they needed infrastructure that could start small and scale as needed • Why AWS? – When they tried out AWS first, they were surprised with the ease of use. AWS functionality and elasticity were critical factors • Benefits for their business – Lower costs substantially using reserved instances – Automation is a key to reduce operational and administration costs. They utilize services such as Amazon SES and Amazon SWF. – They use Redshift for KPI analytics of their services.
- 35. Multiple Data Loading Options Data Integration • Parallel upload to Amazon S3 • AWS Direct Connect • AWS Import/Export • ETL Software • Systems integrators Systems Integrators
- 36. Customers on Performance “Redshift is twenty times faster than Hive” (5x – 20x reduction in query times) link …[Redshift] performance has blown away everyone here (we generally see 50-100x speedup over Hive). link “We saw…2x improvement in query times and a 50% reduction in costs” We regularly process multibillion row datasets and we do that in a matter of hours. link “Queries that used to take hours came back in seconds. Our analysts are orders of magnitude more productive.” (20x – 40x reduction in query times) link “Did I mention it's ridiculously fast? We'll be using it immediately to provide our analysts an alternative to Hadoop.”
- 37. Customers on Cost “We found that Amazon Redshift offers the performance we needed while freeing us from the licensing costs of our previous solution” link “[Redshift] cost saving is even more impressive…Our analysts like [Redshift] so much they don’t want to go back.” (4x reduction in cost over HIVE) link “We saw 50% reduction in costs” “Not only did we avoid 3 months of development work [we] saved approximately $80,000 in labor…Competitive Advantage realized with just a few clicks.” “[Amazon Redshift] took an industry famous for its opaque pricing, high TCO and unreliable results and completely turned it on its head.” link “[Redshift] has reduced our storage and processing costs significantly, helping us to realize another 60-70 percent savings.” link
- 38. Customer on Ease of Use “With Amazon Redshift and Tableau, anyone in the company can set up any queries they like…It’s very flexible.” link “Compared to Hadoop [Redshift] is much easier for analysts to use. What may have been a Hadoop project can become just a query in Redshift.” link “We can spin up an Amazon Redshift cluster, take a snapshot, and scale servers in minutes instead of days.” link “…our team was able to provision Redshift in a matter hours vs. weeks with on-premises servers.” “Amazon Redshift is simple to use and reliable. With one click, we can rapidly scale up or down in real time in alignment with business requirements.” link “Customers can get consistent, accurate, and useful data fast - in weeks not months or years.” link
- 39. AWS Marketplace • Find software to use with Amazon Redshift • One-click deployments • Flexible pricing options http://aws.amazon.com/marketplace
- 40. Questions?
- 41. APPENDIX
- 42. Resources • Detail Pages – – • New Features – – • http://docs.aws.amazon.com/redshift/latest/dg/doc-history.html http://docs.aws.amazon.com/redshift/latest/mgmt/document-history.html Best Practices – – – • http://aws.amazon.com/redshift https://aws.amazon.com/marketplace/redshift/ http://docs.aws.amazon.com/redshift/latest/dg/c_loading-data-best-practices.html http://docs.aws.amazon.com/redshift/latest/dg/c_designing-tables-best-practices.html http://docs.aws.amazon.com/redshift/latest/dg/c-optimizing-query-performance.html Presentations & Webinars: – – – http://www.youtube.com/watch?v=JxLpj_TnisM (2013 SF Summit Presentation) http://www.youtube.com/watch?v=R1m-fwzXMow (Best Practices 1 of 2) http://www.youtube.com/watch?v=7ySzRTOyK6o (Best Practices 2 of 2)
- 43. Amazon Redshift dramatically reduces I/O Column storage Data compression Age State Amount 20 CA 500 345 25 WA 250 678 • ID 123 • 40 FL 125 37 WA 375 • Zone maps 957 • Direct-attached storage • With row storage you do unnecessary I/O • To get total amount, you have to read everything
- 44. Amazon Redshift dramatically reduces I/O Column storage Data compression Age State Amount 20 CA 500 345 25 WA 250 678 • ID 123 • 40 FL 125 37 WA 375 • Zone maps 957 • Direct-attached storage • With column storage, you only read the data you need
- 45. Amazon Redshift dramatically reduces I/O • Column storage analyze compression listing; Table | Column | Encoding ---------+----------------+---------listing | listid | delta listing | sellerid | delta32k listing | eventid | delta32k listing | dateid | bytedict listing | numtickets | bytedict listing | priceperticket | delta32k listing | totalprice | mostly32 listing | listtime | raw • Data compression • Zone maps • Direct-attached storage • COPY compresses automatically • You can analyze and override • More performance, less cost Slides not intended for redistribution.
- 46. Amazon Redshift dramatically reduces I/O • Column storage 10 324 • Data compression 375 623 • Zone maps • Direct-attached storage 637 959 10 | 13 | 14 | 26 |… … | 100 | 245 | 324 375 | 393 | 417… … 512 | 549 | 623 637 | 712 | 809 … … | 834 | 921 | 959 • Track the minimum and maximum value for each block • Skip over blocks that don’t contain relevant data
- 47. Amazon Redshift dramatically reduces I/O • Column storage • Use local storage for performance • Maximize scan rates • Data compression • Zone maps • Automatic replication and continuous backup • Direct-attached storage • HDD & SSD platforms
- 48. Amazon Redshift parallelizes and distributes everything • Query • Load • Backup/Restore • Resize
- 49. Amazon Redshift parallelizes and distributes everything • Query • Load • Backup/Restore • Resize • Load in parallel from Amazon S3 or Amazon DynamoDB or any SSH connection • Data automatically distributed and sorted according to DDL • Scales linearly with number of nodes
- 50. Amazon Redshift parallelizes and distributes everything • Query • Load • Backup/Restore • Backups to Amazon S3 are automatic, continuous and incremental • Resize • Configurable system snapshot retention period. Take user snapshots on-demand • Cross region backups for disaster recovery • Streaming restores enable you to resume querying faster
- 51. Amazon Redshift parallelizes and distributes everything • Query • Load • Backup/Restore • Resize • Resize while remaining online • Provision a new cluster in the background • Copy data in parallel from node to node • Only charged for source cluster
- 52. Amazon Redshift parallelizes and distributes everything • Query • Load • Backup/Restore • • Automatic SQL endpoint switchover via DNS • Decommission the source cluster • Simple operation via Console or API Resize