Introduction to Amazon Redshift
2 likes1,095 views
Amazon Redshift is a fully managed, petabyte-scale cloud data warehouse optimized for high-performance analysis of large datasets. It leverages PostgreSQL and features include massively parallel processing, data compression, and automated backups. Redshift integrates with various ETL and BI tools, making it suitable for traditional enterprise data warehousing and providing scalability at reduced costs.
Introduction to Amazon Redshift
- 2. What's Amazon Redshift? Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the cloud https://aws.amazon.com/redshift/
- 3. Features Petabyte scale, massively parallel Relational data warehouse Fully managed, zero admin SSD and HDD platforms $999/TB/Year
- 4. Architecture
- 5. Client Applications Integrates with various data loading and ETL (Extract, Transform, and Load) tools and business intelligence (BI) reporting, data mining, and analytics tools Redshift is based on industry-standard PostgreSQL, so most existing SQL client applications will work with only minimal changes
- 6. Connections Redshift communicates with client applications by using industry- standard PostgreSQL JDBC and ODBC drivers
- 7. Clusters A cluster is composed of one or more compute nodes Leader Node coordinates the compute nodes and handles external communication
- 8. Leader Node Manage communications with client programs and communications with compute nodes Store metadata Coordinate query execution
- 9. Compute Nodes Execute the compiled code, send intermediate results back to the leader node for final aggregation It has own dedicated CPU, memory, and attached disk storage, which are determined by the node type
- 10. Databases A cluster contains one or more databases User data is stored on the compute nodes Amazon Redshift is a Relational Database Management System (RDBMS) Amazon Redshift is optimized for high-performance analysis and reporting of very large datasets Amazon Redshift is based on PostgreSQL
- 11. Redshift reduces I/O Column storage - read data you need Data compression - analyzes and compress your data Zone Map Keep track of minimum and maximum value for each block Skip over blocks that don't contain data needed for a given query Minimize unnecessary I/O Direct attached storage Hardware optimized for high performance data processing Large data block sizes Large block sizes to make the most of each read
- 12. Redshift runs on optimized hardware Optimized for I/O intensive workloads High disk density Runs in HPC - fast network
- 14. Redshift is easy to use Provision in minutes Monitor query performance Point and click resize Built in security Automatic backups
- 15. Redshift has security built-in SSL to secure data in transit Encryption to secure data at rest AES 256 - hardware accelerated All blocks on disk and in Amazon S3 encrypted No direct access to compute nodes Amazon VPC support
- 16. Redshift backs up your data and recovers from failures Replication within the cluster and backup to Amazon S3 Backup to Amazon S3 are continuous, automatic and incremental Continuous monitoring and automated recovery from failures Able to restore snapshots to any Availability Zone
- 17. Use Cases
- 18. Traditional Enterprise DW Reduce costs by extending DW rather than adding HW Migrate completely from existing DW systems Respond faster to business
- 19. Companies with Big Data Improve performance by an order of magnitude Make more data available for analysis Access business data via standard reporting tools
- 20. SaaS Companies Add analytic functionality to applications Scale DW capacity as demand grows Reduce HW and SW costs by an order of magnitude
- 23. Redshift Implementation High Storage Extra Large (XL) DW Node ETL Activities Approx. 90 minutes including exports from RDBMS, copying to S3, loading stage tables, loading target tables, vacuuming and analysing tables Schema Compression Retention
- 24. DW Anatomy
- 25. Why Redshift works for SkillPages? Scale - MPP Performance - Columnar data access and compression Platform Integration - S3, Dynamo Operational Advantages Ease of Access Cost
- 26. Best Practices Avoid large number of singleton Data Manipulation Language (DML) statements if possible Use COPY for uploading large datasets Choose SORT and DISTRIBUTION keys with care Encode data and time with TIMESTAMP data type Experiment with WLM (Workload Manager) settings