Databases & Analytics AWS re:invent 2019 Recap
- 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Databases & Analytics Sungmin Kim Solutions Architect, AWS
- 2. Data The world’s most valuable resource is no longer oil, but data.* *Copyright: The Economist, 2017, David Parkins “ ”
- 3. Hardware and software installation Configuration, patching, and backups Cluster setup and data replication for high availability Capacity planning, and scaling clusters for compute and storage Managing software on-premises is time consuming and complex
- 4. Customers moving to fully managed services Relational databases Aurora RDS EMR Hadoop and Spark Elasticsearch Service Operational analytics Managed Streaming for Kafka Real-time analytics DynamoDB DocumentDB ElastiCache Managed Cassandra Service Non-relational databases
- 6. AWS Databases – for all your application use cases
- 7. Amazon RDS Managed relational database service with a choice of popular databases Easy to administer No infrastructure provisioning No software installation and patching Built-in monitoring Performant & scalable Scale with an API call or a few clicks Read replicas for increased throughput Automatic Multi-AZ data replication Automated backup, snapshots, and failover Available & durable Secure and compliant Encryption at rest and in transit Network isolation and resource-level permissions
- 8. How do you scale your relational database to support tens of thousands of connections? Serverless applications open and close tens of thousands of connections within seconds Leads to longer query response times that limits application scalability A database proxy server are difficult to deploy, patch, and manage
- 9. Amazon RDS를 위한 완전 관리형 고-가용성 데이터베이스 프록시 기능. 커넥션 풀 및 공유 연결을 통해 애플리케이션 확장성, 데이터베이스 장애에 대한 탄력성 및 보안 향상 DRAFTDatabases Preview – December 3 L E A R N M O R E DAT368: Setting up database proxy servers with RDS Proxy Amazon RDS Proxy RDS Proxy Applications RDS Database Instance Connection Pooling
- 10. Amazon Aurora MySQL and PostgreSQL compatible relational database built for the cloud Performance and availability of commercial-grade databases at 1/10th the cost Performance and scalability 5x throughput of MySQL 3x throughput of PostgreSQL Up to15 read replicas Scale out reads and writes across multiple data centers Fully managed Managed by RDS: no hardware provisioning, software patching, setup, configuration, or backups Availability and durability Fault-tolerant self-healing storage Six copies of data across three AZs Continuous backup to S3 Single Global database with cross- region replication Network isolation Encryption at rest/transit Highly secure
- 11. Generate predictions directly from Aurora queries Models run in SageMaker & Comprehend Use standard SQL, no ML expertise required Suitable for low-latency, high-volume use cases Amazon SageMaker ML Aurora Database Athena Interactive analytics SQL Select from Where ML in Amazon Aurora and Athena Bringing machine learning to data developers and data analysts L E A R N M O R E DAT369 - Deep dive on Amazon Aurora machine learning integration General Availability – November 26 Databases
- 12. Amazon Managed Apache Cassandra Service 높은 가용성과 확장성을 제공하는 서버리스 Apache Cassandra 호환 데이터베이스 완전 관리 서비스 Apache Cassandra 호환성 제공 성능 및 확장성 보장 높은 가용성 및 보안 제공 관리할 서버가 없음 DRAFTDatabases Preview – December 3 L E A R N M O R E DAT324: Overview of Amazon Managed Apache Cassandra Service
- 13. Common data categories and use cases
- 15. Amazon Athena Interactive Query Any type of analytics on the data lake Most comprehensive analytics platform Amazon S3 | AWS Glue Lake Formation Data lake Amazon Elasticsearch Service Amazon Kinesis Amazon MSK Amazon SageMaker Amazon Personalize Amazon QuickSight AWS Data Exchange Operational Analytics Real time Analytics Predictive Analytics RecommendationsVisualizations Data Exchange Amazon Redshift Data Warehousing Amazon EMR Big Data Processing
- 16. Amazon Athena Interactive Query Any type of analytics on the data lake Amazon S3 | AWS Glue Lake Formation Data lake Amazon Elasticsearch Service Amazon Kinesis Amazon MSK Amazon SageMaker Amazon Personalize Amazon QuickSight AWS Data Exchange Operational Analytics Real time Analytics Predictive Analytics RecommendationsVisualizations Data Exchange Amazon Redshift Data Warehousing Amazon EMR Big Data Processing
- 17. Amazon EMR Easily Run Spark, Hadoop, Hive, Presto, HBase, and more big data apps on AWS Low cost 50–80% reduction in costs with EC2 Spot and Reserved Instances Per-second billing for flexibility Use S3 storage Process data in S3 securely with high performance using the EMRFS connector Latest versions Updated with latest open source frameworks within 30 days Fully managed no cluster setup, node provisioning, cluster tuning Easy
- 18. Performance Improvements in Spark for Amazon EMR Performance optimized runtime for Apache Spark, 2.6x faster performance at 1/10th the cost *Based on TPC-DS 3 TB Benchmarking running 6 node C4x8 extra large clusters and EMR 5.28, Spark 2.4 10,164 16,478 26,478 0 5,000 10,000 15,000 20,000 25,000 30,000 Spark with EMR (with runtime) 3rd party Managed Spark (with their runtime) Spark with EMR (without runtime) Runtime total on 104 queries (seconds—lower is better) Runtime optimized for Apache Spark performance 100% compliant with Apache Spark APIs Best performance 2.6x faster than Spark with EMR without runtime 1.6x faster than 3rd party Managed Spark (with their runtime) Lowest price 1/10th the cost of 3rd party Managed Spark (with their runtime) NEW
- 19. Amazon Athena Interactive Query Any type of analytics on the data lake Amazon S3 | AWS Glue Lake Formation Data lake Amazon Elasticsearch Service Amazon Kinesis Amazon MSK Amazon SageMaker Amazon Personalize Amazon QuickSight AWS Data Exchange Operational Analytics Real time Analytics Predictive Analytics RecommendationsVisualizations Data Exchange Amazon Redshift Data Warehousing Amazon EMR Big Data Processing
- 20. Amazon Athena Pay per query Pay only for queries run Save 30–90% on per-query costs through compression Use S3 storage ANSI SQL JDBC/ODBC drivers Multiple formats, compression types, and complex joins and data types SQL Serverless: zero infrastructure, zero administration Integrated with QuickSight EasyQuery instantly Zero setup cost Point to S3 and start querying Serverless, interactive query service
- 21. Preview – November 26Amazon Athena Federated Query Run SQL queries on data spanning multiple data stores Redshift Data warehousing ElastiCache Redis Aurora MySQL, PostgreSQL DynamoDB Key value, Document DocumentDB Document S3/Glacier Run connectors in AWS Lambda: no servers to manage Run SQL queries on relational, non-relational, object, or custom data sources; in the cloud or on-premises Open Source Connectors for common data sources Build connectors to custom data sources Analytics
- 22. Amazon Athena Interactive Query Any type of analytics on the data lake Amazon S3 | AWS Glue Lake Formation Data lake Amazon Elasticsearch Service Amazon Kinesis Amazon MSK Amazon SageMaker Amazon Personalize Amazon QuickSight AWS Data Exchange Operational Analytics Real time Analytics Predictive Analytics RecommendationsVisualizations Data Exchange Amazon Redshift Data Warehousing Amazon EMR Big Data Processing
- 23. Amazon Redshift Best performance, most scalable 3x faster with RA3* 10x faster with AQUA* Adds unlimited compute capacity on-demand to meet unlimited concurrent access Lowest cost Cost-optimized workloads by paying compute and storage separately 1/10th cost of Traditional DW at $1000/TB/year Up to 75% less than other cloud data warehouses & predictable costs Data lake & AWS integration Analyze exabytes of data across data warehouse, data lakes, and operational database Query data across various analytics services Most secure & compliant AWS-grade security (eg. VPC, encryption with KMS, CloudTrail) All major certifications such as SOC, PCI, DSS, ISO, FedRAMP, HIPPA First and most popular cloud data warehouse *vs other cloud DWs
- 24. 컴퓨팅 및 스토리지 비용을 별도로 지불하여 데이터 웨어하우스 비용 최적화 General Availability – December 3 L E A R N M O R E ANT213-R1: State of the Art Cloud Data Warehousing ANT230: Amazon Redshift Reimagined: RA3 and AQUA 기존 클라우드 DW의 3 배 성능 제공 비슷한 가격의 DS2 인스턴스로서 2 배의 성능과 2 배의 스토리지 (주문형) DW 저장 용량을 자동으로 확장 및 최대 8PB의 워크로드 지원 (압축) COMPUTE NODE (RA3/i3en) SSD Cache S3 STORAGE COMPUTE NODE (RA3/i3en) SSD Cache COMPUTE NODE (RA3/i3en) SSD Cache COMPUTE NODE (RA3/i3en) SSD Cache Managed storage $/node/hour $/TB/month Amazon Redshift RA3 instances SEOUL Analytics
- 25. DRAFTAnalytics Preview – December 3 L E A R N M O R E ANT230: Amazon Redshift Reimagined: RA3 and AQUA AQUA (Advanced Query Accelerator) for Amazon Redshift 새로운 하드웨어 기반 가속 캐시를 통해 비용 증가 없이 타 클라우드 데이터 웨어 하우스보다 10 배 빠르게 질의 실행
- 26. DRAFTAnalytics Preview – December 3 L E A R N M O R E ANT230: Amazon Redshift Reimagined: RA3 and AQUA AQUA는 컴퓨팅을 스토리지를 기반으로 하므로 데이터 이동 필요 없음 S3 기반 고속 캐시는 여러 노드에서 동시에 데이터를 처리하도록 수평 확장 AWS 설계 프로세서로 데이터 압축, 암호화 및 데이터 처리 가속화 Redshift 버전과 100 % 호환 AQUA (Advanced Query Accelerator) for Amazon Redshift
- 27. Speed-up queries by orders of magnitude • Joins, filters, aggregations, and projections Analytics Preview – November 27 Simplify and accelerate ETL/BI pipelines • Incremental refresh • User triggered maintenance Easier and faster migration to Amazon Redshift Amazon Redshift Materialized Views
- 28. Amazon Redshift Federated Query DRAFTAnalytics Preview – December 3 L E A R N M O R E ANT230: Amazon Redshift Reimagined: RA3 and AQUA Amazon RDS 및 Amazon Aurora 데이터베이스와 Amazon Redshift 데이터 웨어 하우스 및 Amazon S3 테이블 데이터를 결합할 수 있는 통합 SQL 질의 기능 SEOUL
- 29. 데이터 웨어하우스 기반 데이터를 데이터 레이크로 내보내 다양한 분석 기능을 수행하여 새로운 통찰력 입수 가능 General Availability – December 3 L E A R N M O R E ANT335-R: How to build your data analytics stack at scale with Amazon Redshift Amazon Redshift Data Lake Export SEOUL Analytics
- 30. Amazon Athena Interactive Query Any type of analytics on the data lake Amazon S3 | AWS Glue Lake Formation Data lake Amazon Elasticsearch Service Amazon Kinesis Amazon MSK Amazon SageMaker Amazon Personalize Amazon QuickSight AWS Data Exchange Operational Analytics Real time Analytics Predictive Analytics RecommendationsVisualizations Data Exchange Amazon Redshift Data Warehousing Amazon EMR Big Data Processing
- 31. Amazon Elasticsearch Service Fully managed, scalable, secure, Elasticsearch service Open source Elasticsearch APIs, Kibana, and Logstash Open-source Elasticsearch APIs Managed Kibana Integration with Logstash Scale clusters up/down via a single API call or a few clicks Secured network isolation with VPC, encrypt data at-rest and in-transit Compliant: HIPPA, PCI DSS, and ISO Scalable, secure, and compliant Pay only for what you use Cost-optimized workloads No upfront fee or usage requirement Critical features built-in: encryption, VPC support, 24x7 monitoring Fully managed Deploy Elasticsearch clusters in minutes: simplified hardware provisioning, software installation/patching, failure recovery, backups, and monitoring
- 32. Challenges with analyzing high volumes of data in real-time Storing data is expensive at scale Limits the amount of data retained for analysis Miss out on valuable insights
- 33. Analytics Preview – December 3 Amazon Elasticsearch Service를 위한 저비용의 확장 가능한 웜 스토리지 계층. UltraWarm for Amazon Elasticsearch Service L E A R N M O R E ANT229 - Introducing UltraWarm for Amazon Elasticsearch Service ANT342 - Scaling Amazon Elasticsearch Service with UltraWarm
- 34. Our portfolio Broad and deep portfolio, purpose-built for builders S3/Glacier Glue ETL & Data Catalog Lake Formation Data Lakes Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehose | Kinesis Data Streams | Managed Streaming for Kafka Data Movement Data Lake Business Intelligence & Machine Learning Data Exchange Data exchange NEW QuickSight Visualizations SageMaker ML Comprehend NLP Transcribe Speech-to-text Textract Extract text Personalize Recommendation Forecast Forecasts Translate Translation CodeGuru Code reviews Kendra Enterprise search NEW NEW RDS MySQL, PostgreSQL, MariaDB, Oracle, SQL Server, RDS on VMware Aurora MySQL, PostgreSQL DynamoDB Key value, Document ElastiCache Redis, Memcached Neptune Graph Timestream Time Series QLDB Ledger Database Analytics Databases Managed Blockchain Blockchain Templates Blockchain Managed Apache Cassandra Service Wide column NEW DocumentDB Document Redshift Data warehousing EMR Hadoop + Spark Kinesis Data Analytics Real time Elasticsearch Service Operational Analytics Athena Interactive analytics NEW NEW NEW NEW NEW AQUA EMR on Outposts UltraWarm RDS Proxy RDS on Outposts
- 35. 2019년 출시 서비스 한눈에 보기 ANALYTICS COMPUTE DATABASE MANAGEMENT &GOVERNANCE Amazon EMR on AWS Outposts Deploy Apache Hadoop & Spark Clusters in AWS Outposts. Announcement Amazon Redshift Data Lake Export Unload Amazon Redshift query result to Amazon S3 data lake as Apache Parquet. Announcement Amazon Redshift Federated Query (Preview) Query, analyze data across operational databases, data warehouses, and data lakes. Announcement| Documentation RA3 with managed storage Scale & pay for compute and storage independently to cluster size based on compute needs. Announcement Advanced Query Accelerator (AQUA) Enables Redshift to run up to 10x faster. Amazon UltraWarm for Amazon Elasticsearch (Preview) Store and analyze data using Elasticsearch and Kibana while reducing your cost. Announcement |Documentation Kinesis Video Streams RTC Stream live media with ultra-low latency and enable two-way interactivity for millions of camera devices. Announcement Amazon EC2 Inf1 Instances High performance and the lowest cost machine learning inference in the cloud. Announcement |ProductPage Amazon EC2 M6g, C6g, and R6g Instances powered by AWS Graviton2 Processors Deliver up to 40% price/performance improvement over comparable x86-based instances for a broad spectrum of workloads. Announcement |ProductPage Amazon EC2 Image Builder EC2 Image Builder simplifies creation, maintenance, validation, sharing, and deployment of Linux or Windows Server images for use with Amazon EC2 and on- premises. Announcement|Product Page Amazon EKS for AWS Fargate Amazon Elastic Kubernetes Service (EKS) can now run Kubernetes pods on AWS Fargate, the serverless compute engine built for containers on AWS. Announcement |Blog AWS Wavelength Build applications with single-digit millisecond latencies delivery to mobile devices on 5G. Announcement |Product Page AWS Lambda Provisioned Concurrency Keep functions initialized and hyper-ready to respond in milliseconds. Announcement |Documentation HTTP APIs for Amazon API Gateway (Preview) Make it easy to create, publish, maintain, monitor, and secure APIs at scale. Announcement |Blog AWS Compute Optimizer Uses ML to recommend optimal resources for customer workloads to reduce costs, and improve performance. Announcement |Blog AWS Fargate Spot A new option to run fault-tolerant applications on AWS Fargate with up to 70% discount compared to Fargate prices. Announcement AWS Local Zones A new type of AWS infrastructure deployment that places compute, storage, database and other services closer to customers. Announcement AWSNitro Enclaves AWS Nitro Enclaves enables customers to create isolated compute environments to further protect and securely process highly sensitive data. E.g. PII, health, finance. Product Page AWSOutposts AWS Outposts GA -- a new fully managed service that extends AWS infrastructure, services, APIs, and tools to on-premises environments. Announcement|Blog Amazon Managed Apache Cassandra Service (Preview) Scalable, highly available, managed Apache Cassandra-compatible database service. Announcement |ProductPage Amazon RDS on Outposts (Preview) Deploy managed databases in on- premises AWS Outposts environments. Announcement |FeaturePage Amazon RDSProxy Fully managed, highly available database proxy for Amazon Relational Database Service. Announcement |FeaturePage Simplified Windows and SQL Server BYOL New BYOL experience for software licenses, such as Windows and SQL Server that require dedicated physical servers. Announcement |Product Page | Documentation MIGRATION &TRANSFER End of Support Migration Program (EMP) for WindowsServer Helps customers package legacy applications allowing them to run on newer windows server OS without any code changes. Announcement |ProductPage INTERNET OF THINGS AWS IoT SiteWise (Preview) IoT SiteWise new features and pricing, additional capabilities to collect data, add context to the data, use the data in their custom applications, and view the data. Announcement |ProductPage QUANTUM TECHNOLOGIES Amazon Braket(Preview) Build, test, and run quantum computing applications. Product Page |Blog|Announcement
- 36. MACHINE LEARNING NETWORKING STORAGE Amazon Augmented AI (Amazon A2I) Makes it easy to build workflows required for human review of ML predictions. Announcement |Product Page |Documentation | Amazon CodeGuru(Preview) Machine learning service for automated code review and application performance profiling. Announcement |Product Page | Amazon Fraud Detector (Preview) Easily identify potentially fraudulent online activities such as online payment fraud and the creation of fake accounts. |Announcement |Product Page Amazon Kendra(Preview) Transform the way you search and interact with enterprise data using AI. Announcement |ProductPage |Documentation Amazon Transcribe Medical Quickly create accurate transcriptions from medical consultations between patients and physicians. Announcement|Product Page |Documentation Amazon Rekognition Custom Labels Train custom models to identify objects and scenes in images that are specific to your business needs. Announcement |FeaturePage AWS DeepComposer Gives developers of all skill levels a creative way to experience machine learning with a musical keyboard. Announcement|Product Page AWS DeepRacer Evo Multi-car racing and object avoidance capabilities. Announcement |Product Page | Amazon SageMaker Studio Provides a single, web-based visual interface to perform all ML development steps. Announcement Amazon SageMakerNotebooks One-click Jupyter notebooks that you can start working within seconds. Announcement Amazon SageMaker Experiments Help developers manage and track iterations for machine learning models by automatically storing input parameters, configuration, and results as experiments. Announcement Amazon SageMakerDebugger Debug and profile model training to improve accuracy, reduce training times, and facilitate a greater understanding of machine learningmodels. Announcement Amazon SageMakerAutopilot The industry’s first automated machine learning capability that gives you complete control and visibility into your ML models. Announcement Amazon SageMaker Model Monitor Automatically detect concept drift in deployed models and identify the source of the problem. Announcement SageMaker Operators for Kubernetes Make it easier to use Kubernetes to train, tune, and deploy machine learning (ML) models in Amazon SageMaker. Announcement |Documentation |GitHub| Contact Lens for Amazon Connect Customer centers use of ML to better understand sentiment, trends, and compliance risks of customer conversations. Announcement AWS Machine Learning Embark program Combines training, coaching, and implementation support to launch the company’s machine learning journey. Announcement Accelerated Site-to-Site VPN Improve the performance of VPN connections by intelligently routing traffic through the AWS Global Network and AWS edge locations.Announcement Amazon VPC Ingress Routing Make it Easy to insert Virtual Appliances in the Forwarding Path of VPC Traffic. Announcement |Blog AWS Transit Gateway Network Manager Centrally manage and monitor your global network across AWS and on premises. Announcement SERVERLESS Amazon EventBridge with Schema Registry Stores event structure - or schema - in a shared central location and maps those schema to code. Announcement | Blog |Documentation AWS Step Functions Express Workflows Workflow type capable of orchestrating AWS compute, database, and messaging services at event rates greater than 100,000 state transitions per second. Announcement |Product Page|Blog Access Analyzer for AmazonS3 Ensure access policies provide only intended access to S3 resources. Announcement |Blog |Documentation Amazon S3 Access Points Simplify and manage access at scale for applications using shared data sets on S3. Announcement |Feature Page |Blog | SECURITY, IDENTITY,& COMPLIANCE Amazon Detective(Preview) Easily analyze, investigate, and quickly identify the root cause of potential security issues or suspiciousactivities. Announcement |ProductPage AWS Identity and Access Management (IAM) Access Analyzer Generates comprehensive findings that identify resources that can be accessed from outside an AWS account. Blog |Announcement |FeaturePage DEVELOPER TOOLS Amazon Builders’Library Articles that detail how Amazon builds and operates the technologies that power Amazon.com andAWS. Builders Library 2019년 출시 서비스 한눈에 보기