Cloudera Altus: Big Data in der Cloud einfach gemacht
Download as PPTX, PDF2 likes507 views
The document discusses the advantages of cloud deployments over traditional on-premises systems, highlighting that cloud environments will dominate all workload categories. Cloudera's solutions, particularly Altus, provide efficient data engineering and analytics capabilities by optimizing shared data experiences and supporting rapid job submission without cluster management. It emphasizes the integration of various services for analytics, machine learning, and operational processing in cloud settings, ensuring scalability and strong data governance.
![22© Cloudera, Inc. All rights reserved.
Demo - Retail clickstream analysis
Data
engineering
Analytic
database
Well-formatted
data
semi-
structured
weblogs
cleaning
formatting
filtering
SQL
reporting
Self-service
BI[IP, date, method,
url, http_version,
code1, code2,
dash, user_agent]
Insights](https://image.slidesharecdn.com/altuswebinardach20022018michaelkohsv2-180221142048/85/Cloudera-Altus-Big-Data-in-der-Cloud-einfach-gemacht-22-320.jpg)
Cloudera Altus: Big Data in der Cloud einfach gemacht
- 1. 1© Cloudera, Inc. All rights reserved. ClouderaAltus: Big Data in der Cloud einfach gemacht MichaelKohs | SalesEngineer | mkohs@cloudera.com
- 2. 2© Cloudera, Inc. All rights reserved. Shift to cloud: an analyst view Source: 451 Research, Voice of the Enterprise: Workloads and Key Projects, Cloud Transformation, 2017. Key Points ▪ Cloud deployments will be the dominant environment in every category ▪ Every cloud deployment environment will see increases in every workload category ▪ Analytics and App Development areas expected strong gains
- 3. 3© Cloudera, Inc. All rights reserved. My organization is moving to the cloud, why should we consider Cloudera?
- 4. 4© Cloudera, Inc. All rights reserved. Traditional on-premises deployments perform reasonably well Strong multi-function support Strong shared data experience Strong operational model Moderate cost management Moderate tenant isolation Moderate workload elasticity Weak on self service Weak on speed of deployment Shared Data Experience (Metadata, Security, Governance) One physical cluster provides a shared data experience to multiple workloads and tenants But not good enough for tomorrow
- 5. 5© Cloudera, Inc. All rights reserved. Traditional cloud deployments are strong where on-premises is weak, but at the expense of creating workload silos Moderate multi-function support Weak on shared data experience Weak operational model Moderate cost management Strong on tenant isolation Strong on workload elasticity Strong on self service Strong on speed of deployment Shared Data (S3, ADLS) Cloud This is the experience of cloud house offerings
- 6. 6© Cloudera, Inc. All rights reserved. Only Cloud deployments with SDX optimize for all design goals Shared Data Experience (Metadata, Security, Governance) One logical cluster provides a shared data experience to multiple workloads and tenants SDX makes it possible to transfer on-premises design wins to cloud Shared storage (S3, ADLS) Cloud Strong multi-function support Strong shared data experience Strong operational model Strong on cost management Strong on tenant isolation Strong on workload elasticity Strong on self service Strong on speed of deployment
- 7. 7© Cloudera, Inc. All rights reserved. Cloudera’s Public Cloud Reference Architecture Compute Spark Hive Hive on Spark Impala Solr … Each workload runs in an isolated Workload Cluster Navigator Metastore Hive Metastore Sentry Metastore SDX services run on shared RDS | MySQL Management Cloudera Manager Cloudera Navigator Cloudera Director Each management service supports all Workload and SDX services Storage S3, ADLS Altus
- 8. 8© Cloudera, Inc. All rights reserved. 8 The modern platform for machine learning and analytics optimized for the cloud DATA CATALOG SECURITY GOVERNANCE WORKLOAD MANAGEMENT INGEST & REPLICATION EXTENSIBLE SERVICES CORE SERVICES DATA ENGINEERING OPERATIONAL DATABASE ANALYTIC DATABASE DATA SCIENCE S 3 ADL S HDF S KUD U STORAGE SERVICES Cloudera Enterprise PRIVATE CLOUDBARE METAL INFRASTRUCTURE DEPLOYMENT OPTIONS SERVICES
- 9. 9© Cloudera, Inc. All rights reserved. Cloudera Director for cluster lifecycle management Easy • Single pane of glass for all cloud infrastructure • Create templates to run applications in a pre- optimized manner Flexible • Multi-cloud: AWS, Azure, GCP • Hourly pricing with auto billing & metering • Spot instance/block support Enterprise-grade • Integration across Cloudera Enterprise • Management of CDH deployments at scale • Deeply integrated with Cloudera Manager
- 10. 10© Cloudera, Inc. All rights reserved. Cloudera Altus Multi-cloud foundation for building new cloud services Altus PaaS Foundation DATA ENGINEERING ANALYTIC DATABASE (beta) Altus Platform Services
- 11. 11© Cloudera, Inc. All rights reserved. Director vs Altus - when to use either? Altus Director Automated log saving x Automated Cluster spin up / down (no extra coding) x Data Engineering – Hive, Spark, HoS, MR x x Production Job Driven x Workload Analytics x Cluster Duration Purely Transient Transient OR Persistent Job Development / Exploration x 3rd Party Installations x Full Control of CM x Analytical / Operational - Impala, HBase, Search x Persistent (or Transient) x Grow / Shrink Cluster x
- 12. 12© Cloudera, Inc. All rights reserved. Altus Service Architecture (AWS) ● Runs in Cloudera’s secured and monitored environment ● Manages CDH clusters in customer cloud account ● Customer data does not pass to Cloudera (Workload Analytics requires opt-in log data transfer to Cloudera)
- 13. 13© Cloudera, Inc. All rights reserved. Altus Service Architecture (Azure) ● Runs in Cloudera’s secured and monitored environment ● Manages CDH clusters in customer cloud account ● Customer data does not pass to Cloudera (Workload Analytics requires opt-in log data transfer to Cloudera)
- 14. 14© Cloudera, Inc. All rights reserved. Altus Data Engineering for ETL, machine learning, and data processing • Fast, easy job submission without the cluster management • Built-in Workload Analytics for troubleshooting and optimization • Lower costs with transient resources and pay-per-use pricing • Full benefits of isolation + Shared Data Experience
- 15. 15© Cloudera, Inc. All rights reserved. End-user focused with jobs as first-class objects Workload troubleshooting and analytics ● Troubleshoot jobs after cluster termination through job log and configuration browsing ● Insight into causes of job failure ● Identification and root cause analysis of slow jobs
- 16. 16© Cloudera, Inc. All rights reserved. Capture metadata spanning multiple clusters Persist metadata with Cloudera Navigator ● Export metadata and lineage information from Altus clusters ● Insight into full data management pipeline including transient clusters
- 17. 17© Cloudera, Inc. All rights reserved. Three immediate use cases for Altus Data Engineering ETL FOR ANALYTIC DB BATCH MACHINE LEARNING ETL OFFLOAD Cloud-native batch preparation for Impala on IaaS or, soon, Altus Analytic DB. Scalable compute for massively-parallel batch machine learning training, scoring, or simulation. Offload batch processing jobs from overburdened on-premises clusters. MLData ScienceETL Analytic DB ETL On-Prem
- 18. 18© Cloudera, Inc. All rights reserved. Altus Analytic Database For business analysts: • Query with predictable performance, at any time, without risking SLAs • Bring limitless new users and use cases with instant self-service analytic access • Data available for broad access (SQL, BI tools, Python, R, etc) The first data warehouse cloud service to bring the warehouse to the data - delivering instant analytics to anyone For IT: • Easily and elastically provision isolated resources as and when they’re needed • Simple multi-tenant management including federated identity and consistent governance • Eliminate data movement and copies across workloads
- 19. 19© Cloudera, Inc. All rights reserved. Long Running Cluster: Director Managed Elastic Cluster(s): Director Managed / Altus Managed ETL to Cloud-Native Analytic Database SDX: Schema (HMS), Security (Sentry), Lineage/Audit (Navigator) Data Engineering Workflow: ETL Source Data 1+ Jobs (e.g. Hive) 1+ Jobs (e.g. Spark) Analytic Database (Impala) Source Data BI Tools (e.g. Tableau) SQL Editor (e.g. Hue) Transient Cluster(s): Atlus Managed Workflow, Monitoring, and Scheduling (e.g. AutoSys, ControlM, Airflow, Talend, Informatica) Object Store (ADLS, S3)
- 20. 20© Cloudera, Inc. All rights reserved. The Scenario My Role: Data Analyst at DataCo – a Sports Retailer Business Issue: Experiencing lower than expected website sales. Why? Technical Issues: I have a data warehouse on premise, which contains my sales order data, but it is very old and slow and it is difficult to do ad hoc queries on it. My clickstream data is too big to ingest into my data warehouse Requirements: Need an Analytic Database to do ad hoc queries on order data Need a temporary platform to process weblogs once a day Ability to join processed weblogs to order data
- 21. 21© Cloudera, Inc. All rights reserved.
- 22. 22© Cloudera, Inc. All rights reserved. Demo - Retail clickstream analysis Data engineering Analytic database Well-formatted data semi- structured weblogs cleaning formatting filtering SQL reporting Self-service BI[IP, date, method, url, http_version, code1, code2, dash, user_agent] Insights
- 23. 23© Cloudera, Inc. All rights reserved. Object store Cloudera Director Long-running Kafka cluster Long-running stream processing cluster Long-running Analytic DB cluster (Impala) Click stream data Transient Data Engineering clusters (Altus) Cloudera Altus Cloudera Manager HDFS on premise Self-Service Analytic DB clusters Cloudera Altus Shared Data Experience (Metadata, Security, Governance)
- 24. 24© Cloudera, Inc. All rights reserved. Q&A
- 25. 25© Cloudera, Inc. All rights reserved. Resources Cloudera Altus Cloudera Altus documentation Cloudera Director Cloudera Director documentation Cloudera SDX Try Cloudera Director on Microsoft Azure Try Cloudera Director on AWS with AWS Quickstart Cloudera Reference Architectures for public and private cloud deployments