AWS Data Pipeline Tutorial | AWS Tutorial For Beginners | AWS Certification Training | Edureka
2 likes542 views
This document discusses AWS Data Pipeline, which is a service that helps reliably process and move data between AWS services and on-premises sources at specified intervals. It describes how AWS Data Pipeline can be used to build a data pipeline that collects data from different sources, performs analytics using EMR, and generates weekly reports. The key components of AWS Data Pipeline include the pipeline definition, pipeline, task runner, schedules, and pipeline components like instances and attempts. It also provides an example of using AWS Data Pipeline to import and export data from DynamoDB.
AWS Data Pipeline Tutorial | AWS Tutorial For Beginners | AWS Certification Training | Edureka
- 2. AWS Architect Certification Training www.edureka.co/cloudcomputing Agenda 01 Need for Data Pipeline 02 What is AWS Data Pipeline? 03 AWS Data Pipeline components 04 Demo on AWS Data Pipeline
- 3. Copyright © 2018, edureka and/or its affiliates. All rights reserved. Need of Data Pipeline
- 4. AWS Architect Certification Training www.edureka.co/cloudcomputing A Hypothetical Example Goal 1 Improve business by targeting content Goal 2 Manage application efficiently Goal 3 Improve business faster but at cheaper rate
- 5. AWS Architect Certification Training www.edureka.co/cloudcomputing Problem Statement Huge amount of data in different formats – so processing, storing & migrating data becomes complex Real-time data for registered users Webserver logs for potential users Demographic data & login credentials Sensor data & 3rd party datasets DynamoDB Amazon S3 Amazon RDS Amazon S3
- 6. AWS Architect Certification Training www.edureka.co/cloudcomputing Solution Feasible Solution Optimal Solution Analyse the data & convert from unstructured to structured format Use a data pipeline which handles processing, visualisations & migration
- 7. Copyright © 2018, edureka and/or its affiliates. All rights reserved. AWS Data Pipeline
- 8. AWS Architect Certification Training www.edureka.co/cloudcomputing AWS Data Pipeline AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on- premises data sources, at specified intervals. AWS Data Pipeline Task: Copy log files Daily task Task: Launch data analysis Weekly task EC2 Instance S3 Bucket Amazon EMR
- 9. AWS Architect Certification Training www.edureka.co/cloudcomputing AWS Data Pipeline Data Stores (O/P)Data Stores (I/P) Compute Resources DATA DATA
- 10. AWS Architect Certification Training www.edureka.co/cloudcomputing Example: Launch Data Analysis Collect data from different data sources, perform EMR analysis & generate weekly reports Event data from DynamoDB Bulk data from S3 Daily EMR analytics Daily EMR results Weekly summary report Weekly report in Redshift
- 11. AWS Architect Certification Training www.edureka.co/cloudcomputing Benefits of AWS Data PipelineBenefits of AWS Data Pipeline Provides a drag & drop console Built on distributed, reliable infrastructure Supports scheduling & error handling Distribute work to one machine or many Inexpensive to use Full control over the computational resources
- 12. Copyright © 2018, edureka and/or its affiliates. All rights reserved. AWS Data Pipeline Components
- 13. AWS Architect Certification Training www.edureka.co/cloudcomputing Components of AWS Data Pipeline Pipeline definition Pipeline Task Runner Specifies business logic of data management Data Nodes Activities Schedules PreconditionsResourcesActions
- 14. AWS Architect Certification Training www.edureka.co/cloudcomputing Components of AWS Data Pipeline Pipeline definition Pipeline Task Runner Schedules & runs the tasks to perform defined activities Pipeline Components Instances Attempts
- 15. AWS Architect Certification Training www.edureka.co/cloudcomputing Components of AWS Data Pipeline Pipeline definition Pipeline Task Runner Polls AWS Data Pipeline for tasks & then performs those tasks Pipeline schedules tasks Task runner polls tasks Retries remaining Task runner reports when task is done Did task succeed? Task ends Yes No No Yes
- 16. Copyright © 2018, edureka and/or its affiliates. All rights reserved. Demo – Import & export DynamoDB data
- 17. AWS Architect Certification Training www.edureka.co/cloudcomputing