AWS_Data_Pipeline
Download as PPTX, PDF0 likes344 views
AWS Data Pipeline is a web service that allows users to design data driven workflows to move and transform data between different AWS services reliably and in a cost effective manner. It allows users to schedule, run, and manage recurring data processing workloads. Data Pipeline includes components like pipeline definitions, schedules, task runners, and objects like shell command activities and S3 data nodes to design extract, transform, load (ETL) processes. It works with services like DynamoDB, RDS, Redshift, S3, and EC2. Pipelines are created by composing definition objects in a file and can be accessed through the AWS Management Console, CLI, SDKs, and APIs.
![● EC2 Resource
● Schedule {
"id" : "Hourly",
"type" : "Schedule",
"period" : "1 hours",
"startDateTime" : "2012-09-
01T00:00:00",
"endDateTime" : "2012-10-
01T00:00:00"
{
"id" : "MyEC2Resource",
"type" : "Ec2Resource",
"actionOnTaskFailure" : "terminate",
"actionOnResourceFailure" : "retryAll",
"maximumRetries" : "1",
"instanceType" : "m1.medium",
"securityGroups" : [
"test-group",
"default"
],
"keyPair" : "my-key-pair"
}](https://image.slidesharecdn.com/042154af-5b74-454d-aa23-eb15d8f9533e-160925165131/85/AWS_Data_Pipeline-6-320.jpg)
AWS_Data_Pipeline
- 1. AWS Data Pipeline ~ Ahasan Habib Technical Project Manager, Ixora Solutions Ltd. Dhaka, Bangladesh
- 2. What is AWS Data Pipeline? ● Webservice ● Movement & Data transformation ● Data driven workflow
- 3. Benefits ● Sequence, Schedule, Run, Manage recurring data processing workloads reliably. ● Cost effective ● Easy to design ETL ● Support for both structure and unstructure data ● Support on premises and cloud
- 4. Data Pipeline Components ● Pipeline Definition ● Pipeline Schedules & run tasks ● Task Runner
- 5. Data Pipeline Objects ● ShellCommand Activity ● S3 Data Node { "id" : "CreateDirectory", "type" : "ShellCommandActivity", "command" : "mkdir new-directory" } { "id" : "OutputData", "type" : "S3DataNode", "schedule" : { "ref" : "CopyPeriod" }, "filePath" : "s3://myBucket/#{@scheduledStartTime}.csv" }
- 6. ● EC2 Resource ● Schedule { "id" : "Hourly", "type" : "Schedule", "period" : "1 hours", "startDateTime" : "2012-09- 01T00:00:00", "endDateTime" : "2012-10- 01T00:00:00" { "id" : "MyEC2Resource", "type" : "Ec2Resource", "actionOnTaskFailure" : "terminate", "actionOnResourceFailure" : "retryAll", "maximumRetries" : "1", "instanceType" : "m1.medium", "securityGroups" : [ "test-group", "default" ], "keyPair" : "my-key-pair" }
- 7. Work with Other AWS Services ● Amozon Dynamo DB ● Amaxon RDS ● Amazon Redshift ● Amazon S3 ● EC2
- 8. Accessing Data Pipeline ● Amazon Management Console ● AWS CLI ● AWS SDK ● QUERY API
- 9. Create Data Pipeline ● Compose Pipeline Definition objects in a file ● Definition File Structure { "id": "S3DataInput", "type": "S3DataNode", "schedule": {"ref": "TheSchedule"}, "filePath": "s3://bucket_name", "myCustomField": "This is a custom value in a custom field.", "my_customFieldReference": {"ref":"AnotherPipelineComponent"} }
- 10. Step 1
- 11. Step 2
- 12. Step 3
- 14. Notification ● SNS ● Push Delivery ● Pub/sub Model
- 15. Q & A
- 16. “There's a lot of difference between listening and hearing.” ~G.K. Chesterton THANK YOU