GoDataFest 2019 Data Hubs
- 1. Data HubsData Hubs Architectural Patterns & Security BestArchitectural Patterns & Security Best Practices for Data-Intensive AWSPractices for Data-Intensive AWS PlatformsPlatforms Martijn van DongenMartijn van Dongen AWS Cloud Evangelist | Founder AWSug.nl | AWS APN AmbassadorAWS Cloud Evangelist | Founder AWSug.nl | AWS APN Ambassador 1
- 3. Doing "Consultancy"Doing "Consultancy" Currently @ PathéCurrently @ Pathé 3
- 6. Data HubsData Hubs Architectural Patterns & Security BestArchitectural Patterns & Security Best Practices for Data-Intensive AWSPractices for Data-Intensive AWS PlatformsPlatforms Martijn van DongenMartijn van Dongen AWS Cloud Evangelist | Founder AWSug.nl | AWS APN AmbassadorAWS Cloud Evangelist | Founder AWSug.nl | AWS APN Ambassador 6
- 7. Well Known Analytics & ML ServicesWell Known Analytics & ML Services Athena EMR QuickSight GlueLake Formation Sagemaker Forecast Personalize DeepRacer Rekognition S3 IAM 7
- 8. But, how about these...?But, how about these...? EFS KMS API Gateway RDSDMS VPC VPN Transit Gateway ECS Lambda 8
- 9. High Level OverviewHigh Level Overview AWS Cloud CRM ERP Database Migration Service S3 Data Lake Sagemaker Athena API Gateway Quicksight Analysts Apps 9
- 10. So, AWS Lake Formation?So, AWS Lake Formation? 10
- 11. PermissionsPermissions AWS Account Key Management Service Users S3 Organizations Service Control Policies Key Policies Identity-Based Policies Bucket Policies Resource- Based Policies 11
- 12. Service Control Policies (SCP)Service Control Policies (SCP) Statement: - Sid: "DenyAllOutsideEU" Effect: Deny NotAction: - "s3:*" - "rds:*" - "ec2:*" # ... Resource: "*" Condition: StringNotEquals: "aws:RequestedRegion": - "eu-west-1" - "eu-central-1" 12
- 13. Identity-Based Policies (IAM)Identity-Based Policies (IAM) Statement: - Sid: "AllowUserSeeBuckets" Effect: Allow Action: - "s3:ListAllMyBuckets" Resource: "arn:aws:s3:::*" - Sid: "AllowUserReadDataSet" Effect: Allow Action: - "s3:Get*" Resource: - "arn:aws:s3:::examplebucket/dataset/*" 13
- 14. Identity-Based Policies (IAM)Identity-Based Policies (IAM) Statement: - Sid: "AllowUserReadDataSet" Effect: Allow Action: - "s3:Get*" Resource: - "arn:aws:s3:::examplebucket/dataset/*" Condition: StringEquals: { "s3:ResourceTag/PCI": "False" } 14
- 15. Resource-Based PoliciesResource-Based Policies Statement: - Sid: "DenyWhenNotEncrypted" Effect: Deny Principal: * Action: "s3:PutObject" Resource: "arn:aws:s3:::examplebucket/*" Condition: StringNotLikeIfExists: { "s3:x-amz-server-side-encryption-aws-kms-key-id": "arn:aws:kms:eu-west-1:111122223333:key/*" } 15
- 16. NetworkingNetworking AWS Cloud Datacenter A Datacenter B CRM ERP Data Transit VPC Dev VPC Transit Gateway DMS Sagemaker S3 Data Lake 16
- 17. Data IngestionData Ingestion Source Database Migration Service RDS S3 Staging Glue S3 Datalake Internet 17
- 18. OverviewOverview AWS Cloud CRM ERP Database Migration Service S3 Data Lake Sagemaker Athena API Gateway Quicksight Analysts Apps 18
- 20. OverviewOverview AWS Cloud CRM ERP Database Migration Service S3 Data Lake Sagemaker Athena API Gateway Quicksight Analysts Apps 20
- 21. Sagemaker VPCSagemaker VPC VPC Public subnet Private subnet Data subnet Elastic File SystemNAT Gateway Internet Sagemaker VPC EndpointS3 Data Lake 21
- 22. Now publish that APINow publish that API S3 Datalake Sagemaker Elastic Container Repository Internet Elastic Container Service (ECS) ECS for Kubernetes API Gateway Sagemaker Inference Lambda 22
- 23. Pathé Data HubPathé Data Hub Diederik MeijerinkDiederik Meijerink 23