Amazon Web Services: Lessons for Architecting Data in the Cloud
4 likes2,339 views
The document provides an overview of AWS services such as S3 for data storage and RDS/Aurora for relational database services, highlighting their features, benefits, and use cases. It emphasizes cost-effectiveness, scalability, and ease of operation, addressing common pitfalls and providing tips for effective use. Additionally, it discusses the advantages of APIs and event-driven architectures in cloud-based data delivery solutions.
Amazon Web Services: Lessons for Architecting Data in the Cloud
- 2. Ask us anything! Don Murray Stewart Harper
- 3. Connect Transform Automate FME: By Safe Software
- 4. AWS Data Storage AWS Simple Storage (S3) AWS RDS / Aurora Easy to operate, fault tolerant, scalable Relational Database Service in the cloud. Secure, durable, highly-scalable object storage
- 5. AWS S3 - Overview Secure, durable, highly- scalable object storage 99.999999999% reliability 3+ million requests per second S3 is highly performant and applications can get 100’s of requests a second $0.03 per GB of storage (5TB of data $150 a month) Can store single objects up to 5TB
- 6. AWS S3 - How it works A simple key/value persistent object store. It is not a disk and doesn’t have a directory or folder structure. It can be presented as a folder structure though. Every object is identified with a key.
- 7. You have a large amount of data on premise. When might you use S3? You want to take advantage of features that are not available on premises. You are serving large amounts of data and are having problems scaling. You want programmatic access to your data You wish to build an event driven architecture.
- 8. AWS S3 - Tips & Tricks Host a static website on S3 for next to nothing. Every object in S3 is web addressable. For even higher performance content delivery use in combination with CloudFront. Version your data automatically or transition to Glacier using lifecycle management. Provide time-limited access to objects via query string authentication.
- 9. Easy to operate, fault tolerant, scalable Relational Database Service in the cloud. AWS RDS - Overview AWS Aurora (MySQL compatible) Oracle MariaDB PostgreSQL (PostGIS) Microsoft SQL Server MySQLSupportedDataTypes The code, applications, and tools you use today with your existing databases can be used with Amazon RDS.
- 10. The AWS RDS service lets you... Deploy in minutes Automate backups Apply software patches automatically Scale storage and performance with one click Replicate to enhance availability and reliability
- 11. When might you use RDS? If you don’t want to worry about infrastructure. If you are looking to run a production database with minimal administration, perfectly suited for a small team. If you don’t want to worry about scaling, fault tolerance and enhanced availability as you grow.
- 12. On-Premises vs RDS PostGIS Specification: 16 Cores, 122GB of RAM, Linux, 50TB Storage
- 13. RDS - Tips and Tricks ● You can’t always install all the libraries you want, so check this before you migrate. For example urllib Python lib is not included. ● You don’t have SUPER privileges, which can make it impossible to perform advanced operations against the DB. Native Spatial Support Enterprise Geodatabase Support
- 14. AWS Aurora Database engine runs on EC2 and is compatible with the Open Source MySQL, logging and storage runs on Amazon’s storage layer. This is where most of the gains have come from.
- 15. RDS vs Aurora Aurora only supports the InnoDB storage engine. That rules out spatial index support. The gains are mainly around fault tolerance and performance. If you aren’t running a high throughput transactional database it might not be worth migrating. Aurora is only available on larger instances (r3 family), so it is more expensive if you only need a small database. Cross region replication is not supported by Aurora, it is with RDS.
- 16. Other Key AWS Services Simple Queueing Service Lambda Simple Notification Service API Gateway
- 18. The event driven design pattern Deploying an event-driven architecture is difficult and requires custom development. Lambda, Amazon’s new compute service, coupled with FME gives you an extremely powerful set of tools for the elastic processing of hundreds of datasets. Amazon S3, Amazon Kinesis and Amazon Dynamodb trigger events.
- 19. Automate data processing pattern with AWS Lambda, S3, SNS and SQS
- 20. Event Driven Processing Demo Automatically process data when it is dropped into an S3 bucket, send an email once the data has finished processing and place output in another S3 bucket. Incoming Data S3 Bucket S3 Bucket
- 21. Pelmorex Example
- 22. Pelmorex Example
- 23. Data Delivery
- 24. AWS Architecture for Data Delivery AWS perfectly suited for data delivery solutions: Multiple services for storing large volumes of data cheaply and securely. Costs grows with the popularity of your data delivery service. You can build highly performant workflows with the click of a button.
- 25. Caching a snapshot of your data on S3
- 26. Real time Data Delivery
- 27. Architecture - State Arkansas Data Portal
- 28. Our rack space costs (real estate on data center floor) are $3200 a month. Add to that the hardware costs, etc and you can see why moving to the cloud was a no brainer for us. Projected Costs - State Arkansas Data Portal On Premises AWS Anthony Davis, State Arkansas
- 29. APIs - Next Generation of Data Delivery ● APIs allow businesses to build platforms that partners and customers can use to access core business systems, whenever they want, in a stable and secure way. ● We now have the tools to create a fully functioning scalable API in less than a day, this is going to revolutionize businesses and bring the next wave of API value—creating internal disposable APIs.
- 30. Building an API with no infrastructure or code
- 31. API Gateway - Demo Building an API on top of a database in less than 5 minutes.
- 32. Free Training Hands-on and instructor led. FME Desktop Basic ● Nov 17-18 FME Server Authoring ● Dec 1-2 www.safe.com/training
- 33. AWS Doctor's Office Coming soon - Safe’s AWS experts on hand for 1H.
- 34. Q&A
- 35. Thank you! Download a free trial of FME at www.safe.com