Drive Down the Cost of your Data Lake by Using the Right Data Tiering
0 likes177 views
The document discusses strategies for optimizing data storage costs using Amazon Web Services (AWS) through effective data tiering, classification, and automation with features like S3 intelligent-tiering. It outlines how to monitor usage, assess access patterns, and choose appropriate storage classes to save costs significantly. Additionally, it emphasizes understanding application requirements and organizing data using tags for optimal storage management.
Drive Down the Cost of your Data Lake by Using the Right Data Tiering
- 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Drive Down the Cost of your Data Lake by Using the Right Data Tiering Boaz Ziniman Technical Evangelist - Amazon Web Services @ziniman boaz.ziniman.aws ziniman
- 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Broad portfolio of storage services Amazon Elastic Block Store Amazon S3 Amazon Glacier Data movement OnlineOffline AWS Snow family AWS Storage Gateway Family AWS Direct Connect Amazon EFS File Sync Amazon S3 Transfer Acceleration Storage Partners Amazon Kinesis Data Streams Amazon Kinesis Video Streams Amazon Elastic File System NEW! STORAGE CLASS Object storage S3 Standard S3 Intelligent-Tiering S3 Standard-IA S3 One Zone-IA S3 Glacier
- 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Pillars of cost optimization Application Requirements Data Organization Right Sizing Monitor, Analyze, Optimize
- 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Define application requirements Media master files Big Data File Sharing Content Distribution Archive Data Analytics Backup & Restore Dynamic Websites Mobile sync & backup Disaster Recover Re-creatable data
- 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Organize data with Object Tags and Prefixes CustomerID = 3a24xxyz24 Department = Finance Project = FinancialAnalysis Classification = Confidential Environment = Test Control access, analyze usage, lifecycle and replicate objects Up to 10 mutable metadata tags (key value pair) per object Completely customizable (Dept., Project, Environment, etc.) Tag objects when created, later, or both
- 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Choose the storage class that fits best ≥ 3 AZs 1 AZ 99.99% 99.5% Milliseconds Hours Hours YearsFrequent Infrequent 0 Bytes 5 Terabytes Reduce storage cost > 80% by choosing the storage class option that best fits your use case 2 Regions 99.9%
- 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Monitor, Analyze, and Tier … Monitor with S3 Inventory, Amazon CloudWatch, AWS CloudTrail Tier and expire storage with S3 Lifecycle Policy … or just let S3 Intelligent-Tiering do the work and you save on storage costs automatically Understand access patterns with S3 Storage Class Analysis
- 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman AWS pricing principles No upfront investment Pay-as-you-go approach Pay less by using more Pay less as AWS grows $
- 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Decreasing prices and more storage options 1 2 Decreasing storage prices S3 Standard (2006) Glacier (2012) S-IA (2015) Z-IA (H1-2018) INT (Q4 2018) Accelerating innovation 2006 2018
- 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Your choice of Amazon S3 storage classes Access FrequencyFrequent Infrequent • Active, frequently accessed data • Milliseconds access • > 3 AZ • $0.0210/GB • Data with changing access patterns • Milliseconds access • > 3 AZ • $0.0210 to $0.0125/GB • Monitoring fee per Obj. • Min storage duration • Infrequently accessed data • Milliseconds access • > 3 AZ • $0.0125/GB • Retrieval fee per GB • Min storage duration • Min object size S3 Standard S3 S-IA S3 Z-IA Amazon Glacier • Re-creatable, less accessed data • Milliseconds access • 1 AZ • $0.0100/GB • Retrieval fee per GB • Min storage duration • Min object size • Archive data • Select minutes or hours • > 3 AZ • $0.0040/GB • Retrieval fee per GB • Min storage duration • Min object size S3 INT Amazon Glacier Deep Archive • Archive data • Select hours • > 3 AZ • $0.00099/GB • Retrieval fee per GB • Min storage duration • Min object size
- 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman S3 Storage Class Analysis and S3 Lifecycle Policy U se S3 Sto r ag e Cl ass A nal ysi s to i de nti f y stor age age gr oups that ar e l e ss f r e que ntl y ac c e sse d Se t S3 L i f e c yc l e Po l i c y to tie r stor age to low e r c ost stor age c lasse s and e xp i r e sto r ag e b ase d o n ag e o f o b j e c t G r e at f o r pr e d ic tabl e w or kl oad s ( o b j e c t ag e i ndi c ate s ac c e ss f r e que nc y) F i ne tune anal ysi s b y b uc ke t, p r e f i x, o r o b j e c t tag
- 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman S3 Intelligent-Tiering automates cost savings
- 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman The story behind S3 Intelligent-Tiering S3 Intelligent-Tiering New cloud storage class that automates cost savings for customers Heavy Lifting Fragmented applications, constraints on resources and experience Unmatched experience >1M S3 customers, Trillions of objects, Millions of requests per second Amazon Machine Learning predict future access patterns, inform storage of objects in most cost- effective way + +
- 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman S3 Intelligent-Tiering storage class Automatically moves objects between two storage tiers: • Frequent access tier optimized for frequent use of data • Lower cost infrequent access tier optimized for less accessed data Monitors access patterns and auto-tiers on granular object level No performance impact, no operational overhead Milliseconds access, > 3 AZs, Monitoring fee per Object, Minimum storage duration
- 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Ideal use cases for S3 Intelligent-Tiering Dynamic cost optimization with no performance impact and no operational overhead Big Data, Data Lakes Storage with changing access patterns used by multiple applications Enterprises Storage accessed by fragmented applications from various organizations Startups Constraint on resources and experience to optimize storage themselves Amazon S3
- 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Workload pattern 1 – frequently accessed data W o r kl o ad c har ac te r i sti c s: • F r e q u e n t l y a c c e s s e d s t o r a g e ( > 1 0 0 % o f s t o r a g e r e t r i e v e d ) • S o m e t i m e s s m a l l o b j e c t s ( a v g . o b j e c t s i z e ~ K B ) • S t o r a g e d u r a t i o n s o m e t i m e s s h o r t Co m m o n u se c ase s: • B i g d a t a a n a l y t i c s , d y n a m i c w e b s i t e h o s t i n g , I o T s e n s o r d a t a , D N A s e q u e n c e s , f i n a n c i a l s i m u l a t i o n s , o r i g i n s t o r a g e f o r C D N Sto r ag e c l asse s: • S 3 S t a n d a r d , m a y b e S 3 I N T
- 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Workload pattern 2 – infrequently accessed data W o r kl o ad c har ac te r i sti c s: • O v e r t i m e i n f r e q u e n t l y a c c e s s e d s t o r a g e ( < 1 0 0 % o f s t o r a g e r e t r i e v e d a f t e r 9 0 d a y s ) • L a r g e o b j e c t s ( a v g . o b j e c t s i z e ~ M B ) • S t o r a g e d u r a t i o n l o n g t e r m Co m m o n u se c ase s: • M o b i l e s y n c & b a c k u p , d a t a l o g s , m e d i a a s s e t s f o r g a m i n g , c u s t o m e r g e n e r a t e d c o n t e n t , d a t a s t o r e d f o r d i s a s t e r r e c o v e r y Sto r ag e c l asse s: • L i f e c y c l e f r o m S 3 S t a n d a r d t o S - I A o r Z - I A f o r r e - c r e a t a b l e d a t a • U s e S 3 I N T f o r a u t o m a t e d t i e r i n g • U s e G l a c i e r f o r A r c h i v e
- 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Workload pattern 3 – data with changing access W o r kl o ad c har ac te r i sti c s: • D a t a w i t h c h a n g i n g o r u n p r e d i c t a b l e a c c e s s p a t t e r n s • M i x o f o b j e c t s i z e s ( a v g . o b j e c t s i z e ~ M B ) • S t o r a g e d u r a t i o n l o n g t e r m Co m m o n u se c ase s: • M a c h i n e L e a r n i n g t r a i n i n g d a t a , S a t e l l i t e a n d G e o s p a t i a l i m a g e r y , F i n a n c i a l T r a n s a c t i o n R e c o r d s , A u t o n o m o u s v e h i c l e d a t a , d a t a l a k e s Sto r ag e c l asse s: • S 3 I N T
- 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Workload pattern 4 – unknown access patterns W o r kl o ad c har ac te r i sti c s: • U n k n o w n w o r k l o a d • Y o u o n l y k n o w t h a t o b j e c t s a r e l a r g e ( ~ M B ) a n d s t o r a g e d u r a t i o n i s l o n g ( ~ m o n t h s ) à S 3 I N T W o r kl o ad c har ac te r i sti c s: • U n k n o w n w o r k l o a d • U n k n o w n o b j e c t s i z e a n d s h o r t l i v e d o b j e c t s ( < m o n t h s ) à S t a r t w i t h S 3 S t a n d a r d a n d a f t e r s o m e t i m e l i f e c y c l e l a r g e o b j e c t s i n t o S 3 I N T
- 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman https://bit.ly/AWSCost
- 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Date Tiering Cost Reduction
- 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Pinterest S3 use cases Operational Data Store on AWS S3 Business Intelligence: Reporting and Ad Hoc Analysis Product Analytics and Experimentation Machine Learning ...and all of those fun Pinterest images ...and more standard use cases like online storage backups
- 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Storage usage growth Standard IA Glacier
- 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. @ziniman Putting it all together Understand your application requirements Use tags and prefixes to organize your data Optimize across all storage classes Cost optimize on an object level (or tag, prefix, bucket) S3 Intelligent-Tiering for automated cost savings INT S- IA Z-IA Glacier Std AWS Cloud enables you to be more innovative, agile, and cost effective
- 29. Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Boaz Ziniman Technical Evangelist - Amazon Web Services @ziniman boaz.ziniman.aws ziniman