Media Applications on AWS
Download as PPTX, PDF1 like743 views
The document discusses how media companies can leverage AWS services at each stage of content production and delivery including build, transform, store, deliver, search, archive, understand, notify, improve, and load. It provides examples of how companies like Coursera, Spotify, and PBS have used AWS services like S3, DynamoDB, CloudFront, Glacier, and EMR to scale infrastructure and better serve customers.
Media Applications on AWS
- 1. Media Applications on AWS Danilo Poccia – AWS Solutions Architect
- 2. What?
- 3. from 50,000 users to 17 million in 9 months now ~70 million users
- 4. Longer term, we believe that companies such as ours will divest themselves of all infrastructure in favor of third parties.
- 5. By using Amazon Web Services, Coursera can handle half a petabyte of traffic each month and scale to deliver courses over 3 million students in one year.
- 6. Who? Agency Enterprise Content Provider Broadcaster Web Agency “New Media” Marketing Live Content Traditional Broadcaster “Online” Marketing Online Broadcaster Advertising Agency On Demand Content
- 7. How?
- 8. Build
- 9. Build EU West (Ireland) Choose your Region US East (Northern Virginia) US West (Oregon) US West (Northern California) São Paulo Asia Pacific (Tokyo) Asia Pacific (Singapore) Asia Pacific (Sydney)
- 10. 2 x Availability Zones (AZ) Auto Scaling + ELB RDS Multi-AZ
- 11. RDBMS / NoSQL
- 12. “Amazon DynamoDB … could support over 500,000 writes per second.”
- 13. DynamoDB… Now all we had to do was move there – in 5 days.
- 15. Java Python PHP .NET Ruby Node.js
- 16. AWS Toolkit for Visual Studio AWS Toolkit for Eclipse
- 24. Build
- 25. Store Build
- 26. your content
- 27. your content 99.999999999% durability security & access control 1B to 5TB objects As many as you like
- 28. your content 99.999999999% durability security & access control 1B to 5TB objects As many as you like
- 31. Spotify needed a storage solution that could scale very quickly without incurring long lead times for upgrades… Amazon S3 gives us confidence in our ability to expand storage quickly while also providing high data durability.
- 32. Store Build
- 33. Store Build Load
- 38. Store Build Load
- 39. Store Build Load Transform
- 40. Original Video / Audio File(s) Transcoded Video / Audio File(s) Input Buckets are where you store the files that you want to transcode Output Buckets are where you want the transcoded files to go Transcoding Pipelines are queues that manage your Transcoding Jobs Transcoding Presets contain audio, video and other parameters that specify the output format Transcoding Jobs do the work of transcoding Video files and are contain a filename stored as and S3 a Transcoding Preset(s) objects details
- 43. Build Transform Store Load
- 44. Build Transform Archive Store Load
- 45. Secure and Durable Storage for Data Archiving and Backup
- 46. An organization like ours thinks in centuries when it comes to content retention We are excited to move our archives to Amazon Glacier
- 47. Archive With S3 Lifecycle Management S3 (Online) Move to Glacier (Archive) Expiration (Delete) 1. Store my object on S3 2. After 30 days, move it to Glacier 3. I need it again, restore it for 3 days 4. Then it is back to Glacier 5. (Optional) Delete it after 365 days
- 48. Build Transform Archive Store Load
- 49. Build Transform Archive Store Load
- 50. Build Transform Archive Notify Store Load
- 53. Notify SNS Enpoints SNS HTTP(S) SQS Email Mobile Push Notification SMS
- 54. Build Transform Archive Notify Store Load
- 55. Build Notify Transform Archive Deliver Store Load
- 56. Deliver Seattle New York (3) Los Angeles (2) Jacksonville Dallas (2) St. Louis Miami Palo Alto London (2) Ashburn (3) Newark Dublin Leverage Edge Locations Amsterdam (2) Stockholm Frankfurt(2) Paris (2) Mumbai Singapore(2) Tokyo (2) Hong Kong (2) São Paulo South Bend San Jose Osaka Milan Sydney Hayward Madrid Seoul Chennai
- 57. Deliver From Different Origins CloudFront S3 EC2 Route 53 LBR Custom
- 62. …using Amazon CloudFront is so simple and reliable that the team doesn’t have to think about it. It all just works, freeing us to focus on building cool applications. PBS reaches 120 million people through television and over 29 million people online each month.
- 63. Deliver Live HTTP Streaming Tutorials CloudFront Adobe Media Server IIS Media Services Wowza Media Server
- 64. Since launching Wowza Pro on Amazon EC2 … hundreds of users have signed up and our revenue for Wowza Pro has increased by more than 400 percent in just 6 months.
- 65. Build Notify Transform Archive Deliver Store Load
- 66. Build Transform Deliver Search Archive Notify Store Load
- 67. +
- 70. Faceting
- 71. Range Searching
- 72. Field Searching
- 73. (Meta) Data Indexing Search Requests
- 80. Build Transform Deliver Search Archive Notify Store Load
- 81. Build Transform Deliver Search Archive Understand Notify Store Load
- 82. Understand Analyze Your Data Elastic Beanstalk Log S3 CloudFront Access Log Custom Log Elastic MapReduce Redshift
- 83. Understand Know Your Users Who is it about? What happened? When did it take place? Where did it take place? Why did it happen? The 5 W’s
- 85. Build Transform Deliver Search Archive Understand Notify Store Load
- 86. Store Build Transform Deliver Search Archive Understand Notify Improve Load
- 87. Improve Test Your Content Kaizen* (改善) Content A / B Testing Continuous Integration Measure Feedback Continuous Improvement Continuous Deployment * Japanese for "improvement" or "change for the best” Know Your Users
- 88. Store Build Transform Deliver Search Archive Understand Notify Improve Load
- 89. Store Build Transform Deliver Search Archive Understand Notify Improve Load
- 90. Store Build Transform Deliver Search Archive Understand Notify Improve Load
- 91. Store Build Transform Deliver Search Archive Understand Notify Improve Load
- 92. Store Build Load Transform Deliver Search Archive Understand Notify Improve
Editor's Notes
- #3: To talk about Media Applications on AWS I’d like to begin looking at what you can build on AWS.
- #4: Pinterest is a social network that allows you to “pin” your content in a virtual board. They run on AWS, they managed to grow from 50K to 17M in 9 months, now they have about 70M users. They had to scale the storage to handle the content. And scale the web site to handle the growing number of users. So we have 2 main “point of attention” for media architectures: content storage and a fast, responsive online presence. Both need to be efficient and scalable.
- #5: When the Guardian, the UK newspaper, wanted to add an “Open Platform” to their service, where they could expose via API some of their content, they looked at AWS. It was unclear how many people would use the new platform. And for this kind of “innovative” services you would expect an initial peak of “curiosity” that may settle down after a few weeks. And you don’t want to build your architecture and size your infrastructure on the peak. But you need to serve the peak well! Now they think that – longer term – infrastructure should be a service for a company like them.
- #6: Coursera, they created a new channel for universities to make courses through internet to a global audience. They have a lot of video. And people is using a lot of different devices to participate to the trainings they offer. They have now more than 5M users.
- #7: So who can be interested in building a media application on AWS? …
- #8: And How can you proceed in building your media application?
- #9: Well, you actually need to build your “web infrastructure” where you “app” is running…
- #11: This is a sample web architecture that follows all the usual guidelines we give to our customers… Multi-AZ in front-end and backend. Auto Scaling to adjust your infrastructure to the actual load. Highly Available… To some of you it may look quite familiar, to others can be quite complex… can we simplify it a little bit?
- #12: Well we could (depending on the application – and maybe your “architectural taste”) replace the Relational DB with a fully managed NoSQL service like DynamoDB… With DynamoDB you set your required throughtput (reads and writes) for any table. And change the throughput even during a day… Maybe you need less throughtput during the night on outside of a specific event…
- #13: This is what Shazam did… Maybe you start moving only the critical part fo your RDBMS (maybe 1 or 2 tables) to DynamoDB, and you’re the two repositories together.
- #14: And Tellybug, that is getting the voting feedback fot a lot of UK TV show such as X Factor… etc…
- #15: To simplify the management of the infrastructure, you could use a service like Elastic Beanstalk…
- #16: Where you deploy your own application without any specific changes to be handled by Elastic Beanstalk, being it a Java, .Net, PHP, Ruby, Python or Node.js app…
- #17: You can deploy from Visual Studio o Eclipse or using Git…
- #18: Elastic Beanstalk will prepare all the infrastructure for you…
- #19: From the Elastic Load Balancer, to distribute the load on more than one system in the web tier…
- #20: An Auto Scaling Group, to be able to grow, or shrink, automatically depending on the load on the web tier…
- #21: Deploying instances in the Auto Scaling Group, balanced by the ELB…
- #22: Everything is “wired” by Elastic Beanstalk, and you can monitor your application (not just your infrastructure) from the web console.
- #23: So using Elastic Beanstalk the overall architecture is simpler…
- #24: …let’s say that we use DynamoDB for persistance…
- #25: So we built a service…
- #26: Now we need to store media content… and we want a storage that is efficient and scalable and robust…
- #27: We can use Amazon Simple Storage Service, S3…
- #28: …
- #29: No limits in the number of object you put in an S3 bucket…
- #31: This is why Dropbox is using S3 to store all the files of all its users…
- #32: And Spotify, the music streaming service, decided to use S3 to store their core-asset media library. S3 was scalable, without any “storage upgrade”, with high durability (11 “9”s).
- #33: We know were to store content…
- #34: But we have to load media content , maybe once at the beginning, maybe often, maybe I use user-generated content…
- #35: Let’s add S3 to our architecture
- #36: We can load content through the web application…
- #37: But with S3 we can load content directly, using signed URLs (HTTP POST) or Security Token Service (STS) to create “temporary credentials” that can be used in a client application. A client JavaScript application. Or a mobile application.
- #38: If the media content load initially (or periodically) is too much, you can use the physical AWS Import/Export service… Send your phisical media to us (specs on the website) – we import data and send the physical media back to you. In the same way you can export data…
- #39: We know how to store and load data…
- #40: But we need to adjust the content for all the different devices that our users could possibly use… This is especially true (and expensive) for video… We can build our own transcoding pipeline on EC2… But we probably want to focus our energy in somethig else, in something that can differenciate our business more than trascoding…
- #41: So we can use Elastic Trascoding to do that, it is integrated with S3 as the media storage…
- #43: We add Elastic Trascoder that will read and write from our media content repository on S3.
- #44: And when the transcoded content is ready for our users…
- #45: We probably don’t need the original master video anymore… if not for occasional use…
- #46: We can archive them on Glacier… Same durability as S3… Roughly 1/10-1/9 the cost of S3, but you need 3-5 hours to access archived content.
- #47: This is the storage that the NYPR is using…
- #48: And Glacier can be used as a storage tier with S3 Lifecycle Management…
- #49: So we can archive the content that we don’t immediately need…
- #50: And have it back if we need it… E.g. a new device to transcode for…
- #51: And when new content is ready we probably need to notify our users that they can access it… It depends on the logic of your service…
- #52: Elastic Transcoder can notify when a transcoding job completes (actually depending on the status of the job)…
- #53: We can use SNS to get the notification and manage it, e.g. to send an email to (some) users.
- #54: SNS can be in place be used to notify different targets…
- #55: So our users know when new content (they subscribed?) is ready…
- #56: But we still need to deliver the content in the most efficient way, providing a good user experience…
- #57: We can leverage all the Edge Locations that AWS provides… That are used by our CDN CloudFront and by our distributed DNS architecture, Route 53.
- #58: CloudFront can distribute (caching locally) a lot of different origins, from static content, to dynamic content, to video streaming…
- #60: Here we use CloudFront to distribute bot the static content (on S3, can be an HLS ready segmented video) and the dynamic part of the web site. CloudFront can distribute HTTP content and has recently been extended to accept most HTTP methods, like POST, PUT, DELETE…
- #61: And Route 53 can be the distributed DNS for my service.
- #62: Flipboard, the “virtual magazine” for tablets (and now also for the web) is using CloudFront to distribute its content.
- #63: PBS (Public Broadcasting Service), a no-profit consortium in the US, baked by commercial broadcasters, is using CloudFront. They stream online to 29 million users.
- #64: CloudFront can be easily used to stream Live HTTP content (Apple HLS, Adobe HDS, Microsoft Smooth Streaming). We have tutorials that use our template (CloudFormation) to build an overall streming architecture. And we have agreements on monthly subscription for some of those, like AMS and Wowza.
- #65: In fact Wowza… +400% in 6 months…
- #66: We are ready to deliver…
- #67: But as the media content grows we probably need a way to index our media, based on media metadata. For internal of external (customer facing) use.
- #68: The same engine that powers Amazon.com website can be used as a service.
- #69: Search engines can look easy to implement…
- #70: But there are a lot of features to think of…
- #73: Imagine how many SQL queries you should do if you implement a search engine with a traditional RDBMS… CloudSearch is designed to do that.
- #74: 2 main interfaces…
- #75: So let’s add CloudSearch…
- #76: Here, getting the input from the notification on new content ready.
- #77: SmugMug is a customer using CloudSearch, they have millions of users and billions of images to index and search through.
- #78: Here’s a sample…
- #81: Now se can internally or externally search our media content…
- #82: But it is important we understand how our users are using our content….
- #83: All those logs on S3, we can analyze using EMR or Redshift…
- #84: This is what I mean by “understanding”…
- #85: And this is what Foursquare is doing (among other things) on AWS…
- #86: From understading…
- #87: To improvement!
- #88: Test your content to different population of your user! If you have a main page, try different “ensamble” of contents. If you have “titles” for your content, try different titles! Which one is more engaging for your user?
- #89: So now we’re almost there…
- #90: We did a full loop in implementing our media architecture…
- #91: We’re ready for another loop, building and improving our service!
- #92: All the concepts that we saw, can be taken apart and used “alone”… all the interfaces we use are standard. You can use your one trascoding pipeline, maybe your search engine implementation, and so on…
- #93: But all this concepts are brick in your bag, tools that you can use…
- #94: Building block that you can leverage to have more time to focus on your business! Let’s see what a Customer of ours is doing on AWS!!!