MetaCDN: Enabling High Performance, Low Cost Content Storage and Delivery via the Cloud

Editor's Notes

• #2: MetaCDN is a system that leverages several existing ‘storage clouds’, creating an integrated overlay network that provides a low cost, high performance content delivery network for content creators.
• #4: *Content Delivery Networks (CDNs) such as Akamai and Mirror Image place web server clusters in numerous geographical locations to improve the responsiveness and locality of the content it hosts for end-users. *However, their services are priced out of reach for all but the largest enterprise customers.
• #5: Major CDN providers are notoriously cagey about revealing their prices. Most will only reveal their prices if you are serious customer and are willing to commit to a contract and minimum data usage (as detailed in the previous slide). As such Dan Rayburn @ StreamingMedia.com (a blog run for streaming media and CDN professionals) undertakes an informal sampling of pricing (taken from CDN customers) every few months.
• #6: Numerous ‘storage cloud’ providers (or ‘Storage as a Service’) have emerged that can provide data storage and delivery in several continents, offering S.L.A. backed performance and uptime promises for their services.
• #7: Each storage provider outlines their own cost structure for data transferred in and out of their service, as well as charging for persistent storage. In each of these cases, the costs are in the order of cents per gigabyte. Pricing scales downward based on higher usage for all providers. There is no minimum data usage requirement and no contracts - you only pay for what you store and transfer.
• #8: The providers themselves have very similar core functionality, but there are some key differences, for example, the largest allowable file size, the coverage footprint or specific features.
• #9: It is easy to see why storage clouds provide a compelling alternative to traditional CDNs for content producers that transfer significant amounts of data to their customers.
• #10: Amazon CloudFront offers a CDN-like service that is significantly cheaper than tradional CDNs. Amazon charges different rates depending on where the data is delivered from to reflect the cost of data transfer and operations in different locations.
• #11: 1. MetaCDN is more likely to meet the needs of content creators than a single provider could. 2. There is no ‘unified’ or familiar interface for all storage clouds. Consider Amazon S3, Nirvanix SDN, Mosso Cloud Files and Microsoft Azure Storage Service. These four cloud storage providers have four separate access APIs that a developer would need to learn to access these services. 3. If a content creator attempted to utilise these providers themselves, they would essentially need to perform the load balancing and redirection themselves at their origin sites (complex!)
• #12: The service is presented to end-users as a web portal for small or ad-hoc deployments or as Web Services (currently under development) for integration of customers with more complex and frequently changing content delivery needs. The web portal was developed using Java Enterprise and Java Server Faces (JSF) technologies, with a MySQL back-end to store user accounts, deployments, and the capabilities and pricing of service providers. Introduce connectors, and major components.
• #13: The MetaCDN system integrates with each storage provider via connectors that provides an abstraction to hide the complexity arising from the differing ways each provider allows access to their systems. The connectors encapsulate basic operations like creation, deletion and renaming of files and folders. If an operation is not supported on a particular service, then the connector for that service should throw a FeatureNotSupportedException.
• #14: 1. MetaCDN deploys as many replicas as possible to all available locations. 2. A user nominates regions and MetaCDN matches the requested regions with providers that service those areas. 3. Where MetaCDN deploys as many replicas in the locations requested by the user as their storage and transfer budget will allow, keeping them active until that budget is exhausted. 4. MetaCDN deploys to providers that match specific QoS targets that a user specifies, such as average throughput or response time from a particular location, which is tracked by persistent probing from the MetaCDN QoS monitor.
• #17: The MetaCDN database tracks all pertinent information such as users of the system, credentials for various providers, details about the providers capabilities, pricing and footprint and details of replicas deployed.
• #18: Using the web portal, users can sign up for an account on the MetaCDN system, and enter credentials for any cloud storage or other provider they have an account with. Once this simple step has been performed, they can utilise the MetaCDN system to intelligently deploy content onto storage providers according to their performance requirements and budget limitations.
• #20: A MetaCDN user is required to register the credentials of providers they have accounts with. Once this step is done they do not need to worry about how to interact with each of the providers. Eventually we would like MetaCDN users to not require accounts with specific providers - rather MetaCDN would provide consolidated billing of users for storage and transfer of content.
• #21: The MetaCDN \"Control Panel\" gives easy access to the core features of the service. You can deploy content, view existing deployments (via a high level content view or a detailed replica view), and view a deployment map overlayed onto Google Maps or Google Earth.
• #22: Here we can see an example of geographical-based deployment. A user nominates regions and MetaCDN matches the requested regions with providers that service those areas. The user also specifies the desired lifetime of the deployment, after which the replicas will be removed.
• #23: We can view details of our past deployments. We store and track information such as the origin id (i.e. the original source of the content), a unique GUID, the MetaCDN URL that represents the deployment, the number of times this content has been downloaded, the last time this content was downloaded and how many replicas were generated from this deployment.
• #24: Here we can see the specific replicas that have been generated from our various deployments. For each replica, we can see which provider and location was utilised, the public URL of the replica, the number of times the replica has been downloaded, the last access time of a specific replica and options to modify, delete, or view the replica if we wish to fine tune our deployment.
• #25: We can get a birds eye view of where our replicas are stored, and how many are stored in each location. MetaCDN generates a KML file for each user that is used to overlay on Google Maps (shown here) or we can view our deployments in Google Earth. We expect to overlay more useful information in these views in the near future, such as the cost expenditure at each location and the location of client (i.e. file consumer) hotspots.
• #26: A web service interface is under development that will make all the functionality of the web portal available in a programmatic fashion. Obviously it's not feasible to deploy thousands of files manually via the web portal so we need to prove the facility for advanced customers to scale out easily and rapidly.
• #27: *With multiple sources (and multiple URL’s) the complexity of load balancing is imposed on the origin / content provider *With single namespace we can have coarse and fine-grained control via DNS redirection and layer4/7 load balancing http://www.metacdn.org:8080/MetaCDN/FileMapper?itemid=1 http://www.metacdn.org:8080/MetaCDN/FileMapper?itemid=1&policy=RAN http://www.metacdn.org:8080/MetaCDN/FileMapper?itemid=1&policy=GEO
• #29: During the development phase only, only 1 copy of portal/redirector is running in Melbourne, Australia. The plan is to deploy portals in several locations across US, Asia and Europe. We will see from next slide why this is necessary.
• #30: Let's assume that a consumer in the USA was accessing a replica directly (i.e. it magically knew the best replica to select), or via MetaCDN. Here we can see that there is around 0.4 seconds of overhead, which is predominantly the round-trip time to access the gateway in Australia.
• #31: When a consumer has a gateway that is close to it (in this case, a consumer in Australia is utilising a local gateway) the overhead is significantly smaller, in the order of 0.05 seconds per request. It is obvious that local gateways are needed in key areas to maximise performance.
• #32: In the second half of 2008 we evaluated the two major cloud storage providers at the time, Amazon S3 and Nirvanix SDN. We ran the test over 24 hours from a variety of client and replica locations to see whether the providers demonstrated sufficient performance (i.e. throughput and response times) to act as a \"poor man's\" CDN.
• #33: In 5 out of 6 client locations there were at least 2 replicas each that delivered throughput that is consistent with what we would expect from a traditional CDN service. <Kilobytes per second>
• #34: In 4 our of 6 client locations there was 1-3 replicas that delivered response time that is consistent with what we would expect from a traditional CDN service. It is worth noting here that these times represent the end to end latency and connection time (i.e. a HTTP connection is made), they are not simply ping measurements.
• #36: FTP/WebDav support will be useful in locations where cloud providers do not (and are unlikely to) service. There is a lot of demand from customers to move away from Youtube and Vimeo flash video hosts and host their own streaming content directly on their origin site. This way they control the look and feel and ad monetization of their content.