Embedded CDNs in 2023
0 likes214 views
- Embedded CDNs involve placing CDN servers within ISP networks to serve local end-users. This can improve performance by locating content closer to users. - Many major CDNs and content providers now offer embedded servers, including Akamai, Netflix, Google, Amazon, and Facebook. Traffic from embedded servers accounts for around 14% of total CDN traffic based on one study. - Benefits for ISPs include offloading external traffic and improving performance for end-users. CDNs use techniques like BGP, DNS, and geolocation to map users to the closest embedded server.
Embedded CDNs in 2023
- 1. Embedded CDNs in 2023 ﹣A status Jack Fang June 2023
- 2. What is a CDN? Embedded servers are when some of the CDNs servers are located within ISPs networks and intended to only serve end-users in that network. The servers will use address space originated in the ISPs network and not in the CDNs ASN(s). Clustering is when a number of servers share their storage. This means that the content is distributed over the servers in the cluster and will not be present at all of them. Sometimes servers in different sites can be clustered together. A collections of geographically distributed servers A method to place content on the servers A method to steer end- users to the closest servers
- 3. A bit about the CDN market Growth in the market based on the increased consumer demand for entertainment and OTT services and in the later years on increased demand for e-learning and work from home solutions. The traditional players like Akamai, Lumen, Tata, Edgio etc are well established as reliable and with global reach. The newer major CDNs like Cloudflare, Fastly and Stackpath are offering a more diverse and specialised set of services like security services and DDOS protection, high performance real time delivery services etc. Which btw is also offered by Akamai Large content producers who use comercial CDNs often use multiple CDNs. This both to support their different types of content, to leverage different geographical scope of the CDN, create resilience and finally - more than one provider means better leverage when negotiating contracts…
- 4. A bit about the CDN market The cloud providers allmost all have a CDN to front their content The CDNs are almost all also offering compute and other cloud services So in the end they end up offering the same services but with different specialities and strengths The giant content producers like Netflix, Apple, Microsoft, Amazon, Facebook and Google have all build their own specialised CDN to support their core services ( and some also sell CDN services…) Building a CDNs on a global scale is not small task, so this solution is only for the largest and most resourceful content producers with enough demand for traffic to make the business case work.
- 5. Who offers embedded servers? ● Akamai ● Netflix ● Google ● Amazon ● Facebook ● Cloudflare ● CDN77 ● Microsoft ● Apple ● Qwilt ● And more to come
- 6. So, what is in it for the ISP?
- 7. Statistics • Petros Gigis, Lefteris Mannasakis and more presented at Sigcomm’21 that over 3,500 ASNs globally have embedded servers • Most ISPs in the Kentik traffic sample have 6 or less embedded CDNs • The top 6 in the Kentik sample are: Percentages of the ISPs in the sample that have embedded servers from the CDN Google (25.5%) Facebook (16.2%) Netflix (26.5%) Akamai (11.6%) Apple (3.3%) CDN77 (3.3%)
- 8. Total distribution of traffic marked as CDN traffic ~74% over the external network border ~14% from embedded servers ~7% embedded to embedded internally in the networks ~5% Externally to embedded Average traffic over 24 hours that includes a sunday evening.
- 9. Statistics ● Outside to end-users (39%) ● Embedded to end-users (56%) ● Outside to embedded (5%) ● Embedded to embedded (1%) Google Netflix Facebook Akamai ● Outside to end-users (22%) ● Embedded to end-users (72%) ● Outside to embedded (3%) ● Embedded to embedded (3%) ● Outside to end-users (43%) ● Embedded to end-users (54%) ● Outside to embedded (4%) ● Embedded to embedded (0%) ● Outside to end-users (73%) ● Embedded to end-users (18%) ● Outside to embedded (9%) ● Embedded to embedded (0%)
- 10. Steer to the closest cache ● BGP is used to signal the subnets that a cache or cluster should serve ● The most control is given for the systems where every cache has a BGP session and the steering system respects common BGP parameters. ● A setup where all Servers have an identical session will work for most deployments. ● Prefixes are sent from the ISP to the cache BGP ● End-users are mapped to a cluster based on the DNS server they are using to request the content DNS Server ● Geolocation is used by some as the primary way if mapping an end-user to a cluster. Some use the geolocation of the end-user IP address, but some rely on the geo location of the DNS server the end-user is using to request the content. Geolocation Anycast ● Anycast addresses are announced from the caches and the ISPs routing decides which is the closest
- 11. BGP x.y.z.0/24 x.y.w.0/24 x.y.w.123 B 1 2 3 4 A ● x.y.z.0/24 is announced to A ● x.y.w.0/24 is announced to B 1. Give me movie 2. Go to B and get movie 3. Give me movie 4. Movie This works really well for ISPs who have a very strict regional address plan. It does not work for ISPs where they do not have control over IP address assignments or who do not have regional aggregation
- 12. 1. Where is site.com? 2. Where is site.com? 3. site.com is site.com.cdnsomething.com 4. Where is site.com.cdnsomething.com 5. site.com.cdnsomething.com is A for you 6. Site.com is A 7. Give me Site.com 8. Site.com DNS User in Mumbai Local DNS Authorized DNS CDN node in Mumbai IP address “A” CDN CDN node in Singapore IP address “B” 2 3 1 6 7 8 4 5
- 13. And then some of the magic ● The CDN creates a mapping of ISP DNS servers to POPs or clusters based on latency measurements ● Some CDNs include internet connectivity from the cluster to the DNS servers or IP addresses in their mapping algorithm ● The content owned CDNs also include signals from the content clients in the algorithm Latency and other QoS data ● The load of the individual servers or clusters is part of the decision of exactly which server the end- user will end up using. ● Load balancing can take place in the cluster or in the CDN response to (where is?) Load ● Not all content is on all Embedded servers ● Content that is pushed out is distributed according to CDN magic ● Not all content from a CDN will be cached when the content is pulled ● This means where the content is/can be is also part of the steering decision. Content
- 14. Traffic profiles for different ways of placing content • Centrally calculated placement • Fill window • Optimized to reduce edge fill traffic as much as possible Typical Netflix traffic profile in well dimensioned deployment
- 15. Traffic profiles for different ways of placing content • Proxy and akamai magic • Relatively dynamic content • Not all content will be at the embedded servers Typical Akamai traffic profile in well dimensioned deployment
- 16. Traffic profiles for different ways of placing content • Proxy and Facebook magic • Relatively dynamic content • Only the static is cached • Not all content will ever be at the embedded servers Typical Facebook traffic profile in well dimensioned deployment Feeds Likes Anything that needs to be globally consistent
- 17. Limit who can use the Servers Most embedded Servers will have access list that limits who will be served by the cache The common source to generate these is a BGP feed from the host ISP Some will use one single feed per ISP, other per cluster or per cache. There must be consistency between the user- group that can be mapped to a cluster or server and the user-group that are allowed to use the cluster or server. This means there is a requirement for the CDNs where each server has a BGP sessions that the announcements to each server in a cluster is identical.
- 18. Deploying embedded servers deep into the network • Understand if the end-user mapping is supported by your network implementation • DNS servers • IP address plan • Use of CGN • Be aware of clustering • Off load/hit rate for the local cluster might be lower than for a big cluster because of less disk space in the cluster
- 19. And what about the hardware? 1-2U 10G /50G/100G Interfaces Choice of AC and DC power It varies whether a CDN managed switch is required optional or not applicable. Installation process are more/or less automated, but some still require the ISP tech to do basic configuration of the CDN equipment. Evolves continuously: less space, more throughput, more and faster storage Interfaces supports the industry standard at any given time
- 20. Is it easier or harder in 2023 compared to 2012? ● Large boxes ● Off load unpredictable ● Few players ● No CGN support 2012 ● Small boxes ● More predictable offload ● More predictable and flexible end-user mapping ● Many players ● Some CGN support 2023 CONCLUSION
- 21. What do you experience?
- 22. THANK YOU Jack Fang Solutions Engineer - APAC jfang@kentik.com