Demonstrating 100 Gbps in and out of the public Clouds
0 likes75 views
The document discusses the performance of achieving 100 Gbps data throughput in and out of major public clouds, highlighting the significance of networking in distributed computing. It notes that while compute capabilities are well-documented, networking efficiency remains poorly understood and can greatly impact performance. The study, supported by various NSF grants, demonstrates practical applications of high-speed data movement across cloud providers like AWS, Azure, and Google Cloud, and addresses associated costs of data transfer.
Demonstrating 100 Gbps in and out of the public Clouds
- 1. Demonstrating 100 Gbps in and out of the public Clouds Igor Sfiligoi - University of California San Diego - isfiligoi@sdsc.edu Most of the Cloud costs were covered by credits issued by Amazon, Microsoft and Google. This work was partially funded by US National Science Foundation (NSF) under grants OAC-1941481, MPS-1148698, OAC-1841530, OPP-1600823, OAC-19044, and OAC-1826967. Like it or not, public Clouds are gaining popularity • Especially great for prototyping • But great for compute spikes and urgent compute activities, too • When choosing which resources to use (and pay for) knowing what to expect out of them is important • The compute part of public Cloud resources is well documented • The networking is instead hardly documented at all • Yet networking can make a big difference when doing distributed computing Funding agencies are taking notice • See NSF E-CAS award(s) as an example • SDSC Expanse has Cloud explicitly mentioned • NSF-funded CloudBank • IceCube’s Cloud runs funded by an EAGER award 0 200 400 600 800 1000 1200 1400 1600 1800 2000 0 200 400 600 800 1000 1200 1400 Gbps Number of compute instances AWS Azure GCP AWS Azure GCP Asia – US West 100 Gbps 110 Gbps 940 Gbps US East – West 440 Gbps 190 Gbps 1060 Gbps EU – US East 460 Gbps 185 Gbps 980 Gbps Peak Throughput observed between regions of a single public Cloud provider Peak Throughput observed in a single public Cloud provider region Networking inside public Cloud providers plentiful • Over 1 Tbps in a single region • 100 Gbps to 1 Tbps between regions Importing data into public Clouds to feed compute instances • Using multiplee parallel clients • 100 Gbps doable on production infrastructure • Demonstrated with both storage in the Pacific Research Platform and at University of Wisconsin – Madison ThroughputinBytespersecond ThroughputinBytespersecond Fetching data from public Clouds • Fetching from their Object Storage using multiple parallel clients • 100 Gbps doable on production infrastructure (PRP using 20 nodes) • Over 500 Gbps feeding Cloud instances running in multiple providers AWS Azure GCP US West 100 Gbps 120 Gbps 110 Gbps US Central 100 Gbps 120 Gbps 110 Gbps ThroughputinBytespersecond Networking is a billed commodity •$50/TB – $120/TB egress • $20/TB - $90/TB data movement between Cloud regions • Ingress free Fetching data from Cloud storage Charges on an account used for benchmarking Compute and ingress 15% network waiver for compute intensive workloads