Plenary Talk at ACAT 2010
3 likes1,270 views
This document discusses using Amazon Web Services for scientific computing. It describes how AWS provides scalable, cost-effective, and reliable cloud infrastructure that researchers can use on-demand. Specific AWS services that can benefit scientific applications are mentioned, including Amazon EC2 for flexible computing power, S3 for data storage, and Elastic MapReduce for hosted Hadoop services. Examples are given of scientific projects that have successfully used AWS, such as genome analysis and astrophysics simulations. The document argues that AWS enables new approaches to doing science by providing access to vast, on-demand computing resources and platforms for distributed applications and data sharing.
Plenary Talk at ACAT 2010
- 1. Scien&fic Compu&ng with Amazon Web Services Deepak Singh ACAT 2010. Jaipur, India
- 3. Via Reavel under a CC-BY-NC-ND license
- 9. By ~Prescott under a CC-BY-NC license
- 13. data
- 14. Image: Wikipedia
- 15. Image: Matt Wood
- 18. couldn’t find a good picture for arrays of sensors in the ocean
- 19. Image via image editor under a CC-‐BY License
- 20. years
- 21. weeks
- 22. days
- 23. days
- 24. mi nu tes days ?
- 25. gigabytes
- 26. terabytes
- 27. petabytes
- 28. petabytes
- 29. ex ab y tes petabytes ?
- 33. data management
- 34. data processing
- 35. data sharing
- 37. the cloud
- 39. infrastructure as a service
- 41. Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling
- 42. Payments On-Demand Parallel Processing Messaging Content Delivery Amazon Flexible Workforce Amazon Elastic Amazon Simple Amazon CloudFront Payments Service Amazon Mechanical MapReduce Queue Service (SQS) (FPS) Turk Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling
- 43. Tools Isolated Networks Monitoring Management AWS Toolkit for Eclipse Amazon Virtual Private Amazon CloudWatch AWS Management Console AWS Toolkit for .NET Cloud Payments On-Demand Parallel Processing Messaging Content Delivery Amazon Flexible Workforce Amazon Elastic Amazon Simple Amazon CloudFront Payments Service Amazon Mechanical MapReduce Queue Service (SQS) (FPS) Turk Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling
- 44. Your Custom Applications and Services Tools Isolated Networks Monitoring Management AWS Toolkit for Eclipse Amazon Virtual Private Amazon CloudWatch AWS Management Console AWS Toolkit for .NET Cloud Payments On-Demand Parallel Processing Messaging Content Delivery Amazon Flexible Workforce Amazon Elastic Amazon Simple Amazon CloudFront Payments Service Amazon Mechanical MapReduce Queue Service (SQS) (FPS) Turk Compute Storage Amazon Elastic Compute Database Amazon Simple Amazon RDS and Cloud (EC2) Storage Service (S3) - Elastic Load Balancing SimpleDB - AWS Import/Export - Auto Scaling
- 45. scalable
- 47. go o u s y scalable ay a P cost effective
- 48. scalable cost effective reliable
- 49. scalable cost effective reliable secure
- 50. Amazon EC2
- 52. highly scalable
- 54. elastic
- 55. 3000 CPU’s for one firm’s risk management application 3444JJ' !"#$%&'()'*+,'-./01.2%/' 344'+567/'(.' 8%%9%.:/' 344'JJ' I%:.%/:1=' ;<"&/:1=' A&B:1=' C10"&:1=' C".:1=' E(.:1=' ;"%/:1=' >?,,?,44@' >?,3?,44@' >?,>?,44@' >?,H?,44@' >?,D?,44@' >?,F?,44@' >?,G?,44@'
- 58. “Everything fails, all the time” -- Werner Vogels
- 59. 2.3% AFR in population of 13,250 3.3% AFR in population of 22,400 4.2% AFR in population of 246,000 Source: James Hamilton
- 61. design apps to be resilient
- 63. nothing fails
- 65. elastic IP
- 66. SQS
- 67. US East Region Availability Availability Zone A Zone B Availability Availability Zone C Zone D
- 68. on-demand instances reserved instances spot instances
- 69. data storage
- 70. one size does not fit all
- 72. Amazon S3
- 74. durable
- 75. available
- 76. !"#$%&'()*+ T T T
- 77. scalable
- 78. fast
- 79. simple
- 82. Amazon SimpleDB
- 84. highly available
- 85. schema less
- 86. key-value store
- 87. Amazon Relational Data Service
- 88. single API call
- 89. MySQL database
- 90. automatic backup
- 91. scale up with API call
- 94. hadoop easy and simple
- 95. Amazon Elastic MapReduce Amazon EC2 Instances End Deploy Application Hadoop Hadoop Hadoop Elastic Elastic MapReduce MapReduce Hadoop Hadoop Hadoop Notify Web Console, Command line tools Input output dataset results Input S3 Output S3 Get Results Input Data bucket bucket Amazon S3
- 100. computing platforms
- 104. sudo gem install cloud-crowd http://cyclecomputing.com http://wiki.github.com/documentcloud/cloud-crowd
- 106. Amazon Elastic MapReduce Amazon EC2 Instances End Deploy Application Hadoop Hadoop Hadoop Elastic Elastic MapReduce MapReduce Hadoop Hadoop Hadoop Notify Web Console, Command line tools Input output dataset results Input S3 Output S3 Get Results Input Data bucket bucket Amazon S3
- 113. data distribution
- 116. problem solving
- 118. 3.7 million classifications in just over three days ~15 million in less than a month >2.6 million clicks in 100 hours
- 121. Crossbow: Rapid whole genome SNP analysis Preprocessed reads Map: Bowtie Sort: Bin and partition Reduce: SoapSNP Langmead B, Trapnell C, Pop M, Salzberg SL. Genome Biol 10 (3): R25.
- 122. Crossbow condenses over 1,000 hours of resequencing computa:on into a few hours without requiring the user to own or operate a computer cluster
- 123. doing science
- 125. BLAT @ U. PENN Map 100 million, 100 base paired end reads Quad core with 5 GB of RAM would take 16 days 30 high-memory instances; 32 hours; $195
- 126. GALAXY MAPPING Goal: Create an astrometric catalog of a billion stars with micro arc second precision Gaia satellite launched 2011; observations till 2017; catalog ready 2019 Problem: Single pass through the data for image processing would take 30 years (on one CPU) Solution: Use AWS
- 127. Capacity Capacity Resources Resources Demand Demand Time Time Static data center Data center in the cloud Unused resources
- 128. HEAVY-ION COLLISIONS Problem: Quark matter physics conference imminent but no compute resources handy Solution: NIMBUS context broker allowed researchers to provision 300 nodes and get the simulations done
- 129. BELLE MONTE CARLO Credit: Tom Fifield
- 130. AWS for the sciences
- 131. available resources
- 132. task-based resources
- 133. shared dataspaces
- 137. the cloud works
- 138. today
- 140. Thank you! deesingh@amazon.com Twi<er:@mndoci Presenta?on ideas from James Hamilton and @lessig