Harness the Power of the Cloud for Grid Computing and Batch Processing Applications
Download as PPTX, PDF0 likes292 views
This document summarizes a presentation about harnessing the power of cloud computing for grid computing. It discusses how RightScale provides automated management of grid computing workloads in the cloud, allowing users to easily deploy and control large numbers of servers. Demos show how RightScale enables graceful scaling of server arrays, automated queue handling, and analyzing results to quantify economic benefits like cost savings and increased agility compared to on-premise grid solutions.
Harness the Power of the Cloud for Grid Computing and Batch Processing Applications
- 1. Harness the Power of theCloud for Grid ComputingJune 29, 2010
- 2. Your Panel TodayPresenting:Michael Crandell, CEO
- 3. Josh Fraser, Business Development
- 4. Dave Welch, Sales EngineerQ&A:Matthew Small, Account ManagerPlease use the questions window to ask questions anytime!
- 6. RightScale Customer Use Cases
- 8. Grid Computing in the cloud with RightScale
- 9. Demo #1
- 11. Demo #2
- 16. Scaling events:- 50-3500 servers 3 days- 8000 servers 1 co.- 0 to 35M users/day
- 17. Largest production deployments in the cloud to dateRightScale Cloud ManagementEasily Deploy and Control
- 18. Customers on the CloudCustomers from Web 2.0 to EnterpriseFrom to RightScale Grid Solution PackGrid WorkerGrid WorkerGrid WorkerGrid WorkerGrid WorkerGrid WorkerGrid Worker
- 19. Cloud-based Grid Computing Use CasesPharmaceutical Analysis – Researchers expected a protein analysis comparing 2.5 million compounds to take a week of processing on internal servers • Using hundreds of servers, the job was completed in one dayInsurance Claims Loss Control – Systems for detecting fraudulent or duplicate claims in batches of millions would have required months of processing time to run and millions of dollars to build in the data center • Batch runs finished in a few days at significantly lower cost Web 2.0 – Transcoding images to render video on demand • Processing time was reduced from hours on internal resources to minutesFinancial Data Processing –Back testing environments that analyze data to test new trading strategies • Trading strategies analyzed faster and more cost-effectively by scaling out servers
- 20. Grid Computing & Batch Processing ChallengesFor UsersFor OperationsFor ITHigh capital investment, typically low capacity utilization
- 21. Scheduling conflicts, constant provisioning
- 22. Specialized architectural and operational skills
- 23. Specialized software applications and unique datasetsFor End UsersWait for resources
- 24. Limited to resources available in the datacenter Grid Computing in the CloudGrid Computing in the CloudCloud Computing ModelResolves Scheduling IssuesResources on DemandVirtually Infinite ResourcesSupports Faster ProcessingMatches Costs to Demand Pay as You Go
- 25. Grid Computing managed by RightScaleGrid ComputingManaged by RightScaleRightScaleFast and easy provisioningPreconfigured components and solutionsAutomation and systems managementReduced administrationControl and visibilityUser and cost control
- 26. Grid Computing Solution Pack“Lilly is leveraging RightScale's infrastructure management interface and services against appliance/application stacks in a ‘vending machine’ concept that allows self-service to infrastructure.” - Dave Powers, Assoc. Information Consultant, Eli Lilly & Co.Top 25 Information Managers 2010Preconfigured Framework for Grid ComputingProven, Best Practices Architecture on Amazon Web ServicesLeverages AWS EC2, SQS, and S3
- 27. Multiple AWS Regions and Availability ZonesRightScale Professional Services and Support
- 28. HighlightsComplements existing internal grid engines (Sun Grid, etc.)TBDTBD
- 29. RightScale Grid ArchitectureAutomated server scaling, operational remediation, server cost optimization RightScale Management InterfaceSQS Output Queue SQS Input Queue SQS Error Queue Your codeAmazon S3Worker DaemonAmazon S3Your application or next batch process job consumerBatch jobs from Your job producer applicationScalable cloud servers using RightScale Server TemplatesRightScaleAmazon Cloud InfrastructureCustomer code
- 30. Job CoordinatorInput file Creation and message QueuingS3 BucketsWork Queue INOUTLOGJob Producer creates input file(s) Job Producer constructs a work UnitInput files are placed in an S3 bucketWork_unit is packed into a messageMessage is placed in the SQS work queue.Job Producer CodeInputFileEncoder(Pack Message){Work_Unit }{MSG}LEGEND{}Data Structure Easily schedule with your current scheduler (Tivoli, Condor, other?)Daemon / Worker CodeWork Queue and Work _Unit ProcessingS3 BucketsWorker daemon{Work_Unit }Work Queue INOUTLOGRightScale instantiates serverWorker Daemon copies a messageUnpacks the msg and downloads filesWorker code is called with both the msg_env and original unpacked messageWorker does workOutput files placed in the /out directoryLogs are createdResults returned back to the Daemon/localWorkercode/tmp/in/out/log}{ResultLEGENDAWS Instance{}{}}{Data StructureMSGEnvMSG
- 31. Daemon Results HandlingS3 BucketsElasticity daemonINOUTLOGUpload Output FilesUpload log filesDelete /tmp and /in filesIf worker returns successQueue Results in results_queuePost results to http serverElse queue results to the error queueQueue statistics in Audit QueueDelete Message from input queue/localWorkercode/tmp/in/out/logResults Queue {Results} {Results} Consumer}{ResultError Queue {Results} {Results}ConsumerLEGEND{}Audit Queue {Audit} {Audit}ConsumerData StructureAWS Instance
- 32. Demo #1Pre-configured Grid Architecture
- 34. Graceful Provisioning and De-provisioning
- 35. Automated Queue HandlingAutomated ManagementServer Array ScalingALERTS or ScheduleScale up/down based on monitoring system metricsScale up/down based on set schedule – TOD, DOWQUEUES STATISTICS# of backlogged msgs per instanceTime msgs have stayed in queueAvg. of the last 10 msgsMax length of the last 10 msgsIf idle after 55 mins from launchYour grid processing codeWorker Daemon
- 36. Automated Management Key Benefits
- 37. Demo #2Configuring array automation
- 38. Analyze your results Key Economic BenefitsCost SavingsAgilityNo CAPEXYour job is always first in lineUnlimited resourcesPay only for what is usedPre-configured, proven architecturesSaaS-based management model
- 39. Amazon’s Public Data SetsCentralized repository of public data sets AWS is hosting the public data sets at no charge for the community, users pay only for the compute and storage they use for their own applications.Examples:Economics data, e.g. the 2000 US Census
- 40. Life Sciences data, e.gGenBankhttp://aws.amazon.com/publicdatasets/
- 41. Agility Example: Cost-neutral EquationTotal processing time: 500 hours10,000jobsOutput data2 server cloudTotal processing time: 1 hour10,000jobsOutput data1000 server cloudThis graphic compares running the same 10,000 jobs on 2 servers versus 1000 servers. The cost is the same for either scenario in RightScale using AWS, but the difference in elapsed time is 499 hours.(assuming each server can process 10 jobs/hour)
- 42. Example Cost Savings –1/4 The PriceIn-house Grid – with commercial grid softwareCloud-based Grid Computing
- 43. Q&A - Getting StartedContact us:sales@rightscale.com
- 44. (866) 720-0208
- 45. Twitter: @rightscaleGrid Computing Whitepaper: www.rightscale.com/gridsolutionWebinar Archive: www.rightscale.com/webinars
- 46. Thank You!