DOES SFO 2016 - Avan Mathur - Planning for Huge Scale
Download as PPTX, PDF0 likes635 views
The document discusses strategies for designing scalable and resilient pipelines for large-scale applications, emphasizing the importance of planning for long-term scale and establishing success criteria. Key focus areas include infrastructure, data management, pipeline design, and monitoring, with recommendations for architecture and disaster recovery. Effective monitoring of metrics such as CI cycle times and deployment frequency is crucial for anticipating system demands and ensuring smooth operations.
DOES SFO 2016 - Avan Mathur - Planning for Huge Scale
- 1. electric-cloud.com #DOES16 Planning For Huge Scale: Designing Your Pipelines for Scalability and Resiliency Avan Mathur | Product Manager amathur@electric-cloud.com | @avantika_ec
- 2. electric-cloud.com #DOES16 What do I Mean by Huge Scale? • A Fortune 5 Bank • Deploying to 140K endpoints • 6000s of applications on-boarded at accelerated pace • Chip Manufacturer • 100K Jobs/Day • Constant load on the system • High job throughput
- 3. electric-cloud.com #DOES16 Plan Ahead – Know Your End Goal • Estimate and plan for long term scale • Define your success criteria • # of deployments per window • Max duration of a deployment • Number of concurrent deployments • Benchmarking • Validate numbers with benchmarks • Testing real workloads in real environments
- 4. electric-cloud.com #DOES16 Metrics to Measure • CI Cycle Times • Deployment Frequency and Duration • Concurrent Deployments • Release Frequency
- 5. electric-cloud.com #DOES16 Focus Areas • Infrastructure • Data • Pipeline Design • Monitoring / feedback loop
- 6. electric-cloud.com #DOES16 Infrastructure • High Availability • Horizontal Scalability • Distribute load and components • Database sizing and reliability • Understand and plan for bottlenecks and hardware limitations
- 7. electric-cloud.com #DOES16 Recommended Architecture Commander Server Commander ServerClustered Orchestration Servers Web Server Database AgentAgentAgentAgentAgent Workspace Repository 1 Browser AgentAgentAgentAgentAgent Web Server Site 2 WAN Workspace Repository 2 Browser Web Server Site 3 Repository 3 Site 1 Browser AgentAgentAgentAgentAgent Workspace HA/ HS
- 8. electric-cloud.com #DOES16 Disaster Recovery Architecture Commander Server Commander ServerClustered Orchestration Servers Web Server Database AgentAgentAgentAgentAgent Workspace Repository 1 DR Site HA/ HS Commander Server Commander ServerClustered Orchestration Servers Web Server Database AgentAgentAgentAgentAgent Workspace Repository 1 Primary Site HA/ HS Disk Replication WAN DB Mirroring
- 9. electric-cloud.com #DOES16 Managing Data • Each process generates data Metadata, log files, build & test data, monitors, artifacts … • Understand growth rates and plan accordingly Different areas to monitor o Database o Filesystem o Artifact Repository • Establish Data Retention & Archiving Policies early
- 10. electric-cloud.com #DOES16 Pipeline Design: Scaling Across an Organization 1. Plan and monitor for performance at scale at every step 2. Build a generic model to standardize across apps 3. Reusable components that can be used across projects 4. Automated onboarding of new projects 5. Plan and Define Access Controls and Roles early in the process
- 11. electric-cloud.com #DOES16 Monitoring • Monitor key metrics Understand patterns Foresee upcoming peaks in usage Plan ahead for hardware expansion or distribution • Best Practices Unobtrusive Automated Not only in production One pane of glass
Editor's Notes
- #5: Ci Cycle time – more explanation – not only build time but including test that needs tobe run
- #8: Say that it is not ElectricFlow specific Give it more example – site1 is NA, site2 is ASIA, site3 is AWS Artifact Repositoru – put a replication policy in place – only replicate what is required for that site Distaster Recovery
- #9: Say that it is not ElectricFlow specific Give it more example – site1 is NA, site2 is ASIA, site3 is AWS Artifact Repositoru – put a replication policy in place – only replicate what is required for that site Distaster Recovery
- #11: #1 EXAMPLE – for example, instead of making a separate model for each and every dployment. Identify the processes and components that are being used across apps, and #2 EXAMPLE – what are the common pieces that can be reused #5 -- needs concrete example. Transition to next slide. How would you design a taxing processdifferently for better throughput
- #12: Because its is mission critical – driving your enditre software delivery pipelines. Single dashboard centralized for monitoring all of the activities across all of the tools and activities. Shared data being read from different tools and processes in one place. One server and one agent is easy to monitor – once you scale to 1000s of deployments, oyu need one central place where you can see all activity Monitors can feed into delivery platform to react to events
- #13: HP – who is also talking at the conference - feeds monitoring data back into ChatOps to alert people or perform automated actions WRAP UP – the system is mission critical to many of our customers – if it goes Questions slides at the end. Add community edition link at the end.