Lessons from an AWS outage and how to detect root cause of cloud service disruptions
1 like485 views
The document analyzes a recent AWS outage caused by compromised BGP prefixes, resulting in services being down for about two hours and significant collateral damage. It outlines key takeaways for cloud outages, emphasizing the importance of updated monitoring practices and recognizing external dependencies while adapting to a cloud readiness lifecycle. Additionally, it discusses best practices for managing cloud-centric IT operations and tools for enhancing visibility and accountability across cloud services.
Lessons from an AWS outage and how to detect root cause of cloud service disruptions
- 2. @ThousandEyes • Analysis and demo of a recent AWS outage • 5 key takeaways from cloud outages • Monitoring best practices for the cloud • Q&A Agenda Archana Kesavan Director of Product Marketing @archana_k7
- 3. @ThousandEyes About ThousandEyes Network Intelligence solution that helps you understand performance from every user to every application over any network User App Enterprise, Endpoint and Cloud Agents Routing End-to-End Performance Data User Experience App Performance Routing Topology Network Topology Network Connectivity Device Performance 20/25 top SaaS companies 6/7 top US banks 65+ Fortune 500 companies 8/10 top global software companies
- 4. @ThousandEyes Software and Services Financial Services Retail and eCommerce Healthcare Media and Entertainment Education Communications Cloud Infrastructure Gaming Relied on by Industry Leaders
- 5. @ThousandEyes What’s a Cloud Outage? DDoS Attack Hijack & Leaks DNS Outage Cable Cut Public Cloud
- 6. @ThousandEyes • AWS Route 53 service suffered an outage • Immediate cause was wrong BGP prefixes propagated through a handful of ISPs • Services affected for ~2 hours • Millions in collateral damage (Instagram, CNN, …) A Story of a BGP and DNS Exploit
- 7. @ThousandEyes How Did They Do It? Attackers compromised XLHost (eNET) – a hosting provider in Ohio Handful of ISPs propagate false BGP Route DNS entries for popular services hijacked and redirected to a malicious server Services compromised for almost 2 hours Hijacked Amazon’s Route 53 DNS service using false (more specific) BGP announcements
- 8. @ThousandEyes The Real Target: Cryptocurrency Heist X
- 10. @ThousandEyes DNS Servers Become Unavailable Availability map indicates problem areas DNS Servers unavailable for ~ 2 hours
- 11. @ThousandEyes DNS Traffic Blackholed at an Unknown ISP
- 12. @ThousandEyes BGP Prefixes Hijacked and Propagated • Unknown ISP announces a more specific BGP prefix • Two ISPs propagate the prefix
- 13. @ThousandEyes 5 Key Takeaways From Cloud Outages
- 14. @ThousandEyes 1. External Dependencies: To Infinity and Beyond! 2. Traditional Monitoring Flatlines in the Cloud 3. Find and Fix Evidence and Escalate 4. Methodology Shift: See What You Don’t Own 5. Mindset Shift: Embrace a Cloud Readiness Lifecycle
- 15. @ThousandEyes It Used to Be that You Controlled Everything On-premises, IT-controlled monolithic Branch Office Branch Office Branch Office Branch Office Data Center CRM ERP Flow PCAP SNMP Productivity
- 16. @ThousandEyes External Dependencies: To Infinity and Beyond! CRM ERP PCAP SNMP Productivity Flow Branch Office Branch Office Branch Office Branch Office Data Center CRM ERP Flow PCAP SNMP Productivity Customer SaaS IaaS CDN DNS DNS Provider Security Provider API IaaS Data Center Cloud opportunity comes with vast new performance, security and operations risks that most businesses are unprepared for
- 17. @ThousandEyes Traditional Monitoring Flatlines in the Cloud FlowPCAPSNMP
- 18. @ThousandEyes Pre-Cloud Monitoring Stack Monitoring Category IT Management Domain Presentation Layer All IT Service Management All APM Internally Developed Apps & Services IT Infrastructure Mgmt (SNMP) Internal Data Center & WAN infrastructure Network Perf Mgmt & Diagnostics (Flow/PCAP) Network Capacity Mgmt (Flow) Log Mgmt All
- 19. @ThousandEyes Cloud-Sized Hole In Your Monitoring Stack Monitoring Category IT Management Domain Presentation Layer All IT Service Management All APM Internally Developed Apps & Services IT Infrastructure Mgmt (SNMP) Internal Data Center & WAN infrastructure Network Perf Mgmt & Diagnostics (Flow/PCAP) Network Capacity Mgmt (Flow) Log Mgmt All Internal Data Center & WAN infrastructure Cloud-Specific Mgmt (CloudWatch) Cloud Infrastructure Missing Visibility Digital Experience ISP, DNS, CDN, DDoS. CASB providers SaaS providers SD-WAN Internet transport Internet Routing IT Infrastructure Mgmt (SNMP) Network Perf Mgmt & Diagnostics (Flow/PCAP) Network Capacity Mgmt (Flow)
- 20. @ThousandEyes Cloud Native Tools: Helpful But Not Enough Amazon CloudWatchAmazon CloudTrail Best for security analysis, resource change tracking, compliance auditing Best for gaining visibility into resource utilization, app performance and operational health of your AWS service Azure Network Watcher Best for packet level analysis and flow monitoring between VMs
- 21. @ThousandEyes 92% of NetOps Teams See Challenges with Native Cloud Monitoring Services 28% 24% 24% 23% 18% 18% 16% 14% 8% Resiliency - monitoring service goes down when cloud provider goes down No end-to-end view across provider's regions/availability zones Internal network monitoring tools cannot effectively collect/integrate data from these services Lack of industry standards for data/metrics Relevance - metrics do not offer enough networking- related insights Business value - too expensive for the visibility it provides No end-to-end view across multiple cloud providers Granularity - monitoring intervals are too long None - we perceive no weaknesses
- 22. @ThousandEyes Operations Processes Change in the Cloud IT Assets You Don’t Control Evidence Escalate?? IT Assets You Control Find Fix
- 23. @ThousandEyes 3 Boston, MA Dallas, TX Hong Kong 182.50.78.169 182.50.78.41 San Francisco, CA Methodology Shift: See What You Don’t Own
- 24. @ThousandEyes Evidence: Keep Your Cloud Providers Accountable Your Network ISP Cloud Provider SnapshotsDashboards / ReportsAlerts
- 25. @ThousandEyes Mindset Shift: Embrace a Cloud Readiness Lifecycle Baseline performance and define success KPIs READY Improve MTTR by 90% prevent service desk costs from spiking OPERATE Improve cloud, SaaS and SD-WAN project success rates DEPLOY
- 26. • Industry’s first vendor- neutral, data-based research • Empowering data-driven decisions in the cloud PUBLIC CLOUDPERFORMANCEBENCHMARKREPORT 2018 Edition
- 27. @ThousandEyes Best Practices for A Cloud-Centric ITOps Team 1Update your cloud monitoring stack 2 Establish a common monitoring platform across teams 4Integrate with and Automate 3 Revamp your operations process
- 28. @ThousandEyes Understand Experience for Any User and App Lightweight software-based agents easily installed on your own network, in data centers, branch offices & VPCs. Enterprise Agent Endpoint Agent Browser-based plugins installed on end-user laptops and desktops. End User ExperienceInternal Vantage Points Cloud Agent Globally distributed agents installed and managed by ThousandEyes in 175+ POPs around the world. External Vantage Points
- 29. @ThousandEyes Turning Data into Actionable Intelligence Capabilities Real-time Visualizations Timeline, Path Viz, Route Viz Trending and Aggregation Reports Dashboards Anomaly Detection Alerts, Notifications Eco-System Integration Native REST API, Integrations Algorithmic Layer Global Inference Engine Cross-Correlation and Collective Intelligence Data Sources Application Layer Tests HTTP server, Page Load, Transaction, FTP server, DNS, RTP, SIP End User Experience Endpoint Agent Tests BGP Routing Global BGP RIB Feeds Network Layer Tests L3 Paths with hop-by-hop metrics Device Layer LLDP and SNMP Solutions Digital Experience Cloud AdoptionModern WAN
- 30. @ThousandEyes Resources • Interested in more outage analysis? Sign up for our blog at www.blog.thousandeyes.com • Benchmark and start monitoring your cloud with a free ThousandEyes trial www.thousandeyes.com/contact • Read more: – EMA report – 2018 Public Cloud Performance Report – https://www.thousandeyes.com/resources 30