AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale

Expedia lessons from the trenches -
Managing AppDynamics at scale
Bhadri Govindarajan, Sr. Application Engineer
Vishal Singla, SDE

Bio
•  Vishal Singla
–  SDE, Ecommerce Platform
–  Over 8 Years of Product Development Experience
•  Bhadri Govindarajan
–  Sr. Application Engineer, Ecommerce Platform
–  Over 12 Years Of Experience with Infrastructure and Application
Management
Copyright © 2015 AppDynamics. All rights reserved. 2

Agenda
•  Background
•  AppDynamics Rest API
–  To Automate AppDynamics Transaction
Snapshot
–  For Health Rule Automation

Your Ultimate Travel Companion
Behind The Book Button
•  Complex Distributed System
•  Multiple Datacenters
•  Several Teams
•  Different Technologies, Different Platforms
–  Web Services
–  Messaging Solutions
–  SQL, NO SQL Solutions

Your Ultimate Travel Companion
Behind The Book Button
•  One Booking
–  37 Web Service
–  56 Database
–  11 Queues
–  3 Caching System

AUTOMATE APPDYNAMICS
TRANSACTION SNAPSHOT
Vishal Singla

Wait Begins
•  INC67095
•  ORD-7234
•  PAY-4325
•  SEARCH-4444
•  …..

Problem(s)
•  Dissatisfied customer
•  Response time of the ticket to reach/identify right owners
–  Trouble shooting time

Possible Fixes & Limitations
•  Grow team(s)
–  Not Scalable & Efficient
–  $$$$$
•  Systems transactions link up with common identifier
–  Difficult to maintain contract among 100+ application
interactions
–  $$$
•  Application Performance Management Solution
–  $$

Solution Evaluation Criteria
“If you can’t explain it simply, you
don’t understand it well enough.”
- Albert Einstein
Things we were looking for
–  Ability to deep dive in to applications
for troubleshooting
–  Easy to deploy
–  Visual Representation

Transaction Snapshot
•  Visual representation of the code paths
•  Depicts a set of diagnostic data, taken at a certain point in
time
•  Code-level visibility for troubleshooting problems in
environment

Design & Workflow

What went Wrong?

HEALTH RULE AUTOMATION
Bhadri Govindarajan

Typical Operational Problems
•  Server Crashed
•  Service Hung, Not responding,
Slow
•  Latency, Performance Problems
•  Errors, Exceptions
•  Garbage Collection Issues
•  Disk IO, Space Issues
•  Traffic Patterns
Unhappy Customers
Financial Loss $$$

Operational Efficiency ==> Customer Delight
•  Effective Monitors & Alerts
–  With Troubleshooting Guides
–  Eliminate Noise
•  Be Predictive
•  Detect Early, Restore Quickly
•  Auto Recover

Solution – Year Before Last Year

A Year Later
•  Time Consuming
•  Human Errors
•  Standards Drop
•  Inconsistent
•  Work Load Increase
•  Priority Changes
•  Audit Failures

Solution - Health Rule Automation
•  Templatized Approach
–  Template Health Rule in the Controller
–  Integrated with JIRA for Change Management
–  Jenkins Jobs
Download
Template
Replace
Tier
Upload
Create
Policy

Health Rule Automation – Jenkins Job

Design & Workflow

Health Rule Automation

Gains
•  Created over 600 Health Rules for more
than 70 Applications in 2 hrs
•  Standardized Configurations
•  Decreased Overall MTTR
•  Reduced Cost of Maintaining Monitors
and Alerts
•  Enable/Disable Alerts programmatically
–  During Deployments/Maintenance
–  Reduce Noise

Key Takeaways
•  Visualize Problem
•  Rapid Troubleshooting
•  Easy To Implement
•  Scalable Solution
–  Infrastructure Monitoring
–  Database Monitoring
–  Backend Monitoring
–  Business Transaction Monitoring

Useful Links
Configuring Data-Collector
•  https://docs.appdynamics.com/display/PRO14S/Configure+Data+Collectors
Configuring transaction snapshot
•  https://docs.appdynamics.com/display/PRO14S/Configure+Transaction
+Snapshots
AppDynamics REST API documentation
•  https://docs.appdynamics.com/display/PRO14S/Use+the+AppDynamics+REST
+API

Questions

AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale

More Related Content

What's hot (20)

Viewers also liked (10)

Similar to AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale (20)

More from AppDynamics (20)

AppSphere 15 - Expedia Lessons from the Trenches: Managing AppDynamics at Scale