Developer and Fusion Middleware 1 _ Christian Ledwidge _ Application Performance Monitoring - Improve User Experience reduce user abandonment.pdf
- 1. Applica'on Performance Monitoring: Improve user experience, reduce user abandonment Chris&an Ledwidge Director Solu&ons Architecture Monitoring Quest So9ware The most comprehensive Oracle applications & technology content under one roof
- 2. Going to tell you a li<le story before we begin • Established well established travel company wants to get into the online travel booking engine business – They have an established brand in the package holiday market with their own fleet of aircra9 to service – Margins are under the gun and they need to find new areas for revenue genera&on • Its seems an obvious choice they know the travel business a9er all • Have a good name that should help drive users to their new website • They ramp up a big team of marke&ng, IT, and business owners to kick this project off • Some solid ini&al success and word of mouth spreads and site hit counts move in the direc&on they had hoped for The most comprehensive Oracle applications & technology content under one roof
- 3. Story con&nued • BUT! • Booking numbers are missing forecasts by a solid margin • Calls to their call center are going thru the roof – Business: these call are ea&ng up our margin aren't they? • Nobody in IT seems to be able to tell the business why they are not conver&ng the solid site hit rate into completed transac&ons • Lots of finger poin&ng and conspiracy theories are geSng thrown around – It’s the network! – No it’s the DB! – Its that new Java applica&on that THEY wrote! – Why did we ever go Virtual? • Only thing the business cares about it why are they not conver&ng all of these poten&al customers into paid clients? The most comprehensive Oracle applications & technology content under one roof
- 4. Part One User Experience Management (UEM) Indicators • What are they and what do they do • Five key indicators and how they can be used • What sources can be used to obtain indicators • Important points to remember The most comprehensive Oracle applications & technology content under one roof
- 5. UEM Indicators: Bridge the Gap between Business and IT Useable? UEM Indicators Stable? Right Content? • User counts Right Features? • Changes in user behavior Fast Enough? Business • Geographical variations • Problem impact • Service level agreements Web User Web App Back-end Systems Available? Fast Enough? Short MTTR? Proactive Problem Solving? IT The most comprehensive Oracle applications & technology content under one roof
- 6. UEM Indicator: User Counts Twenty-five users have registered in the last 15 Use Case minutes, but normally at this time, I see 200 or more people registering. Business IT Did our promo&onal campaign stop Are there any outages impac&ng producing results or is the link broken? registra&ons that we need to communicate to the business? Is the reduc&on in registra&ons isolated to How many users are being impacted and a specific region or is it across the board? from which region(s)? The most comprehensive Oracle applications & technology content under one roof
- 7. UEM Indicator: Changes in User Behavior Typically 25% of users complete this transaction, but Use Case today we see fewer than 15%; why the changes in user behavior? Business IT Did the new site rollout hide the submit Are problems with the applica&on causing bu<on to where some users can find it problems with this transac&on to where anymore? users can’t complete it? Where along the path are we losing the How can we help the business restore users and when did it change? service and measure the business impact of the change? The most comprehensive Oracle applications & technology content under one roof
- 8. UEM Indicator: Geographical Varia&ons We see unexpected geographical variations in user Use Case volumes from our EMEA region. Business IT How many users are being impacted and Is our European proxy server having for how long? performance issues? Is this a problem we are causing or is it one Is there a specific service provider that we of our applica&on delivery partners? or an ISP that our users are using experiencing issues? The most comprehensive Oracle applications & technology content under one roof
- 9. UEM Indicator: Problem Impact A user just called in and reported a problem that our Use Case monitors didn’t pick up; what is the problem impact? Business IT When did the problem start and what Is there a specific web server area of the site is being impacted? throwing these excep&ons? Is the error being displayed to end users or Is the right web administrator is it just an image failing that they don’t involved to get this fixed or really see? diagnosed? The most comprehensive Oracle applications & technology content under one roof
- 10. UEM Indicator: Service Level Agreements Use Case SLA goals are not being met? Business IT Do the performance agreements that are Are the IT systems mee&ng the agreed-‐ in place guarantee a sa&sfactory user upon delivery of services to end users? experience? When the SLA viola&ons started occurring Were our Web, App, or Database systems were there devia&ons in transac&on at fault or was it an external interface? conversions? The most comprehensive Oracle applications & technology content under one roof
- 11. User Experience Management Data Sources • User Counts • Changes in User Behavior • Geographical Variations • Problem Impact • Service Level Agreements Data Sources • Web Log Files User Experience • Synthetic Transaction Robots Management • Client-Side Tagging • Web Traffic Analysis Appliances Indicators • Content Capture Appliances • User Surveys The most comprehensive Oracle applications & technology content under one roof
- 12. Web Log Files Data Source Usefulness Limitations • Error tracking • Difficult to identify • User click stream analysis individual user sessions. • Demographic analysis • Data set is partial due to caching. • Performance The most comprehensive Oracle applications & technology content under one roof
- 13. Synthetic Transaction Robots Data Source Usefulness Limitations • Measures transaction • Measures only a small performance and percentage of locations. availability via scripted • Exercises a small transactions executed percentage of the from remote locations at technology stack in larger set intervals. environments. • Sometimes referred to as a “clean room” measurement. • Good for service level reporting. The most comprehensive Oracle applications & technology content under one roof
- 14. Client-Side Tagging Data Source Usefulness Limitations • Good for measuring true • Inserting and managing the scripts adds client performance complexity and overhead • Gathering browser-based to the application events needed to • Data security concerns understand Web 2.0 user interactions The most comprehensive Oracle applications & technology content under one roof
- 15. Web Traffic Analysis Appliances Data Source Usefulness Limitations • 100% sampling of • Asynchronous applications make it transaction performance, difficult to see cause and volume, and errors. effect. • Analysis has many • Analyzed data set is perspectives including by partial due to external caching. server, by geography, by ISP, and even by user. The most comprehensive Oracle applications & technology content under one roof
- 16. Content Capture Appliances Data Source Usefulness Limitations • Stores the full user • Storage capacity sessions, including the concerns data returned from the • Privacy and security web server. concerns • Session replay and other perspectives on the session that are attainable only when full payloads are accessible The most comprehensive Oracle applications & technology content under one roof
- 17. User Surveys Data Source Usefulness Limitations • True voice-of-the- • Typically only small customer samples are gathered, • Can validate metric- and small samples may based hypotheses represent extremes. • Answer questions like: o Why are customers happy? o Why are customers ready to leave? The most comprehensive Oracle applications & technology content under one roof
- 18. Part Two • Measuring user experience in web 1.0 and 2.0 environments • Top 5 performance measurement challenges • Applying approaches to address the challenges • Key elements to remember The most comprehensive Oracle applications & technology content under one roof
- 19. User Experience Management Performance Measurements Internet Web App Back-end Systems Request Response Client Time Network Time Processing Time • Web 1.0 – Simple requests and responses • Web 2.0 – Asynchronous requests and responses • CDNs and content hos&ng – third-‐party effects The most comprehensive Oracle applications & technology content under one roof
- 20. Web 1.0 – Simple Page Request No Third-‐party Connection 1 Connection 2 Connection 3 1. Request ini&ated Client Server Client Server Client Server GET main.html 2. Server gets first 1 packet 2 GET style.css 3. Server sends last GET frame1.html packet 4. Final response GET image1.gif received by client GET news.asp 3 4 The most comprehensive Oracle applications & technology content under one roof
- 21. Web 2.0 – Asynchronous Requests Connection 1 1. Request ini&ated Client Server GET item.js 2. Server gets first 1 packet 2 3. Server sends last packet 3 4. Final response 4 received by client The most comprehensive Oracle applications & technology content under one roof
- 22. CDNs and Remote Content – Third-‐Party Effects Connection 1 Connection 2 Connection 3 Connection 4 Client Server Client Server Client Akamai Client Akamai GET main.html GET style.css 1 GET frame1.html GET perf.js GET news.asp 2 3 4 The most comprehensive Oracle applications & technology content under one roof
- 23. User Experience Management Measuring Performance to Solve IT Challenges “The user tries to explain her “We initially can’t tell if the problem, but it really doesn’t “We don’t know how many issue is on the client, in the help me resolve it.” and which users are having network, or buried in the this experience.” backend infrastructure.” “We learn about most “We lack historical data to problems only when a user proactively detect unwanted calls in” performance trends and their IT root cause.” Internet Web User Web App Back-end Systems The most comprehensive Oracle applications & technology content under one roof
- 24. Challenge One “We learn about most problems when users call” Management Approach Minimal Coverage BeCer Coverage Best Coverage • Log parser-‐-‐ alerts on • Log parser-‐-‐ alerts on • Synthe&c package-‐-‐to errors or performance errors or performance add transac&on perspec&ve • Synthe&c package-‐-‐to • Selec&ve page add transac&on instrumenta&on-‐-‐ perspec&ve captures third party content provider issues • Passive sniffer-‐-‐low overhead capture all solu&on The most comprehensive Oracle applications & technology content under one roof
- 25. Challenge Two “User explana'on is not useful for resolu'on” Management Approach Minimal Coverage BeCer Coverage Best Coverage • Call the user(s) back and • Enlist support teams to • Selec&ve page inquire about issue recreate issue instrumenta&on-‐-‐ captures desktop events • Scrape web logs for • Use online desktop • Passive sniffer-‐-‐captures traces of informa&on sharing to record full content and about the issue customer scenario recreates scenario The most comprehensive Oracle applications & technology content under one roof
- 26. Challenge Three “Can’t isolate issue to client, network, or backend” Management Approach Minimal Coverage BeCer Coverage Best Coverage • Look for anomalies in • Look for anomalies in • Selec&ve page web logs-‐-‐dismiss web logs-‐-‐dismiss instrumenta&on-‐-‐ problem if no anomalies problem if no anomalies captures third-‐party content performance • Use synthe&c solu&on to • Passive sniffer-‐-‐ isolate issue-‐-‐if it shows differen&ates client from up network &me for requests The most comprehensive Oracle applications & technology content under one roof
- 27. Challenge Four “Don’t know how many users experiencing issue” Management Approach Minimal Coverage BeCer Coverage Best Coverage • Count anomalies in web • Count anomalies in web • Selec&ve page logs-‐-‐es&ma&ng impact logs-‐-‐es&ma&ng impact instrumenta&on when from counts from counts third-‐party content being served • Use synthe&c solu&on to • Passive sniffer-‐-‐assess judge impact according impact of all first-‐party to bad loca&ons requests The most comprehensive Oracle applications & technology content under one roof
- 28. Challenge Five “No historic info to detect performance anomalies” Management Approach Minimal Coverage BeCer Coverage Best Coverage • Customer sa&sfac&on • Web log analysis tools • Selec&ve page surveys with long term reten&on instrumenta&on when third-‐party content being served • Web log analysis tools • Synthe&c solu&on with • Passive sniffer-‐-‐for first with long term reten&on base lining capabili&es party base-‐lining and anomaly detec&on The most comprehensive Oracle applications & technology content under one roof
- 29. Important Points to Remember • User Experience Management encourages a common understanding between IT and the business • Common indicators enable collaboration • Sources of User Experience Management information are many—use those that best fit your needs The most comprehensive Oracle applications & technology content under one roof
- 30. Part Three • Why Manage Transac&on Conversions • Web Analy&cs vs. Transac&on Conversion Analysis • Measuring Transac&on Conversions • Top 5 Challenges to Managing Conversions • Points to Remember The most comprehensive Oracle applications & technology content under one roof
- 31. Why Manage Transac&on Conversions • Why do we manage transac&ons the way we do? • Example Transac&on Types ü Shopping Carts ü Ad campaigns ü Customer registra&on ü Login The most comprehensive Oracle applications & technology content under one roof
- 32. Why Transac&on Analysis—We already have Web AnalyCcs? Web Analy'cs Transac'on Analysis • Primary goal: How effec&ve your • Primary goal: What kind of marke&ng is… experience your users have… • Answers “How effec&ve is my • Answers “What influenced user marke&ng?” transac&on conversions?” • Observe how users use your • Observe how user transac&on website: conversions are affected by: – What transac&ons they use and how – Content o9en – Applica&on func&onality – Click stream analysis – Applica&on useability – Applica&on stability Complementary solutions The most comprehensive Oracle applications & technology content under one roof
- 33. User Experience Management TransacCon Conversion Analysis The most comprehensive Oracle applications & technology content under one roof
- 34. Measuring Transac&on Conversions Session Page 1 Page 2 Page 3 Page 4 Hit Hit Async Hit Hit Async Hit Hit Async Hit Hit Async Transaction Start Step 2 Step 3 End Capturing Measurements • Unique Request ID (Off the wire) • JavaScript Instrumenta&on – URL, Page Title, Content, Variable, Loca&on, – Captures page or request level etc.., informa&on per session – Persists current transac&on state per session in – Reports transac&on progress back to memory central monitoring server – Result sent to a third or first party The most comprehensive Oracle applications & technology content under one roof
- 35. User Experience Management Increasing Conversion Rates through Conversion Analysis “We know where people “I need to understand bad “We can’t identify how are abandoning the conversion trends immediately many and which users website but we don’t to reduce lost business.” are associated with understand why.” conversion issues.” “How much money have “When people abandon the we lost because of this site, they often call customer problem?” service – which drives up our Business cost of service.” The most comprehensive Oracle applications & technology content under one roof
- 36. “Revenue loss -‐ a Key Indicator example” 1. Recognized add-‐to-‐cart page My Transac'on Record 2. Page analyzed My Transac'on Record Transac&on Name Buy Funnel • Parse $$ amount Transac&on Name Buy Funnel 3. Updates Transac&on Record Username Mike Smith Username Mike Smith • Add to current Cart Amount Browser IE 8 4. Transac&on Completes Successfully Browser IE 8 Cart Amount $625 5. Increment Sale Amount Cart Amount …. • Set Lost Sale Amount to 0 Transac&on State Ac&ve Transac&on State Complete Success Sale Amount N/A Sale Amount $625 Lost Sale Amount N/A The most comprehensive Oracle applications 0 Lost Sale Amount & technology content under one roof
- 37. “Understanding Bad Conversion Trends” My Transac'on Record Transac&on Name Buy Funnel Username Mike Smith Browser IE 8 Cart Amount $625 Transac&on State Ac&ve Sale Amount N/A Lost Sale Amount N/A • Quan&ta&ve measures available in Analy&cs and UEM • Allows us to form a hypothesis about conversion trends • Qualita&ve valida&on is the next step to validate • Session replay is that valida&on The most comprehensive Oracle applications & technology content under one roof
- 38. “The 5 W’s of Transac&on Abandonment” • Where are users dropping? My Transac'on Record My Transac'on Record • Which users are dropping? Transac&on Name My Transac'on unnel Transac&on Name Buy F Record Username My Transac'on unnel Buy F Record Mike Smith • Why are users dropping? Transac&on Name Username Username My Transac'on unnel Buy F Record Mike Smith Browser Transac&on Name 8 My Transac'on unnel IE Buy F Record Mike Smith Browser Transac&on Name 8 IE Buy Funnel • When did users start dropping? Cart Amount Username Cart Amount Username Transac&on State Browser $625 Browser Transac&on Name 8 IE $625 Mike Smith Ac&ve IE 8 Buy Funnel Mike Smith Cart Amount Username $625 Mike Smith • What were users doing when they dropped? Transac&on State Browser Sale Amount Amount Cart Transac&on State Browser Sale Amount Amount Cart Ac&ve IE 8 N/A N/A $625 Ac&ve IE 8 $625 Lost Sale Amount Transac&on State N/A Ac&ve Sale Amount Amount Cart N/A $625 Lost Sale Amount Transac&on State N/A Ac&ve Sale Amount N/A Lost Sale Amount Transac&on State N/A Ac&ve Sale Amount N/A Lost Sale Amount N/A Sale Amount N/A Lost Sale Amount N/A Lost Sale Amount N/A The most comprehensive Oracle applications & technology content under one roof
- 39. “People calling customer service drive up costs” • Relying on customers to call in when problems with transac&ons is costly – Lost revenue – Damage to brand—reduces repeat purchases – Call center personnel are expensive • Transac&on Conversion Analysis provides insight to: – Spot unwanted conversion trends before they impact the business – Increases efficiency of call-‐center personnel —look at a user session to recreate the problem The most comprehensive Oracle applications & technology content under one roof
- 40. Points to Remember ü Transaction analysis is insurance for achieving and maintaining desired business outcomes ü Weigh the costs of not doing versus doing conversion analysis ü Is analytics enough or do you need something more to make you analytics actionable? The most comprehensive Oracle applications & technology content under one roof
- 41. Thank you for your &me Please drop by the Quest booth in the vendor hall if you would like to have a look at our approach to Applica&on Performance Management The most comprehensive Oracle applications & technology content under one roof