How Allscripts Streamlined Root Cause Analysis - AppSphere16
0 likes515 views
Allscripts is a healthcare IT company that provides software solutions to over 180,000 physicians. Jeff Brady manages a large program at Allscripts focused on implementing regulatory initiatives. He discusses how Allscripts uses AppDynamics to streamline root cause analysis and improve system performance. AppDynamics allows engineers to more quickly identify issues, often reducing root cause times from weeks to hours. It provides visibility into specific code methods and SQL queries slowing performance. Allscripts has seen 200% increases in transaction speeds and 665% decreases in SQL query times after optimizing systems using AppDynamics.
How Allscripts Streamlined Root Cause Analysis - AppSphere16
- 1. How Allscripts streamlines root cause analysis – The gift of time Jeff Brady Allscripts
- 2. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 2 Allscripts • Brand New Allscripts Tower opening in 2017 in Raleigh NC • Allscripts serves more than 180,000 physician users, solutions in 2700 hospitals and 13,000 extended care organizations users on 48 different solutions • Supporting hybrid on-prem and SaaS systems
- 3. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 3 About Jeff • “Program of Programs” manager working within the Common Components and Solutions (CCnS) • Focus is the timely certification and implementation of an industry wide regulatory initiative know as Meaningful use. • Coordination of the activities for 28 inter related development areas that requiring approx. 1000 team members to execute. • Married, 1 Grandson, 5 Children, and an old Dog
- 4. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 4 “To improve the interconnectivity of patient data while remaining in compliance with Regulatory initiatives our systems are becoming increasingly interconnected. The ability to assess the health and performance of these system arrays is critical and needs to evolve at a pace faster than the products.” Problem Statement
- 5. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 5 Requirements from the business/challenges for IT • Continuous performance improvement – Better, Faster, Cheaper • SME focus on reactive actions – Expertise to root cause – Junior engineers may not have the full view of the application – Just working in their one area of the array • Minimized exposure when incidents occur; Extended time to root cause – Forced separation of development and hosting due to PHI – SCOM (indication of a deeper issue) – Difficulty in identification of problem source – Log file analysis – Issue triage and ownership – War rooms
- 6. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 6 Growth is necessary, but it can be expensive System Expansion (organic or net new opportunity) – New regulatory initiatives • Growth with limited additional income – Data analytics • Data warehousing of ever expanding data elements • Timely retrieval of data and report execution (key to client satisfaction) • Expansion of reporting base – System expansion • State by state rollouts • Additional market opportunities • Feature expansions • Onboarding of additional customers – Shifts in market needs • Transition of core business model
- 8. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 8 Relieve the pressure on SMEs Transition subject matter experts (SME) into proactive behaviors – Streamline root cause and detection – Elevate (top line) junior engineers Augmentation of existing tools – Symbiotic relationship between tools such as SCOM and AppDynamics – Performance lab capability enhancements Hosting SCOM Enhancements SCOM Trouble Shooting Architectural Validation Production insights / Details Development QA Feedback Performance Feedback Upgrade Comparison Development Insights Architectural Validation Dev Ops Communications Improved inter-group communications – High fidelity targeting of the source of the issue or performance bottleneck • Exact SQL Query that is limiting database performance • Precise .NET Method and Class Enhance current engineering processes – Architectural validation – Production details easily conveyed to development team Improved system performance
- 9. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 9 Smart Growth • Economic scaling evaluations – Extensibility of performance lab into value optimization – Ability to numerically validate various design options performance • System configuration • Software design options – Determination of best fit hardware scaling versus software scaling – Augmentation of existing performance lab tooling makes this possible BEFORE production deployment. B1 B2 B3 B4 A1 A2 C1 C2 B1 B2 B3 B4 A1 A2 C1 C2 Capacity Doubling Via Hardware B1 B2 B3 B4 A1 A2 C1 C2 Baseline Deployment B1 B2 A1C1 B3 B4 A2C2 8x Capacity Via Config & Software
- 10. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 10 Action plan • Identify stakeholders and understand their expectations • Start small with a well understood solution and solve problems • Form a knowledge sharing community or working group of dedicated and naturally curious engineers • Align the deployment of AppDynamics from development through production on your pilot • Augment your existing tools, the relationship is symbiotic
- 11. Value seen after changes
- 12. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 12 -- “ had an issue with “Change Password” and they were hunting it for 2 weeks, so they applied AppD to it. They found it was not in their code but in an unmanaged library. AppD to provided visibility into the Method and Class hierarchy where the failure was occurring put them on it in a single run. Issue solved, it was related to a security update from Microsoft.…” --This was root caused within 1 hour -- “downloads chase request files from our payers was one of our longest running processes. After drilling down into the Tx.ChaseRequests we were surprised to see that DNS resolution was consuming over 90% of our execution time.” --”Our SQL.Open calls were often the most "expensive" calls performed by the message handlers. Often these calls took 300ms and up to 800ms. “-- -- Analysis and development are in-process this sprint. Value – AppDynamics “Win Wire” -- “We are working to change our traffic models to reflect a production like mix as well as enhancing test scripts” -- “This should go a long way to addressing the gaps noted below.”
- 13. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 13 Value – “Smart Growth” Example Sales opportunity – “Double throughput required” – Applied AppDynamics to load test lab • Augmentation of existing tool set to more accurately identify improvement areas • Extended performance lab into value optimization – 2 Optimization runs • Utilized AppDynamics database monitoring capability to determine most expensive SQL calls • Utilized AppDynamics ability to compare different releases to numerically assess impact – Est. of 4 hours of engineering development / test • Resulted in the ability to meet the sales request w/o the need for an additional system • Systems of this complexity can easily exceed $1MM in total cost – This approach completely changed the business case for this opportunity “Save Result” “Check EMPI” Engineering effort Baseline 450ms 130ms 1 hour issue ID Round 1 250ms 17ms 2hrs to corrective action Round 2 150ms N/A Next Morning~ 2 hrs to Corrective action Impact 200% Increase 665% Increase Second Round
- 14. Summary
- 15. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 15 Recap • Started with a small well understood solution to stabilize our AppDynamics understanding • To understand impacts to inter-team communication we expanded to a more complex product and folded AppDynamics into the product development progression, development through production. • To better understand the full range of AppDynamics capabilities we expanded to a larger product base, mostly at the development level.
- 16. Copyright © 2016 Allscripts Healthcare Solutions, Inc. 16 2016 #1 Top Ambulatory EHR Vendor in user satisfaction poll from Black Book Rankings Company benefit • Simple Identification of performance limitations – By the most junior of engineers, freeing SME to higher level tasks – Preproduction deployment optimization • Direct and measurable performance impacts – Client satisfaction increase where response time matters • Feedback from Production systems into Development • Time to Root Cause, Time to Corrective actions • System performance – Ability to rapidly measure overall performance of design options – Optimize solution SW and HW to maximize value to the company “AppDynamics is the only tool that we have that looks at the specific WCF calls to identify Current and Future issues, in the Web Code”
- 17. Please give us your feedback—Session B5619 • Complete the online survey you'll receive via email later today or via text at: Text this number: 878787 Text this word: APPSPHERE • Every time you submit a session survey, your name will be entered in a random drawing. We're giving away Amazon Echos to 5 lucky winners! • Thank you for your input APPDYNAMICS CONFIDENTIAL AND PROPRIETARY 17 Win!
- 18. Thank you