Managing Software Risk with CAST
2 likes3,331 views
The document discusses the increasing risks associated with software failures in business technology and how CAST technology can help mitigate these risks through effective software analysis. It outlines the causes of software failures, the impact on business operations, and outlines steps for risk management including identifying, stabilizing, hardening, and optimizing software systems. CAST provides tools and approaches for assessing and improving software quality to minimize potential defects and enhance organizational resilience.
Managing Software Risk with CAST
- 1. Managing Software Risk with CAST Building Resilient Software to Support Business
- 2. CAST Confidential 1 Webinar goal and content Goal: Understand how CAST can help avoid software glitches Content Review of state of software risk in business technology industry Analysis of reasons that software fails Explanation of CAST technology for software analysis Examples of potentially-lethal software CAST has uncovered How to implement CAST as a quality gate to lower software risk
- 3. CAST Confidential IT risk has become a serious concern 2 How IT Risk Impacts Business Percent of respondents identifying each business element Source: 2012 IBM Global Reputational Risk and IT Study n = 427 What Drives Reputation Risk
- 4. CAST Confidential System outages have never been easy to control 3 Sources: The Register – 2008 Risk & Resilience Study, IDC Software Quality Study 2011 n = 200 Number of defects requiring patches in 12 months after production rollout 21% of project managers report over 50 defects in the first 12 months after rollout
- 5. CAST Confidential Incidence of software “glitches” is clearly on the rise 4 Software is the primary culprit in system outages Software glitches in live business systems happen frequently Most of the time we don’t find out, but recently there’s more in the news Trading platforms & exchanges Airlines Sources: Wall Street Journal, Bloomberg, The Register – 2008 Risk & Resilience Study
- 6. CAST Confidential Incidence of software “glitches” is clearly on the rise 5 Responsible for 10% of North America trading by volume $440 million loss in 45 minutes
- 7. CAST Confidential Air traffic control system Ticketing self-service website 6 Past forensics related to similar outages Variable not sized properly, limited to 50 days of operation IT procedure to reboot system every 30 days reset timer almost 3 weeks before it ran out Until that procedure was changed A user accidentally types a URL into the wrong field Thousands of personal, records leaked all over the internet Website service suspended for months until new version released
- 8. CAST Confidential 7 Are we just getting used to software failure?
- 9. CAST Confidential 8 Why does this happen? System complexity keeps increasing Too many applications to track Hitting limits of doing more with less Turnover and short-term-ism Sourcing complexity & offshore Speed of software production Inadequate approach to QA No institutionalized product oversight at the structural level
- 10. CAST Confidential 9 Analyst perspectives on the problem, and solution “There is a balance between ‘just get it done’ and ‘do it the right way.’A few additional quality measures help you find that balance.” “Addressing technical debt is really a risk decision for IT executives. I can invest in fixing some of the technical quality problems now, or risk that they result in outages, breaches or other problems that can cost far more.” The architectural assessment of design consequences (on software performance, stability, adaptability, maintainability, and security vulnerabilities) is an area in which CAST excels and successfully differentiates from static analyzers.”
- 11. CAST Confidential Defects in poor systems turn into software failures Software delivered contains 5 potential defects per FP Many defects are dormant in the code Technical debt continues to mount Source: Capers Jones. Data collected from 1984 through 2011;About 675 companies (150 clients in Fortune 500 set); About 35 government/military groups; About 13,500 total projects; New data = about 50-75 projects per month; Data collected from 24 countries; Observations during more than 15 lawsuits. 1. Design defects 17.00% 2. Code defects 15.00% 3. Structural defects 13.00% 4. Data defects 11.00% 5. Requirements creep defects 10.00% 6. Requirements defects 9.00% 7. Web site defects 8.00% 8. Security defects 7.00% 9. Bad fix defects 4.00% 10. Test case defects 2.00% 11. Document defects 2.00% 12. Architecture Defects 2.00% TOTAL DEFECTS 100.00% Severity 1 = total stoppage; Severity 2 = major disruption Defect Origin % Severity 1 or 2 Defects 10
- 12. CAST Confidential 11 Industry starting to pay attention to code quality But code quality & hygiene is only a small part of the solution Component-level Violations Architecturally Complex Violations Dev Test 83% 10% Operations 2% 13% % of violations crossing a phase boundary 8X worse 6X worse 60,700 83,000 168,000 2009 2010 2011 Searches for code quality Violations that cause defects Sources: Li, et al. (2011). Characteristics of multiple component defects and architectural hotspots: A large system case study. Empirical Software Engineering
- 13. CAST Confidential 12 Measurement based on standards Consortium for IT Software Quality Characteristic Architectural & System Level Flaws Coding & Component Level Flaws RELIABILITY Multi-layer design compliance Software manages data integrity and consistency Exception handling through transactions Class architecture compliance Protecting state in multi-threaded environments Safe use of inheritance and polymorphism Patterns that lead to unexpected behaviors Resource bounds management, Complex code Managing allocated resources, Timeouts, Built-in remote addresses PERFORMANCE EFFICIENCY Appropriate interactions with expensive and/or remote resources Data access performance and data management Memory, network and disk space management Centralized handling of client requests Use of middle tier components versus stored procedures and database functions Compliance with Object-Oriented best practices Compliance with SQL best practices Expensive computations in loops Static connections versus connection pools Compliance with garbage collection best practices SECURITY Input validation SQL injection Cross-site scripting Failure to use vetted libraries or frameworks Secure architecture design compliance Error and exception handling Use of hard-coded credentials Buffer overflows Broken or risky cryptographic algorithms Missing initialization Improper validation of array index Improper locking References to released resources Uncontrolled format string MAINTAIN- ABILITY Strict hierarchy of calling between architectural layers Excessive horizontal layers Tightly coupled modules Unstructured and Duplicated code Cyclomatic complexity Controlled level of dynamic coding Encapsulated data access Over-parameterization of methods Hard coding of literals Commented out instructions Excessive component size Compliance with OO best practices www.it-cisq.org
- 14. CAST Confidential 13 Technical debt is related to software risk Most technical debt measures do not categorize the debt There’s a lot of debt out there, many questions about “when to pay it off?” and “which to debt focus on?” It turns out only about 30% of technical debt has any immediate risk component Source: CRASH Report for 2011-2012, CAST Research Labs Distribution of Technical Debt n = 756 applications (365 million lines of code)
- 15. CAST Confidential 14 CAST approach to software risk management (1/2) IDENTIFY Risk reduction starts with identification of risks to understand the scale and scope of risks across an organization Identification using automated tools for consistency and objectivity Output of “Identify” stage should include portfolio view & high profile risks STABILIZE Prioritized list provides an action plan Focus on immediate, short-term risks to critical business systems – Security risks – Production defects Reassess to validate that short term risks have been addressed IDENTIFY STABILIZE HARDEN OPTIMIZE Risk Perspective Immediate-Risk Long-Term Risk Assessment Level Portfolio Critical Systems Application Application
- 16. CAST Confidential 15 CAST approach to software risk management (2/2) HARDEN Move beyond short term, immediate risks to address the “long tail” Focus on performance, robustness, security Improving brittle systems to become responsive, adaptable OPTIMIZE Shift to long-term thinking Shift from process thinking to product thinking Focus on improving maintainability and transferability of systems Address organizational or process issues for long-term improvements Technical debt management and reporting strategy IDENTIFY STABILIZE HARDEN OPTIMIZE Risk Perspective Immediate-Risk Long-Term Risk Assessment Level Portfolio Critical Systems Application Application
- 17. CAST Confidential Analysis strategy for typical IT application portfolio 16 Effort(ManDays/Year) Importance to Business Highest Lowest Critical Apps Entire Application Portfolio CAST AIP Deep Structural Analysis Risk Detection Lean Application Development Function Points & Productivity Vendor Management Continuous Improvement CAST Highlight Fast Cloud-based Delivery No source code aggregation Key Metrics on Entire Portfolio Size, Complexity and Risk analytics Annual/Quarterly Benchmark
- 18. CAST Confidential Portfolio risk review with Highlight 17 Risk vs. Application Criticality This chart examines business criticality against the risk level of the applications. 40 applications are situated in the high risk zone. These 40 applications require detailed assessment and planning for ongoing improvement.
- 19. CAST Confidential ArchitectureCompliance Enterprise IT applications require depth of analysis 18 Intra-technology architecture Intra-layer dependencies Module complexity & cohesion Design & structure Inter-program invocation Security Vulnerabilities Module Level Integration quality Architectural compliance Risk propagation simulation Application security Resiliency checks Transaction integrity Function point & EFP measurement Effort estimation Data access control SDK versioning Calibration across technologies System Level Data FlowTransaction Risk Code style & layout Expression complexity Code documentation Class or program design Basic coding standards Program Level Propagation Risk Java EJB PL/SQL Oracle SQL Server DB2 T/SQL Hibernate Spring Struts .NET C# VB COBOL C++ COBOL Sybase IMS Messaging Java Web Services 1 2 3 JSP ASP.NETAPIs
- 20. CAST Confidential CAST going well beyond static analysis Static Analysis Behavioral Simulation Dependencies Code Pattern Scanning Data Flow Architecture Checker Rule Engine Transaction Finder Function Points Aggregation & Consolidation Understanding of language syntax and grammar using source code parsing Analysis of some run-time behaviors to understand dynamic behaviors of applications Understanding of cross-layer and cross-technology links between application components Finding patterns and anti-patterns in application control flow Tracking the use of the content of variables such as user inputs along static and dynamic call stacks Identification of invalid calls and references between application architectural layers Analysis of knowledge base against quality rules, metrics and constraints to identify violations (non- compliant objects or situations) Identification and configuration of cross-layer and cross-technology transactions from UI down to data entities Estimation of Function Points functional sizing, relying on data entities and Application-wide transactions Aggregation and calibration of results along the quality model and consolidation across applications Intelligent Configuration Capability to build object sets based on object properties, links, etc. to support layers, modules, and scope definition Content Updater Adjustment of analysis results to better match application advanced behaviors 19
- 21. CAST Confidential Simulating runtime behavior to resolve links in code 20 Behavioral Simulation Emulating some run-time behaviors to understand dynamic behaviors of applications Consider “Select Title from Authors where Author = ” as a SQL statement Use (select) link between Java method “f()” and SQL table “Author” quasi-runtime behavior
- 22. CAST Confidential Multi-tier analysis for dependencies (1/2) Capability to handle cross-layer and cross-technology links between Application components Create links between Java Class and Sql Table Hibernate mapping.dtd Table oracle address Address.java Dependencies 21
- 23. CAST Confidential Multi-tier analysis for dependencies (2/2) 22 Create links between JSP page and Action mapping Create links between Action mapping and Java class Struts-config.xml Payment.jsp ActionPaymentMethod.java Capability to handle cross-layer and cross-technology links between Application components Dependencies
- 24. CAST Confidential 23 AIP counts of framework diagnostics Frameworks are the link between components in a well- architected system There are also rules to using such constructs effectively Framework Rule Counts Struts 1.x 21 Struts 2.x 9 Spring 3 Hibernate/JPA 23 EJB 8 JSF 1 Servlet 2 Tiles 1
- 25. CAST Confidential Data flow – cross distributed architecture 24 Capability to track along static and dynamic call stacks the use of the content of variables such as user inputs (1) (2) (3) (4) SQL injection vulnerability – CWE-89 Data Flow
- 26. CAST Confidential Configuring rules specific to enterprise architecture 25 Capability to identify invalid calls and references between Application architectural layers Architecture Checker
- 27. CAST Confidential Security breach due to architecture misuse For example: banking application, for monitoring reasons, all database calls must go through specific stored procedures Investigations showed: – Many transactions developed offshore did not comply with secure architecture framework – Without automation, this could not be monitored • 100 UI elements (250 kloc) • 2000 mid-tier programs (1 mloc) • 250 tables, 350 kloc of PL/SQL Use of Architecture Checker – to define the desired architecture – To generate and enforce the appropriated quality rules 26
- 28. CAST Confidential “UPDATE” trigger causing big problems at a global services provider In reservation system Java application must access legacy main- frame to finalize transaction. In production, a performance issue occurred when a volume of transactions occurred at one time. Investigation showed: – Abnormal activity on the database due to an "on update" trigger that was fired too frequently. – The Hibernate ‘show SQL property’ revealed that the trigger was firing even if the data had not changed. Error was due to a specific parameter in Hibernate: select-before- update on the entity that was set to false. When set to false, Hibernate updated the table systematically. MY_ENTITY A B C D MyUpdateTrigger Always fired 27
- 29. CAST Confidential Real, measurable performance improvement numbers after fixing open/close inside loops. We get around 90% performance improvement. 28 90% performance improvement in large mainframe batch process
- 30. CAST Confidential 29 Application shows a potentially dangerous lack of data control Reduce risk – better use of safe components
- 31. CAST Confidential 30 Violation with the largest impact on the rest of the application, regarding Robustness, Performance, or Security LogicLayerDataLayerGUILayer Propagated Risk Index (PRI) explained
- 32. CAST Confidential 31 Allows to rapidly identify the most significant critical violations related to a Health Factor PRI is based on – Violation Index (VI) which assesses the quality issues a defective object for a specific Health Factor – Risk Propagation Factor (RPF) which assesses the number of call paths of a defective object Violation ViewContext (software / Health Factor) Propagated Risk Index – Prioritize findings
- 33. CAST Confidential 32 Transaction Risk Index (TRI) Identify the riskiest transactions for pen testing, remediation Sum of Violation Indices (VIs) of the objects along a specific transaction: Robustness, Performance or Security. Transaction View Transaction Details View
- 34. CAST Confidential Transaction Weight Risk Index explained 33 GUILayerLogicLayerDataLayer Transaction with largest number of Robustness, Performance or Security violations
- 35. CAST Confidential Stabilizing a multi-tier IT application Missing error handling block across all layers User Interface - Flex Business Logic – C# .NET Data Access – SQL Server (T-SQL) 34
- 36. CAST Confidential Securing a multi-tier IT applications Multiple violations across the same transaction make warfighter / broad end-user facing applications more vulnerable Input validation - 4 form fields without validator in user interface Architecture design - action class talking to data access object bypassing business layer Database access security - multiple artifacts accessing and modifying data on the LOAN table potentially containing confidential data 1 1 2 2 3 3 35
- 37. CAST Confidential 36 Making risk management actionable Identify and stabilize are the tactical steps To harden and optimize is a move towards proactive risk management Requires inserting some actionable processes into the application lifecycle IDENTIFY STABILIZE HARDEN OPTIMIZE Risk Perspective Immediate-Risk Long-Term Risk Assessment Level Portfolio Critical Systems Application Application
- 38. CAST Confidential Measuring risk is important, but not enough At some point, inserting proactive prevention into application lifecycle 37
- 39. CAST Confidential 38 Cost vs. risk tradeoffs If you have Technical Debt – so what? Technical Debt SoftwareRisk L H H L
- 40. CAST Confidential IT risk management is an area of investment 39 IT executives expect to spend more on IT risk IT, and IT risk, is a C-level concern Who has responsibility for reputational risk due to IT? If you’re working on code quality, your efforts should be tied to managing software risk
- 41. CAST Confidential Market leader in Software Analysis & Measurement 40 Ambitious Mission Rock Solid Foundation Market Leader Introduce fact-based transparency into application development and sourcing to transform it into a management discipline Broad market presence in Europe, North America and India Strongly endorsed by software industry gurus and long term investors Over $100 million of investment in R&D, driven by top talent in computer science and software engineering Pioneer and recognized market leader since 1999 CAST Research Labs, the world’s largest R&D facility dedicated to the science of software analysis & measurement (SAM) “CAST metrics have become the de facto standard for measuring the quality and productivity of application services.” – Helen Huntley, Research VP, Gartner
- 42. CAST Confidential Driving software measurement in the ADM industry 41 Key Influencers Recognize CAST 250 Global Leaders Rely on CAST Institutions Engage CASTSIs Resell CASTSIs Use/Resell CAST Top technology First in business IT Biggest benchmark DB
- 43. CAST Confidential CAST dashboards, reports & benchmarks 42 CAST Highlight Portfolio Analysis Size Complexity Risk Technical debt estimation Zero Deployment No centralized source code collection Portal results Full analysis report CAST Application Intelligence Platform Risk Drivers Robustness Performance Security Cost Drivers Transferability Changeability Alerts, trending, root cause analysis Discovery Portal Automated App Blueprint Discover, modernize and change applications Function Point Manager • Automated FP counts • Technical Sizing • Effort Estimation Function Point Changes Due to a Sequence of Change Requests 0 5 10 15 20 25 30 35 40 0 50 100 150 200 Cumulative Effort (Staff Hours) #FunctionPoints 1 52 3 4 Benchmarking Services Compare to industry business process and technology
- 44. CAST Confidential 43 Year end assessment offer from CAST Immediate, actionable insight into a business critical application regarding: – Resilience and stability risk – Performance risk – Portfolio risk assessment How it works: – An assessment will typically take 3 weeks, the longest part of that is collecting all the source files – Can be delivered by CAST or a certified AI Services partner – Typically $10k to $50k for an assessment, depending on the size and complexity of the application Contact Pete Pizzutillo for more information
- 45. CAST Confidential Contact Information Pete Pizzutillo p.pizzutillo@castsoftware.com www.castsoftware.com blog.castsoftware.com linkedin.com/company/cast @OnQuality slideshare.net/castsoftware