FDA ASQ presentation metrics April 2012
1 like1,040 views
This document discusses metrics and how they can drive both positive and negative behaviors if not implemented properly. It provides examples of good metrics that are fact-based, objective, and focus attention on key goals like customer satisfaction and regulatory compliance. Good metrics also reflect both internal and external factors. The document outlines characteristics of effective metrics and provides examples for regulatory purposes and quality improvement. It emphasizes the importance of having metrics that are close to the problem source and drive accountability.
FDA ASQ presentation metrics April 2012
- 1. Presented for the ASQ/FDA joint meeting April 25, 2012 Presented by: Maureen McAllister
- 2. “It was the best of metrics. It was the worst of metrics.”
- 3. Metrics Metrics drive behavior…..for better or worse “Good” metrics link to desired performance Revenue/margins (senior mgt attention) Regulatory compliance Customer satisfaction (what the customer cares about) Growth/future progress
- 4. Metrics can also drive you crazy! “Bad” metrics Side track attention from key performance goals Can be one-sided (not a good mix) Confuse/obfuscate/overwhelm “monster dashboards” Strictly hind-sighted Driving, looking just in the rearview mirror
- 5. Metrics – In Context Good metrics reflect a mix of internal and external factors Customers Suppliers Regulatory /statutory constituents Others , including investors Where in the “food chain”? OEM versus contract manufacturer? What tier in the supply chain? How critical are the products/services? No formulas; no “one size fits all” approach
- 6. Metrics for Regulatory Purposes Examples: No. of MDRs. No. of late MDRs No. of complaints No. of complaints open more than 60 days No. of CAPAs open (compliance vs. others) No. of CAPAs past due No. of CAPAs open (patient safety) No. of CAPAs past due (patient safety) Total no. of CAPAs open Total no. of recall/field-initiated CAPAs
- 7. Some Characteristics of Good Metrics 1. Reflect the “critical few” performance goals – especially customer line-of-sight issues A good mix of metrics 2. Fact-based = objective data 3. Close to the source (not “filtered”) Close in time – fast to know Close in responsiveness – action at the source of the problem 4. Consistently defined Across time Across product lines Across locations/business units
- 8. Example 1: Complaints vs. Shipments Quantity-based Lagged by an appropriate time period Segmented by product line, work cells, locations, etc. Pareto the problems Drives attention to problem areas/product lines
- 9. Example 2: Internally: defects vs. sampled Internally focused: PPM as produced (not as shipped) Quantity focused Defect codes/ root causes Total Root causes 83 Top 10 Root causes 48 58% Drives a comparison to what the customers are complaining about => impact inspection/test criteria and processing points of inspection/test
- 10. Example 3: Aging of Complaints & CAPAs Rating system , based on severity 1 to 3 1 to 10 (AIAG Risk Priority Numbers) Aligned with reportable events Number overdue vs. total open Aging past the 30 day investigation window Drives responsiveness Drives consistency in definition and escalation practices
- 11. Example 4: Residual risk vs. warrantee/service Risk assessment at new product release vs. service activities (rolling 3 or 6 months) RGAs (out-of-box failures vs. modifications-related) Supplier issues (purchased finished product) Internal issues (assembly or manufactured defectives) Subcontractors (modified/customized)**** Drives attention to what the customer is experiencing directly – especially safety Drives attention to supply chain issues
- 12. Example 5: New product launch incidents Leading indicators No. of reportables in first 8 weeks No. of reportables in first 100 installations Service responsiveness: Mean Time Between Unscheduled Service Call (Open) Mean Time Between Unscheduled Warranty Call Drives attention back to initial risk assessment Immediate responsiveness – what the customers are experiencing
- 13. Some “contrarian” metrics Metrics that may drive quality and safety improvements indirectly Revenue/worker-hour vs. scrap costs Drives hiring decisions – permanent vs. temps Drives training resources/commitment On-time delivery Fines for shipping late or short shipping => drives increased attention to planning (contract review)
- 14. Metrics: pictures vs. spreadsheets??? Supplier, 62% Customer Complaint, 3% Internal Audit, 27% Continual Improvement, 5% Operation, 3% CAPAs: Open and overdue by Source
- 15. In Warranty Root Cause Code Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Full Year 1001 Wax in Air System 1 1 6 2 4 2 - 4 3 1 - - 24 1014 Rotary Valve Errors 4 2 1 3 2 4 1 2 2 - - - 21 1038 Blown Fuses - 2 1 2 2 1 2 4 3 1 - - 18 1028 Xxxxx Issues 2 3 1 3 2 - 1 - 3 - - - 15 1020 Power Supply Problems - - - 2 2 1 5 2 3 1 - - 16 1026 Miscellaneous Fault Causes - 5 - 1 2 1 3 3 - 1 - - 16 1027 Loose Air Lines - 2 1 1 4 - 3 - 3 - - - 14 1035 Faulty / Leaking Liquid Valves - 1 2 3 2 2 - 2 1 - - - 13 1003 User Errors / Customer Training 3 1 - - 1 2 1 1 4 1 - - 14 0000 Invalid Cause Code - 2 3 1 6 - - - - - - - 12 1034 Faulty Air Valves 1 2 - 2 - 1 1 1 - - - - 8 1005 Unable to Classify 1 - - - 2 - - 1 3 - - - 7 1011 Service Errors 2 1 - 1 1 - 1 - - - - - 6 1030 Hard Disk Failures - 1 1 1 - 1 - - 1 - - - 5
- 16. Where from here? Start with the “painful” areas Listen to your customers – external and internal Go to the source Root cause investigations at the source Tie findings back to your metrics Change your metrics to capture reality Have a mix of high-level and lower-level metrics => business goals => operational problems/pain => two metrics really worth having