Launching Successful Applications
0 likes398 views
The document outlines a systematic framework for launching successful applications, emphasizing the selection and measurement of key performance indicators (KPIs) that align with strategic goals, enhance transparency, and build goodwill among users. It describes how to identify, track, and evolve these KPIs to maintain relevance and effectiveness as goals change. Additionally, it discusses the importance of leveraging existing frameworks and audit logging for accurate data collection.
![How To Count
Anatomy Of An Audit Log Statement
• Items:
– Timestamp
– Action
– “key=value” pairs supplying context
• Format:
– TIMESTAMP action=ACTION [key1=value1 ...]
– Exclude anything the is deducible or implicit, e.g. server name
– $64k Question: Can you parse it with a regular expression?
Successful Applications](https://image.slidesharecdn.com/kpitalk-110421172549-phpapp02/85/Launching-Successful-Applications-56-320.jpg)
Launching Successful Applications
- 2. Successful Applications A systematic framework for launching successful applications Chris K. Chew Enterprise Infrastructure Team Pearson eCollege chrisch@ecollege.com @chrischew
- 20. Launching Successful Applications 1. What To Count – Key Performance Indicators – KPI’s in software – Choosing your indicators 2. How To Count – Finding the right places to measure – Frameworks that can help 3. Count Out Load – Transparency builds goodwill – Advertising success creates momentum 4. When To Evolve Your Measurements Successful Applications
- 21. What To Count Key Performance Indicators Successful Applications 1
- 29. Key Performance Indicators KPI’s are a small number of measurable objectives that will add the most value to the organization for the foreseeable future. Presentation Title runs here l 00/00/00
- 30. Key Performance Indicators • Targeted towards your strategic goals • Must be objectively knowable • Each measurement must clearly add value to the organization • Articulate which scenarios are included in each measurement • The fewer the better Successful Applications
- 31. Key Performance Indicators • Quantitative: Something that is a number • Practical: Something that ties into existing business processes • Directional: Specifying whether something is improving, trending • Actionable: Identifying a problem that is within the organization's ability to change • Comparative: Comparing yourself with a "peer universe", e.g. industry. Also referred to as “financial” Successful Applications
- 32. Key Performance Indicators Etsy’s Indicators • Quantitative: How many items were sold last month? • Practical: How many new members signed up? • Directional: Comparison from January 2011 and February 2010 • Actionable: Number of complaints or returned items? • Comparative: What are ebay’s trends for the same month? Successful Applications
- 33. Key Performance Indicators In Software KPI’s are a small number of measurable objectives that will add the most value to the organization your application for the foreseeable future. Successful Applications
- 34. Key Performance Indicators In Software • Quantitative: Measure the occurrences and durations of activities and transactions • Practical: Measuring strategically-targeted usage patterns • Directional: Did recent changes help or hurt performance? • Actionable: Any error should be an actionable indicator • Comparative: Compare your application to similar applications in your installation Successful Applications
- 35. Key Performance Indicators In Software • Functional: Measuring the frequency and duration which features are used. – Examples: ○ Minutes in class ○ Click-throughs ○ SSO hand-offs ○ Page views Successful Applications
- 36. Key Performance Indicators In Software • Non-Functional: Measuring the transactional throughput of an application. – Examples: ○ Connections per server ○ Response times ○ Page load times Successful Applications
- 37. Key Performance Indicators In Software • Transactional: Measuring the frequency and duration which business transactions are executed. – Examples: ○ Messages ○ Enrollments processed ○ Courses viewed ○ Thread responses Successful Applications
- 38. Key Performance Indicators In Software How To Identify Your KPI’s • What are the nearest-term goals of your application? • How will you know when you have reached your goals? Successful Applications
- 39. Key Performance Indicators In Software Example: Prospero Message Bus Strategic Goal: To provide an always-on message bus that guarantees delivery with very low message publication latency. What To Measure: • Availability • Message loss • Delivery latency Successful Applications
- 40. Key Performance Indicators In Software Example: Prospero Message Bus Key Performance Indicators: • Messages produced (per second and day) • Number of times it takes longer than 500ms to publish a message • Number of delivery problems • Production-to-Delivery latency • Linux “load factor” of servers Successful Applications
- 41. Key Performance Indicators In Software Example: Anton.io Browser-Channel Test Strategic Goal: To feel comfortable recommending the use of web sockets and flash sockets for realistic event-driven web UI. What To Measure: • Connection reliability per transport • Connection density and server behavior of Node.js Successful Applications
- 42. Key Performance Indicators In Software Example: Anton.io Browser-Channel Test Key Performance Indicators • Connections succeeded and failed • Transport used (web sockets vs. flash vs. XHR polling) • Connection time in milliseconds Successful Applications
- 43. How To Count Collecting and presenting your application’s statistics Successful Applications 2
- 44. How To Count The Golden Rules Of Audit Logging • Log transactions, not errors and debug statements • Find the right places to place the hooks • Record the transaction as concisely as possible • Leverage existing frameworks Successful Applications
- 45. How To Count The Golden Rules Of Audit Logging • Log transactions, not errors and debug statements • Find the right places to place the hooks • Record the transaction as concisely as possible • Leveraging existing frameworks Successful Applications
- 46. How To Count The Golden Rules Of Audit Logging • Log transactions, not errors and debug statements – Error logs are for identifying code and system problems – Debug statements are for development and last-resort debugging – Audit logs need to be noiseless and easily parsed Successful Applications
- 47. How To Count The Golden Rules Of Audit Logging • Log transactions, not errors and debug statements • Find the right places to place the hooks • Record the transaction as concisely as possible • Leveraging existing frameworks Successful Applications
- 52. How To Count Finding The Right Places To Measure ...is all about knowing what pieces are included in the measurement Successful Applications
- 55. How To Count The Golden Rules Of Audit Logging • Log transactions, not errors and debug statements • Find the right places to place the hooks • Record the transaction as concisely as possible • Leveraging existing frameworks Successful Applications
- 56. How To Count Anatomy Of An Audit Log Statement • Items: – Timestamp – Action – “key=value” pairs supplying context • Format: – TIMESTAMP action=ACTION [key1=value1 ...] – Exclude anything the is deducible or implicit, e.g. server name – $64k Question: Can you parse it with a regular expression? Successful Applications
- 57. How To Count Anatomy Of An Audit Log Statement - Prospero Example 2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4 2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default 2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce 2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f- f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0 NOTE: This is word-wrapped...always do one statement per line for easier parsing! Successful Applications
- 58. How To Count Anatomy Of An Audit Log Statement - Prospero Example 2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4 2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default 2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce 2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f- f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0 NOTE: This is word-wrapped...always do one statement per line for easier parsing! Successful Applications
- 59. How To Count Anatomy Of An Audit Log Statement - Prospero Example 2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4 2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default 2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce 2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f- f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0 NOTE: This is word-wrapped...always do one statement per line for easier parsing! Successful Applications
- 60. How To Count Anatomy Of An Audit Log Statement - Prospero Example 2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4 2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default 2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce 2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f- f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0 NOTE: This is word-wrapped...always do one statement per line for easier parsing! Successful Applications
- 61. How To Count The Golden Rules Of Audit Logging • Log transactions, not errors and debug statements • Find the right places to place the hooks • Record the transaction as concisely as possible • Leveraging existing frameworks Successful Applications
- 62. How To Count Leverage Existing Frameworks • Log files work well: – Rotated with logrotate or CHOMP – Many search utilities (grep, sed, awk) – Easily processed by any language – log4j and log4net write asynchronously – They can always be latter parsed and inserted into a database Successful Applications
- 63. How To Count Leverage Existing Frameworks – Try our audit logging library available on GitHub at ○ https://github.com/PearsonLearningStudio/audit-logging – Java – Coming soon: ○ .NET ○ Node.js ○ Erlang Successful Applications
- 68. Count Out Loud Transparency and advertisement Successful Applications 3
- 71. Count Out Loud, Be Transparent • Being transparent with successes and failures builds goodwill towards your application. – Users are more likely to trust your application – Users will be more patient when problems occur Successful Applications
- 73. Count Out Loud, Advertise Your Results • Advertising your results will help keep users aware of your application after the excitement of the initial launch dies down. – Product folks are more likely to prioritize improvements to the application – Potential users of the application will feel more comfortable adopting it – The data you share may help your peers Successful Applications
- 75. When To Evolve Your Measurements Metrics change as goals change Successful Applications 4
- 77. When To Evolve Your Measurements • Goals will change either because strategies change or because the goal is reached and new ones must be set. • KPI’s should be adjusted at the same time when goals change Successful Applications
- 78. When To Evolve Your Measurements • Goals will change either because strategies change or because the goal is reached and new ones must be set. • KPI’s should be adjusted at the same time when goals change • But it is also important to listen for negative feedback because you might be measuring the wrong things. Successful Applications
- 79. Wrong KPI? • Inadvertently incentivize undesirable behavior • Start measuring the negative feedback to determine if the problem is real • Either replace the old KPI with the new one or add it to the squad Successful Applications
- 80. Buffer Bloat: Measuring The Wrong Thing • Bufferbloat is the existence of excessively large (bloated) buffers in systems, particularly network communication systems Successful Applications
- 81. Buffer Bloat: Measuring The Wrong Thing • Bufferbloat is the existence of excessively large (bloated) buffers in systems, particularly network communication systems • Some experts believe that bufferbloat is the cause of much of the poor performance and human pain experienced using today’s Internet Successful Applications
- 82. Buffer Bloat: Measuring The Wrong Thing • Bufferbloat is the existence of excessively large (bloated) buffers in systems, particularly network communication systems • Some experts believe that bufferbloat is the cause of much of the poor performance and human pain experienced using today’s Internet • If true, this is a quintessential example of picking the wrong industry KPI – Networks should probably use “Round Trip Latency” instead of “Dropped Packets” because excessive work to not drop packets negates the effectiveness of TCP/IP Successful Applications
- 83. Buffer Bloat: Measuring The Wrong Thing • See http://www.bufferbloat.net for details Successful Applications
- 84. Buffer Bloat: Measuring The Wrong Thing • See http://www.bufferbloat.net for details • Wager: Phở is on me if Buffer Bloat is not part of the general networking conversation by the end of 2012. Successful Applications
- 86. Launching Successful Applications 1. What To Count – Key Performance Indicators in software – Choosing your indicators Successful Applications
- 87. Launching Successful Applications 1. What To Count – Key Performance Indicators in software – Choosing your indicators 2. How To Count – Finding the right places to measure – Frameworks that can help Successful Applications
- 88. Launching Successful Applications 1. What To Count – Key Performance Indicators in software – Choosing your indicators 2. How To Count – Finding the right places to measure – Frameworks that can help 3. Count Out Load – Transparency builds goodwill – Advertising success creates momentum Successful Applications
- 89. Launching Successful Applications 1. What To Count – Key Performance Indicators in software – Choosing your indicators 2. How To Count – Finding the right places to measure – Frameworks that can help 3. Count Out Load – Transparency builds goodwill – Advertising success creates momentum 4. Evolve Your Measurements – When goals change – Negative feedback Successful Applications
- 90. Thank you Chris K. Chew Enterprise Infrastructure Team chrisch@ecollege.com @chrischew