Documentation Example Testing: Value & Impact
- 1. Documentation Example Testing: Value & Impact Dallas C. Kennedy Ab Initio Software LLC Society for Technical Communication – New England InterChange Conference 2017 UMass Lowell
- 2. What Is Doc Example Testing All About? • In the context of software … although you might do similar things elsewhere …. • Examples are essential to users in any documentation and structured learning beyond the most superficial. Explain it to me … Show me … Involve me … … in ascending order of effectiveness • Testing your doc examples … … keeps examples working, … keeps examples relevant, … tests new and updated features as they appear … and much more!
- 3. More about the speaker • Technical writer in the Boston area since 2000 • Currently at Ab Initio Software in Lexington* … … Previously at MathWorks in Natick • Wide experience in documenting complex software systems for advanced technical and data transformation applications • Co-author of multiple documentation sets … … sole author of several significant product documentation suites and books • About 15 years experience in testing documentation examples • Previous experience in physics and astrophysics research, including creating, testing, and using numerical programs for analysis and prediction * Talk not sponsored or endorsed by Ab Initio
- 4. Testing Your Documentation and Products
- 5. What are the value and impact of example testing? As you test and validate your doc examples, you are doing something vital. You are: • Keeping your examples working and relevant • Pre-emptively removing a major source of user frustration and support requests, saving your company money and time • Testing new features, and upgrades to existing features, as they emerge from development • Developing insight into products, features, tasks, and customers
- 6. Deeper cumulative impact Pursued consistently and long term, doc example testing accomplishes less obvious and more profound things: • It shapes how you choose and construct examples. • Can your example be tested properly? If not, why? If not, maybe it shouldn’t be a doc example, or maybe it needs to be rewritten. • It keeps you from tolerating bad examples whose weaknesses are obscured by fancy pictures and words. • It checks your products for regressions – if your examples are supposed to continue to work as new versions of software appear. • Developers can be surprised by this, but they shouldn’t be: Realistic examples and edge cases often test high-level, complex functioning missed by simple tests (like unit tests).
- 7. How Doc Example Testing Works in Practice
- 8. Where do documentation examples come from? Examples have many sources: • Development • Quality engineering and its test cases … … especially more complex cases and work flows • Technical marketing • Training • Customers Lots of “street cred” when you publish examples and applications openly co-authored with customers
- 9. Two broad approaches to testing … • Automated – Convenient, scalable, fast • Examples can be automatically co-tested with demos and the product itself. • Automated testing must often meet important restrictions. • Manual – Sometimes more appropriate for complex examples: complex workflows often most valuable tests, most difficult to implement. … and related in a deep way to doc usability testing – testing that checks the usability, not of your product, but of your documentation and specifically of your examples. • Doc usability testing is time-consuming and requires willing, live human testers in a usability lab. Done right, it is almost always worthwhile. Tests user’s reaction to examples and features. Several things happening at once worth disentangling: • The product and various discrete features • The example • How well the doc presents the product and feature • How well the doc presents the example
- 10. Automated testing • Most easily implemented with code or code-like examples. • Most sophisticated versions of testing automatically extract and reformat examples from the documentation and pass the example code to a product test harness for execution and evaluation. • Requires coordination of processes: product and feature development and testing, and documentation authoring and build. Doc source Extracted and reformatted examples Product test harness Example test results
- 11. Automated testing (continued) • Graphical interface (GUI) testing more difficult to implement, fewer tools to automate, although not impossible • GUI-based examples: various approaches available • If scriptable, entire example can be automated with quality testing on the development side in the engineering test suite. Some GUI-based products have scripting or similar languages behind the GUI application. Others can provide handles to graphical objects. • In other cases, can use automated GUI tools like AutoIt to develop a macro- or script-like automated test, often with real system scripts. • Can always be tested manually, at cost of consuming more time. • Co-test with training and marketing videos and other materials.
- 12. Different levels of testing rigor • Basic and least rigorous – Verification of procedures: Are prerequisites met? Does example finish? Does it freeze or crash? • Deeper rigor – Parsing of syntax, compilation of code, verification of structure and elements of example • Deepest rigor – Evaluation of specific results – numeric, alphanumeric, graphical, etc. Sets of baselines. • Baselines = Numeric and other specific test results Baseline testing is tricky in an environment where results can change from run to run; for example: • Dynamic engineering and scientific simulations • Generation of random numbers or sequences Dependence on platform and machine state
- 14. The other side of example testing • Example testing checks products and examples. • It also builds relationships in your organization. • Technology work is usually divided into functional areas: • Development = primary engineering of products and features • Quality or Testing = verification and validation of product functionality • Documentation = explanation to users of product • Marketing* = showing current and potential customers what your products can do and how they might solve customers’ problems • Training* = educating new and current users about products and features • Example testing touches each of these areas and requires cooperation among all of them, especially between Documentation and Quality. * Customer-facing
- 15. Creates and enhances flow of information about products, features, and examples Products Features Examples Development Quality Documentation Marketing Training
- 16. Mutually reinforcing supports of each: • Development generates products, features, in response to projected or actual customer needs examples • Quality builds product test harness, generates test cases examples test cases from Development, Doc, Marketing • Documentation documents examples, tests examples and test cases with Quality more test cases • Marketing studies and forecasts customers’ problems test cases feedback from customers • Training uses and possibly generates examples examples and test cases for Doc and Quality feedback from experience with users
- 17. Building most important relationship … … with customers! • Important to concentrate documenting and testing examples on more complete, more realistic examples … … not unit tests and other basic functional tests. Basic testing important for Development and Quality, not so much for Doc, Marketing, or Training. • Such examples are what customers need and want. Documenting and validating such examples – keeping them fresh, relevant, and functional – together form an essential source of value to your product’s users.
- 18. Some References • Software testing: • B. Laboon, A Friendly Introduction to Software Testing • M. Feathers, Working Effectively with Legacy Code • No good book sources on testing documentation examples – Limited material available on doc usability testing • See latest and greatest online courses from Coursera, EdX, Udacity, MS Virtual Academy, and so on, concerning software testing and especially writing test cases. Email: dalet@stanfordalumni.org Email: dkennedy@abinitio.com LinkedIn: linkedin.com/in/dckennedy/