Know Your Users: Improving Learning Content by Connecting with Users

Editor's Notes

• #2: I am also a member of the Council for our STC chapter. Today I will share why you should get to know your users, and how you can go about it. Please hold questions to the end?
• #3: A few years ago, the technical writing team was feeling…cloistered. Isolated and cut off from our users. We would occasionally receive a comment from a user sent over the wall by carrier pigeon, but we had no direct contact with them, and very little interaction of any kind. We didn’t know whether they used our learning content, found it helpful, needed more from us? We just didn’t know…
• #4: So we set out to break out of the walls that enclosed us, and create connections with our users. Today I’d like to share some of these experiences with you, because when you get to know your users, you can realize many benefits. In this presentation, I’ll describe 4 reasons WHY you should get to know your users, and 10 different ways to approach it.
• #5: Before we begin, think about where you currently get feedback about your documentation: Developers QA SMEs Users? If you do not get feedback from your users… How do you know what information they need? Or that your content meets their needs? Can they find the information they need? Can they use it? Developers, QA, and SMEs are not good enough. While they may be smart and experienced and highly qualified for their jobs, these people are not users. They simply do not have the same day-to-day tasks, pressures, context, and real world to deal with. They might be able to serve as pseudo-users in certain cases. When newly hired, they provide “fresh meat” for user testing for a while (somewhere in the 1-12 months range, depending on the product), but eventually they acquire a corporate bias and lose their new-user shine.
• #6: So let’s look at a few reasons why it’s important to connect with actual users, real honest-to-goodness users of your products or services.
• #7: First, when you connect with users, you start to understand their pain points. Where do users get confused? in the software? in your help/learning content? What types of information do they need? What types of information don’t they need? What are their real world challenges? Concerns? Applications of the product? And when you understand your users’ pain points, you will better understand how to help them.
• #8: …visual representation of the results of a user study that I performed on the information architecture… shows the path that users followed to locate the correct answer to a question on where they would go to locate information... The larger the circle, the more users clicked it. Click-through rates: 145 clicks on Home, but only 97 clicks on Document the Project. [CLICK] We lost 1/3 of the users there. Next: From 97 clicks, we went down to only 32 clicks for Share the Design. [CLICK] Here we lost 2/3 of the users! Users lost the scent of information there. And I would never have known this is an issue without connecting with my users. After seeing these results, we made changes to the information architecture. In a follow-up test, over 80% of users were able to navigate directly to the correct topic. When you know your users, you can make your learning content user-centric. Design it around them, to meet their needs.
• #9: When you connect with users, they feel part of a team. By connecting with you, users feel that they are not just on the receiving end of the help or learning content. They have a voice. When you respond to their comments and questions, or they participate in surveys or user testing, they feel heard and important to your company, and that their voice matters. It’s no longer a one-way flow of information. Instead, it becomes a two-way street. Also, when you interact with users or you reply to their questions and comments: For the user, it puts a person on the other end of otherwise anonymous help. They come to understand that real people are working on those help topics, not just The Corporation. For the writer, it puts an identity on the other end of The User. These are real people with real problems trying to get real work done. So it’s no longer THE CORPORATION vs. THE USER. Instead, it becomes US, we’re all in this together, part of a team.
• #10: Another reason to connect with users? Gain credibility in your organization. When you make it known that you know the users, your peers and management will recognize you as a high-quality source of information, and they’ll come to you. Use analytics/metrics to prove that customers and prospective customers use the help, so management understands that it is worth investment. Show improvements in search engine ranking for the help content to prove that your work matters. In discussions with management, you will be able to use direct quotes from customers. You become a reliable voice of the customer within your organization.
• #11: So now that you know Why it’s important to connect with users…
• #12: …let me share with you 10 different ways to do it. I’m sure some of you can think of additional methods, and I look forward to hearing about them during Q&A.
• #13: Before venturing out to connect with your users, [CLICK] check to see whether your organization has guidelines around direct interactions with customers. Learn these guidelines and follow them. [CLICK] If you feel anxious about responding to user comments or other interactions, find a buddy who is experienced in such things and can coach you along. For instance, you might partner with someone from Customer Support, User Experience, or Product Management who regularly rubs elbows with users and knows how to respond to complaints and feature requests. For example, this buddy might look over your shoulder to review an email or social media comment before you hit Send. Or she can provide advice on the best way to address a particular issue with a customer. Are you an introvert? This might feel a bit scary at first. But you can do it! Be prepared to step outside your comfort zone a bit while you gain new skills. Rely on your buddy to help and support you. Over time, as you gain confidence in your new-found customer interaction skills, you’ll find that you rely on your buddy less and less.
• #14: Also, recognize that some methods of connecting with users require less effort [CLICK], while others require considerably more effort. [CLICK] In another dimension, there are varying degrees of interaction with users. Some methods involve indirect connections with users [CLICK], and other methods require that you interact directly with users. [CLICK] If connecting with users is new to you, and especially if you tend to be a bit of an introvert, you might find it helpful to start with methods that are lower on the effort/interaction scale. Then, as you become more comfortable and gain new muscles and skills in this area, you can move into the higher end of the scale for effort and interaction. After describing the 10 methods of connecting with users, I’ll display them on this chart so you can see where they land on the matrix of effort vs. interaction.
• #15: Customer Support is your friend. These folks are on the front lines, interacting with users every day. Granted, these folks are not The User. But they know where the pain points are, because it’s their job to listen to the users and help them. And Customer Support knows how to solve the users’ pain points. A few months ago, I talked to Customer Support and asked, “Where do users experience a lot of pain? Where can we make life easier for them?” Without hesitation, they said, “These 2 error messages keep one support person employed full time. If we can improve those error messages, we can free him up to help users with other problems.”
• #16: So they provided me with some details, and here is what I learned. [CLICK 1st message; read it out loud] Recovery file? What’s a recovery file? And how do I create one? [CLICK through bullet list] Then I talked to Management and made a case for getting a little help from Development, and this is what we did. [CLICK to 2nd message] We added instructions on how to create the recovery file (without using that jargon), and added a link to a help topic that provides instructions. Our help is translated into 12 languages, so this information is now available to everyone at the time when they need it. But don’t make this a one-time event. Have regular conversations with Customer Support. Meet with them weekly or monthly to ask “What is the biggest pain point for our users now?” Use this information to improve the quality of your learning content. And when possible, take steps to alleviate that pain in a more direct way.
• #17: Your company likely sponsors online communities for your users. If not, the users may have created communities of their own. Seek them out. Find the active communities. Then start to listen in. GOOD: Lurk to observe, listen, and start to learn the pain points. BETTER: Participate in conversations. Ask clarifying questions. Point them to existing resources when available. BEST: Investigate the info they need to solve their problems and respond to their queries. Then fold this information into your learning content and make it findable so others can find answers to the same questions.
• #18: Ask users what they think about… Your learning content, and the type of information it contains How they access your learning content When/why they use your learning content Other places they go for information These days, SurveyMonkey and similar tools make it relatively easy to reach out to existing customers and get their input. There are lots of guidelines around designing good surveys, so find a good resource and use it. Run an in-house pilot test of your survey before sending it to customers, and be sure to get management approval before sending it out. GOOD: conduct a user survey to get their input BETTER: use that input to inform decisions about your learning content BEST: use that input to make a sea change in the way you produce and deliver learning content
• #19: About a year after our user guide was made available as WikiHelp, we surveyed users and learned this: ** More than 40% of them looked at online help before making a purchase decision. ** With this kind of information, we can get the attention of management to make further investments in our online help system.
• #20: More recently, we performed another user survey about our videos. [CLICK] We have been including short 1- to 3-minute videos with our help system for a couple of years now. We had a new video tool available to use called ScreenCast. [CLICK] It includes a timeline that shows when individual commands were executed, so the user can jump to that part of a video, if needed. But we weren’t sure how effective it would be as a learning tool. Would our users like it? So we designed an A/B test that allows the users to see the same task demonstrated using each tool, along with a survey to ask our users: Which do you prefer? Why?
• #21: And we got an answer. Most preferred the new ScreenCast tool. We wouldn’t have known this without performing this type of testing. But we also learned of other preferences around videos, and we have used that feedback to improve our process and our video output. ** More details, if they ask: (Chronicle=Screencast)** Both styles of videos scored high as learning tools. 7 out of 9 participants preferred the chronicle video over the standard video Pointer highlighting: continuous mouse pointer highlighting and effects on click Narration: enthusiastic voice, and casual pace, more like a teacher; avoid a distinctive accent and lingo Navigation: left pane thumbnail navigation and command line As a result, we tried to implement the first 2 bullets in our standard videos (continuous mouse cursor and more enthusiastic, casual narration). Due to the recording/editing/hosting drawbacks of producing Chronicles, we have not yet switched to that method.
• #22: A diary study, or longitudinal study, tries to understand users’ behavior and needs over a longer period of time. In a diary study, users self-report their activities at regular intervals to create a log of their activities, thoughts, and frustrations. It’s a useful approach for capturing organic feedback on activities that are repetitive, long, or unpredictable. The diary study is less about getting feedback on one particular thing, but instead is about looking for trends, or larger issues. It is great for discovering habits and motivations that users might not be able to give during other types of testing. For example, you might use this type of study to learn: When do users go to the help? (or other sources of information) What are they doing that prompts it? Are they able to find the information they need? Which types of information do they find most useful? (videos? Conceptual topics? Step-by-step instructions? Troubleshooting info?)
• #23: We recently conducted a diary study to learn why and when people need information about our software, and and where they go to find information. Instead of an analog-type diary with pen and paper, we chose to use an online tool called dscout. With dscout, a study participant, called a scout, uses a tablet or smartphone to make diary entries, called snippets. The snippets can include a photo or a video, and scouts answer a short series of questions to provide specific information. We recruited 20 users and asked them to submit at least 12 snippets over 4 weeks, providing information about incidents where they needed information about our software. We rewarded customers for participating with Amazon gift cards.
• #24: Here is a sample snippet. (talk through this example)
• #25: (talk through this example)
• #26: (talk through this example)
• #27: (talk through this example)
• #28: So we collected snippets from our 20 users over 4 weeks. When we didn’t hear from someone for a while, we would send them a gentle reminder. After 4 weeks, we evaluated the results.
• #29: When we looked at the results, we learned that users overwhelmingly look for information when they are trying to complete a task. Knowing this, we can work to make sure that our help content is task-oriented.
• #30: Also, they use many different resources when trying to learn about the software: Primarily web searches, followed by turning to a neighbor or office expert. Knowing that most folks perform web searches to look for information, we can work to make sure that our help content rises to the top of relevant search results. We also gleaned specific comments from their questions and issues, and used these to create user stories in our Agile rapid board to improve the quality of our help content. Furthermore, we learned about areas of the software that are problematic for our users, and passed that information to our development teams, so they can investigate how to make the software more usable. So by using a diary study, you can get lots of great information over time that you might not be able to gain otherwise.
• #31: If your product help is available online and provides a user commenting feature, get involved! For our product with thousands of users, our team of 5 writers responds to the questions and comments that English users make about our help topics. We probably average a couple of hours per week each, responding to them. But this relatively small amount of time invested reaps large rewards in improving the quality of our help, its findability, and its usefulness.
• #32: For example, for each help topic, our users can provide feedback on whether a particular topic is helpful and why. They can also leave general comments or ask questions. GOOD: Respond directly to user questions. BETTER: Use their input to improve the help going forward.
• #33: BEST: Update the current help based on their feedback, and let them know it. On my team, we update the online help for our current version on a regular basis, based on user comments and questions. We list these updates on a page so that users can find new or changed information AND so they know that we are listening to them.
• #34: Whenever possible, attend events where you can meet and talk to actual users, such as: Local user group meetings Annual user conferences Beta testing events Shake their hands, look them in the eyes, learn their names, make personal connections Listen to their stories; be alert for information about their pain points Let them know what you do and that you are open to their input and feedback
• #35: Usability testing generally involves giving users one or more tasks, observing them as they try to perform those tasks, then interviewing them to get direct feedback. It can take many forms, and we have done both extremes: hiring professionals to do the usability testing for us, and doing simple, inexpensive testing ourselves. For example, a few years ago we hired the team at the User Experience Center at Bentley University to do some testing around videos: How long should videos be? Should they be narrated or have captions? Should they show step-by-step instructions? Or illustrate higher level concepts? The Bentley team did a fabulous job, and we learned lots of useful information as a result of that testing. (Videos should be 1.5-3 minutes max, narrations are good, step-by-step or conceptual depends on the need and the content.)
• #36: However, usability testing does not have to be complicated or expensive. We have also performed less expensive usability testing using online tools with remote users. For example, I recently used a tool called Treejack by Optimal Workshop to test the information architecture of our help system. I wanted to learn whether our users can locate information easily. Treejack is an online tool, similar to SurveyMonkey, where you send a link to users and they answer questions. With Treejack, they can also perform tasks by trying to locate information in your information hierarchy.
• #37: So I used Treejack to ask users: Where would you look for *this type* of information? And they navigated the help hierarchy to indicate the place they thought they would find it. Treejack records every click up and down the hierarchy, as well as the topics that users nominate as the correct answer.
• #38: (Talk about the results provided in this summary.)
• #39: Treejack provides easy, visual analysis of the results, helping you to pinpoint issues in the hierarchy.
• #40: Recruit users who are interested in improving the quality of your learning content. Ask if they would be willing to provide occasional feedback on a small set of new help topics, a new video, or other learning content. This effort likely requires management approval and NDA. When you have a few topics ready about a new feature, or have revamped existing help, send it to them and ask for feedback…quick turn-around…direct feedback…sanity check…meeting their needs? Example 1: local user group meeting…met some customers… answered their questions. Asked them “Would you be interested in providing occasional feedback on learning content for new features?” Example 2: A user often left comments in our online help. One comment provided valuable insights about his learning style and what he finds most helpful. I contacted him directly by email, and asked him to be one of my user-reviewers. This feedback helps me to improve the help even before most users see it.
• #41: If you are practicing Agile, you have regular sprint demos to demonstrate whatever you have finished in that sprint. Some teams invite users to attend their sprint demos to see what they are working on. The same users attend throughout that release cycle or project, and provide regular feedback. When users attend a demo, we make sure that we attend. We ask the team to include in the demo the learning content that we have developed for the feature, and we ask the users for feedback on the learning content. When users see the learning content in context with the new feature, we often get insights that we don’t get when they see the learning content alone. During the demo, we listen carefully to the questions they ask. Afterwards, we check the help topics and other learning content to make sure we have addressed those questions. This type of direct feedback is highly valuable and improves the quality of the information provided in the learning content.
• #42: Go on site visits with UX or product management. A site visit can expose you to the real everyday challenges that your users face, the conditions they work in (distractions, office environment, and so on) It provides you with additional context which may be useful to inform your design decisions. Whether it’s in the office…
• #43: Or out in the field
• #44: In team meetings
• #45: Or on the factory floor, you can learn valuable information from a site visit. Sometimes just seeing the work conditions can help you make better decisions about your learning content. For example, suppose you want to create a narrated video for a particular task. When you visit the plant, however, you realize that the factory floor is so noisy, the video narration will not be heard. As a result, you decide to provide captions instead. These are the kinds of insights that help you to improve the quality of your learning content, making it more usable and more useful for your customers.
• #47: I mentioned earlier that some methods of connecting with users require less effort [CLICK], while others require considerably more effort. [CLICK] In another dimension, there are varying degrees of interaction with users, from indirect contact such as [CLICK] lurking or email or surveys, to more direct contact via [CLICKl] online conversations, telephone, or face-to-face meetings. So here I will plot the various methods I’ve described on this matrix. [build out the slide and discuss as you go] If connecting with users is new to you, and especially if you tend to be a bit of an introvert, you might find it helpful to start with methods that are lower on the effort/interaction scale. Then, as you become more comfortable and gain new muscles and skills in this area, you can move into the higher end of the scale for effort and interaction.
• #48: [build out the slide]
• #49: Just to recap, it is important to connect with your users to: Understand users’ pain points Gain insights to improve the quality and findability of your learning content Put users on your team (no longer The Corporation vs. The User, but “we’re all in this together.”) Gain credibility in your organization