Remote Fieldwork: How observational studies elevated usability at AutoTrader.com

Editor's Notes

• #2: Hi. My name is Emily Schroeder, and I’m a Sr. Usability Analyst at AutoTrader.com.Today I’m going to share with you how we tried a new approach – “Remote Fieldwork” – and how it has helped elevate our usability practice beyond the immediate value the research provided.Here’s what you can expect from today:
• #3: I’ll start with a quick introduction about our usability practice at AutoTrader.com, and the type of work we were doing early last year. I’ll then share how a perfect storm of events set us up to try something new that was a positive experience for us and our internal customers, while continuing to elevate our practice in ways we couldn’t even predict at the time. I’ll share the details of our study planning and how we conducted the study, as well as and how we communicated those findings. Finally, I’ll share the immediate positive reception our approach received, the long-term benefits our practice has seen, and how you can try something new in your own practice, no matter what challenges you think you may face.Throughout this presentation, I’ll take breaks to take any questions you may have. There will also be a chance for questions at the end, for you to ask anything you forgot or were too afraid to ask earlier on.
• #4: As I mentioned, I work at AutoTrader.com. Our site is a consumer-facing site of automotive classifieds that allows people who sell cars – dealers and private sellers – to reach people who buy cars.
• #5: When our story takes place, way back in early 2011, we were doing mostly task-based evaluative usability tests: Can the user do that thing we want them to do using this interface?Because we got the chance to test most changes before they went live on our site, we kept very busy with these types of studies.We also were lucky to have a small but dedicated usability team, our own lab and a budget that made it possible to do so many studies and expose lots of people to how real users use websites and other UIs.We had also recently begun doing most of our studies remotely from our lab, and this had become standard practice primarily for recruiting purposes.
• #6: Despite how well things were going for us, we wanted more!We wanted to…Do a wider variety of studies, so that we could utilize a larger part of the UX research toolkit-and-Be more involved in projects from the beginningSo that we couldProvide more value to our organization – our team, our department, and our company-and-Continue to grow as a practice and as usability professionals
• #7: In order to do the things we wanted to do, we needed to identify the right opportunity.We needed the right type of problem to solve, specifically, the type of questions that need to be asked at the beginning of a projectWe needed the right solution, one that would work within our organization and with our teamFinally, we needed the right attitude to actually do what we wanted to do and step out of our comfort zonesLuckily, last spring, we had the perfect opportunity to make that move.
• #8: The business was looking to kick off a huge not-yet-project, and wanted to do it “the right way” by considering user needs based on research from the beginning.This problem aligned very well with our desire to do more up-front UX research, and we had the perfect solution…
• #9: We wanted to be able to follow people at home and at dealerships while they shopped for and bought vehicles, known as contextual inquiry.We knew that this type of research would give us the early insights into how users use tools – online and off – to decide what car to buy and follow through with that purchase. It would help the product managers and designers ground their solutions in the actual needs and behaviors of usersIt was a great idea, but we knew it wasn’t feasible in our environment.Things that got in the way:Time – We are typically in the lab at least two weeks a month, and traveling would cut in to our ability to do other projectsMoney – While our budget easily supports in lab studies, it would have been harder to maneuver travel costsResources – Our team wasn’t large enough to keep other things running while we were in the field conducting this study Lack of clarity between this and the ethnographic interviews that Market Research does in the field andToo big of a departure from our existing methodsSo we took a step back and began figuring out a new solution.
• #10: As with everything else, we wanted to start with our most important goals :Something that can and should be done early to inform the very beginning of a projectSomething more observational in nature so that we can see how users actually behaveAnd then consider our restrictions, the afore mentioned time, money, resources, and fear of the vastly unknown.
• #11: We were already doing lots of remote studies. We had the logistics pretty much figured out, they were an accepted part of our practice, and they were definitely time, money, and resource-friendly.So, why couldn’t we do REMOTE! Contextual Inquiry
• #12: Which we also called Remote Fieldwork
• #13: And Remote Observational User StudiesAnd probably a lot of other silly things, but they all meant the same thing:
• #14: Observe the user in their natural environment remotely, using the same tools we use for remote usability studies and the same general approach used in contextualinquiry
• #15: There were a few things that this solution accomplished that made it the right solution for us, with this project, at this time. It was no more expensive than our average remote study, where we just ad recruiting and incentive costs It gave us the opportunity to see users completing their own tasks, using their own web browser, their own computer, and whatever sites they chose Using our lab, we were able to let many observers see user behavior first-hand Finally, it wasn’t too far a departure from the setup we were used to and that our stakeholders were used to seeing from us, so much less scary fear-inducing for everyone involved.
• #16: We were lucky enough to find ourselves in the perfect storm to try something new.We were at the right stage in the right project – Not only did the project happen to exist, but we recognized it for the opportunity it presented us to conduct early UX researchProject stakeholders who liked working with us and trusted us – In retrospect, this was way bigger than I realized at the time, and had been a product of many months of working to build collaborative relationships with product managersThe right attitude – We were prepared to not have a perfect study, if it meant that we could try something new and learn from that experienceWillpower – We just did it! We made room in our schedule to make this happen, and while we kept lots of people informed along the way, we didn’t wait for explicit permission to move forward
• #17: So far, we’ve talked about our usability practice and our ambitions, and then the situation we found ourselves in that allowed us to act on those ambitions: the right problem, followed first by the wrong solution, then the right solution, all in the right situation for us to make our move.Any questions about any of that so far?Next up, planning our study.
• #18: And then the situation we found ourselves in that allowed us to act on those ambitions: the right problem, followed first by the wrong solution, then the right solution, all in the right situation for us to make our move.Any questions?Next up, planning and conducting our study.
• #19: Next up, planning and conducting our study.
• #20: Once we decided, in general, what we were doing, we went ahead and did it.In the beginning, it looked a lot like the start of any study looked.First, we needed to figure out what, exactly, we planned to do. The questions that would help inform the project we were working on were:How do people decide on major purchases online?How do they use various interface tools as they come across them?Which design patterns support decision making? Which hurt it?What don’t we know that we don’t know? – A key for us with any type of exploratory study. Be open to new learnings that you never expected.
• #21: And then structured a study around answering those questions. We built a test plan around two main “tasks.”To understand what prompts the need to make a decision and how the decision making process proceeds, we conducted a short interview with participants before observing their behaviors. This helped us to understand where they were coming from, what they brought to the decision making process, and the larger scope beyond what we could learn in an hour phone call. While we typically begin all usability studies with an interview, those are focused more on confirming screener questions. For this study, we took a bit more time with the goal of learning a bit more about the user beyond confirming if they were qualified to participate.The bulk of the study would be focused on observing users interacting with the actual interfaces they used to accomplish their goal. We used a lot of the same methods we use in our usability studies: asking the user to think aloud while completing a task and keeping our mouths shut! The only difference was that we asked participants to show us what they had done on whatever sites they used, rather than assigning them discrete tasks to complete on an interface we chose for them. If there was time at the end of the study, we would sometimes direct them to a site that they hadn’t used and may not have been familiar with to see how they would use different types of interfaces to accomplish their goals.
• #22: Next, we figured out who can best help us answer these questions. When we do evaluative studies, we are always talking to people who are either buying or selling a car. Finally, a chance to talk to people who aren’t doing something car related! Our big requirement was people who had made major purchase decisions online recently.The really great thing about opening things up is that we would get a chance to see other sites beyond ours and our competitors, which honestly, we knew pretty well.
• #23: So if we think about how this study differed from our typical studies,A typical study for us was task-based, where we defined specific tasks for participants to complete,Whereas this study was planned to be much more open-ended.Our typical study looked at a defined interface, usually a new design that was planned for our site,While in this study, we expected to see a wide variety of interfacesMost of our studies were evaluative, where we were hoping to determine how successful a new design was,While this was far more observational, with no goal of evaluating any specific interfaceFinally, we would recruit a very specific audience – new car buyers, used car sellers, etc. – for our typical studies,But for this study, we were open to a variety of different verticalsDespite these differences, both our typical study and this study wereConducted remotelyFrom our labUsing think aloud methodology
• #24: Once we knew what we were doing, we had to go through the standard setup steps that we do for every usability study.Recruit: We use external recruiters, and have a good relationship with one. We sent our account manager our detailed screener, and they took care of the rest. (Thank God.) We recruited 12 scheduled participants and 6 floaters. The floaters were expected to be available for two sessions, should we run into cancellations or technical difficulties with a scheduled participant.Write the detailed test plan, and review it with stakeholdersSet up web conferences – we were using GoToMeeting, but any web conference software will workEmail participants with links to our online release form and a link to the web conferenceInvite observers to come down to the lab and see as many sessions as they could – those directly involved attended most, if not all of the sessions, but we had room for others to come and observe as wellOnce all of that was done, we were ready to conduct our study.
• #25: Then we did the study! It was all very similar to our standard remote studies once the planning was done and the study was underway.I got comfortable in the lab while our observers and note taker got comfortable in the observation room.In the lab I had two computers. The first was running the web conference and Morae to record the session. The second was for me to take notes and communicate with the note taker in the observation room.
• #26: In the observation room, observers could see the computer with the web conference thanks to a big TV screen and could hear me thanks to microphones and, when those failed, conference phones.I also left post-it notes and pens out on the tables so that observers would have an easy way to record things they wanted to remember to share with the rest of the group after the session.
• #27: Also in the observation room was another usability analyst who served as my primary note taker. She typed detailed notes, answered questions that came up from the observers, and communicated any questions that the observers had to me via insistent messenger.
• #28: When it’s time, I just used the speaker phone to call the participant and followed my detailed test plan.Introduce myself, make sure I am talking to the right personConfirm availability – Can you talk for an hour, are you at a computer with high-speed internet accessReview release – If they haven’t already signed it, get it signed; Explain that it allows me to record the session and use their feedback; As soon as I have the signed release, I begin recording the session using MoraeSet expectations – That I’m going to ask them some questions, and then have them walk me through the process they went through to decide on a recent purchase online by thinking aloud; while the details were novel to me, they were no more novel to my participants than a standard usability study isRemote setup – Make sure they have the link and have clicked it; coach them through any technical problems
• #29: Once we were all set up and ready to go, I spent about 10 minutes on the interview.Confirm screener information: I understand that you recently bought an [item.] What did you buy? – Checking to see that they know enough about their purchase to be believableWhen did you buy your new [item]? – Should have completed the purchase within the past monthHow did you decide which [item] to buy? – Expect them to at least mention online sourcesHistory:Why did you decide to buy a new [item]?Would you consider this purchase a want or a need?When you decided to buy a new [item], did you have any ideas about what one you’d chose to buy?Decision making processDid you consult with anyone else (when deciding which [item] to buy)? If so, when? Who did you consult?Priorities: What was more important to you: the [item] or the price?Next steps:How long was it from the time that you decided to buy a new [item] and when you made the purchase?Where did you buy your [item])?After you made your purchase, did you do anything online as a result, like right a review, sign up for email updates, or anything else?
• #30: Once all of that was out of the way and I understood where they were approaching the decision making process from, I went to the one big “task” of this study.As with traditional usability studies, I simply let the participant take it from there with as little guidance as necessary. If they needed help getting started, I asked them to think back to the very first thing they did online and start there. I reminded them to think aloud, and stayed quiet as much as possible, though I did ask probing questions as necessary and appropriate.This part took 30 minutes, and we saw all kinds of interesting things, like…
• #31: How users started: Did they start with a specific website or retailer in mind? Did they type that URL into their browser, or use a search engine? What did they search for?What they did when they got to a site: How did they navigate to the things they were looking for? Search? Global navigation? Did they get lost?How and why they used specific interface tools: Search, filters, sort, change views, images, compare, etc.When and why they left a site: met goal, gave up, had new goalAnd what they did next: talked to someone, went to a store, went to another websiteAt what point they had made their decision
• #32: The way we wrap up our studies is short and sweet.Thank the participant for their timeClose the web conference, confirm that it’s closed on their end and that I will never be able to see their screen againConfirm when and how they will be paidBid them adieu
• #33: Once the session is done, I went back to the observation room to discuss what we saw with everyone.Observers got a chance to share any notes they had taken and discuss their observations with me and with each other.Given the nature of this study, there tended to be very robust discussions after each session that were very different in nature from what’s discussed during task-based usability tests. Specifically, we talked more general observations – what we saw that participant to and why they did it – and less about discrete findings, where the goal is to provide a clear example of how a specific interface element doesn’t work and needs to be changed.In total, we did that 12 times, with participants who had made a decision in one of our five verticals
• #34: Now we’ve talked about how we planned for the study, from the goals, general approach, and participants to the nuts and bolts of recruiting, plan writing, and setup.
• #35: As well as the details of how the study was conducted, form the setup, to the interview and observation, and finally the closing and the observer debrief.Any questions about any of that so far?
• #36: Next, I’ll talk about how we communicated the study findings.
• #37: When we were done, it was time to figure out how to communicate our findings.For the vast majority of our studies, we do very detailed findings reports that take the screens that we tested and enumerate the findings on each one. Due to the breadth and unfocused nature of this study, that approach wasn’t appropriate.I was honestly unsure of the best way to document findings. This study was entirely about actually observing user behavior, which I was struggling with how to communicate in a readable report.
• #38: I decided instead to conduct an analysis workshop with the stakeholders – product manager, designers – all of whom had seen a majority of, if not all of, the sessions and were familiar with the general goals of user observation.
• #39: I structured the workshop around five questions:What are users trying to do? – Get a lay of the land? Pick a specific product? Figure out where to go to make the purchase?What information do they need to do that? – Specs, pictures, retail locations?How do they start? – Google? Specific manufacturer? Specific retailer?When are they done? – They met their goal; they gave up; they ran into new goals that can’t be done onlineHow do they use different tools and interfaces?
• #40: While going through those questions, the discussion ended up focusing on:Key findings, as intended, specifically how users did and didn’t use specific interface elementsUnanswered questions, such as how frequency of certain behaviors we saw and the divers behind that different behavior (audience segment, drivers, level of comfort online, etc.)Next steps – quantitative research to determine how prevalent some of our observations are; qualitative research to evaluate design iterations
• #41: We also discussed if formal documentation was important for this study. The stakeholders agreed that the most value was had by observing the sessions directly, but they did state that having a formal documentation of key findings and planned actions would help later on as it became time to defend the designs.But as with many things we say that we’ll do later, it never got done, lost in a sea of other, more urgent work. This is one of my biggest regrets about this study. Even though I was unsure of how to document such rich information, I wish I had just taken a stab at it. Like everything else, the format could have been evolved over time.
• #42: Any questions about our analysis workshop and our lack of a report?
• #43: Any questions about our analysis workshop and our lack of a report?Next, I’ll share how this study was received at its conclusion and the long-term benefits our team saw as a result of trying something new.Finally, I’ll share some tips on taking the leap and trying something new in your own practice.
• #44: After we discussed findings and next steps, I took a few minutes to discuss what worked and didn’t work about this approach with the stakeholders.What we heard:The designers felt that this was the right research at the right time: worthwhile to actually see users interact with different interface elements, and early enough to impact the very earliest designsEveryone liked that we looked at different verticals and saw websites that we typically don’t evaluate; in fact, some people wished we had cast our net wider, looking at other verticals and smaller purchasesProduct managers especially wished they had been more dedicated to observing the sessions, rather than having the opportunity to multi-task and attend other meetings.First-hand observation was keyThis was the first time that having our own lab was a hindrance, because it made it too easy for observers to do other thingsWhen this is done in the future, wants them to last longer and possibly happen off site, even if still conducted remotely
• #45: While the positive reception after the study was great, the real benefits our team saw were long term.Proved that we have value beyond task-based, evaluative usabilityEstablished us as an active part of the project team – attended bi-weekly team meetings, providing design feedback throughout and staying in the loopBoth of which allowed us the opportunity to propose and conduct other new types of research
• #46: All in all, despite making some mistakes (no findings report!) and doing some guessing, we have no regrets about trying something new.You may think that’s fine and dandy, but it just won’t work for you. That’s what we thought, too. And then we ignored that thought and did it anyway.You have more power than you think you do, soJust do it! Try something new! If you have good relationships with your stakeholders, they will forgive you for moving full speed ahead on something new, even if they don’t quite understand what’s happening in the beginningTo get buy-in, work within the confines you can – for us, that was controlling costs and making this study not too different in setup from other studies we doBut ignore the confines you can’t – or are unwilling – to work with, like the false believe that change isn’t possibleSet proper expectations – “Let’s try this. If it doesn’t work, we won’t do it again.” Position it as an experiment, and build the experiment in such a way that there is little real riskFinally, learn. Not just from the study itself, but from the experience of doing the study. Evolve based on those learnings.
• #47: I have shared how we not only got positive feedback coming out of the study, but how the experience propelled our practice to get to try new things.
• #48: I’ve also hopefully given you some inspiration to try some new things yourself, even if you feel like it’s not possible
• #49: It looks like we have [x] minutes or so for questions. Any questions about the reception, benefits, how to get started, or anything else we’ve discussed today?Thank you!