From UI to UX: Building Ethnographic Praxis in a Usability Engineering Culture

Editor's Notes

• #2: We’re visiting from San Francisco, home of the Golden Gate Bridge.
• #3: Recently released a redesign of our core sales product. This has been such a huge success for the company, and this work all started with ethnographic field work. But it was a long road to get there.
• #4: Lots of companies have gone through the growing pains of embracing design thinking. If you haven't yet you should read the fast company article on “how Google got Desing.” These are important stories. This is our story of how we our research team evolved from the team who created the product on the left, and became the UX team who created the product on the right.
• #5: So, what changed? How’d we get here? You’re probably familiar with the stories of “how Google got design”, and how Intuit grew to have X# of Chief Design officers. Today, we’re proud to share the success story of how the scope of research, and UX in general, has grown in the past few years.
• #6: To describe that cultural shift, we find it helpful to reference the design maturity stages model. (explain levels)
• #7: Our founders respected great UX at each of those levels from the start. Even when they were working out of this studio apartment, and couldn’t afford to have researchers or designers on staff, they contracted that work out to make sure it was high quality.
• #8: So using the design maturity model, this is what our research portfolio looked like when the company was founded. Where, executives were doing site visits to understand what customers wanted and how they were using our products. Then the UI design and usability tests were outsources to top notch agencies.
• #9: But, as the company grew, UX research had become focused on incremental feature improvements, and researchers were becoming typecast at “the usability people.”
• #10: Research had become synonymous with usability testing. People on the team tended to have HCI backgrounds, and were using many evaluative methods, like cog-tool and eye tracking. These are all important and powerful tools, but are limited in the types of problem they can help you solve. This mindset of evaluating and optimizing concepts falls short when applied to undeveloped products, or new markets, or users whose expectations are unclear, or unknown technical constraints. This is what we’re calling the “fixing bugs mindset”: limiting to role of research to only evaluative methods.
• #11: Our goal was to balance our research portfolio, to have research inform each level of design strategy
• #12: Understanding the history of the team.
• #13: In 2004, a former usability analyst was hired as the founding member of the UX team. The role of the “usability analyst” was introduced. Team staffing dynamics were actually supporting that model. PMs are usually staffed to a particular feature for years, unlike user experience (UX) researchers who are more commonly staffed to a feature for a weeks or months. Because of this was abnormal for these subject matter experts, our PMs to seek out UX researchers for insights on user behaviors or industry trends.
• #14: In 2008, the role bifurcated. “UX researchers” using generative methods like site visits, and diary studies. But, this model only lasted a few years because the org structure and processes didn’t match this model. PMs are usually staffed to a particular feature for years, unlike user experience (UX) researchers who are more commonly staffed to a feature for a weeks or months. Because of this was abnormal for these subject matter experts, our PMs to seek out UX researchers for insights on user behaviors or industry trends.
• #15: In 2010, the company hired the first director of UX research and consolidated the team under the name “Usability Team.” Not everyone changed their title. Not everyone stopped doing ethnographies and reporting on user behaviors. But, during this time, members of the team were spending most of their time in usability labs.
• #16: Early in 2012, the pendulum swung in the opposite direction. An executive requested for the team to provide insights to inform future releases,and the Usability Team was rebranded the User Research Team, and IC’s titles were updated to reflect the change. This change to the team’s mandate was really the beginning of the growing pains that help get us where we are today. It was in that time of uncertainty that we joined the team and our began our work. You can see this tension of describing the scope of the work and who should play where has a long history. These were all natural shifts in the process of growing as a company and as a research team. But, this was the context we walked into. We know that if we were going to contribute at a “problem framing” level we’d need to understand the needs and challenges of the stakeholders working in that space. At the time, it felt like frustrating uncertainty. operational changes that we needed to Would the team get more headcount to cover this additional work? Should researchers deny work requests from the scrum teams they’d been supporting to prioritize time for future-oriented work? This also happened to be when we joined the team.
• #17: To begin our process of organizational change we pulled lessons from the work of
• #18: Armenakis and Bedeian who break down organizational change into the issues of content, context, process, and criterion.
• #19: And we pulled from the work of other practitioners like Jump associates and Second Road, who have published great models for understanding cultural context and providing proof of the value of design thinking.
• #20: And we pulled from the work of other practitioners like Jump associates and Second Road, who have published great models for understanding cultural context and providing proof of the value of design thinking.
• #21: Tsoukas and Chia work on organizational becoming.
• #22: Before we could start making changes, needed to understand the terrain, and start from solid ground. to do that, we needed a shared understanding of the goals of the team, (which had to be informed/decided by the team and our stakeholders). turned tools of research on ourselves.... history of research at salesforce stakeholder interviews team card sorting visualizing gaps in research (circle product development process image)
• #25: Next, we turned our attention to executive stakeholders, and again turned the tools of research on ourselves. After conducting about 25 interviews with the company's most senior executives to learn what information sources they trust, and the types of insights they wished they had, we found that what this audience wanted most was usage data based on large sample sizes. This was problematics for a few reasons: (1). the team’s new mandate was to provide generative, future facing insights, a research problem that ethnography is well suited to inform, but wasn’t at all the information what this group of stakeholders was eager for (2) our SLAs make usage data had to get, (we take customer privacy very seriously), so history weren’t collecting much usage data (3). even though our SLAs had evolved to allow collecting anonymized usage data, most features didn’t have the instrumentation to collect that data. So, in some ways, the new mandate of our team was even more ambiguous than when we started! But, in analyzing these interviews, we also learned that we as a team we didn’t have shared language or expectations about our teams work.
• #26: Next, we turned our attention to executive stakeholders, and again turned the tools of research on ourselves. After conducting about 25 interviews with the company's most senior executives to learn what information sources they trust, and the types of insights they wished they had, we found that what this audience wanted most was usage data based on large sample sizes. This was problematics for a few reasons: (1). the team’s new mandate was to provide generative, future facing insights, a research problem that ethnography is well suited to inform, but wasn’t at all the information what this group of stakeholders was eager for (2) our SLAs make usage data had to get, (we take customer privacy very seriously), so history weren’t collecting much usage data (3). even though our SLAs had evolved to allow collecting anonymized usage data, most features didn’t have the instrumentation to collect that data. So, in some ways, the new mandate of our team was even more ambiguous than when we started! But, in analyzing these interviews, we also learned that we as a team we didn’t have shared language or expectations about our teams work.
• #27: So that became our next step, to align as a team about the scope of our work. To to that, we setup a workshop and had the team basically use a card sort method to categorize all the different the types of reports and types of insights our team was producing. We tacitly knew there were lots of different ideas about what “good” research was and this workshop created the tools and forum for us to discuss those differences.
• #28: The outcome of that work was this framework, which unified language and explanation for how vastly different approaches to research fit together. Gave us a way to have conversations. For example, we started discussing career paths, and explaining that it’s fine to specialize within areas or to play the whole field.
• #29: Most importantly, that framework gave us, a shared language to have more effective conversations about the scope of our work and the goals of our team. From - Undocumented tribal knowledge at the outset of projects (domain expertise was tacit knowledge of AEs and Execs, hard for UX teams to make use of), - Blurry shifts from Ideation through Optimization phases (we work fast, and these lines were very blurry, that’s hard for people just starting to work together). - Limited usage data for Evaluation To - Documented Problem Framing and project briefs - Structured time for Ideation, Experimentation and Optimization - Usage data to evaluate success This was the model that allowed us to align as a team about the research services we wanted to provide as a team.
• #30: Shared language so that we could have productive conversation about what our team goals were. Set consistent expectations with stakeholders. shared goals to “build” as a team
• #31: with that shared ground, we could start building.....
• #32: To start the process of building our foundation, we needed a proof of concept, an example of how this type of research would be valued within the current company context. We were looking for a project with these characteristics: succeed at a team - collaborate across a variety of UX roles qual and quant - to satisfy stakeholder expectations for large sample sizes, and highlight the qualitative research skills of our team visual presentation (company has high standards for marketing :) executive champions who were very invested in using the research for strategy So… SUPER DUPER ADMIN SURVEY Our product managers often want to know whether we’re building the right things for the right people. (development teams often had cyclical, unproductive conversations). So we wanted to know more about how Salesforce administrators do their jobs: are we building the right administrative tools for our administrators? We used a combination of quantitative and qualitative work to answer this--and then we involved designers and product managers and executives in the creation of our deliverable.
• #33: The organizational culture we’re working in today is hungry and eager for all types of research. For example, this is from a segmentation report that went viral in the company. The day after we first presented it there were literally hundreds of people in the dec and we really did get comments on 113… This work sparked strategic conversations with teams we’d never even heard of.
• #34: (we’ve blogged about this). Great example of problem framing project because new language and tools for approaching the problem. for product dev teams to use to work together more effectively. After we presented our results, product managers were asking, “Wait! My feature wasn’t included! How do I measure my sadmin/gladmin ratio?!?!?” We’re pretty stoked that our UX team invented the sadmin/gladmin ratio. The designers we brought in to help build and tell our story actually coined the terms sadmin and gladmin. An accessibility engineer who had gone to grad school for cartooning drew our illustrations. This was such an important project for us because it changed the conversation about what our team “does.” This work didn’t “fixed any bug.” It gave teams new tools for decision making and empowered them to make more informed decisions.
• #35: Key features like the opportunity workspace, came from our researchers going into the field, and seeing that sales reps were creating kanban boards to work their leads.
• #36: within under a year, we had two foundational projects to demonstrate how research could change the way we approach problems and inspire better solutions.
• #37: Again, the way to choose a good foundational project is to make sure it has: UTILITY: immediately useful for both tactical and strategic decisions PARTNERS: succeed through partnerships with people across the organization. designers aren’t there just to make your slides pretty--they are there to help you build the arc of the story, too. VISIBILITY: visible enough to build traction across the organization. market however you need to: email, chatter, brown bags, whatever. get everyone involved in the project to spread the project through their networks.
• #38: we are here.
• #39: Expanding the scope “research deliverables” to include things like behavioral models and needs statements.
• #40: We’re still experimenting. Expanding the idea of “research deliverables” to include things like workshop and journey maps.
• #41: When things work, formalize the process so more people can repeat that success. Continue to focus on outcomes, highlighting the decisions that can be made the product improvements. at this point you can start to Experiment. You don’t want that first proof of concept project to be an experiment! once you have a track record of success, it’s much easier to develop new processes with your partners in the organization
• #42: that brings us back to where we are today, where research is an integral part of the earliest phases of new product development.
• #44: “Design Maturity” implies Style and Form are “less mature,” that stylization is the avenue to being hip and cool. We argue that there are strategic opportunities at each level. Our goal was never to lose rigor in researching the form and function of products.  Ways to contribute strategically at each level. (Jan Schmiedgen)
• #45: As a UX team we’re very proud of our recently released style guide and think it is an example of design strategy at it best. Besides empowering the ecosystem of independent software vendors who build products on our platform, the style guide improved design operations making it easier for teams to work quickly, it provided the standards for designers to follow so that ux researchers don’t have to police consistency.
• #46: Today, we’re proud to have a more balanced research portfolio… where research is.
• #47: Today, there’s not just one way to get across the bridge. There is a lot of movement happening in both directions, where our executives are getting more involved in nuanced UI decisions, and our UX team is becoming more involved with long-term product strategy. Remember, when you want to build a strong bridge, take the time to excavate the land and lay a foundation. ethnography--on ourselves and on our clients--helped us build the foundational bridge to go from a fixing bugs mindset to a building businesses mindset. What will your bridge look like?