Adjusting the Focus: Usability Study Aligns Organization Vision with Community Needs

Editor's Notes

• #11: Thank you Dr. Barnum, and thank you all for being here today. I’m going to talk about how the teams approached testing. The theme you are going to hear is that we had 2 teams, there were some commonalities in our approaches, then we diverged , but we still ended up with overlapping results, leading to a more focused vision for the E4C website.As Dr. Barnum mentioned, we had two teams of 4 working on this study. Both teams attended the project kickoff meeting, had access to the same background information, and sponsor concerns. We both structured our user research in the manner that Dr. Barnum described, creating personas and conducting heuristic evaluations. However, almost immediately the team approaches began to diverge. Let’s start with the commonalities. Common process:planning for testing-Test planning:personas,heuristics,pre-test questions,scenarios/taskspost-test questionsParticipants:screening, recruitingTest day:lab, moderator script, test toolsAnalyzed dataReported and presented findings and recommendations
• #12: Starting with our user research, we created personas. Using input from the sponsor and data from the E4C user profiles, we all agreed the typical users included American and international professional engineers and engineering students. They came from a variety of engineering disciplines and were overwhelmingly interested in using their skills to solve problems that could benefit others. Once the personas were completed, the teams began to diverge in their approaches. Given the time constraints, each team could conduct testing for only one user type. Team 1 chose to test professional engineers to give the sponsor test results from the view point of their primary users. They planned to test the website from the POV of their persona, Elsie Manning, a 52 year old manufacturing engineer. Team 1 recognized they may face problems finding professional engineers willing to participate in the study but they were confident that their personal networks would provide a supply of recruits. Team 2 chose to test engineering students, who are a high percentage of secondary users, this also gave the sponsor an alternate view point. They planned to test from the POV of their persona, Michael Samford, a 22 year old civil engineering student. Team 2 knew would have no problem recruiting participants later since they had a large student population they could draw from at SPSU.
• #13: Using this typical user information, we conducted personal interviews of people that match the types and came up with personas. Although more than one persona exists for the e4c site, each team could only choose one and here the teams diverged in their choices. Once the personas were completed, the teams began to diverge in their approaches. Given the time constraints, each team could conduct testing for only one user type. Team 1 chose to test professional engineers to give the sponsor test results from the view point of their primary users. They planned to test the website from the POV of their persona, Elsie Manning, a 52 year old manufacturing engineer. Team 1 recognized they may face problems finding professional engineers willing to participate in the study but they were confident that their personal networks would provide a supply of recruits. Team 2 chose to test engineering students, who are a high percentage of secondary users, this also gave the sponsor an alternate view point. They planned to test from the POV of their persona, Michael Samford, a 22 year old civil engineering student. Team 2 knew would have no problem recruiting participants later since they had a large student population they could draw from at SPSU.
• #14: The next step was for each team to perform a heuristic evaluation. The teams followed a common approach which was to choose a set of heuristics and each member on the team was to independently evaluated the E4C website based on the heuristics chosen and from the viewpoint of their user type. That being said, the teams immediately diverged and chose different sets of heuristics. Team 1 chose Nielsen’s 10 heuristics which are well known but also for the specific and objective evaluation criteria and Team 2 chose Quesenbery’s5Es – they provided the same comprehensive assessment while also providing enough flexibility to provide a holistic evaluation of the various site elements and functions. Although the decisions were made independently, as it turned out, the choices provided an opportunity to test the ‘process’..where would the different heuristic choices take us?
• #15: Here is a list of the heuristics for anyone not familiar with them and where you can access them online
• #16: These were possible show stoppers – these categories are repeated in findings/recommendations sectionsThe heuristic evaluations led each team to identify problem areas for further testing, and the teams drafted scenarios or stories with tasks embedded for participants to walk through in order to test their reactions – the two teams came up with different criteria for drafting those scenarios, due to the difference in the heuristics chosen
• #17: It was time to start identifying qualified participants for testing. For the commonalities, both teams used a pre-screening questionnaire or a selector to begin qualifying participants- they asked basic demographic questions, confirmed that candidates were engineers, either professionally or as students and which engineering discipline they were involved in, asked about their interest or participation in community service, verified they were technically literate, and verified they were available to attend the testing sessions, and were open to being recorded for this study. We also let them know that a $25 would be provided by the sponsor to selected participants. Again, at this point the teams diverged.Team 2 on the other hand had no trouble finding candidates that fit their user profile, as you can see they received over 60 responses indicating interest in participating.The user types the teams chose immediately impacted the recruitment process and results – team 1 immediately faced challenges attracting a pool large enough to be selective –due to a lack of response we modified the participant requirements - while we preferred a combination of men and women engineers, aged 45 and older, we modified the criteria to accept any age as long as they were a practicing engineer and community service minded.
• #18: Despite the differences encountered, both teams recruited 6 participants – 1 practice participant and 5 participants whose data points were collected, recorded, and evaluated as part of the study. Note the limitation of the Professionals participant pool – all males in similar age bracket
• #19: Once test day arrived, both teams used the same student usability lab – we had access to the same materials you see here – Note that each team conducted mulitple testing days that took place over the course of several weeks
• #20: And we followed similar procedures – although the questions, scenarios, and tasks varied by team
• #21: We both used a moderator script for consistency
• #22: John Weaver will nowaddress the findings
• #26: Team organized cards based on comments.Insert image of cards laid out on a table or maybe an image of a few cardsShould we make a comment how people generally try to be positive
• #29: Participants commented positively on the look of the E4C SITE, including imagery and media Participants recognized the good that E4C organization works toward. Participants commented favorably towards the wealth of information and content available. Over half of the participants expressed an interest in returning to the site.
• #34: After determining the issues, both groups provided recommendations, which fell into the following high-level categories:Navigation format/options, language, homepage changes, registration/membership, social media promotion.Recommendations to client based on data and verified by video- Recommendations were arrived at using qualitative and quantitative tools, which produced qualitative and quantitative findings. Our 2 groups using different tools to arrive at same insights (similar but different paths, we were not guessing – proven methodologies) Hufflepuff did not test/ask questions about Social Media.
• #35: Change the slidingmenu so that all major links are visible at once.
• #36: Change the sliding/hidden menu so that all major links are visible at once. Allow users to search E4C members by name and/or user name, and/or location in the members directory.
• #37: Expose all links (goes back to the sliding menu issue)Use conventional F-Pattern – possibly add navigation for specific groups or purposes Eyetracking shows not much viewing of links, more of picture and text
• #38: The current mission statement is long and difficult to read. Create a concise, and prominent mission statement on the home page.Use more meaningful pictures that show engineering projects in order to visually communicate the mission statement. Ensure all images relate to their corresponding text. Change the sliding/hidden menu so that all major links are visible at once.Rename primary links to communication actions that can be taken on the E4C website.
• #39: Consider renaming button to “Join this workspace” or “Work on this project.”Create labels based on the user’s languageConduct a card sort to determine the most meaningful button name.
• #40: Change the sliding/hidden menu so that all major links are visible at once. Evaluate link names and conduct a card sort to determine the most meaningful link names. Consider renaming “Solutions Library” to “WorkspaceLibrary” or “Projects Library”. Add a clear, concise, and meaningful statement that describes the purpose of these areas on their main pages.
• #41: Evaluate link names and conduct a card sort to determine the most meaningful link names. Consider renaming “Workspaces” to “Projects”. Use their language.Add a clear, concise, and meaningful statement that describes the purpose of a workspace on the main workspaces page.
• #42: Focus on purpose and identity, fewer words, more pictures and accessible links, Fix script error
• #43: Change the sliding menu so that all major links are visible at once.
• #44: Allow users to pause the scrolling pictures. Slow down the scroll speed.
• #45: Reduce the number of elements on the home page.Evaluate the purpose of the map function and whether it belongs on the home page. Create clear labels and visual cues for actions that can be taken from the home page. Consider color changes for contrast
• #46: Provide instant feedback at the field level.Evaluate the need for highly complex password requirements. Make instruction easier to read with larger, darker text.
• #47: Evaluate the need for this level of security.
• #48: Eliminate the map and location field on the registration page. Location information is useful; allow users to enter it into their member profiles after completing registration.
• #49: Provide a reason for users to register for the siteExplain how information is going to be displayed and who has access to itProvide confirmation email upon successfully registering
• #50: Move social media links to more prominent location.Embed YouTube video into homepage as an additinal visual aid to explain purpose of E4C and why/how they should participate.
• #51: Test each improvement as you implement.Use a more diverse participant pool.
• #52: Let’s listen now to our client’s thoughts about testing and plans for the future.
• #53: If Dr. Barnum wants to make comments before opening up to questions, we can leave this slide here…if questions before wrap up then we’ll switch this with questions slide.