Might Not Be a Magic Number but There Are Magic Ranges
MeasuringU
UX research and software supporting all stages of the user and customer experience.
“What sample size do I need?”
We’ve all been trained from years of math education to expect a single answer to that question—a single sample size number.
But earlier, we warned against the quixotic quest to identify the one true sample size to use for UX research—the “magic number.”
Because sampling error is real but not insurmountable, you can and should have a plan for coming up with sample sizes. However, a single magic number is not the answer.
The right number (sample size) is a function of the research goal. In UX research, most research questions can be categorized into one of three goals:
But even within each of these broad goals, there still isn’t one magic number. In other words, there is no magic sample size number for any type of UX research.
There are, however, Goldilocks Zones (“magic ranges”) that balance the various statistical and logistical forces that drive appropriate sample sizes for different goals:
Avoid looking for one magic number that always works (like 5 or 30 or 999). Instead, start with the typical (“magic”) ranges for the research goals shown in Table 1.
In this article, we show how we arrived at these “magic ranges” by balancing the logistical (e.g., study cost) and statistical (e.g., data types) aspects of sample size planning.
Read the full article on MeasuringU's Blog
Summary and Discussion
These graphs and tables make it very clear that there is no “magic number” for UX research studies. There isn’t even one “magic range.”
By considering competing statistical and logistical considerations, we’ve recommended ranges for three of the most common high-level UX research goals: discovery, estimation, and comparison.
When planning a UX research study, the first step is to identify the research goal, which establishes the magic range (5 to 20 for discovery, 30 to 300 for estimation, 40 to 400 for comparison).
The magic ranges are too wide for precise planning, so the next step is to look inside the magic range to find the best balance between precision and cost for your specific study. Only if you can’t find that balance inside the magic range should you look outside the range.
For example, suppose you need to conduct a between-subjects comparison of two groups and have a budget for only 15 participants per group. Consulting Table 5, this total sample size of 30 means you can detect, with 90% confidence, a binary difference of 46% or a rating scale difference of 23 points. If that level of precision is satisfactory, go ahead and run the study. If not, then you should save your money. As David Salsburg wrote in his book on the history of statistics, The Lady Tasting Tea (p. 265):
A careful examination of resources available often produces the conclusion that it is not possible to answer that question with those resources. I think that some of my major contributions as a statistician were when I discouraged others from attempting an experiment that was doomed to failure for lack of adequate resources. For instance, in clinical research, when the medical question posed will require a study involving hundreds of thousands of patients, it is time to reconsider whether that question is worth answering.
On the other hand, there might be a legitimate reason to plan for a binary margin of error with 90% confidence of ±2.6% or ±1.3 points, even though that would require n = 1,000 (Table 3). When the need justifies the expense, run the study.
And don’t you wish you could go back to those math tests and provide a range instead of that one number? The answer is between 30 and 300!
Read the full article on MeasuringU's Blog
20 Years of MeasuringU: Celebrating with the UX Community
Join us as we celebrate 20 years of innovation and partnership in UX Research on Thursday, August 7th, from 6:00 PM - 9:00 PM in the RiNo neighborhood of Denver.
We're excited to invite you to our anniversary happy hour - a special gathering for our valued customers, clients, and the entire UX community to connect, reminisce, and look to the future.
Expect an evening of great networking opportunities, drinks, and appetizers, along with brief remarks from Jeff Sauro and Jim Lewis to mark this milestone.
Please register on our website by August 1st so we can finalize arrangements. Spots are limited and will be open to the general public July 9th. Full event details will be provided after registration.
Click here to register: MeasuringU 20th Anniversary UX Happy Hour
This is an awesome reference, definitely going to refer back to this in the future.