ARF foq2 Router Focus Group Report

Report on Focus Group Findings
ARF FOQ 2 Router Initiative
ABSTRACT
This report will detail the findings of a focus group conducted on behalf of the Advertising Research
Foundation on August 8, 2012 in New York City. The discussion was led by Steve Gittelman.
Those taking part in the discussion were chosen from a group of online sample users including account
managers and researchers drawn from a random sample of the top 50 Honomichl companies. Eight
companies participated in the forum. All participants met the following criteria:
1. Have direct client contact
2. Be responsible for proposing and or designing research projects that involve online sample
3. Be accountable to clients for online research projects conducted by their firms
4. Have some say in the design and online sample employed for projects
5. Have some familiarity with routers
While participants were not required to have employed routers for studies for which they were
accountable, or to have ever used routed sample, they had to be familiar with them. Also, some
participants were familiar with routers, but had rejected their use.
The subject of interest was participants’ thoughts and comments on the use of sample routers, a
technology-based approach to assigning sample to surveys. Routers typically:
1. Maximize the likelihood that anyone who wants to do a survey has the chance to do so
2. Increase the chances of filling all quotas and delivering within project schedules
3. Automate sample frame design to incorporate a set of well thought-out rules so that the
process is less ad hoc than in the past (when this was done manually, often by project managers
and in inconsistent ways)
4. Centralize decision making about how to optimize use of the available pool of respondents and
provide metrics so that decision makers are aware of how the router is performing.
Participants recognized that routers could increase the number of interviews that a sample provider can
yield, thereby lowering costs. They also understood that routers could provide a better experience for
respondents since there are fewer screen outs and a higher degree of participation in surveys – both of
which can lower respondent frustration. However, participants expressed apprehension regarding the
use of routers as the bias that routers can introduce into the sample frame remains unclear.
Although it was accepted as fact by the participants that there were benefits to be gained by the use of
routers, the researchers were alarmed by the lack of transparency from sample or technology providers

regarding their use. Many have surrendered to the concept of having sample providers manage sample
frames despite the lack of training that the technologists and sample provider project managers receive
in this area. However, when asked what diagnostic information they believed would be useful for them
in managing and overseeing the process, they expressed a desire for information but also sense of
frustration in that even if they received the information that they were asking for, they were not
confident that they knew how to use it to manage the process and alleviate any potential bias in the
sample frame.
Prepared by:
FOQ 2 Router Initiative Team, including primary authors:
Steve Gittelman, MKTG Inc.
Efrain Ribeiro, Lightspeed Research
Direct quotes from focus group participants have been reflected as well as audio recording allowed.

Discussion
Sample Sources
One of the first topics discussed was where participant companies obtained their sample. The general
consensus was that they obtained their sample from a multitude of sources and they were willing to cite
some of the names of their providers (sources). Some chose to single-source studies, allowing the
sample provider to use partners to fulfill quotas when needed. This was most evident with researchers
who were employed by large companies possessing their own internal sample sources (which may be in
a different entity within the larger company). One participant offered: “Supplier X currently handles
about 60 something percent of our volume. And then they decide on partners.”
The main domestic players cited included Lightspeed, SSI, Research Now, Federated, uSamp, GMI,
Borderless Access and M3. For international research, Research Now was cited as the principal player
mentioned.
Consistency was most frequently mentioned as a reason for selecting a sample source. “That’s a big
concern of ours, when we do our work, is [to] make sure that we have consistency in the data and what
we’re actually sourcing [… ].“
Another reason given for choosing a sample provider is control of the process. As one participant stated
“We don’t drop off our sample order and pick up our completes the next day. Our selection process is
based on partnerships that allow us to drive [...] panels in the way that we need to deliver a particular
data set to a client.”
Feasibility
Feasibility was also cited as an important factor to consider when choosing sample providers. One
participant indicated she would be “Very unhappy if my supplier can’t tell me upfront that they’re not
going to be able to achieve it [quota].” Although she admitted that she had limited experience with
routers, she believed that routers could help with feasibility.
For one participant, panel selection is based primarily on feasibility. That participant indicated that they
would rather go with a panel supplier who can complete the entire study then to have to split it up.
“Our approach is simply based on feasibility. If you know, based on the target audience […] if one panel
can do it for the life of the study then they do it simply because we haven’t seen data one way or the
other that proves or disproves that more sources [are better]. And, then based on the point of
consistency [...] is extremely important to us”.
Another participant mentioned that at the end of the day, when she needs to find the last few
respondents to finish the study, she will turn a blind eye and tell the sample provider to just finish the
job and find them. “[…] hide my eyes, close my mouth -- get me those 50 people, and I don’t care how
you do it.”

One participant expressed that router usage aids in feasibility, “[A router] undoubtedly helps with
feasibility”, but was concerned that routers can “[…] actually knock my measurements out.” Another
participant pointed out “So there are two things that routers do. One is to allow you to go to multiple
samples, multiple sources, so that’s definitely going to increase your feasibility. But also increase the
capacity of the internal panels as well because you’re optimizing people to go into certain surveys. So
everybody’s going to get a survey rather than not […].”
Types of sample used
Those companies that are not single-sourcing their online sample tend to seek diversity in the sources
they use. Similarly, those that are single-sourcing hope that the providers they use are drawing from a
diversity of sources. Some preference was expressed indicating that no single sample source dominates
the pool of sources used to fulfill the sample needs of any individual study. One concern of those
blending sources was that the same respondents do not repeat surveys. One participant stated, “When
we’re looking at sources, we’re looking at how often we have seen those people, how often they have
come in [to our survey]. And if we can apply that methodology on top of a router [...] control
[duplicates] no matter where they come from [...] you can actually control the quality you get.”
The overall sentiment of the participants was that with greater diversity came an increased likelihood
that a sample would be representative. “To me, I think you want to have as broad of a pipeline […]
beginning to end.” Companies want to use panels, but want the sourcing of the panel to be highly
diverse. They asked, “[if they} are coming on to the panel through River at first? If they are, that first
survey they answer will be different from when they get the next survey [...]. The key is to know how to
control the sources and consistently be able to put these measures into the study.” The participants all
seemed to agree with one participant’s statement that they “Think it’s much more difficult now to keep
representation than 10 years ago when everything was through email”.
Using social media as a source raised a security concern for one participant. She expressed an inability
to “train” respondents to avoid discussing the survey they had just taken with others. She feared the
possibility that Facebook participants could spread the word about the content of her surveys.
Routers
As indicated above, all participants had some degree of knowledge about of routers, while not all were
using them. All participants were in agreement that the use of routers can increase feasibility, but the
question remains “at what cost?”
Some stated that they currently use routers in their research and that they use the routers to control
the percentage of sample provided by the different sample sources and types. Those decisions along
with blending decisions are based upon the client and study. There are no hard and fast rules for these
blends; they seem to be determined on either a project by project, or company-wide basis.

One participant expressed that “Routers can be a respondent engagement tool”. It enables a
respondent who wants to take a survey to actually be able to do so. Some of the other benefits
mentioned included efficiency, cost savings and increased feasibility.
While all seemed to agree that routers had some definite benefits, there was also a great deal of
trepidation regarding their use. Trust was an issue brought up by a few of the participants. One
participant expressed that he doesn’t have trust in some of the routing providers to do it “the right
way.” Another explained that the problem she has with the routing providers is that they are not
researchers and their main priority is profit, stating “What they’re doing there -- their bottom line is to
make money based on selling the completes.”
There is also a good deal of concern with the number and types of studies that are being fed sample by a
router simultaneously. One participant was concerned by the impact that business and management
decisions of the sample provider might have on the prioritization of the studies within the router. She
felt that the router’s randomization technology worked, but if a particular study needed to be finished
and was then given a higher priority, data shifts could occur. She believed that the longer you leave a
project in field, the more random the resulting survey data would be. One participant was concerned by
the sizes of the various studies running in the router at one time and how that might impact the
“random” flow of sample. Another thought that a router would cease to be random if the percentages
of the different sample sources going through the router are changed so that one source contributed a
disproportionately large percentage of the sample flowing into the router process. She also said that
she “[…] did not realize I was buying a panel’s excess,” and concluded, “That’s horrible because it is no
longer random.”
Many of the participants expressed concern about the prescreening process and how it might impact
the survey data. The prevailing sentiment among participants was that between 5 and 6 was the
appropriate number of prescreening questions, although one participant said that “just a couple” was
appropriate. Others indicated they weren’t aware of the number of questions respondents were being
asked, but thought it was less than 10 with one stating “I mean, it might be 5-10. I can’t imagine they’re
much more than that.” There was uneasiness expressed with how many screeners a respondent would
see before they got to their survey.
Another participant wanted to know how many [full] questionnaires each respondent completed before
reaching her study. She thought she could add more screening criteria to adjust the routing assignment
and get more control. One participant looked at prescreening in an entirely different way and felt it was
good that the respondents could still take a survey if they were disqualified from a previous study. She
thought this would help avoid respondent frustration and provide a better experience for the
respondent. Yet participants were still “[…] concerned about [prescreening] taking us away from
randomization.”
Participants settled on a symbolic example that represented potential problems in a router. That
example was having a smoking and a toothpaste study running concurrently. All things being equal, one
can expect that essentially all of the population use toothpaste while a much smaller percentage of the

population smokes. An example was cited where the smoking study needed to finish and was prioritized
as the higher of the two studies. Once the respondent was pushed through the router for the smoking
study, assuming they qualified, they were removed from the sample pool for toothpaste. In a sense, the
toothpaste study could be completed by a lower number of people who smoked.
While most participants conveyed at least some concern over the hazards of using a router, none of the
participants had actually received any metrics from the sample providers that could show whether or
not their fears were justified.
Consistency
The number one concern expressed by participants was the maintenance of sample consistency. The
inability to produce a truly probabilistic sample is an inherent problem with all online sampling. The lack
of the ability to produce a probabilistic sample has left the researcher looking for some way to ensure
sample quality. The researchers in this group have turned to consistency.
They spoke about the consistency of survey data, the consistency in the blending of sample sources, and
the consistency of the sample sources themselves. They also discussed some of the methods they used
to maintain consistency. They admitted that they were not doing enough, but that having some control,
even if it is a limited, was better than no control. “That’s a big concern of ours when we do our work, is
to make sure we have that consistency in the data and what we’re actually sourcing.”
Several methods for sustaining consistency were cited. One strategy appears to be testing the
consistency of the sample coming out of the router. This method used both outside and internal
reference points, as well as consistency measures within the questionnaire as benchmarks. Brand usage
in particular seemed to be a prevalent benchmark.
A number of the participants preferred sample blends. One blend uses between 10%-15% river sample
and the remaining 85%-90% opt-in panel. These participants felt that maintaining a consistent blend of
sample sources would give them consistent survey data. One panelist cited “90% panel, 10% river […]
that’s my percentage […] as long as I know I can keep it that way, I know my data is staying consistent.”
Another thought the more sources used, the less chance there was for a sample to be biased. They
stated “[…] it is the law of averages protecting you a little bit. The more sources you have coming in, the
better off you are. If you can, you know, and using the router example again, if you could literally end
up with a million sources, then you’re better off.”
Another method discussed was the concept of pre-testing the panels before using them. While a
participant mentioned pretesting and comparing resulting survey data to sales data. He stated “The
other aspect on confidence that we have is we do validity in pretesting and in on our ongoing tracking
[...] at the end of the day they’ll take our pretesting results and they’ll compare it to our, you know, to
[...] sales data.”

One participant cites doing an analysis of his data by dividing it between different suppliers. If an
inconsistency is found, he would bring it up to the provider. He further expressed that although these
types of discussions with clients can be “difficult” on a project level, clients tended to appreciate these
types of conversations on an overall relationship level. If pretesting looks good, then they feel more
comfortable using a router [internal router] so there is more “hands on” management of the different
sources going through it.
Others are using clustering by asking the respondents both behavioral and demographic questions
during the survey. “The important thing is once you have defined those different clusters, the
consistency then from that point forward on every single study [...] what it comes down to is even on
trackers which is what Research Company X is really based off of – consistency is #1.”
Another participant described his company’s use of parallel tests -- essentially testing against a similar
study that did not use a router. “Well, in a sense, we just let them use the router to the full extent and
compared it to what we did previously when we didn’t use a router.”
While all the participants seemed to have a plan on how they are dealing with the use of routers, at
least some were willing to admit that what they were doing was probably not enough. They said that
their methods were not foolproof, but that they would rather do something than to do nothing. One
stated “So it kind of gives you a false sense of control, which I still prefer over no control.”
They were looking for some way to confirm that their methods of sustaining consistency actually
worked. They expressed their view that the needed information was either unavailable or conflicting. “I
think that’s one of our issues is that is the level of education that’s being provided by the owners of
these particular products. […] No one can come up with a valid answer to [help] alleviate some of those
concerns. We obviously have lots of people [...] that are all providing wide varieties of answers and you
know, [to] these particular questions.”
Others preferred to use a single panel as a way to maintain consistency, “I know what I’m getting. [I’m]
getting a sense of being able to replicate it.”
Metrics:
When asked which metrics they wanted to be made available to help them manage the routers and
sample sources, the participants had much to say. The key point was that the participants want to know
more about how the router technology itself works, including the sample sources being fed into them.
The problem is that they don’t know exactly what questions to ask.
Without knowing what else was happening within the router (other studies, volume, etc.) there was a
sentiment that they were unable to tackle the task themselves. One participant suggested that this was
the sample provider’s responsibility, while others thought that the panel companies were not
researchers and wouldn’t know if there was a bias being created in a survey by using a router. “They

don’t consider themselves necessarily research experts.” Another participant suggested that the sample
providers she worked with would like to do better, but needed help.
However, all could come up with at least a few metrics they wanted to know. One participant suggested
that she wanted to know “all of it,” and indicated that then she might be able to figure out what she
really needed to know, or perhaps there was so much going on that it doesn’t make any difference --
“Beats the hell out of me [...] yeah, I want to know everything. I also want to know why [...] it’s
important. Is all these different things [...] type of studies [...] does it make a difference? Maybe there’s
so much going on all the time that it doesn’t make a difference. I don’t know.”
A concern expressed was randomization vs. prioritization in the router and how the distinction impacts
the sample coming through to their study. Others wanted to know what the rules are, how many
screeners were used, how many surveys can a respondent take, prioritization, and the method for
invitation of or entry into the router. Another asked how and why the sample providers came up with
their rules, and seemed to be implying that their rules may be what works best for the sample provider’s
business model but not necessarily for the collection of quality survey data. They stated this is “… a
problem area. I don’t think you can trust the technology [provider] to come out and say here’s quality
attainable metrics that […] might influence your sample.”
Another thought “that it is too much information, and only the sample provider can manage the sample
itself, as you will never be able to know all the studies going through the router, how they impact yours,
how many are going through etc. […] is a tricky thing because you still have even with all of that
information […] there are still things that you just aren’t going to see.”
Participants want to know who their respondents are, and who are completing surveys and who are not.
Is their study going to be representative? Where is sample being sourced from? One participant
wanted some sort of certification, but wasn’t sure if that could actually be done. “I think it would be
nice if it eventually got to a point where it was standard so you could be accountable… Certification of
some sort would be even better, but I don’t know if you can get there.”
Summary
In general, the group was undecided regarding router use, and did not know if the benefits outweighed
the costs. While routers may provide benefits to respondents and projects as a whole, a better
understanding of their application and methods is needed to assess any impacts from their usage.
The group felt a responsibility to their clients to try and ensure the quality of the survey data they were
collecting, and better understanding routers would help achieve that. However, they felt compelled to
leave it up to the sample provider when it came to online sampling and the use of routers due to a lack
of understanding around sourcing and practices. The absence of knowledge complicates the situation,
and gives them even less control over the sample frame. In many cases the group didn’t even know
which questions to ask to gain such knowledge. While most seemed resigned to the fact that routers

are a technology that they are already using (and will likely need to use more in the future), some still
preferred to use a “pure panel” because they felt that they could better achieve consistency from a
known panel and its apparently less complex practices.
In the end, it was agreed that more knowledge and transparency about how routers worked was needed
by the industry. This includes an understanding of the methods used by routers, and how those
methods may or may not affect the data. Without this information, it is difficult for researchers to
manage their projects to ensure the quality of their data provided to clients.

ARF foq2 Router Focus Group Report

More Related Content

What's hot (20)

Similar to ARF foq2 Router Focus Group Report (20)

More from Federated Sample (12)

Recently uploaded (20)

ARF foq2 Router Focus Group Report