Writing an engaging, pointed questionnaire designed for today’s online respondent is only step one in setting your online survey up for success. Step two is defining your sample design.
What is a sample design? Your sample design is the framework for the audience you’ll invite to participate in your online survey and how you’ll qualify them to achieve the desired data output. This includes aspects like sampling methodology, sources, quota design and qualifying criteria.
Optimizing the sample design of your project is a step that shouldn’t be overlooked. It often results in cost-savings, improved representation for your research, faster turn-around and better-quality data.
Here are five statistics to be aware of and advice on how best to utilise them to direct your framework. So that you get the most from your online survey projects.
50% of people who screen-out from a survey are doing so from only 10% of studies where less than 20% qualify
How do you improve incidence rates when you want to interview a niche audience?
Utilize respondent profiling information. Profilers can help identify the best sample population for your survey, thus reducing the number of respondents you invite but don’t qualify to complete your survey.
For example, if you’re looking for people who buy grain-free cat food, you can target those on a panel who are identified as cat owners. Similarly, if you’re looking for people who own a particular electric bicycle brand, you might try targeting specific urban regions paired with an income threshold. This type of targeting will improve the rate your audience will qualify, and in turn, reduce the number of screen-outs and cost of each interview.
Pre-profiling not only increases the qualification rate of a survey, it can also speed up fieldwork time – making your overall project more efficient.
30%+ of people who start a survey are screened out before they can complete it
How do you improve completion rates?
Screen-outs are a by-product of the research process. While some are not avoidable, there are others that can be avoided. For example; Industry screening questions should not be used as standard. Use them in exceptional circumstances when it’s critical for your research. If they are necessary, we recommend making the industry specific and allowing a maximum of 5% through.
Category screener questions (i.e. “Have you taken part in a beverage study in past 3 months?”) should be reconsidered. Instead, exclude people who participated in research you’ve already conducted.
Depending on the sample source and supplier you worked with previously, you can exclude inviting past research participants from your sample population in future studies.
And as a best practice for online survey research, don’t terminate respondents with the category screener, “have you taken part in any research in past 3 months?”. If that’s a necessary requirement, consider if online is the best platform for your research.
70%+ of online respondents are recruited through a mobile device
How do you capture a mobile audience?
Ensure your sampling plan does not block mobile starts or have caps on the percentage of mobile completes. This is critical to gain a representative sample that reflects todays mobile-connected world.
The survey itself must be compatible with mobile devices. For best results, it should be designed for a mobile user’s optimal experience. This means each question should be optimized for the device rather than simply compatible with it.
Beyond sample design, make sure mobile respondents will stick with your survey through completion. Design your surveys to be fast, frictionless and enjoyable. Most respondents (75%) have told us 20 minutes is too long for a survey.
If people are bored, disinterested or distracted then the data you get back will a reflection of that. When designing, think like a respondent and ask yourself: “would I take this entire survey?”.
The optimal length is 15 minutes. That said, a badly designed 15-minute survey is worse than a well-designed 20-minute survey in terms of data quality. Consult a survey design and online respondent engagement expert before launching your study.
10% of online respondents who start a survey dropout before they finish
How do you lower your incomplete rates to less than 10%?
Ensure your surveys work on all devices so that the audience you invite and qualify from your sample frame can complete your survey.
Research has shown that you may miss important information from those who dropped out of a survey early.
For example, during a real concept test, it was found that those who drop out during the survey were more likely to perceive the concept being tested negatively. This means that by the end of the survey, those completing provide a more positive response towards the concept than reality. Therefore, the higher the incomplete rates, the higher the over claim on the performance of the concept.
50% of survey invites for a study are used for the last 5% of completes
How do you avoid over filling quotas?
At Kantar, we recommend applying at least a 5% quota tolerance on all quota targets in your sample design. Depending on your sample size and specifications, we often recommend increasing that to 15%.
High volumes of quota fails in a survey is a warning sign on the representivity of those who have completed. However, applying quota tolerances provide minimal risk to data and it can save 20%+ of your total sample, reducing both time and the total cost of your online research.
Without any tolerance management, collecting the last few percent of completes often turns into “finding a needle in a haystack” – thus increasing field times and introducing bias.
For example, looking to complete a project with only 18–24-year-olds earning over $100,000. Since a respondent meeting that criteria is rare, you would potentially be oversampling this group against the proportion of this group in the population.
We acknowledge that quota setting as not an exact science. It is a way of providing a reasonable spread across key demographics that are believed to have some influence on results. Therefore, when you’re fielding your survey, we recommend you thoroughly review your quota tolerances established in your sample design and adjust where necessary as the project progresses.
Learn more
By being aware of these statistics and applying the suggested approaches to your sample designs, you’ll see improvements in your data returns.
Keen to know more? Reach out to Kantar’s expert sampling team today.
Sign up using the form below to receive research tips on the first Thursday of every month. Topics include design, engagement, sampling and data enhancement. You’ll also be the first to hear when new Online Survey Training modules are released.
