5 Crucial Steps to Choosing a Panel Provider
When running a research study the audience form the foundation of quality analysis. Typically researchers need to work with panel providers to find quality respondents to take their surveys. There are many challenges of working with panel providers which can cause bias such as their coverage of your target population, different rates of response amongst sub-groups of respondents, duplicate responses and attrition rates. The good news is that lots of great analysts have studied this before us and there’s plenty we can do to ensure a quality study.
AdExperiments recommend these 5 steps when choosing your panel provider:
Define your audience
It’s important to view the panelists in the context of your business or subject matter. You might just care about a national representative sample but are there other things that you know about your audience composition? Basic demographics are the “go to” parameters to assess but perhaps there are additional audience definitions of importance to you - High earners? City dwellers? Or perhaps you have already defined a bespoke audience.
Ask for their response to "ESOMAR’s 28 Questions"
These are a set of questions aligned with best practices for panel recruitment, maintenance and fielding. Their recommendations are driven by experiments conducted by various research bodies that seek to understand panel quality. All experienced panel providers should have a copy of these to hand. Pay attention in particular to the profiling information held by the provider and how well this corresponds to your identified audience in step 1.
Test them out
Whatever is promised to you, the reality may well be different. Run a quick survey to sample a few hundred respondents and see how well their sample matches national representation. If you do have a bespoke audience criteria that is not captured by the panelists profile data, ask a question which specifically tests for this. Ideally you should have a reference point of the expected incidence rate of this audience in the true population which you can then compare to the test sample.
Choose only one
At least per a market. If you’re running multiple studies and want to avoid duplicate responses this is a must as most respondents sign up to multiple panel providers. Using multiple suppliers will mean losing the ability to de-duplicate.
Compensate for quality
Everyone likes a bargain but bear in mind that panel respondents are typically exchanging their time for an incentive of some kind (reward points or cash). A poorer incentive has been proven to make for a poorer quality sample. As a rough guide the panel provider should be rewarding the respondent something roughly equivalent to normalised market minimum wage, calculated by the median survey time. So whilst it is always a good idea to source multiple quotes and negotiate - if the price looks to good to be true, it probably is.
We hope that helps and if you need help designing and running a study get in touch.
Our own full process for choosing our panel providers globally for our digital creative testing platform can be found here.