The 'Seven Veils' of Research
Monday, 07 September 2009
The 'Seven Veils' is a dance elaborating on a biblical tale.
Panels are a completely new way of doing marketing research. Just as probability sampling has always been a convenient way of finding out how wrong our estimates are, so panels have always been a rigorous way of measuring cause and effect. The difference is that panels have only recently become convenient through the advance of technology. There seem to be a lot of good reasons why panels should be run on a probability sampling basis, thereby combining the best of both scientific worlds. The goal would be the rigour and reliability of clinical trials in the marketing environment in a much shorter turnaround time and at a much lower cost.
Why is this not happening? SA is about 8 years behind the UK in adopting this approach, and more than 15 years behind the US. Here, we visit 7 factors that veil the adoption of this most scientific, quick, clever and inexpensive methodology.
The 'lie' about Internet penetration.
AMPS has been the main instrument upon which research agencies and other commentators have depended. AMPS continues to ‘show’ very slow penetration in the middle and lower LSM’s. Actually, that’s not true! AMPS actually reveals (to those who use it properly) that access to Internet has dramatically increased over the past 4 years.
This has profound implications for existing research practitioners and the way that market research could (should) develop in South Africa.
Panel Services Africa (initially through Marketing Science (Pty) Ltd) have been working for the past 4 years at perfecting the application of these ‘new’ procedures and processes in the practice of marketing research.
So how do we run our panels? And, why...
Our panels are run strictly according to the ESOMAR guidelines and (according to our information and our competitors' websites) we are the only company in South Africa following those guidelines. This means that...
Veil 1: The panel is never static, so panel size is a misleading measure.
Lifting the veil
- We remove panellists who do not adequately comply with our panel relationship measures.
- We recruit constantly through face2face, advertising in print and radio, and our panels grow through referrals. We target our recruitment efforts to segments that we have the most requirement for. In fact, we run different panels depending on the degree of specialisation that we may be asked to provide.
Veil 2: 'Panel profile' has limited usefulness in evaluating panels in a similar way that size is not a useful measure.
- We sample randomly, or on quota controls (usually demographics), as well as screeners where we have the relevant information from our panellists.
- It is a statistical reality that the size of a sample is independent of the size of a sampled universe and is only related to the variance in the universe. So, sampling across panels has the same effect as stratification in probability sampling designs.
Veil 3: Unique stringent metrics are used.
- Compliance rates, being the percentage of completed surveys to which a panellist has been invited, is used to manage the panel and reward panellists who participate regularly. No other methodology has a similar measure that assesses the extent to which the researcher can be persuaded that the respondent is not providing garbage.
- Reliability is measured regularly in our panels to evaluate the extent to which respondents give the same answers at different times to dimensions that are unlikely to have changed. Reliability is a scientific requirement for assessing validity – the nearest procedure in other methodologies is 'back checking' – it is completely absent in qualitative market research.
Veil 4: Relationship management
A fundamental point of departure between panels and probability and other kinds of surveys is the respect afforded to the respondent. 'Typical' market research is only interested in the information contained in the questionnaire. Respondents are, at best, treated as objects to be measured and discarded. This is a serious disservice that probability theory has visited on the marketing research community. Primarily, the reason is logistical – it is expensive to find exactly the same person more than once (ask any honest researcher - back-checks are notoriously difficult and expensive). More insultingly, the truth is that the researcher has not attempted to establish a relationship with the respondent, has not provided a reward of recognition to them as sentient beings and often does not even bother to answer questions that may be directed to the research organisation.
We, on the other hand, employ a large and skilled department of people (tellingly, 4 times the size of the sales department) to manage the relationship with panellists, and also to craft communications and nurture cultures that panellists can relate to. This requires ongoing daily communication, constant revision to questionnaires, invitations, reminders, websites and many more skills that are absent in research organisations.
Henk Pretorious covered many of these issues from a psychology measurement point of view in his recent excellent paper. This SAMRA 2009 'Best Paper' award winner should be required reading for all researchers.
Veil 5: Incentives
- We have learned from a number of empirical experiments that lotteries, prize draws and other 'chance' based rewards not only damage the credibility of research and the integrity of the panel, but directly affect profiles of respondents as well as response rates.
- We maintain response rates of over 60% and reach as high as 90% as a result of a deliberate policy of paying panellists cash for every survey they complete. It's worth mentioning in passing that response rate is therefore not a useful metric in relation to panels, and seems to measure the incentive used and not the panel effectiveness.
Veil 6:. Conditioning
A common criticism of panels is that they condition respondents in a way that may influence the results. Yes, that is the point of panels. The point of panels is to get rid of the sampling bias and error, to get rid of the interviewer bias and error, to get rid of the procedural bias and error. In fact, the point is to get rid of the logistics of research and focus on enabling the facts to be revealed.
Veil 7:. Sensitivity to change
Lift the last veil to reveal all! We rarely measure to quantify, with quantification in mind. Even Censii realise that their measures are imperfect. However, when we compare different measures then we are measuring change – it just so happens that we are measuring the same imperfect construct the same way. Let’s call this tracking research where we measure the same things repeatedly.
The presumption of such research is that changes detected are a consequence of changes in the marketplace. The fiction and foolishness of this is manifest when considering the following two points:
- Sampling is rarely probability based – there simply are no sampling frames sufficiently well maintained. The only reason why this matters is that if sampling was random, it would be possible to determine how wrong each measure was likely to be from the previous wave. Therefore probabalistic significance tests would provide a crutch to the researcher. If the sampling is quota based, as it usually is, then it is anybody's guess as to what has caused any change in the measures. Nobody can know or estimate.
- Dr Jannie Hofmeyr of Conversion model fame taught us 20 years ago that the Chaos theory (aka ‘The Butterfly Effect’) is far superior in predicting change than trend based measurement. Also known as the ‘Importance of Insignificant Events’, the principle exhorts its disciples to research the outliers not the averages. Tracking research focuses on the averages and creates the opportunity to obfuscate rather than reveal opportunities.
Panel research has been conducted for many, many years - longer than market research has existed. However, it is only recently that the processes, procedures, theory, mathematics and technology has permitted it’s rapid, inexpensive, accurate utilisation in the marketing context.
The road we've taken is a long and arduous one. As in all endeavours, there are the sycophants to the existing order and the sceptics. Our goal remains to bring the rigour and reliability of clinical trials in much shorter turnaround times and much lower cost to the marketing environment.
