A little more than two years ago, just when online research had become a major source of data for marketers looking to keep their fingers on the pulse of consumers, Procter & Gamble's marketing research leadership dropped a bomb: They said that two online surveys from the same provider conducted a week apart delivered diametrically opposed results.
P&G is the world's biggest buyer of survey research, so this got the industry's attention! Then similar reports came from others. Suddenly, what had been taken for granted turned into a question: "Can marketers trust online research?"
Two years ago leading marketers (imagine Procter and Unilever in the same room) and research companies formed working committees at The Advertising Research Foundation (ARF) in New York to address this showstopper. In Oct/Nov 2008, the industry supported a $1 million study, involving over 100,000 interviews among 17 different leading online panels, to understand under what conditions online research can produce inconsistent results.
The good news: Each of the 17 panels replicated their own results on benchmarking questions (like owning a home, or ever smoked) when testing the same survey a few weeks apart.
The bad news: There were big inconsistencies across the 17 panels.
Notable causes for differences:
- How long someone was on a research panel affected their answers. "Newbies" were more favorable to a new product idea than those who were on the research panel longer. Some panels are filled with newbies; some are not.
- While most fill out surveys to be heard, those who take surveys for the money turn out to be less diligent about filling out the survey. Certain research panels tend to give outright cash gifts which attract more of those people.
Because the same research supplier can actually vary where they get their respondents from (even if they have their own panel), this alone can blow data comparability out of the water. We asked buyers if they were explicitly controlling for this with their research suppliers. Mostly, they were not.
To provide needed structure to the conversation that research buyers and sellers have to manage data consistency, the ARF introduced something we are calling the "Quality Enhancement Process". QeP has a series of 10-plus templates to address how the online panel is generally recruited and run, the source of respondents for a given research program, and the survey itself.
Eight big companies are bullish on the QeP and plan to pilot test it. They are: Unilever, Coca-Cola, Microsoft, General Motors, Kraft Foods, General Mills, Bayer, and Capital One. They hope to find the path back to trust--assurance that online research results are comparable (e.g. did our new product idea get good scores?) and trendable (e.g. is my brand equity really changing?).
Joel Rubinson is Chief Research Officer at The ARF, where he directs the organization's priorities and initiatives on behalf of 400+ advertisers, advertising agencies, associations, research firms, and media companies. Joel is a frequent speaker at industry conferences and an active blogger. He holds an MBA in statistics and economics from the University of Chicago and a BS from NYU and never leaves home without his harmonica. Please visit his main blog, CRO-ing About Research, for more. Follow him on Twitter: @joelrubinson.