That's one messed-up scale

Ahem. Getting back to marketing research, now.

I received an e-mail at work last week, inviting me to participate in the "Research Industry Trends 2006" survey. It was being sponsored in part by the NY-AMA (American Marketing Association), publishers of the GreenBook Marketing Research Directory, with which obviously I am familiar. Addtional sponsors were DialTek (vaguely familiar), Pioneer Marketing Research (not familiar), and the mysteriously named Rockhopper Research.

I almost always participate in surveys directed at our own research industry. I like to voice my opinions (obviously), and I'm always hopeful that a top-line summary of the research might be made available to me as "compensation". I think our marketing research industry could use some more objective research about what makes the most successful companies hum. What are clients looking for? Is the industry truly polarizing to the extremes of crude data collection (order takers) offset by consultative strategic partners, leaving those in the middle on tenuous ground?

But then I get a survey like this, and I'm just disappointed. Take a look at the following graphic (click it to open in a larger view), a screen-shot of one question in the study that made me shake my head:


What was bothersome about this question? Let me count the ways!

First, the respondent is supposed to have a strong enough idea about the growth potential in each of ten different global regions to be able to form an opinion about each. You're not allowed to "Don't Know", because that's just not an option.

Second, what if I anticipate that there will be a decline in one of the areas? Again, that's just not an option. I guess the survey authors felt that there couldn't possibly be a retrenchment in research anywhere in the world, even the Middle East. I'm sure they'll just be pouring out the marketing research studies over in Baghdad throughout 2006.

Third, look at the five-point scale. At the top end, a "5" reports "Very High Growth" (nevermind that we seem to be forgetting about "potential" any more). At the bottom end, there is "1" anticipating "Little to No Growth". So, what would you expect in the middle, at point "3"? The survey says "No Change". WHAT?! If you're going to have "No Change" in the middle of the scale, then 4 and 5 should be positive change, but 2 and 1 need to be negative change. They're not! It's all growth, growth, growth.

Fourth, I have to marvel that, when a question is this patently awful in design and execution, how did it POSSIBLY pass through the quality checking process of (presumably) all FOUR sponsoring research entities? Were they all asleep at the wheel, or was this the disasterous result of a project by committee, where this rating scale was the queer result of some kind of compromise among disagreeing sponsors?

It just looks bad for them, either way.

That's my criticism. How about making it constructive? What should they do now? In my opinion, the study should be pulled from the web and sent through a more thorough review process. Additional completes then can be solicited on a second, better version. Heck, with the right mea culpa, those who already completed the survey could be invited to take another crack at it.

I would always rather have 50 completed interviews on a well-designed question than 150 completed interviews on one that is seriously flawed. I guess I'll notify GreenBook, DialTek, Pioneer, and Rockhopper about this now.


Tags: AMA, DialTek, GreenBook, marketing research, Pioneer Marketing Research, rating scale, rating scales, Research Industry Trends