Monday, April 10, 2006

That's one messed-up scale

Ahem. Getting back to marketing research, now.

I received an e-mail at work last week, inviting me to participate in the "Research Industry Trends 2006" survey. It was being sponsored in part by the NY-AMA (American Marketing Association), publishers of the GreenBook Marketing Research Directory, with which obviously I am familiar. Addtional sponsors were DialTek (vaguely familiar), Pioneer Marketing Research (not familiar), and the mysteriously named Rockhopper Research.

I almost always participate in surveys directed at our own research industry. I like to voice my opinions (obviously), and I'm always hopeful that a top-line summary of the research might be made available to me as "compensation". I think our marketing research industry could use some more objective research about what makes the most successful companies hum. What are clients looking for? Is the industry truly polarizing to the extremes of crude data collection (order takers) offset by consultative strategic partners, leaving those in the middle on tenuous ground?

But then I get a survey like this, and I'm just disappointed. Take a look at the following graphic (click it to open in a larger view), a screen-shot of one question in the study that made me shake my head:

What was bothersome about this question? Let me count the ways!

First, the respondent is supposed to have a strong enough idea about the growth potential in each of ten different global regions to be able to form an opinion about each. You're not allowed to "Don't Know", because that's just not an option.

Second, what if I anticipate that there will be a decline in one of the areas? Again, that's just not an option. I guess the survey authors felt that there couldn't possibly be a retrenchment in research anywhere in the world, even the Middle East. I'm sure they'll just be pouring out the marketing research studies over in Baghdad throughout 2006.

Third, look at the five-point scale. At the top end, a "5" reports "Very High Growth" (nevermind that we seem to be forgetting about "potential" any more). At the bottom end, there is "1" anticipating "Little to No Growth". So, what would you expect in the middle, at point "3"? The survey says "No Change". WHAT?! If you're going to have "No Change" in the middle of the scale, then 4 and 5 should be positive change, but 2 and 1 need to be negative change. They're not! It's all growth, growth, growth.

Fourth, I have to marvel that, when a question is this patently awful in design and execution, how did it POSSIBLY pass through the quality checking process of (presumably) all FOUR sponsoring research entities? Were they all asleep at the wheel, or was this the disasterous result of a project by committee, where this rating scale was the queer result of some kind of compromise among disagreeing sponsors?

It just looks bad for them, either way.

That's my criticism. How about making it constructive? What should they do now? In my opinion, the study should be pulled from the web and sent through a more thorough review process. Additional completes then can be solicited on a second, better version. Heck, with the right mea culpa, those who already completed the survey could be invited to take another crack at it.

I would always rather have 50 completed interviews on a well-designed question than 150 completed interviews on one that is seriously flawed. I guess I'll notify GreenBook, DialTek, Pioneer, and Rockhopper about this now.

Tags: , , , , , , ,


At 3:59 AM, May 11, 2006, Anonymous Dotan Mazor said...

Just wondering. You said: "I guess I'll notify GreenBook, DialTek, Pioneer, and Rockhopper about this now"

Well, tell us what evolved. That may prove to be even more interesting than the mistakes done in the survey ;-)


At 1:50 PM, August 03, 2006, Blogger O-Maps said...

Come on! let's hear it!

At 3:38 PM, August 03, 2006, Blogger Gregory Kohs said...

As our parents told us when we were growing up . . . "If you can't say something nice about someone, don't say anything at all."

The party I spoke to about the problems with the survey seemed to agree that the questionnaire had flaws, but pretty much took a "it's too late to do anything about it now" attitude.

At 6:55 AM, July 22, 2008, Anonymous Adrian said...

Great analysis, Gregory! I agree with you completely!

But on the bottom line, why on Earth they asked growth rate to the respondents? Were they asking respondents' opinion? I think they wanted to know the real rate, not respondents' opinion. Therefore, I think they should use secondary data, not respondents' opinion.

At 11:21 AM, December 21, 2010, Anonymous Arthur Weiss said...

The problem isn't just the scaling questions but also the geographies on the left of the question. What happens if you are in Australia or New Zealand or South Africa (or Kenya, Nigeria...)? What countries represent the Pacific Rim versus the Far East. If you are based in Egypt are you in the Middle East or North Africa (or both). Is the Czech Republic in Eastern Europe, Central Europe as well as Europe? Or should you include it for all 3 areas....


Post a Comment

<< Home