Lies, Statistics, and Peer-Reviewed Journals

Let's talk for a moment about the article by Nicholas DiGiuseppe and Bonnie Nardi that discusses character selection in world of warcraft(1). The discussion is interesting and the conclusions are theoretically intriguing, however the whole thing is a load of balogna. I'm not saying that I disagree with their conclusions. They could be right. Here is the problem I have, the data the paper is based on is worthless. "Why do you say that?" or "How can their data be wrong in a peer reviewed journal?" you may ask?

Here is why it is wrong.

Determination of gender was based, in part, on a highly fallible (and stereotype laden) method. Listening to the sound of the player's voice is hardly a reliable method for determining gender. But that's not even the worst error made in this supposedly peer reviewed article (What does this say about the peers?). From a statistical viewpoint there is insufficient data to draw any conclusions about the overall population. A sample of several hundred is needed in order to determine the percentage of characters chosen. Even if we accept that error (8%, plus or minus) isn't that bad, the breakdown for male vs female in each character class is unacceptable. In order to be about 70% confident that our results are within 10% of reality (For you non statistics people, this level of confidence is only slightly better then using a coin toss to pick the right answer on a true false test.) a sample of 25 individuals for each character type needs to be surveyed. The most people sampled of any class in this article is 7!

Go ahead, check my math...

So this article is worthless. What's worse is that it supposedly past a peer review. This article provides evidence of the reasons to use wiki based reference. When you limit who can review an article you are limiting what errors can be found. By opening up who can review an article you can actually increase the reliability of information presented. Academia doesn't seem to like this since I've heard countless professors say that wikipedia can't be used as a source. However it's doubtful that the type of egregious errors in this article would have lasted long on a site like wikipedia. </rant>

(1) DiGiuseppe, Nicholas, and Bonnie Nardi. "Real Genders Choose Fantasy Characters: Class Choice in World of Warcraft." First Monday 12.5 (2007). Print.