Trust and Critical Thinking in Science Reporting: A Case Study
If you’ve been paying attention, you’ve heard me say before that I’m not a science blogger. However, over the weekend, I authored a guest post that was not merely science blogging but also blogging on a peer-reviewed publication. I wasn’t thinking about it at the time, but it was an opportunity to apply some of my thoughts regarding my upcoming session on Trust and Critical Thinking for ScienceOnline, which seeks ideas on how to report science in a way that teaches readers to interact with information skeptically.
Given that, I thought I’d capture what I set out to do in my post. Mind you, all these strategies involve modeling critical thinking. I have no data on how effective modeling may be, but it’s the best idea I have right now and it’s fairly easy to do as a writer.
Use the Controversy
This is something that a lot of science writers do. Controversy is conflict is the basis of story. Stories order information, making it more accessible, and stories get remembered. I only hope I did it right.
I used the conflict between those who want us to believe that IQ testing differences between racial categories is indicative of some underlying, immutable, fundamental difference between races and those who find the concept abhorrent. I also used the conflict between a researcher who showed up to tell a bunch of science geeks and some scientists that they were incompetent to understand his field and all the people he stepped on. I used the first so that people would know they were dealing with competing claims that would have to be analyzed and the second to find a study that people would be interested in analyzing.
What I didn’t do: I didn’t generate controversy where none existed (except by writing one of the posts to which said researcher objected). I didn’t report on the controversy (they say this, but they say that). I didn’t suggest the researcher had any political reason to produce the results he did, because I don’t have any way of knowing his motivation.
Check the Tools
The first thing I noticed when the author recommended his paper was that he was using a tool (reaction time testing) that I had seen used for intrapersonal testing (looking at the effect of situational variables) but not interpersonal testing (looking at the effect of variables intrinsic to the person). So I read up on the tool.
It turns out that I was mostly correct. The majority of uses for the tool involve things like attentional priming and measures of distraction, although some trends in individual differences due to age and sex have been found. A good chunk of my post is giving the reader a summary of the background needed to understand the use of this tool, as well as resources for further understanding.
Check the Controls
Once I understood how the tool was used and what results it had produced in the past, I understood what variables affected it. I saw that age and sex had been controlled for and noted that in the post. I also noted some that could plausibly also vary with race and noted that they hadn’t been measured, much less controlled for.
Check the Claims
This was where things really fell apart and where I think much reporting of scientific findings falls apart. The researcher was making claims online about what his study proved that weren’t part of the Discussion section of the paper and weren’t supported by the citations in the paper. That didn’t mean they were wrong, but it did mean they were well worth investigating.
In the end, I contrasted the study’s findings with the researcher’s assertions by setting them next to each other. I presented the strongest support I could find in the literature for the leap being made by the researcher and explained where and why it still fell short of bridging the gap.
Untangle the Logic
This one came up in the discussion on my post. Someone asked me to evaluate the overall evidence for there being genetic differences between races that lead to differences in intelligence as measured by IQ tests. I think this person was looking for a simple summation.
Instead, they got an explanation of why it isn’t a simple question, as I broke the large hypothesis down into smaller hypotheses that would each, individually, need to be proved in order to prove the large one. I identified six, but there are almost certainly more. Making the steps explicit hopefully exposed some of the leaps of logic required by those who still say that “of course” these differences are real and genetic.
Identify the Biases
Also in the comments, someone noted that I was setting a high bar for evidence on this particular topic. I agreed, noting it wasn’t entirely an academic question, but I also pointed out that I was wary of accepting weak evidence because we’ve identified cognitive biases that make us more likely to believe the race/IQ hypothesis instead of the appropriate null hypothesis, which is that there is no connection.
We make a whole host of attributional errors on a regular basis. That is to say, we are much better at seeing how environment affects us than others and groups of which we’re a part than those we’re not. In each case, we’re more likely to look at “the other” and ascribe behavior to fundamental features of the other instead of to environmental factors. Race is one of the mostly highly “othering” factors in our society, and I pointed out that counteracting that bias (not even a one-race-good, other-race-bad bias) requires a great deal of skepticism.
All right, despite what I said above, this one doesn’t involve modeling critical thinking. There is a statement toward the end of the Discussion of the paper I blogged on that is pure assertion without experimental support. Nothing in the study addressed the question, and there was no citation.
I didn’t point it out. I don’t know how many people will read the paper in full, but those who do will have enough information after my post to have a little moment of discovery of their own when they read that. They will have figured out for themselves that something is wrong. I hope they find that as rewarding as I do and that it offers encouragement to continue thinking critically.
Okay, that’s it for my ideas. For those of you who read my guest post, were these strategies effectively modeled? And more importantly, did you identify the Easter egg statement in the original paper?
This entry was posted on Monday, December 28th, 2009 at 6:23 pm and is filed under Science, Stephanie Zvan. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.