In a development psychology course, the professor told us to find two things:
1) A newspaper article about a research study in child development.
2) The actual study itself (written up in an academic research journal).
We then had to do the following:
– Read both the newspaper article and the research paper.
– Evaluate the strengths and flaws of the study. (Some examples: How did the researchers select the sample, and was the sample size too small? How did the researchers define the concepts, phenomena, or behavior they were studying? What were the weaknesses in the statistical analyses?)
– Note discrepancies between the claims made in the study and the way the newspaper article reported these claims.
This was an eye-opening assignment. It helped show me the profound effects of study design and statistics on research findings. And how newspaper articles can misrepresent these findings, usually in the headline and opening paragraph of the article – the parts needed for grabbing attention through bold claims. Also, the parts people usually don’t read past.
I recommend this as an exercise in critical thinking. Research papers are often behind paywalls, but not always (sometimes, a professor will have a copy on their site). And if you’re already a college or graduate student, you may be able to access journal papers for free using school library privileges.
In a class I took a few years ago, the professor assigned readings every week and instructed the students to come up with some comments or discussion questions in response. The readings were primarily research articles in psychology and neuroscience.
At one point the professor brought to our attention that most of the time our comments were negative and critical. “The researchers could’ve done XYZ but they didn’t” or “You can’t use an ANOVA for these data, can you?” or “They didn’t perfectly control for XYZ so their results are less conclusive.” These comments usually weren’t followed up on with alternate suggestions, so the professor would try to coax them out of people. “How would you have improved on the study?” she’d ask. But what’s more, she wanted a substantive discussion of the bigger picture questions. She started to demand more questions larger in scope and accompanied by people’s own ideas for experiments. The discussion had a different intensity then, more energetic and thought-provoking than when students just sat around picking at other people’s work.
Don’t get me wrong. It’s important to pick apart ideas and recognize a study’s limitations and flaws, whatever they happen to be: abuse of statistics, a poorly chosen subject population, a set of conclusions that’s too bold given the relatively weak results. That’s all a necessary part of critical thinking. Regardless of whether you’re a scientist or not you need to be able to evaluate people’s claims and see what merit they have.
But it’s also important to think in a more positive sense – generating ideas, asking questions, relating one topic to another and considering the implications of different findings. Fewer student comments referred to the strengths of any given study, only the weaknesses.
I remember at the time thinking of why negative remarks naturally dominated our discussions until the professor stepped in:
- We were afraid to look stupid. If we offered our own ideas they could get shot down and maybe show the workings of an immature mind. What did we know? We didn’t want to take risks. Picking at other people’s mistakes protected us from the most part from criticism, and this was important because we worried too much about what others thought of us.
- Some of us wanted to look like hotshots in a game of one-upmanship. It was less about the research, more about scoring points off of other people.
- We were emulating certain professors. The one who ran the class wasn’t like this, but over the years I’ve known other professors who liked to devote their seminars to shredding the work of academic rivals in a mix of scholarly rigor and personal enmity (recently I watched a movie that explores this toxic mix).
- We were on the receiving end of frequent critical evaluation, sometimes of a very negative kind, so we liked being able to dish it out. It gave us a feeling of power.
- Making small focused negative remarks took less effort than also trying to think of solutions or come up with new ideas or questions to investigate. Granted, our critical thinking, even if it was mostly negative criticism, took more mental effort than just blindly accepting or rejecting something without justification; we did our homework. But for lack of time, training, knowledge or willingness to put in the effort, we stuck to picking things apart.
One reason I respected the professor who taught that class was her balanced approach to criticizing other people’s work. She looked for flaws, but also for possibilities. She encouraged debate and discussion but didn’t permit nasty remarks. The idea was that we were supposed to take risks, and think more widely and broadly than a purely negative approach would allow, while also being perceptive enough to delve into the nitty-gritty details of a research study and understand its limitations.
Staying purely negative would have been a safer option. In playing the part of ‘superior critic’ we wouldn’t have had to confront our fears, insecurities and weaknesses as much, or take as many risks. And the discussions wouldn’t have been nearly as productive and inspiring.