Research is an essential part of practically every field of study. Biology, economics, anthropology – you name it. However, research is not always perfect. Researchers sometimes focus on data that “shows what they were looking for – findings that are only true if they look at information a certain way,” according to Kevin Loria.
In the field of psychology, research is necessary to better understand human behavior and mental illnesses. But researchers in psychology have begun to face a major problem – a potential replicability crisis.
Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. Put more simply, replicability is being able to repeat a study and get consistent results. If a study is able to be replicated, its results are seen as much more reliable than the results of studies that are unable to be replicated.
A replicability crisis means that studies that are repeated may lead to different results. In order to figure out how prevalent the crisis really is, Brian Nosek (along with 270 other psychologists) repeated 100 studies published in three of the top psychology journals and published their findings in the journal Science.
Fig. 1. Science.
Originally, 97% of the studies were statistically significant. However, only 36% of the studies were still significant when repeated (Fig. 1).
The psychologists also looked at how much of an effect there was in the study. As Ed Young explains, “the effect values measure the strength of a phenomenon; if your experiment shows that red lights make people angry, the effect size tells you how much angrier they get.” The effect size changed after repeating the studies; the median effect size for most studies was less when the studies were repeated, and on average, the effect sizes of the replications were half those of the originals (Fig. 2). In some studies, there was even a negative effect, meaning that the new results were opposite those of the original study.
Fig. 2. Science.
Many are questioning the reliability of studies in psychology. While the replicability crisis is a problem, this does not mean that you should not believe in psychology at all. Importantly, failed replications do not discredit the original studies. There are numerous reasons why different results may have been produced. It could be due to random chance. The original study might be flawed, or the replicated study could be flawed. There could be differences in the people who volunteered for both experiments or differences in the way in which those experiments were conducted.
According to Nosek, “This doesn’t mean the originals are wrong or false positives. There may be other reasons why they didn’t replicate, but this does mean that we don’t understand those reasons as well as we think we do. We can’t ignore that. We have data that says: We can do better.”
The study conducted by Nosek is an important step toward understanding research in the field of psychology. While it is important to maintain a healthy level of skepticism when reading research studies, do not assume every study is a false positive.
An important factor to consider when examining psychology studies is that people’s behaviors change with generations. Their attitudes, the way they approach certain issues will change over time. We simply have to adjust our models and understandings to current data sets, and recognize that the results may change over time as new generations are born with. It certainly doesn’t mean that psychology is pointless, but warns us not to make reckless assumptions about human behavior.
Psychology isn’t the only field of study that is struggling with misleading studies—it’s hard to trust the headlines of major studies because so many have conflicting results—but much of the confusion in psychology is due to our limited understanding of the human brain. It’s amazing that we’ve landed on the moon but still can’t be sure what piece of our brain causes certain mental illnesses. There’s so much that we haven’t learned yet that researchers are bound to misinterpret things, which is why we try to replicate studies. Making mistakes is part of the learning process, so those who study psychology can’t broaden our understanding without disproving each other’s studies sometimes. It’s great to hear a reminder that science isn’t perfect and reading studies is a practice in balance. It’s important not to discount everything while also taking everything with a grain of salt.
This was a great way to approach research. I find it interesting how many different ways research can be done in the field of psychology and how there is so many different things to look into. I agree that we cannot just assume every study is a false positive because it’s done within this field. I am interested to see how psychology research will continue now with their being more of a focus on mental health and researching its affects.