Verified by 
The internet is full of psychologists arguing about whether psychology is “in a crisis” due to many results not giving the same results when they are repeated. One of the conversations that I’ve seen repeated many times goes something like this:
Reformer: Another famous effect didn’t replicate! I don’t know which results to trust anymore.
Anti-reformer: Actually, I wouldn’t expect most psychology experiments to replicate, because they all depend on so many factors specific to that cultural and historical moment in time.
My typical response is to dismiss this argument as being made in bad faith because that criticism isn’t applied outside the context of replication. For example, they don’t read new results—or report their own new results—with the caveat that we have no idea if the effect found will ever occur again because it was done in a unique historical moment. They assume that new results are interesting because they capture some general information about people that should hold in the future, not because they’re just a historical summary of a thing that happened in a lab once. An effect seen in a study might not hold all the time or for all people, but, broadly speaking, psychologists act as if they expect effects to be true when studying new people.
I feel that this is used against reformers in bad faith because it seems like the argument is raising a deep issue that all psychology research needs to address—but instead, once the reformer has shut up (because maybe they are considering the depth of the issue) the anti-reformer can just move on and continue doing the same kind of research as before, as if the point had never been made.
But what if the argument wasn’t made in bad faith? What if it wasn’t a tactic used to dismiss reformers, but an honest question about how to think about psychology? What if we actually tried going down this philosophical rabbit hole?
Last year, Prof. Phil Hammack came to my university and gave a really interesting talk about a major project he’s running that examines the sexual and gender identity across different regions of California. The project examines (among other things) the ways that queer teenagers are respond to and make sense of their communities and lives. One of the big takeaways from his talk was that masculine “macho” type ideals—not being emotional, being tougher than other guys—was something faced even by men who weren’t straight. Across many interviews, he described teenage boys saying things to the effect that it was ok to be gay, but not to be “sissy.” That’s an interesting finding, and it’s totally bound up in the historical moment. If we waited five to ten years and tried to see if this finding “replicated,” I would expect views to have changed.
Prof. Hammack’s research provides a perfect example of the way psychology—especially social psychology—can be historical. Several of his projects track the lives of LGBTQ+ people over time specifically to see how they are interacting with the broader culture. In a way, part of his research is figuring out what *isn’t* replicable—what unique changes have happened in American culture that influence the health and well-being of the LGBTQ+ community. This is valuable work, but it doesn’t neatly fit into our current narrative of what good science should look like.
One scientific approach would be to look for the “invariant features” across situations that are associated with an outcome. So maybe a particular historical moment can’t be completely captured, but specific aspects of it can be, such as media representation of figures with a certain gender identity. Maybe the effect of representation on feelings of well-being or acceptance is consistent over time, even if the specific representations change over time (e.g., Will and Jack in Will and Grace versus Mitch and Cam on Modern Family). This relationship of representations to well-being is what we hope is “invariant” across contexts; it’s something that will hold in different periods in time (and maybe even for different marginalized groups in other societies).
In trying to find invariant features, the researcher is implicitly saying that the scientific knowledge comes in what can be counted on to happen reliably. To do science on these questions successfully, the researcher needs to recognize what kind of questions can be productively addressed by the scientific process. The research process therefore may not resemble the classic formula of (1) coming up with a hypothesis, (2) designing an experiment that tests it, and (3) collecting and analyzing data to find out if the hypothesis was right. Instead, the research process might resemble what Philosopher of Science Ian Hacking calls “stabilizing the phenomenon.”
Hacking talks about stabilizing the phenomenon in terms of lab studies in physics. Researchers might observe that an interesting effect occurs when a current is run through a particular type of metal, for example. But that effect may not be easy to achieve. Sometimes it doesn’t quite work, or maybe they can’t get it to work in another lab or when another researcher operates the machine. Stabilizing the phenomenon involves seeing that interesting effect and then tinkering with all the possible things that could be influencing it, working out which are necessary to get it to work. The effect is stabilized when the researcher has worked out exactly what parts of the experimental set-up are needed to get the effect nearly every time the experiment is run.
In psychology, we might stabilize our phenomenon by running multiple versions of the same study over and over again, each time trying to vary small aspects of it to see what’s needed to get the effect. For example, a person studying the effect of bright colors on the ability of people to recognize certain words might vary text size, font, whether the words were printed or on a screen, whether the task was done alone in a room or across a table from an experiments, etc. By trying out all these different variants of the experiment, the researcher would get a sense of when the effect will occur and when it won’t.
In social psychology, Stanley Milgram’s obedience to authority experiments are a good example of this. Milgram tested whether a normal individual from off the street would be willing to shock a complete stranger to the point of severe pain (no real people were harmed, there was just a tape recorder playing sounds of a person getting shocked in the other room). Milgram tested whether having an authority telling you to shock the person in the same room or another room mattered, whether one or two authorities mattered, whether doing the study in a dingy rented room or a famous university mattered, and lots of other factors. The story we hear about in textbooks about Milgram’s work is the punchline: people were willing to shock others in response to authority. But the deeper lesson that we might want to take is that we can get people to shock others only in specific circumstances. The effect can be stabilized in certain versions of the task, but disrupted in others. Milgram stabilized his phenomenon by spending years figuring out what was and wasn’t important to get his effect to work.
So what could we do if we wanted to, for example, stabilize the effect of representation on well-being among homosexual men? We would need to find a way to examine the effect many times, using many slightly different techniques, on many different people. Ultimately, we would need to converge on a specific procedure that would consistently produce the effect (e.g., improved well-being for homosexual men). We might be able to do that in the moment, by studying how people they say right after watching a media representation, and changing around the lab set up, the person portraying in the video, the popularity of the media figure, etc. We might even be able to do it over brief time spans, by assigning people to watch certain shows in certain contexts regularly for a week or a month. But these approaches have real limitations.
Ultimately, we can’t control culture and create different versions of 2000’s cultures where Will and Grace does and doesn’t exist, or does and doesn’t get popular. There are going to be certain effects that we just can’t study through repetition in a lab setting. They rely on so many external factors, or occur on such a long time scale, or require the input of so many people, that we just can’t stabilize their effects. And so while I think that repeating experiments and stabilizing effects are hugely useful and can give us insight into a lot of different important psychological processes, I ultimately don’t think that this scientific framework can address all the questions psychologists want to ask. In the end, when we ask the good faith question of “aren’t certain effects not going to replicate due to changes in the historical moment?” I think the answer is yes. Some important parts of psychology are unavoidably historical, and so some of psychology will always be history and not science.
