The Trouble with Crime Statistics

It’s surprisingly hard to say what makes crime go up or down.
Graph pie charts and various infographics made with pills weapons and crime scenes.
Criminologists sometimes describe crime as a “chaotic system,” and countless factors contribute to it.Illustration by Nick Little

Last year, lawmakers in New Jersey and Pennsylvania proposed legalizing recreational marijuana in their states. A debate ensued. Some argued that legalizing pot would make crime go up; others claimed that it would make crime go down. There is evidence to favor the optimists: a recent paper in the Journal of Economic Behavior & Organization reports that, after Washington State legalized recreational marijuana, in 2012, rapes there decreased by as much as thirty per cent, and thefts by about twenty per cent.

And yet there are plenty of pessimists about legalization, too; many of them work in law enforcement. Curious about their views, I reached out to more than seventy-five county sheriffs in California, Colorado, Maine, Massachusetts, Michigan, Nevada, Oregon, Vermont, and Washington—states where recreational marijuana is legal. (It’s also legal in Alaska.) Of the twenty-five sheriffs who got back to me, half said they hadn’t noticed a trend, and the rest were certain that legalizing marijuana had made crime go up. “We can just tell you from our experience that any time you’re around marijuana, or the marijuana industry, the likelihood that you’ll be the victim of some type of crime is higher,” Ray Kelly, a sheriff’s sergeant in Alameda County, California, which is home to Oakland, said. Paul Bennett, a captain in the Riverside County Sheriff’s Department, in California, told me, “I can certainly say that cops in the field, on the streets, and specifically narcotics officers, have experienced an increase in violent crime, all related to marijuana trafficking, sales, and cultivation, both legal and illegal.” I asked the sheriffs about the paper in the Journal of Economic Behavior & Organization. “Whoever gave you those statistics is so full of crap that they can’t even see how ludicrous these statements are—you can quote me on that,” Kendle Allen, the sheriff of Stevens County, Washington, said. Frank Rogers, the sheriff of Okanogan County, Washington, had a different hypothesis: “Maybe when they wrote it they were indulging in a little of the green stuff themselves.”

Whether smoking marijuana causes crime is an important question. It informs opinion on whether smoking marijuana should be a crime. According to the F.B.I., there were more than six hundred thousand arrests for marijuana possession in 2018—about six per cent of all arrests nationally. Even if an arrest doesn’t lead to imprisonment, it creates a criminal record, disrupts work and family life, and piles up legal fees and other costs; the enforcement of anti-marijuana laws is, moreover, overwhelmingly focussed on poor communities of color.

Given the high stakes of the question, it’s tempting to take sides: either legalizing pot leads to more crime or it doesn’t. And yet the truth may be unknowable. “We do not have a good mechanism in place for tracking why a person commits crime,” Timothy Tannenbaum, a sheriff’s lieutenant in Washington County, Oregon, told me. “I’m not sure most of the data you seek is available.” In an e-mail, the spokesman for Sheriff Joseph McDonald, of Plymouth, Massachusetts, cautioned that “it’s often hard to identify marijuana as either the cause or the deterrent for criminal conduct.” I brought all these responses to David Weisburd, a criminologist at George Mason University. “The sheriffs raise an important question,” Weisburd said. In his view, marijuana’s effects on crime are likely to remain hazy; in fact, the effect of pretty much anything on crime is rarely crystal clear.

Certainly, we know a few things about what causes and prevents crime. The “Handbook of Crime Correlates,” from 2009, a reference book compiled by three criminologists, lists more than a hundred demographic, economic, relational, institutional, cognitive, and biological risk factors; in aggregate, they suggest that young men in hard times find trouble. A 2015 report from the Brennan Center for Justice identifies a dozen plausible explanations for the major decline in crime that unfolded across America from 1990 to 2010—among them, more police officers, a decline in alcohol consumption, a stronger economy, and the adoption of CompStat, a statistics-based approach to managing police departments, pioneered by the N.Y.P.D. But each of these factors can explain only a few per cent of the broader change. After analyzing a hundred and sixty-nine criminology studies published from 1968 to 2005, Weisburd found that, on average, each study—despite combining many variables—could explain only a third of a given change in crime. A 2018 report in the Annual Review of Criminology concluded that the findings in one out of ten crime studies couldn’t be replicated, and that another fifteen per cent were only partially replicable.

“The world is complicated,” Weisburd said. Many people are sure that they know how to reduce crime. They urge the adoption or repeal of laws based on that conviction. But crime and crime statistics are more mysterious than they seem.

The first problem with understanding crime is that measuring it is harder than it sounds. The Department of Justice approaches the problem in two ways. The F.B.I.’s Uniform Crime Reporting Program, or U.C.R., solicits data from about twenty thousand law-enforcement agencies around the country. Simultaneously, the Bureau of Justice Statistics’ National Crime Victimization Survey, or N.C.V.S., interviews about a hundred and fifty thousand nationally representative citizens, asking them whether they have been victims of a crime.

Both datasets have problems. An obvious one is that there’s no consensus about what counts as criminal activity. In some jurisdictions, only offenses worthy of incarceration are considered crimes. In others, fined infractions also count. (Is speeding a crime? What about manspreading, for which one can be fined seventy-five dollars in Los Angeles?) Because the U.C.R. draws its data from investigators, and the N.C.V.S. relies on victims, they can present starkly different pictures of crime. According to the U.C.R., the incidence of rape nearly doubled from 1973 to 1990. The N.C.V.S., by contrast, shows that it declined by around forty per cent during the same period. Researchers at Vanderbilt University looked into the discrepancy; they found that the upward trend in the U.C.R. data correlated with upticks in the number of female police officers, and with the advent of rape crisis centers and reformed investigative styles. It could be, in short, that a modernized approach to the policing of rape drastically increased the frequency with which it was reported while reducing its incidence. But coherent stories like these only sometimes emerge from the conflicting data.

In 2016, a panel convened by the National Academies of Sciences, Engineering, and Medicine on modernizing the nation’s crime statistics concluded that we need to incorporate additional data that complements the U.C.R. and N.C.V.S.—data that is more fine-grained and carefully categorized. But the panel’s chair, Janet Lauritsen, a criminologist at the University of Missouri-St. Louis, acknowledged that collating and reporting detailed statistics require resources and training that aren’t widely available in police departments. In the meantime, insights derived from one dataset often prove incompatible with the other. In 2007, for example, a study conducted by Jessica Reyes, a professor at Amherst College, concluded that the removal of lead from gasoline in the late nineteen-seventies and early nineteen-eighties could explain fifty-six per cent of the reduction in violent crime between 1992 and 2002. (Reyes drew upon a large body of literature that theorized that lead, by damaging children’s brains, had made them more likely to become criminals.) That study used U.C.R. data. When Lauritsen and her colleagues re-ran the analysis with N.C.V.S. data, they found that leaded gasoline had approximately zero effect on violent-crime trends.

Even if we had perfect measurements of crime rates, we’d need to sort correlation from causation. Drug use might be a risk factor for committing crime—a behavior that’s correlated with it, and so helps predict it. And yet drugs themselves might not cause crime; drug use and crime might both have other causes (unemployment, say, or under-policing). The connection could also be random. Criminologists sometimes describe crime as a “chaotic system,” and countless factors contribute to it.

The passage of time makes it especially difficult to sort correlation from causation. Statisticians are always on the lookout for a phenomenon they call “regression toward the mean.” If crime randomly goes up, it may soon return to average levels on its own; the increase and subsequent reduction might be a statistical hiccup. And yet a fortuitously timed intervention—a new law, say, passed in the wake of the random increase—may appear, incorrectly, to be what brought crime under control. “There’s empirical evidence for this in speeding enforcement,” Mark Kleiman, a public-policy researcher at New York University, who died in July, told me. “Tougher traffic rules tend to get enacted right after spikes in highway fatality, and the mean reversion makes it appear as though that toughness worked.” Measuring the effects of new punishments, moreover, also requires figuring out whether people know about them. “What matters for deterrence is not how much punishment actually happens but how much punishment potential offenders think will happen,” Kleiman went on. “If perception lags punishment, which presumably it does, then you’ve got another modelling mess on your hands.”

Mechanisms matter, too: in addition to knowing that something works, we want to know why it works. The authors of the Washington State marijuana paper suggest that legalizing pot could reduce crime by sedating people; by substituting for alcohol or other drugs; by diminishing the allure of black markets; and by allowing police to focus on other crimes. And yet describing mechanisms is easier than proving their existence. Criminologists have sometimes taken cues from medical research, implementing randomized controlled trials. But Robert Sampson, a sociologist at Harvard University, argues that, because police departments and neighborhoods are not laboratories, such trials are often imperfectly designed and executed. Many also fail to track the long-term effects of the anti-crime programs they study.

In some cases, decades must pass before the effects of an intervention become visible. Criminology’s first large-scale randomized controlled trial, the Cambridge-Somerville Youth Study, took place between 1939 and 1945. Researchers matched two hundred and fifty-three pairs of boys who lived in Boston-area youth homes on age, intelligence, home stability, and other factors, then tossed a coin to see who in each pair would receive counselling, tutoring, and a trip to summer camp. Initial results showed that the interventions had little effect. Then, in the nineteen-seventies, a criminologist named Joan McCord tracked down nearly all of the men. She found that the counselled men were actually more likely to have committed multiple crimes, and they had higher rates of alcoholism and mental illness.

Camp, it seemed, had backfired. (Perhaps bringing many at-risk kids together in one place had deepened their problems.) In 1992, a meta-analysis of four hundred and forty-three published studies on juvenile-delinquency programs found that a third of them had done more harm than good. Evidence suggests that D.A.R.E. and Scared Straight—modern-day programs similar to the Youth Study—may have been counterproductive, too.

In some cases, findings reverse, then reverse again. In the early nineteen-eighties, the Minneapolis Domestic Violence Experiment found that the mandatory arrest of offenders reduced the incidence of further violence against the victims by a third. Many states enacted laws requiring domestic-violence arrests. In the following decades, though, six replication studies in different cities found mixed effects; some even suggested that arrests encourage revenge against the victims. In 2002, a trio of criminologists published a meta-analysis of those replications in Criminology & Public Policy. They discovered that their colleagues in the eighties had been on the right track: the policy worked after all.

Beyond measuring crime and determining its causes, a third difficulty lies in predicting the effects of interventions. Peter Grabosky, a political scientist, wrote that “the tendency to overgeneralize” might be “the most common pitfall” in the study of anti-crime interventions: “What works in Wollongong might fail on Palm Island,” he said. Neighborhoods, cities, and states are different.

The converse is also true: crime isn’t a purely local phenomenon, and interventions in one place may affect criminal behavior in another. Legalizing pot in one state, for example, could reduce theft there while increasing drug trafficking somewhere else. “State legalization is not the same as national legalization,” Jonathan Caulkins, a public-policy professor at Carnegie Mellon University, told me. Even in a single location, decreases in one kind of crime can create increases in others. “The biggest effect of marijuana legalization has been the increase in people driving under the influence of marijuana,” Will Reichardt, the retired sheriff of Skagit County, Washington, said. Because detecting marijuana requires a blood or saliva test, which are often administered in a hospital, rather than a breathalyzer, effectively policing the roads has become more time-consuming and expensive. Such indirect effects of changes in the law are hard to predict. For this reason, Sampson argues, “practitioners (e.g., cops on the beat) may be better ‘theorists’ of what policy changes will trigger on the ground than academic criminologists who theorize at a considerable remove.”

Sampson thinks that criminologists should spend less time trying to figure out what causes crime—in many cases, it’s an impossible task—and turn, instead, to investigating the effects of law-enforcement policies. Such research might provide politicians and voters with a more contextual view of proposed crime-fighting measures. As a discipline, Sampson points out, criminology isn’t narrowly focussed on criminals and their motives; it also studies communities, the economy, and the kinds of trade-offs citizens are willing to make. In February, 2019, for example, a paper in the American Sociological Review looked at the effect of the N.Y.P.D.’s Operation Impact—a multi-year effort to increase policing in the city’s highest-crime neighborhoods—on two hundred and fifty thousand students between the ages of nine and fifteen. The study found that, although the operation probably led to a reduction in crime, it significantly reduced the test scores of African-American boys, who are stopped on the street by police officers more than any other group. Such research suggests that anti-crime interventions can’t be contemplated in isolation. They affect society as a whole.

Criminologists may disagree on questions of causality, but they agree that outsiders underestimate the complexity of criminology. “Everyone thinks they know what causes crime,” Sampson told me. Lauritsen concurred: “Everybody has a strong view that some factor is responsible, whether it’s video games, bad music, or sexist attitudes,” she said. Kleiman complained about the “very primitive models people have in their heads” when it comes to crime: “Most of those models imply that more severity of punishment is better, which is almost certainly false.” He went on, “Anyone who hasn’t studied this professionally has more confidence than they ought to have. You have to really look at it hard to see how confusing it is.”

Criminologists face a problem that’s common in many fields: overdetermination. Why does someone commit a crime? Was it peer pressure, poverty, a broken family, broken windows, bad genes, bad parenting, under-policing, leaded gasoline, Judas Priest? “You could just keep stepping back and back and back and back, and you wonder when, ultimately, you’re going to draw a line,” Lauritsen told me. “It could be drawn at probably thousands of points.” Criminologists aren’t the only researchers who study overdetermined subjects: biologists, who have long sought specific genes for diseases, have come to realize that many traits and illnesses may be “omnigenic”—determined by countless genes. The sociologist David Matza summed up the difficulty, in 1964: “When factors become too numerous, we are in the hopeless position of arguing that everything matters.”

Still, it’s human nature to prefer comprehensible stories to uninterpretable complexity. We like simple stories—and we prefer some stories over others. A 2007 study in the European Journal of Social Psychology showed that people found explanations for events such as forest fires more satisfying when those explanations involved people. (Arson may be a senseless act, and one that causes only a few per cent of wildfires, but it’s more intuitively sensible than an inferno sparked by sunlight focussed through broken glass—in part because one can punish a person, but not a bottle.) Other research shows that we shy away from explanations that entail solutions we don’t like. In 2014, researchers at Duke University asked some gun-control supporters to read an article containing fake evidence that guns often help homeowners during home invasions. After reading the article, which seemed to justify gun ownership, they lowered their estimates of how often home invasions occur. And moral principles can shape the factual beliefs on which they should be based: a 2012 study at the University of California, Irvine, found that shifting people’s feelings about the moral acceptability of capital punishment affected their beliefs about its practical effectiveness as a criminal deterrent.

Crime is morally charged, and so our stories about it are, too. When Joan McCord, the researcher who tracked down the Cambridge-Somerville men, first published her findings on the project’s failure, in 1978, she received threatening phone calls and shouted insults from people who didn’t want to believe her findings: the programs she was investigating fit a redemptive, positive narrative that people found hard to set aside. As late as 2017, the Attorney General, Jeff Sessions, was still touting the success of D.A.R.E.

Perhaps, whenever someone offers up an especially compelling explanation for a rise or fall in crime, we should be wary. We might recognize that criminology—a field with direct bearing on many charged issues—is also slow and confounding, with answers that may come decades late or not at all. The moral and social complexity of crime makes simple accounts of it all the more appealing. In hearing an explanation for its rise or fall, we might ask: What kind of story is its bearer trying to tell?