Did Sturgis Motorcycle Rally Cause 260,000 Coronavirus Cases? Here Are Problems With That Study

Some things in life are simple. For example, avocado toast is tasty. Other things like the spread of the Covid-19 coronavirus pandemic are a lot more complex. That’s why estimating the number of new Covid-19 cases that resulted from a mass gathering like the 80th annual Sturgis Motorcycle Rally is not a simple thing to do.

Nevertheless, a team of economists tried to do just that and their results went viral, not exactly in a Covid-19 coronavirus way but in a social media type of way. Just look at how many re-tweets the following tweet by one of the team members got:

Holy Batman! Or in this case Holy Economist Man! Did the motorcycle rally that occurred from August 7 to August 16 in Sturgis, South Dakota, actually lead to over 250,000 new Covid-19 cases and potentially $12.2 billion in public health costs? Those certainly wouldn’t be numbers to sneeze at or cough at or diarrhea at or whatever else you might do towards them. Well, that’s what the team’s publication posted on the IZA Insitute of Labor Economics website said. And that’s what he (Friedson) said.

Certainly, you’d expect a mass gathering of people from multiple states where many people weren’t wearing face masks or practicing social distancing to produce more SARS-CoV2 infections. But how accurate are these estimates from the study?

Let’s take a closer look, to borrow the words of Seth Meyers. The team consisted of four economists: Dhaval M. Dave, a Research Fellow at Bentley University, Andrew I. Friedson an Assistant Professor of Economics at the University of Colorado, Denver, Drew McNichols, a Postdoctoral Research Fellow at the University of California San Diego, and Joseph J. Sabia, a Research Fellow San Diego State University. The study was essentially an analysis of two general sets of data.

The first set was anonymized cellphone data from SafeGraph Inc. collected between July 6 and August 30, 2020. This data allowed the researchers to determine where people were located and traveled during that time period because surprise, surprise, your cell phone can help people track where you go. Using this data, the research team could tell the origins and return destinations for people visiting Sturgis as well as track visits to restaurants, bars, stores, and other establishments in Sturgis. They could also determine what percentage of Sturgis-area residents avoided the rally and stayed at home versus continued their typical daily activities.

Analysis of this data showed that, yes indeed, lots o’ people seemed to come from different states to visit Sturgis during the motorcycle rally. At the same time, Sturgis-area residents didn’t appear to significantly change their movements, meaning that they continued to go out and potentially interact with the visitors.

The second set of data was the number of reported Covid-19 cases for each State and county from June 6, 2020 to September 2, 2020. This data came from the Centers for Disease Control and Prevention (CDC) via the Kaiser Family Foundation, the New York Times, and Johns Hopkins University. The key word here is “reported.”

Using this data, the team of economists graphed the trends of new Covid-19 cases in different states and counties before the Sturgis rally and tried to create statistical equations that matched these trend lines. They then used these equations to extend the trend lines and predict how many cases would have occurred in each location after the August 7 to August 16 time period had no Sturgis rally occurred.

Then they checked what really happened to the number of reported new Covid-19 cases after the motorcycle rally in both the Sturgis area and any counties that people seemed to return to after traveling back from the motorcycle rally. The difference between these actual numbers and the numbers predicted from the statistical equations then was supposed to give the increase in reported Covid-19 cases associated with the Sturgis rally. Here “supposed to” are the key words.

Of course, not all states have maintained the same pandemic policies such as shelter-in-place orders and mask wearing mandates because this is America in 2020 where the federal government is not coordinating the pandemic response in this manner . To address this issue, the team created and incorporated into the statistical equations a mitigation index to account for differences in such policies. This index sorted each state into either Stricter Mitigation or Weaker Mitigation categories. Not exactly a detailed representation of the specific types of precautions different people may have been taking.

The difference between actual reported Covid-19 cases and the predicted case yielded the following: 3.6 new Covid-19 cases per 1,000 population. Multiplying this out by the relevant populations resulted in a total estimate of 266,796 new cases. Multiplying this number by the estimated $46,000 cost of a Covid-19 case from one of their previous studies produced the grand total of $12.2 billion.

Sounds simple, right? Well, that’s the problem. All of these steps didn’t take into account the many complexities of the pandemic.

First of all, when considering the data used, remember the key word “reported.” There are always differences between reported infectious disease cases and actual cases just like there are differences between the number of guys who admit that they sing Celine Dion’s “My Heart will Go On” in the shower and the number of guys who actually do. But the discrepancy between reported and actual cases has been even greater for the Covid-19 coronavirus. After all, not everyone is getting tested, and testing policies and coverage seem to be quite different among different states. Moreover, delays can occur between a positive test occurring and that result being reported to the county or the state. Plus, all of this has been changing over time. Therefore, basing future predictions on past available Covid-19 data can be like a mixture of milk, ice cream, and flavorings: at best shaky.

Secondly, you can’t just take the trend in cases before a point in time and then extend the trend to predict how many cases may occur in the future. Imagine trying to do that with the stock market, the weather, your significant others’ feelings towards you, or something important like the sales of One Direction songs. What if you said to your significant other, “your feelings seemed to be growing for me when we traveled around the world and I happened to not shower much. So I decided to continue the ‘not showering much’ and assumed that your love would continue to grow.” Making such an assumption may be a bit too simplistic because other things may be growing besides love when you don’t shower much. Similarly, many other factors can affect the number of new Covid-19 cases in the future. The past does not necessarily predict the future.

In fact, if you look at the curves that appear in the publication, the predicted trend lines over time look a bit too linear. that is, too much like straight lines. In actuality, transmission dynamics are much more complex. The number of new cases doesn’t necessarily increase at the same rate over time. When enough cases build up in an area, the rate of new cases per day can shoot up as seen in New York City in March. That’s what happens when you get to the steeper part of the epidemic curve because if each new case can subsequently infect two to four new people, things can multiply pretty quickly. Thus these trend lines could have underestimated the number of cases that would have occurred without the rally and thus overestimated the number of cases caused by the rally.

Finally, the study barely scratched the surface of things that could have increased the number of new Covid-19 coronavirus in the Sturgis area and the other counties and states. The motorcycle rally isn’t the only thing that occurred in the United States during that time. Many businesses and schools were re-opening in mid-August. People may have been getting more lax with social distancing and mask use. Different pandemic mitigation strategies may have been changing in ways that weren’t fully reflected by a simple Stronger or Weaker Mitigation index. Other types of inter-state travel may have increased. Heck who knows, maybe more people were sticking their faces in urinals. OK, the last thing probably wasn’t a big factor. But you get the picture. SARS-CoV2 transmission and the course of the Covid-19 coronavirus pandemic is a lot more complex than what relatively simply statistical equations and correlations can show.

This is why it isn’t enough to say that a “model” showed something. A “model” can mean so many different things. A model can be simple or complex or something in between. It could be a statistical model that helps analyze data or a computer simulation model that actually tries to represent the mechanisms of a virus spreading and recreate what may be occurring in the real world. Or the model could be Gigi Hadid. By itself, the word model is just too broad and generic to mean much.

Therefore, when presented with model results, always look under the hood. Look for the details about the model. Determine what kind of model was developed and used and how complex it may be. After all, if someone said, “eat this, because this is food,” would you just swallow it? Nah, you’d want to know what’s actually in the “food.” Or if someone told you that a clinical trial showed that a drug is effective, wouldn’t you want to know details such as the set-up of the trial and the number and type of people enrolled? After all, a randomized, double-blinded placebo controlled trial with 30,000 people is quite different from testing something on two guys in Batman suits who happened to be sitting in the local Arby’s.

Again, this Sturgis study used a statistical model. Statistical models take data and try to figure out trends and correlations from the data. Thus, they are highly dependent on the quality of the data, really can’t prove cause-and-effect, and only really show associations. This pandemic has seen a number of such statistical models go viral, so to speak. Some have tried to predict the number of Covid-19 deaths that may occur in the near future. You may have noticed how these predictions have turned out to be wrong. That’s because past trends do not necessarily hold into the future. If you could simply use the past to predict the future, you probably could make a whole lot more money off of the stock market and buying real estate, and that relationship with your ex may have turned out quite differently.

The Sturgis rally certainly could have led to a surge in Covid-19 coronaviruses cases. But how much of a surge is still unclear. Finding the answer may be difficult. Ultimately, many aspects of life including the Covid-19 coronavirus pandemic are a lot more complex than what statistical correlations may show. Therefore, there is often a need for more complex computer models that actually represent the detailed mechanisms involved to address questions about the Covid-19 coronavirus. Unless, of course, you are talking about whether you should eat avocado toast.