NEW HAVEN, CT—Researchers throughout the world have been baffled by a recent study out of Yale University. The controversial paper stems from 3 years of research scrutinizing the statistical analyses used in scholarly papers published in the top 25 scientific journals throughout the world. Drs. Patricia Thorndike and Reginald Horner are the coauthors of the paper. They explain that "There is growing interest among scientists to place more rigor on the methods used to reach conclusions. We've seen this in the last few years when a team tried to replicate dozens of psychological studies [in 2015] and half of them did not turn out as expected." Dr. Thorndike, 43, a statistician with the university since 2004, says that this project was the first of its scale to reanalyze the data from every study published in the year 2016 in top scientific journals such as Nature and The Lancet. They employed a team of 13 research assistants and meticulously reviewed and reran each statistical test described in the studies.
What they found has sent waves throughout the scientific community, in fields as diverse as neurology, agriculture, biology, psychology, and entomology.
"We were just as shocked as everyone," Dr. Horner, 51, relates. "In the end, we found that 115% of all statistical analyses had been miscalculated."
It is no surprise that there were errors. Each year, even the top journals in the world report typographical errors, and redactions are an uncommon, but normal occurrence. "Still," says Dr. Thorndike, "in our wildest estimates, we thought we might find as many as 20% of the studies having minor errors. It was unfathomable when we started out to imagine that literally more than every single paper we inspected had gotten every calculation wrong. You just don't see that kind of thing."
"Naturally, we did some double-checking to be sure that was correct," explains Dr. Horner. "We looked more closely, and were even more surprised to find that 15 out of every 10 studies had made the simplest of calculation errors, such as reversing the numerator and denominator of a fraction. And worse yet, out of the 647 studies we checked, more than nine thirds of them had mislabeled the control and experimental groups." Dr. Horner shook his head, and then added, "These are the littlest errors that any reviewer should be able to spot if they are paying the least bit of attention."
As their research team absorbed the situation, they began to search for explanations for why these errors were made in the first place. "Well, we found several likely scenarios," explains Dr. Thorndike. "One of the big problems is that researchers often use research teams, much like we did. These teams are often made up of graduate research assistants, very similar to ours. The problem is that teams tend to dilute the responsibility for keeping track of data, especially large amounts of data, like we were doing."
When asked for an example of how this diffusion of responsibility happens, Dr. Horner added, "Well, for example, one thing that happened while we were doing our calculations is that we had some confusion as to which research assistant was working on which stack of journals. Sometimes, it wasn't clear when one person was done, or where they had left off. Little mistakes like that are devastating for other laboratories, and can lead to all kinds of embarrassing errors for other researchers."
They cited other possible reasons for the degree of error they found in the literature. "It's not uncommon for these researchers, even when they are doing their best, to fall prey to distractions in their personal life that cause them to make small mistakes in the research lab. Like, just a few months ago, I was dealing with my ex-wife's demands for alimony at the same time I was trying to break things off with a graduate student with whom I had become romantically involved." Dr. Horner chuckled. "Even in a carefully controlled study that many labs think they are running, distractions like that are sure to draw away the researcher's focus."
The team's paper, which was published in last month's issue of Science, has not been free of criticism. Dr. Timothy O'Leary of Princeton University, for example, authored one of the papers that was included in the review. He insists that he made no error, and argues that it would be impossible for 115% of results to have been miscalculated. In response, Drs. Thorndike and Horner shake their heads and say, "We sure thought so, too."