A crisis of reproducibility

Today, high-ranking scientists are expected to volunteer their time to review work in their field before a journal will publish it. This reviewing  is the foundation of trust that science today is built on. But how sturdy is that foundation? Are there parts that should be done differently? Brian Uzzi is here from Northwestern University, he researches the sociology of science, and he discussed the subject with Chris Smith...

Brian - The process of peer review works very well in some ways, but there's definitely room for improvement. So peer review basically tries to do two things. One, it wants to make sure that the findings are presented and the study was done correctly. Were the right statistics used? Were the right inferences drawn? Etc. And then the other thing that peer review tries to do is to make sure that the result is reproducible, that it can be replicated; do that's something that is published today, will work for the public tomorrow, for a week from now, for a month from now, even over an entire lifetime.

Chris - And how reproducible is the science then? If the whole process is working really well, we should have really, really reproducible papers. Are they?

Brian - Well, the issue here is that we're beginning to see that scientific papers reproduce at a rate lower than expected. In psychology, economics, and some parts of medicine and biology, we're learning that about 60% of the papers do not replicate.

Chris - Goodness me! So 60% of the science, if I take a paper off the shelf and I copy what the scientist who wrote the paper did, and I try to repeat their work, I get a different answer. Is that what you're saying?

Brian - That's precisely what I'm saying. So you use exactly the same procedures, you do exactly the same experiment, but maybe you only change the subjects in the experiment - and you find out that the first result was a fluke, not a fact.

Chris - But that sounds like a disaster!

Brian - Well, this is one of the reasons why people are beginning to look at this problem in much more detail; trying to find ways in which to improve the process of science so that more papers will replicate, and trying to find ways to predict whether a paper will replicate or not before it gets into the public domain.

Chris - But do we know why this is happening Brian? Has anyone unpicked the process that's leading to a paper just not working? Is it deceit on the part of scientists? Are they just publishing fraudulent science, or is something else going on?

Brian - Good question. The first thing that people looked at was, is this being driven by deceit? And there's very little evidence that it is. It appears that these are honest mistakes that are occurring in the research process itself. Now what you have to remember is: some papers will not replicate, you can't expect a hundred percent replication. Science is an innovative field; you're going to have experiments that end up on the cutting room floor, so to speak. But 60% is too high. And currently people are trying to bring that level down by understanding better the research process itself, and training people to learn how to make sure that their research replicates before they submit it for peer review.

Chris - Who are the worst culprits?

Brian - Currently, we can't give an answer to that because we haven't really begun to look at replication in all branches of science. Science is an amazingly diverse area of study, everything from the very hard sciences where you might look at, like, power station reliability; all the way over to mental health. What we do know is that the areas that we have looked at so far do vary in their levels of replication. So psychology and economics, medicine, some areas of biology - up to 60% of the papers appear to be failing. In other areas like engineering it appears to be lower.

Chris - How are you actually studying this? And I have to put it to you, is your research reproducible?

Brian - I could answer one of those questions! No, just joking. One way to do this is to improve procedures that scientists would use to make sure that their papers will replicate. Another way to do it is to develop procedures that allow someone who is reviewing someone else's paper to know whether the paper will replicate or not. My approach to that has been to develop an artificial intelligence system that reads papers, finds cues in the papers that human beings otherwise miss when they're reviewing the paper, but the AI system can tell us correlate with an accurate prediction about whether the paper will or will not replicate.

Chris - And does it work?

Brian - It works approximately 80% of the time. But the important thing is that for the papers that it feels most sure about - because the artificial intelligence system gives you a level of confidence in its prediction - for the top 10% of its most confident papers, it's right a hundred percent of the time. So that gives you real security in knowing that if you were to go to these papers and use them for whatever - investment purposes, what to build on in future research - you can be fairly convincingly secure that you're building on a fact, not a fluke  in the paper.