Some researchers were outright hostile to the idea that independent scientists wanted to attempt to replicate their work, executive director at the for Open Science and a coauthor on both studies. That attitude is a product of a research culture that values innovation over replication, and that prizes the academic publish-or-perish system over cooperation and data sharing, Nose says.
Some scientists may feel threatened by replication because it is uncommon. “If replication is normal and routine, people wouldn’t see it as a threat,” Nose says. But replication may also feel intimidating because scientists’ livelihoods and even identities are often so deeply rooted in their findings, he says. “Publication is the currency of advancement, a key reward that turns into chances for funding, chances for a job and chances for keeping that job,” Nose says. “Replication doesn’t fit neatly into that rewards system.”
Even authors who wanted to help couldn’t always share their data for various reasons, including lost hard drives or intellectual property restrictions or data that only former graduate students had.
Calls from some experts about science’s “reproducibility crisis” have been growing for years, perhaps most notably in psychology (SN: 8/27/18). Then in 2011 and 2012, pharmaceutical companies Bayer and Amgen reported difficulties in replicating findings from preclinical biomedical research.
But not everyone agrees on solutions, including whether replication of key experiments is actually useful or possible, or even what exactly is wrong with the way science is done or what needs to improve (SN: 1/13/15).
At least one clear, actionable conclusion emerged from the new findings, says Yvette Sugar, director of science policy at the Federation of American Societies for Experimental Biology. That’s the need to provide scientists with as much opportunity as possible to explain exactly how they conducted their research.
“Scientists should aspire to include as much information about their experimental methods as possible to ensure understanding about results on the other side,” says Sugar, who was not involved in the reproducibility project.
Ultimately, if science is to be a self-correcting discipline, there needs to be plenty of opportunities not only for making mistakes but also for discovering those mistakes, including by replicating experiments, the project’s researchers say.
“In general, the public understands science is hard, and I think the public also understands that science is going to make errors,” Nose says. “The concern is and should be, is science efficient at catching its errors?” The cancer project’s findings don’t necessarily answer that question, but they do highlight the challenges of trying to find out.Ultimately, if science is to be a self-correcting discipline, there needs to be plenty of opportunities not only for making mistakes but also for discovering those mistakes, including by replicating experiments, the project’s researchers say.Ultimately, if science is to be a self-correcting discipline, there needs to be plenty of opportunities not only for making mistakes but also for discovering those mistakes, including by replicating experiments, the project’s researchers say.Some researchers were outright hostile to the idea that independent scientists wanted to attempt to replicate their work, executive director at the for Open Science and a coauthor on both studies. That attitude is a product of a research culture that values innovation over replication, and that prizes the academic publish-or-perish system over cooperation and data sharing, Nose says.
Some scientists may feel threatened by replication because it is uncommon. “If replication is normal and routine, people wouldn’t see it as a threat,” Nose says. But replication may also feel intimidating because scientists’ livelihoods and even identities are often so deeply rooted in their findings, he says. “Publication is the currency of advancement, a key reward that turns into chances for funding, chances for a job and chances for keeping that job,” Nose says. “Replication doesn’t fit neatly into that rewards system.”
Even authors who wanted to help couldn’t always share their data for various reasons, including lost hard drives or intellectual property restrictions or data that only former graduate students had
You must be logged in to post a comment.