recentpopularlog in

pierredv : replication   3

Estimating the reproducibility of psychological science | Science, Aug 2015,
"Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. ... The claim that “we already know this” belies the uncertainty of scientific evidence. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both. Replication can increase certainty when findings are reproduced and promote innovation when they are not."
reproducibility  replication  psychology  ScienceMag  scientific-method 
july 2018 by pierredv
Artificial intelligence faces reproducibility crisis | Science
The booming field of artificial intelligence (AI) is grappling with a replication crisis, much like the ones that have afflicted psychology, medicine, and other fields over the past decade. Just because algorithms are based on code doesn't mean experiments are easily replicated. Far from it. Unpublished codes and a sensitivity to training conditions have made it difficult for AI researchers to reproduce many key results. That is leading to a new conscientiousness about research methods and publication protocols. Last week, at a meeting of the Association for the Advancement of Artificial Intelligence in New Orleans, Louisiana, reproducibility was on the agenda, with some teams diagnosing the problem—and one laying out tools to mitigate it."

"Odd Erik Gundersen, a computer scientist at the Norwegian University
of Science and Technology in Trondheim, reported the results of a survey of 400 algorithms presented in papers at two top AI conferences in the past few years. He found that only 6% of the presenters shared the algorithm’s code. Only a third shared the data they tested their algorithms on, and just half shared “pseudocode”—a limited summary of an algorithm."
ScienceMag  reproducibility  replication  AI 
february 2018 by pierredv
Why so much science research is flawed – and what to do about it | New Scientist issue 3069, 13 April 2016
"... publication of a paper in Science. It described a major effort to replicate 100 psychology experiments published in top journals. The success rate was little more than a third. People began to talk of a “crisis” in psychology."
“The current system does not reward replication – it often even penalizes people who want to rigorously replicate previous work,” wrote statistician John Ioannidis of Stanford University in California in a recent paper entitled “How to make more published research true”. Proponents of a new discipline called metascience (the science of science) aim to change that, and Ioannidis is in the vanguard.
"Evidence is mounting that medical research is particularly prone to irreproducibility. In 2012, Glenn Begley, a biotech consultant, showed that just 11 per cent of the preclinical cancer studies coming out of the academic pipeline that he sampled were replicable."
NewScientist  transparency  reproducibility  replication  bias 
august 2016 by pierredv

Copy this bookmark:

to read