A Swansea University psychologist is part of a team of international academics putting research credibility to the test.
Dr Alex Jones and more than 860 colleagues from across the world have been sharing their insights in new research as part of the Systematizing Confidence in Open Research and Evidence (SCORE) programme.
It offers new empirical evidence on the reproducibility, robustness, and replicability of research across the social and behavioural sciences, and the predictability of replicability. The programme examined the capability of humans and machines to predict the replicability of research findings. Its results have just been published in Nature as a collection of three papers and five additional preprints.
The programme sampled claims from 3,900 papers published between 2009 and 2018 in 62 journals spanning criminology, economics, education, health, management, psychology, political science, sociology, and other fields.
Dr Jones, from the School of Psychology, has been involved in research around the theme of robustness, examining happens when many experienced researchers are presented with a dataset and a simple hypothesis, and, using their skills in data analysis and statistical inference, tested that hypothesis.
The authors highlight the importance of an approach called multiverse analysis, essentially running every possible analysis. By looking across the range of analyses presented, the authors found a huge range of analysis types, as well as conclusions from the same data and hypothesis.
Dr Jones contributed one of these analyses, using a type of statistical analysis called a Bayesian hierarchical model to test the original hypothesis and also reviewed other researchers’ analyses.
He said: “This project highlights the uncertainty in conclusions that arise from how statistical analyses are used in science. Experienced researchers can easily come to different conclusions when testing the same hypothesis on the same dataset. This shows just how subjective the practice of science is.
“Multiverse approaches can help quantify this subjectivity, but this work also shows the vital importance of having a clear theoretical model of our questions, which is often missing in social science.”
The programme’s results reveal that there is no single indicator of the repeatability of evidence, or research credibility more generally.
One of the SCORE project leaders Tim Errington said: “The main message of SCORE is a simple one: research is hard. And, in some ways, the hard work begins after making a discovery. A tremendous amount of work is needed to verify and have enough confidence in new discoveries to build foundations for further discovery.”
Learn more about the School of Psychology