Reproducibility

Our researchers' publications on reproducibility, transparency and reduction of bias and waste in research.

Reproducibility means that research data and code are made available so that others are able to reach the same results as are claimed in scientific outputs. Closely related is the concept of replicability, the act of repeating a scientific methodology to reach similar conclusions. These concepts are core elements of empirical research.

2020-

Designing, conducting, and reporting reproducible animal experiments
Wilson, E., Ramage, F.J., Wever, K.E., Sena, E.S., Macleod, M.R. and Currie, G.L. (2023) ‘Designing, conducting, and reporting reproducible animal experiments’, Journal of Endocrinology, 258(1), e220330. Available at: https://doi.org/10.1530/JOE-22-0330.
 
The Open Brain Consent : Informing research participants and obtaining consent to share brain imaging dataBannier, E., Barker, G., Borghesani, V., Broeckx, N., Clement, P., Emblem, K.E., Ghosh, S., Glerean, E., Gorgolewski, K.J., Havu, M., Halchenko, Y.O., Herholz, P., Hespel, A., Heunis, S., Hu, Y., Hu, C., Huijser, D., Iglesia Vayá, M., Jancalek, R., Katsaros, V.K., Kieseler, M., Maumet, C., Moreau, C.A., Mutsaerts, H., Oostenveld, R., Ozturk‐Isik, E., Pascual Leone Espinosa, N., Pellman, J., Pernet, C.R., Pizzini, F.B., Trbalić, A.Š., Toussaint, P., Visconti Di Oleggio Castello, M., Wang, F., Wang, C. and Zhu, H. (2021) ‘The Open Brain Consent: Informing research participants and obtaining consent to share brain imaging data’, Human Brain Mapping, 42(7), pp. 1945–1951. Available at: https://doi.org/10.1002/hbm.25351. 
From BIDS-Formatted EEG Data to Sensor-Space Group Results: A Fully Reproducible Workflow With EEGLAB and LIMO EEG
Pernet, C.R., Martinez-Cancino, R., Truong, D., Makeig, S. and Delorme, A. (2021) ‘From BIDS-Formatted EEG Data to Sensor-Space Group Results: A Fully Reproducible Workflow With EEGLAB and LIMO EEG’, Frontiers in Neuroscience, 14, p. 610388. Available at:
https://doi.org/10.3389/fnins.2020.610388
.
 
Framework for advancing rigorous research
Koroshetz, W.J., Behrman, S., Brame, C.J., Branchaw, J.L., Brown, E.N., Clark, E.A., Dockterman, D., Elm, J.J., Gay, P.L., Green, K.M., Hsi, S., Kaplitt, M.G., Kolber, B.J., Kolodkin, A.L., Lipscombe, D.,
MacLeod, M.R.
, McKinney, C.C., Munafò, M.R., Oakley, B., Olimpo, J.T., Percie Du Sert, N., Raman, I.M., Riley, C., Shelton, A.L., Uzzo, S.M., Crawford, D.C. and Silberberg, S.D. (2020) ‘Research culture: Framework for advancing rigorous research’, eLife, 9, e55915. Available at:
https://doi.org/10.7554/eLife.55915
.
 
Research Culture and Reproducibility
Munafò, M.R., Chambers, C.D., Collins, A.M., Fortunato, L. and Macleod, M.R. (2020) ‘Research culture and reproducibility’, Trends in Cognitive Sciences, 24(2), pp. 91–93. Available at:
https://doi.org/10.1016/j.tics.2019.12.002
.
 
Issues and recommendations from the OHBM COBIDAS MEEG committee for reproducible EEG and MEG researchPernet, C., Garrido, M.I., Gramfort, A., Maurits, N., Michel, C.M., Pang, E., Salmelin, R., Schoffelen, J.M., Valdes-Sosa, P.A. and Puce, A. (2020) ‘Issues and recommendations from the OHBM COBIDAS MEEG committee for reproducible EEG and MEG research’, Nature Neuroscience, 23(12), pp. 1473–1483. Available at: https://doi.org/10.1038/s41593-020-00709-0. 
UK consensus on pre-clinical vascular cognitive impairment functional outcomes assessment: questionnaire and workshop proceedings
McFall, A., Hietamies, T.M., Bernard, A., Aimable, M., Allan, S.M., Bath, P.M.,
Brezzo, G.
, Carare, R.O., Carswell, H.V., Clarkson, A.N.,
Currie, G.
, Farr, T.D.,
Fowler, J.H.
, Good, M., Hainsworth, A.H., Hall, C.,
Horsburgh, K.
, Kalaria, R., Kehoe, P., Lawrence, C., Macleod, M., McColl, B.W., McNeilly, A., Miller, A.A., Miners, S., Mok, V., O’Sullivan, M., Platt, B.,
Sena, E.S.
, Sharp, M., Strangward, P.,
Szymkowiak, S.
, Touyz, R.M., Trueman, R.C., White, C., McCabe, C., Work, L.M. and Quinn, T.J. (2020) ‘UK consensus on pre-clinical vascular cognitive impairment functional outcomes assessment: Questionnaire and workshop proceedings’, Journal of Cerebral Blood Flow & Metabolism, 40(7), pp. 1402–1414. Available at:
https://doi.org/10.1177/0271678X20910552
.
 

2010-2019

TitleCitation
Longitudinal multi-centre brain imaging studies: guidelines and practical tips for accurate and reproducible imaging endpoints and data sharing
Wiseman, S.J.
,
Meijboom, R.
,
Valdés Hernández, M.D.C.
, Pernet, C.,
Sakka, E.
,
Job, D.
,
Waldman, A.D.
and
Wardlaw, J.M.
(2019) ‘Longitudinal multi-centre brain imaging studies: guidelines and practical tips for accurate and reproducible imaging endpoints and data sharing’, Trials, 20(1), p. 21. Available at:
https://doi.org/10.1186/s13063-018-3113-6
.
Reproducibility of preclinical animal research improves with heterogeneity of study samples
Voelkl, B., Vogt, L., Sena, E.S. and Würbel, H. (2018) ‘Reproducibility of preclinical animal research improves with heterogeneity of study samples’, PLOS Biology. Edited by E.-J. Wagenmakers, 16(2),  e2003693. Available at:
https://doi.org/10.1371/journal.pbio.2003693
.
Cerebrovascular reactivity measurement in cerebral small vessel disease: rationale and reproducibility of a protocol for MRI acquisition and image processing
Thrippleton, M.J.
, Shi, Y., Blair, G., Hamilton, I., Waiter, G., Schwarzbauer, C., Pernet, C., Andrews, P.J.,
Marshall, I.
,
Doubal, F.
and Wardlaw, J.M. (2018) ‘Cerebrovascular reactivity measurement in cerebral small vessel disease: Rationale and reproducibility of a protocol for MRI acquisition and image processing’, International Journal of Stroke, 13(2), pp. 195–206. Available at:
https://doi.org/10.1177/1747493017730740
.
Standardized mean differences cause funnel plot distortion in publication bias assessments
Zwetsloot, P.-P., Van Der Naald, M., Sena, E.S., Howells, D.W., IntHout, J., De Groot, J.A., Chamuleau, S.A., MacLeod, M.R. and Wever, K.E. (2017) ‘Standardized mean differences cause funnel plot distortion in publication bias assessments’, eLife, 6, e24260. Available at:
https://doi.org/10.7554/eLife.24260
.
Risk of bias reporting in the recent animal focal cerebral ischaemia literature
Bahor, Z., Liao, J., Macleod, M.R., Bannach-Brown, A., McCann, S.K., Wever, K.E., Thomas, J., Ottavi, T., Howells, D.W., Rice, A., Ananiadou, S. and Sena, E. (2017) ‘Risk of bias reporting in the recent animal focal cerebral ischaemia literature’, Clinical Science, 131(20), pp. 2525–2532. Available at:
https://doi.org/10.1042/CS20160722
.
Increasing value and reducing waste in stroke research
Berge, E.,
Salman, R.A.-S.
, Van Der Worp, H.B., Stapf, C., Sandercock, P., Sprigg, N., Macleod, M.R., Kelly, P.J., Nederkoorn, P.J., Ford, G.A., Arnold, M., Berge, E., Diez-Tejedor, E., Jatuzis, D., Kelly, P.J., Krieger, D.W., Nederkoorn, P.J., Sandercock, P., Stapf, C., Weimar, C. and Ford G.A. (2017) ‘Increasing value and reducing waste in stroke research’, The Lancet Neurology, 16(5), pp. 399–408. Available at:
https://doi.org/10.1016/S1474-4422(17)30078-9
.
Ensuring transparency and minimization of methodologic bias in preclinical pain research: PPRECISE considerations
Andrews, N.A., Latrémolière, A., Basbaum, A.I., Mogil, J.S., Porreca, F., Rice, A.S.C., Woolf, C.J.,
Currie, G.L.
, Dworkin, R.H., Eisenach, J.C., Evans, S., Gewandter, J.S., Gover, T.D., Handwerker, H., Huang, W., Iyengar, S., Jensen, M.P., Kennedy, J.D., Lee, N., Levine, J., Lidster, K., Machin, I., McDermott, M.P., McMahon, S.B., Price, T.J., Ross, S.E., Scherrer, G., Seal, R.P., Sena, E.S., Silva, E., Stone, L., Svensson, C.I., Turk, D.C. and Whiteside, G. (2016) ‘Ensuring transparency and minimization of methodologic bias in preclinical pain research: PPRECISE considerations’, Pain, 157(4), pp. 901–909. Available at:
https://doi.org/10.1097/j.pain.0000000000000458
.
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) ‘Estimating the reproducibility of psychological science’, Science, 349(6251), p. aac4716. Available at:
https://doi.org/10.1126/science.aac4716.
Improving functional magnetic resonance imaging reproducibility
Pernet, C. and Poline, J.-B. (2015) ‘Improving functional magnetic resonance imaging reproducibility’, GigaScience, 4(1), p. 15. Available at:
https://doi.org/10.1186/s13742-015-0055-8
.

If you are a CMVM-affiliated author and would like your publication on reproducibility featured on this page, please get in touch at CMVMopenaccess@ed.ac.uk.