How reproducible and reliable is geophysical research?

A review of the availability and accessibility of data and software for research published in journals


  • Mark Ireland Newcastle University
  • Guillermo Algarabel Department of Physics, Durham University
  • Michael Steventon Shell Research Ltd
  • Marcus Munafò School of Psychological Science, University of Bristol



reproducibility, data availability, FAIR


Geophysical research frequently makes use of agreed-upon methodologies, formally published software, and bespoke code to process and analyse data. The reliability and repeatability of these methods is vital in maintaining the integrity of research findings and thereby avoiding the dissemination of unreliable results. In recent years there has been increased attention on aspects of reproducibility, which includes data availability, across scientific disciplines. This review considers aspects of reproducibility of geophysical studies relating to their publication in peer reviewed journals. For 100 geophysics journals it considers the extent to which reproducibility in geophysics is the focus of published literature. For 20 geophysical journals it considers a) journal policies on the requirements for providing code, software, and data for submission; and b) the availability of data and software associated for 200 published journal articles. The findings show that: 1) between 1991 and 2021 there were 72 articles with reproducibility in the title and 417 with reliability, with an overall increase in the number of articles with reproducibility or reliability as the subject over the same period; 2) while 60% of journals have a definition of research data, only 20% of journals have a requirement for a data availability statement; and 3) despite ~86% of sampled journal articles including a data availability statement, only 54% of articles have the original data accessible via data repositories or web servers, and only 49% of articles name software used. It is suggested that despite journals and authors working towards improving the availability of data and software, frequently they are not identified, or easily accessible, therefore limiting the possibility of reproducing studies.


AGU. (n.d.). Data and Software for Authors.

American Journal of Political Science. (2019). A.J.P.S. Verification Policy. In American Journal of Political Science.

Arnold, B., Bowler, L., Gibson, S., Herterich, P., Higman, R., Krystalli, A., Morley, A., O’Reilly, M., & Whitaker, K. (2019). The Turing Way: A handbook for reproducible data science. Zenodo.

Behnke, J., Mitchell, A., & Ramapriyan, H. (2019). NASA’s Earth Observing Data and Information System – Near-Term Challenges. 18(1), 1.

Beyreuther, M., Barsch, R., Krischer, L., Megies, T., Behr, Y., & Wassermann, J. (2010). ObsPy: A Python Toolbox for Seismology. Seismological Research Letters, 81(3), 530–533.

Boeker, M., Vach, W., & Motschall, E. (2013). Google Scholar as replacement for systematic literature searches: Good relative recall and precision are not enough. BMC Medical Research Methodology, 13(1), 131.

Borgman, C. L. (2010). Research Data: Who Will Share What, with Whom, When, and Why? SSRN Electronic Journal.

British Geophysical Association. (2014). What is geophysics?

Caelleigh, A. S. (1993). Role of the journal editor in sustaining integrity in research. Academic Medicine, 68(9), 23–29.

Carr, T. R., Buchanan, R. C., Adkins-Heljeson, D., Mettille, T. D., & Sorensen, J. (1997). The future of scientific communication in the earth sciences: The impact of the internet. Computers & Geosciences, 23(5), 503–512.

Childe, S. J. (2006). What is the role of a research journal? Production Planning & Control, 17(5), 439–439.

Christensen, G., Dafoe, A., Miguel, E., Moore, D. A., & Rose, A. K. (2019). A study of the impact of data sharing on article citations using journal policies as a natural experiment. PLOS ONE, 14(12), 225883.

de Groot, P., & Bril, B. (2005). The open source model in geosciences and OpendTect in particular. In SEG Technical Program Expanded Abstracts 2005 (pp. 802–805). Society of Exploration Geophysicists.

Dembe, A. E., Partridge, J. S., & Geist, L. C. (2011). Statistical software applications used in health services research: Analysis of published studies in the U.S. BMC Health Services Research, 11(1), 252.

European Commission. (2016). G20 Leaders’ Communique Hangzhou Summit.

Evangelou, E., Trikalinos, T. A., & Ioannidis, J. P. (2005). Unavailability of online supplementary scientific information from articles published in major journals. The FASEB Journal, 19(14), 1943–1944.

Figshare. (n.d.). Figshare API User Documentation.

Geophysics. (n.d.). GEOPHYSICS instructions to authors.

Glynn, E., Fitzgerald, B., & Exton, C. (2005). Commercial adoption of open source software: an empirical study. 2005 International Symposium on Empirical Software Engineering, 2005, 10 pp.-.

Gomes, D. G. E., Pottier, P., Crystal-Ornelas, R., Hudgins, E. J., Foroughirad, V., Sánchez-Reyes, L. L., Turba, R., Martinez, P. A., Moreau, D., Bertram, M., Smout, C., & Gaynor, K. (2022). Why don’t we share data and code? MetaArXiv.

Goodman, S. N., Fanelli, D., & Ioannidis, J. P. (2016). What does research reproducibility mean? Science Translational Medicine, 8(341), 341 12-341 12.

Hager, B. H., & Clayton, R. W. (1989). Constraints on the structure of mantle convection using seismic observations, flow models, and the geoid.

Hamman, J. (2017). xarray: N-D labeled Arrays and Datasets in Python (Vol. 5).

Harvey, M. J., Mason, N. J., & Rzepa, H. S. (2014). Digital Data Repositories in Chemistry and Their Integration with Journals and Electronic Notebooks. ACS Publications; American Chemical Society.

Harzing, A. W. (2007). Publish or Perish.

Harzing, A. W. (2010). The publish or perish book. Tarma Software Research Pty Limited Melbourne.

Hauge, Ø., Ayala, C., & Conradi, R. (2010). Adoption of open source software in software-intensive organizations – A systematic literature review. Information and Software Technology, 52(11), 1133–1154.

Hodson, S., Jones, S., Collins, S., Genova, F., Harrower, N., Laaksonen, L., Mietchen, D., Petrauskaité, R., & Wittenburg, P. (2018). Turning FAIR data into reality [Techreport].

Houtkoop, B. L., Chambers, C., Macleod, M., Bishop, D. V., Nichols, T. E., & Wagenmakers, E. J. (2018). Data sharing in psychology: A survey on barriers and preconditions. Advances in Methods and Practices in Psychological Science, 1(1), 70–85.

Ireland, M. (2022). Reproducibility in Geophysics.

Jun, H., & Cho, Y. (2022). Repeatability enhancement of time-lapse seismic data via a convolutional autoencoder. Geophysical Journal International, 228(2), 1150–1170.

Konkol, M., Kray, C., & Pfeiffer, M. (2019). Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study. International Journal of Geographical Information Science, 33(2), 408–429.

Lepak, D. (2009). Editor’s Comments: What is Good Reviewing? Academy of Management Review, 34(3), 375–381.

McCullough, B. D., & Heiser, D. A. (2008). On the accuracy of statistical procedures in Microsoft Excel 2007. Computational Statistics & Data Analysis, 52(10), 4570–4578.

McCullough, B. D., & Wilson, B. (2002). On the accuracy of statistical procedures in Microsoft Excel 2000 and Excel XP. Computational Statistics & Data Analysis, 40(4), 713–721.

McCullough, B. D., & Wilson, B. (2005). On the accuracy of statistical procedures in Microsoft Excel 2003. Computational Statistics & Data Analysis, 49(4), 1244–1252.

Mélard, G. (2014). On the accuracy of statistical procedures in Microsoft Excel 2010. Computational Statistics, 29(5), 1095–1128.

Mesirov, J. P. (2010). Accessible Reproducible Research. Science, 327(5964), 415–416.

Muenchow, J., Schäfer, S., & Krüger, E. (2019). Reviewing qualitative GIS research—Toward a wider usage of open‐source GIS and reproducible research practices. Geography Compass, 13(6), 12441.

National Academies of Sciences. (2016). Statistical challenges in assessing and fostering the reproducibility of scientific results: Summary of a workshop.

National Academies of Sciences. (2019). Understanding Reproducibility and Replicability. In Reproducibility and Replicability in Science.

Nature. (2014). Journals unite for reproducibility. Nature, 515(7525).

Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., & Christensen, G. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425.

Nüst, D., & Pebesma, E. (2021). Practical Reproducibility in Geography and Geosciences. Annals of the American Association of Geographers, 111(5), 1300–1310.

Oguntimilehin, A., & Ademola, E. O. (2014). A Review of Big Data Management, Benefits and Challenges. A Review of Big Data Management, Benefits and Challenges, 5(6), 6.

Oren, C., & Nowack, R. L. (2018). An overview of reproducible 3D seismic data processing and imaging using Madagascar. Geophysics, 83(2), 9–20.

Pendlebury, D. A. (2009). The use and misuse of journal metrics and other citation indicators. Archivum Immunologiae et Therapiae Experimentalis, 57(1), 1–11.

Piwowar, H. A., Day, R. S., & Fridsma, D. B. (2007). Sharing Detailed Research Data Is Associated with Increased Citation Rate. PLOS ONE, 2(3), 308.

Poldrack, R. A., & Gorgolewski, K. J. (2014). Making big data open: Data sharing in neuroimaging. Nature Neuroscience, 17(11), 11.

Pop, M., & Salzberg, S. L. (2015). Use and mis-use of supplementary material in science publications. BMC Bioinformatics, 16(1), 237.

Rallison, S. (2015). What are Journals for? Annals of The Royal College of Surgeons of England, 97(2), 89–91.

Reese, R. J. (1965). Recent Applications of Digital Computers to Geophysical Problems. AAPG Bulletin, 49(7), 1089–1089.

Robinson, E. A., & Treitel, S. (2000). Geophysical signal analysis. Society of Exploration Geophysicists.

SCImago. (n.d.). SCImago Journal & Country Rank. In SCImago.

Starr, J., Castro, E., Crosas, M., Dumontier, M., Downs, R. R., Duerr, R., Haak, L. L., Haendel, M., Herman, I., Hodson, S., Hourclé, J., Kratz, J. E., Lin, J., Nielsen, L. H., Nurnberger, A., Proell, S., Rauber, A., Sacchi, S., Smith, A., & Clark, T. (2015). Achieving human and machine accessibility of cited data in scholarly publications. PeerJ Computer Science, 1, 1.

Steventon, M. J., Jackson, C. A., Hall, M., Ireland, M. T., Munafo, M., & Roberts, K. J. (2022). Reproducibility in subsurface geoscience. Earth Science, Systems and Society, 12.

Tedersoo, L., Küngas, R., Oras, E., Köster, K., Eenmaa, H., Leijen, Ä., Pedaste, M., Raju, M., Astapova, A., Lukner, H., Kogermann, K., & Sepp, T. (2021). Data sharing practices and data availability upon request differ across scientific disciplines. Scientific Data, 8(1).

Tenopir, C., Allard, S., Douglass, K., Aydinoglu, A. U., Wu, L., Read, E., Manoff, M., & Frame, M. (2011). Data Sharing by Scientists: Practices and Perceptions. PLOS ONE, 6(6), 21101.

van Rooij, S. W. (2011). Higher education sub-cultures and open source adoption. Computers & Education, 57(1), 1171–1183.

Waage, M., Bünz, S., Landrø, M., Plaza-Faverola, A., & Waghorn, K. A. (2018). Repeatability of high-resolution 3D seismic data. Geophysics, 84(1), 75–94.

Walker, R., Gill, S. P., Greenfield, C., McCaffrey, K., & Stephens, T. L. (2021). No demonstrated link between sea-level and eruption history at Santorini. In Earth arXiv.

Wildman, G., & Lewis, E. (2022). Value of open data: A geoscience perspective. Geoscience Data Journal.


Additional Files



How to Cite

Ireland, M., Algarabel, G., Steventon, M., & Munafò, M. (2023). How reproducible and reliable is geophysical research? A review of the availability and accessibility of data and software for research published in journals. Seismica, 2(1).