Multiplicity Eludes Peer Review: The Case of COVID-19 Research
2021
Gutiérrez-Hernández, Oliver | García, Luis V. | Gutiérrez-Hernández, Oliver [0000-0003-2580-5465] | García, Luis V. [0000-0002-5514-2941] | Consejo Superior de Investigaciones Científicas [https://ror.org/02gfc7t72]
Multiplicity arises when data analysis involves multiple simultaneous inferences, increasing the chance of spurious findings. It is a widespread problem frequently ignored by researchers. In this paper, we perform an exploratory analysis of the Web of Science database for COVID-19 observational studies. We examined 100 top-cited COVID-19 peer-reviewed articles based on p-values, including up to 7100 simultaneous tests, with 50% including >34 tests, and 20% > 100 tests. We found that the larger the number of tests performed, the larger the number of significant results (r = 0.87, p < 10−6). The number of p-values in the abstracts was not related to the number of p-values in the papers. However, the highly significant results (p < 0.001) in the abstracts were strongly correlated (r = 0.61, p < 10−6) with the number of p < 0.001 significances in the papers. Furthermore, the abstracts included a higher proportion of significant results (0.91 vs. 0.50), and 80% reported only significant results. Only one reviewed paper addressed multiplicity-induced type I error inflation, pointing to potentially spurious results bypassing the peer-review process. We conclude the need to pay special attention to the increased chance of false discoveries in observational studies, including non-replicated striking discoveries with a potentially large social impact. We propose some easy-to-implement measures to assess and limit the effects of multiplicity.
Show more [+] Less [-]Peer reviewed
Show more [+] Less [-]AGROVOC Keywords
Bibliographic information
This bibliographic record has been provided by Instituto de Recursos Naturales y Agrobiología Sevilla