The Pitfalls of Using Open Data to Develop Deep Learning Solutions for COVID-19 Detection in Chest X-Rays

Rachael Harkness, Geoff Hall, Alejandro F. Frangi, Nishant Ravikumar, Kieran Zucker

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Since the emergence of COVID-19, deep learning models have been developed to identify COVID-19 from chest X-rays. With little to no direct access to hospital data, the AI community relies heavily on public data comprising numerous data sources. Model performance results have been exceptional when training and testing on open-source data, surpassing the reported capabilities of AI in pneumonia-detection prior to the COVID-19 outbreak. In this study impactful models are trained on a widely used open-source data and tested on an external test set and a hospital dataset, for the task of classifying chest X-rays into one of three classes: COVID-19, non-COVID pneumonia and no-pneumonia. Classification performance of the models investigated is evaluated through ROC curves, confusion matrices and standard classification metrics. Explainability modules are implemented to explore the image features most important to classification. Data analysis and model evalutions show that the popular open-source dataset COVIDx is not representative of the real clinical problem and that results from testing on this are inflated. Dependence on open-source data can leave models vulnerable to bias and confounding variables, requiring careful analysis to develop clinically useful/viable AI tools for COVID-19 detection in chest X-rays.

Original languageEnglish
Title of host publicationMEDINFO 2021
Subtitle of host publicationOne World, One Health - Global Partnership for Digital Innovation - Proceedings of the 18th World Congress on Medical and Health Informatics
EditorsPaula Otero, Philip Scott, Susan Z. Martin, Elaine Huesing
PublisherIOS Press
Pages679-683
Number of pages5
ISBN (Electronic)9781643682648
DOIs
Publication statusPublished - 6 Jun 2022
Event18th World Congress on Medical and Health Informatics: One World, One Health - Global Partnership for Digital Innovation, MEDINFO 2021 - Virtual, Online
Duration: 2 Oct 20214 Oct 2021

Publication series

NameStudies in Health Technology and Informatics
Volume290
ISSN (Print)0926-9630
ISSN (Electronic)1879-8365

Conference

Conference18th World Congress on Medical and Health Informatics: One World, One Health - Global Partnership for Digital Innovation, MEDINFO 2021
CityVirtual, Online
Period2/10/214/10/21

Keywords

  • Computing Methodologies
  • Data Science
  • Respiratory Tract Infections

Fingerprint

Dive into the research topics of 'The Pitfalls of Using Open Data to Develop Deep Learning Solutions for COVID-19 Detection in Chest X-Rays'. Together they form a unique fingerprint.

Cite this