The potential of text mining to reduce screening workload in systematic reviews: a retrospective evaluation

Background:The task of identifying relevant studies for systematicreviews in an unbiased way is increasingly time consuming. Text miningmay be able to assist in the screening process in two ways: (1) byprioritising the list of items for manual screening so that the studies atthe top of the list are those that are most likely to be relevant (‘screeningprioritisation’; Fig. 1); (2) by using manually-assigned include/excludedecisions in order to ‘learn’ to apply such categorisations automatically(‘semi-automatic classification’; Fig. 2).Objectives:To evaluate the performance of two text mining methods to reduce screening workloadby assessing their performance in completed reviews.Methods:Data from ten previous reviews covering health care and public healthwere entered into the system. Data included the record’s title andabstract text plus reviewer decisions on whether to exclude the studyor not. We ran simulations of the screening process (ten times foreach condition) testing how the following affected the performance ofthe two text mining processes: the size of the initial training sample(5, 10, 20 or 40 studies); the method and frequency of re-running thesearch/training the classifier (after 5 or 20 includes were identified,or after 25, 250 or 500 records were screened). Performance wasassessed using accuracy, precision, recall/sensitivity, F-measures, andArea Under a (ROC) Curve and burden.Results:There was somevariability between the performance of the tools between reviews, butconsistency within reviews. Screening workload can be reduced, butat the risk of missing potentially relevant studies.Conclusions:A balance needs to be drawn between the number of studies it is feasibleto screen manually and the gain that accrues from screening thousandsof ultimately irrelevant studies. Text mining is likely to have a role toplay, though further evaluation is required.