TY - JOUR
T1 - Why rankings of biomedical image analysis competitions should be interpreted with care
AU - Maier-Hein, Lena
AU - Eisenmann, Matthias
AU - Reinke, Annika
AU - Onogur, Sinan
AU - Stankovic, Marko
AU - Scholz, Patrick
AU - Arbel, Tal
AU - Bogunovic, Hrvoje
AU - Bradley, Andrew P.
AU - Carass, Aaron
AU - Feldmann, Carolin
AU - Frangi, Alejandro F.
AU - Full, Peter M.
AU - van Ginneken, Bram
AU - Hanbury, Allan
AU - Honauer, Katrin
AU - Kozubek, Michal
AU - Landman, Bennett A.
AU - März, Keno
AU - Maier, Oskar
AU - Maier-Hein, Klaus
AU - Menze, Bjoern H.
AU - Müller, Henning
AU - Neher, Peter F.
AU - Niessen, Wiro
AU - Rajpoot, Nasir
AU - Sharp, Gregory C.
AU - Sirinukunwattana, Korsuk
AU - Speidel, Stefanie
AU - Stock, Christian
AU - Stoyanov, Danail
AU - Taha, Abdel Aziz
AU - van der Sommen, Fons
AU - Wang, Ching Wei
AU - Weber, Marc André
AU - Zheng, Guoyan
AU - Jannin, Pierre
AU - Kopp-Schneider, Annette
N1 - Funding Information:
We thank all organizers of the 2015 segmentation challenges who are not co-authoring this paper (a list is provided as Supplementary Note 3) and all participants of the international questionnaire (a list is provided as Supplementary Note 4). We further thank Angelika Laha, Diana Mindroc-Filimon, Bünyamin Pekdemir, and Jenshika Yoganathan (DKFZ, Germany) for helping with the comprehensive challenge capturing. Many thanks also go to Janina Dunning and Stefanie Strzysch (DKFZ, Germany) for their support of the project. Finally, we acknowledge support from the European Research Council (ERC) (ERC starting grant COMBIOSCOPY under the New Horizon Framework Programme grant agreement ERC-2015-StG-37960 as well as Seventh Framework Programme (FP7/2007-2013) under grant agreement no 318068 (VISCERAL)), the German Research Foundation (DFG) (grant MA 6340/10-1 and grant MA 6340/12-1), the Ministry of Science and Technology, Taiwan (MOST 106-3114-8-011-002, 106-2622-8-011-001-TE2, and 105-2221-E-011-121-MY2), the US National Institute of Health (NIH) (grants R01-NS070906, RG-1507-05243, and R01-EB017230 (NIBIB)), the Australian Research Council (DP140102794 and FT110100623), the Swiss National Science Foundation (grant 205321_157207), the Czech Science Foundation (grant P302/ 12/G157), the Czech Ministry of Education, Youth and Sports (grant LTC17016 in the frame of EU COST NEUBIAS project), the Engineering and Physical Sciences Research Council (EPSRC) (MedIAN UK Network (EP/N026993/1) and EP/P012841/1), the Wellcome Trust (NS/A000050/1), the Canadian Natural Science and Engineering Research Council (RGPIN-2015-05471), the UK Medical Research Council (MR/ P015476/1), and the Heidelberg Collaboratory for Image Processing (HCI) including matching funds from the industry partners of the HCI.
Publisher Copyright:
© 2018, The Author(s).
PY - 2018/12/6
Y1 - 2018/12/6
N2 - International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
AB - International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
UR - https://www.scopus.com/pages/publications/85058062868
U2 - 10.1038/s41467-018-07619-7
DO - 10.1038/s41467-018-07619-7
M3 - Article
C2 - 30523263
AN - SCOPUS:85058062868
SN - 2041-1723
VL - 9
SP - 1
EP - 13
JO - Nature Communications
JF - Nature Communications
IS - 1
M1 - 5217
ER -