A Case Study in the use of ROC curves for Algorithm Design

We describe the development of a vision system to detect natural events in a low-resolution image stream. The work involves the assessment of algorithmic design decisions to maximise detection reliability. This assessment is carried out by comparing measures and estimates made by the system under development with measures obtained independently. We show that even when these independent measures are themselves noisy, their independence can serve to guide design decisions and allow performance estimates to be made. Although presented here for one particular system design, we believe that such an approach will be applicable to other situations when an image-based system is to be used in the analysis of natural scenes in situations where a precise ground truth is not available.