E. Trucco, A. Ruggeri, T. Karnowski, L. Giancardo, E. Chaum, J. P. Hubschman, B. al-Diri, C. Y. Cheung, D. Wong, M. Abramoff, G. Lim, D. Kumar, P. Burlina, N. M. Bressler, H. F. Jelinek, F. Meriaudeau, G. Quellec, T. MacGillivray, B. Dhillon



Publication year



Investigative Ophthalmology & Visual Science

Periodical Number






Author Address

Trucco, E Univ Dundee, Sch Comp, VAMPIRE CVIP, Dundee, Scotland Univ Dundee, Sch Comp, VAMPIRE Project, Dundee, Scotland Univ Padua, Dept Informat Engn, Padua, Italy Oak Ridge Natl Lab, Oak Ridge, TN USA Ist Italiano Tecnol, Genoa, Italy Univ Tennessee, Ctr Hlth Sci, Memphis, TN 38163 USA Jules Stein Eye Inst, Los Angeles, CA 90024 USA Lincoln Univ, REVIEW Grp, Lincoln, England Singapore Eye Res Inst, Singapore, Singapore ASTAR, Inst Infocomm Res, Singapore, Singapore Univ Iowa, Dept Ophthalmol & Visual Sci, Iowa City, IA USA Univ Iowa, Dept Elect & Comp Engn, Iowa City, IA 52242 USA Univ Iowa, Dept Biomed Engn, Iowa City, IA 52242 USA Iowa City VA Med Ctr, Dept Vet Affairs, Iowa City, IA USA Natl Univ Singapore, Sch Comp, Singapore 117548, Singapore RMIT Univ, Melbourne, Vic, Australia Johns Hopkins Univ, Appl Phys Lab, Laurel, MD USA Johns Hopkins Univ, Wilmer Eye Inst, Baltimore, MD 21218 USA Charles Sturt Univ, Albury, NSW, Australia Univ Burgundy, Le2i, Le Creusot, France INSERM, U1101, SFR ScInBioS, Brest, France Univ Edinburgh, Clin Res Imaging Ctr, VAMPIRE Project, Edinburgh, Midlothian, Scotland Univ Edinburgh, NHS Princess Alexandra Eye Pavil, VAMPIRE Project, Edinburgh, Midlothian, Scotland

Full version

This paper concerns the validation of automatic retinal image analysis (ARIA) algorithms. For reasons of space and consistency, we concentrate on the validation of algorithms processing color fundus camera images, currently the largest section of the ARIA literature. We sketch the context (imaging instruments and target tasks) of ARIA validation, summarizing the main image analysis and validation techniques. We then present a list of recommendations focusing on the creation of large repositories of test data created by international consortia, easily accessible via moderated Web sites, including multicenter annotations by multiple experts, specific to clinical tasks, and capable of running submitted software automatically on the data stored, with clear and widely agreed-on performance criteria, to provide a fair comparison.