An empirical evaluation of ranking measures with respect to robustness to noise

Daniel Berrar

Research output: Contribution to journalArticle

4 Citations (Scopus)


Ranking measures play an important role in model evaluation and selection. Using both synthetic and real-world data sets, we investigate how different types and levels of noise affect the area under the ROC curve (AUC), the area under the ROC convex hull, the scored AUC, the Kolmogorov-Smirnov statistic, and the H-measure. In our experiments, the AUC was, overall, the most robust among these measures, thereby reinvigorating it as a reliable metric despite its well-known deficiencies. This paper also introduces a novel ranking measure, which is remarkably robust to noise yet conceptually simple.

Original languageEnglish
Pages (from-to)241-267
Number of pages27
JournalJournal of Artificial Intelligence Research
Publication statusPublished - 2014 Feb 1


ASJC Scopus subject areas

  • Artificial Intelligence

Cite this