An empirical evaluation of ranking measures with respect to robustness to noise

Daniel Berrar

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Ranking measures play an important role in model evaluation and selection. Using both synthetic and real-world data sets, we investigate how different types and levels of noise affect the area under the ROC curve (AUC), the area under the ROC convex hull, the scored AUC, the Kolmogorov-Smirnov statistic, and the H-measure. In our experiments, the AUC was, overall, the most robust among these measures, thereby reinvigorating it as a reliable metric despite its well-known deficiencies. This paper also introduces a novel ranking measure, which is remarkably robust to noise yet conceptually simple.

Original languageEnglish
Pages (from-to)241-267
Number of pages27
JournalJournal of Artificial Intelligence Research
Volume49
Publication statusPublished - 2014 Feb
Externally publishedYes

Fingerprint

Statistics
Experiments

ASJC Scopus subject areas

  • Artificial Intelligence

Cite this

An empirical evaluation of ranking measures with respect to robustness to noise. / Berrar, Daniel.

In: Journal of Artificial Intelligence Research, Vol. 49, 02.2014, p. 241-267.

Research output: Contribution to journalArticle

@article{62f4fb03d9ea4a889042f62675f74d0c,
title = "An empirical evaluation of ranking measures with respect to robustness to noise",
abstract = "Ranking measures play an important role in model evaluation and selection. Using both synthetic and real-world data sets, we investigate how different types and levels of noise affect the area under the ROC curve (AUC), the area under the ROC convex hull, the scored AUC, the Kolmogorov-Smirnov statistic, and the H-measure. In our experiments, the AUC was, overall, the most robust among these measures, thereby reinvigorating it as a reliable metric despite its well-known deficiencies. This paper also introduces a novel ranking measure, which is remarkably robust to noise yet conceptually simple.",
author = "Daniel Berrar",
year = "2014",
month = "2",
language = "English",
volume = "49",
pages = "241--267",
journal = "Journal of Artificial Intelligence Research",
issn = "1076-9757",
publisher = "Morgan Kaufmann Publishers, Inc.",

}

TY - JOUR

T1 - An empirical evaluation of ranking measures with respect to robustness to noise

AU - Berrar, Daniel

PY - 2014/2

Y1 - 2014/2

N2 - Ranking measures play an important role in model evaluation and selection. Using both synthetic and real-world data sets, we investigate how different types and levels of noise affect the area under the ROC curve (AUC), the area under the ROC convex hull, the scored AUC, the Kolmogorov-Smirnov statistic, and the H-measure. In our experiments, the AUC was, overall, the most robust among these measures, thereby reinvigorating it as a reliable metric despite its well-known deficiencies. This paper also introduces a novel ranking measure, which is remarkably robust to noise yet conceptually simple.

AB - Ranking measures play an important role in model evaluation and selection. Using both synthetic and real-world data sets, we investigate how different types and levels of noise affect the area under the ROC curve (AUC), the area under the ROC convex hull, the scored AUC, the Kolmogorov-Smirnov statistic, and the H-measure. In our experiments, the AUC was, overall, the most robust among these measures, thereby reinvigorating it as a reliable metric despite its well-known deficiencies. This paper also introduces a novel ranking measure, which is remarkably robust to noise yet conceptually simple.

UR - http://www.scopus.com/inward/record.url?scp=84894418956&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84894418956&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:84894418956

VL - 49

SP - 241

EP - 267

JO - Journal of Artificial Intelligence Research

JF - Journal of Artificial Intelligence Research

SN - 1076-9757

ER -