On the noise resilience of ranking measures

Daniel Berrar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Performance measures play a pivotal role in the evaluation and selection of machine learning models for a wide range of applications. Using both synthetic and real-world data sets, we investigated the resilience to noise of various ranking measures. Our experiments revealed that the area under the ROC curve (AUC) and a related measure, the truncated average Kolmogorov-Smirnov statistic (taKS), can reliably discriminate between models with truly different performance under various types and levels of noise. With increasing class skew, however, the H-measure and estimators of the area under the precision-recall curve become preferable measures. Because of its simple graphical interpretation and robustness, the lower trapezoid estimator of the area under the precision-recall curve is recommended for highly imbalanced data sets.

Original languageEnglish
Title of host publicationNeural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings
PublisherSpringer Verlag
Pages47-55
Number of pages9
Volume9948 LNCS
ISBN (Print)9783319466712
DOIs
Publication statusPublished - 2016
Event23rd International Conference on Neural Information Processing, ICONIP 2016 - Kyoto, Japan
Duration: 2016 Oct 162016 Oct 21

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9948 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other23rd International Conference on Neural Information Processing, ICONIP 2016
CountryJapan
CityKyoto
Period16/10/1616/10/21

Fingerprint

Learning systems
Statistics
Experiments

Keywords

  • AUC
  • Classification
  • H-measure
  • Noise
  • Precision-recall curve
  • Ranking
  • Robustness
  • ROC curve
  • TaKS

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Cite this

Berrar, D. (2016). On the noise resilience of ranking measures. In Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings (Vol. 9948 LNCS, pp. 47-55). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9948 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-46672-9_6

On the noise resilience of ranking measures. / Berrar, Daniel.

Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings. Vol. 9948 LNCS Springer Verlag, 2016. p. 47-55 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9948 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Berrar, D 2016, On the noise resilience of ranking measures. in Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings. vol. 9948 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9948 LNCS, Springer Verlag, pp. 47-55, 23rd International Conference on Neural Information Processing, ICONIP 2016, Kyoto, Japan, 16/10/16. https://doi.org/10.1007/978-3-319-46672-9_6
Berrar D. On the noise resilience of ranking measures. In Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings. Vol. 9948 LNCS. Springer Verlag. 2016. p. 47-55. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-46672-9_6
Berrar, Daniel. / On the noise resilience of ranking measures. Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings. Vol. 9948 LNCS Springer Verlag, 2016. pp. 47-55 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{32f18427c09846ea8367189e03fe7d98,
title = "On the noise resilience of ranking measures",
abstract = "Performance measures play a pivotal role in the evaluation and selection of machine learning models for a wide range of applications. Using both synthetic and real-world data sets, we investigated the resilience to noise of various ranking measures. Our experiments revealed that the area under the ROC curve (AUC) and a related measure, the truncated average Kolmogorov-Smirnov statistic (taKS), can reliably discriminate between models with truly different performance under various types and levels of noise. With increasing class skew, however, the H-measure and estimators of the area under the precision-recall curve become preferable measures. Because of its simple graphical interpretation and robustness, the lower trapezoid estimator of the area under the precision-recall curve is recommended for highly imbalanced data sets.",
keywords = "AUC, Classification, H-measure, Noise, Precision-recall curve, Ranking, Robustness, ROC curve, TaKS",
author = "Daniel Berrar",
year = "2016",
doi = "10.1007/978-3-319-46672-9_6",
language = "English",
isbn = "9783319466712",
volume = "9948 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "47--55",
booktitle = "Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings",

}

TY - GEN

T1 - On the noise resilience of ranking measures

AU - Berrar, Daniel

PY - 2016

Y1 - 2016

N2 - Performance measures play a pivotal role in the evaluation and selection of machine learning models for a wide range of applications. Using both synthetic and real-world data sets, we investigated the resilience to noise of various ranking measures. Our experiments revealed that the area under the ROC curve (AUC) and a related measure, the truncated average Kolmogorov-Smirnov statistic (taKS), can reliably discriminate between models with truly different performance under various types and levels of noise. With increasing class skew, however, the H-measure and estimators of the area under the precision-recall curve become preferable measures. Because of its simple graphical interpretation and robustness, the lower trapezoid estimator of the area under the precision-recall curve is recommended for highly imbalanced data sets.

AB - Performance measures play a pivotal role in the evaluation and selection of machine learning models for a wide range of applications. Using both synthetic and real-world data sets, we investigated the resilience to noise of various ranking measures. Our experiments revealed that the area under the ROC curve (AUC) and a related measure, the truncated average Kolmogorov-Smirnov statistic (taKS), can reliably discriminate between models with truly different performance under various types and levels of noise. With increasing class skew, however, the H-measure and estimators of the area under the precision-recall curve become preferable measures. Because of its simple graphical interpretation and robustness, the lower trapezoid estimator of the area under the precision-recall curve is recommended for highly imbalanced data sets.

KW - AUC

KW - Classification

KW - H-measure

KW - Noise

KW - Precision-recall curve

KW - Ranking

KW - Robustness

KW - ROC curve

KW - TaKS

UR - http://www.scopus.com/inward/record.url?scp=84992669640&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84992669640&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-46672-9_6

DO - 10.1007/978-3-319-46672-9_6

M3 - Conference contribution

SN - 9783319466712

VL - 9948 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 47

EP - 55

BT - Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings

PB - Springer Verlag

ER -