On the noise resilience of ranking measures

Daniel Berrar

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

Performance measures play a pivotal role in the evaluation and selection of machine learning models for a wide range of applications. Using both synthetic and real-world data sets, we investigated the resilience to noise of various ranking measures. Our experiments revealed that the area under the ROC curve (AUC) and a related measure, the truncated average Kolmogorov-Smirnov statistic (taKS), can reliably discriminate between models with truly different performance under various types and levels of noise. With increasing class skew, however, the H-measure and estimators of the area under the precision-recall curve become preferable measures. Because of its simple graphical interpretation and robustness, the lower trapezoid estimator of the area under the precision-recall curve is recommended for highly imbalanced data sets.

Original languageEnglish
Title of host publicationNeural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings
PublisherSpringer Verlag
Pages47-55
Number of pages9
Volume9948 LNCS
ISBN (Print)9783319466712
DOIs
Publication statusPublished - 2016
Event23rd International Conference on Neural Information Processing, ICONIP 2016 - Kyoto, Japan
Duration: 2016 Oct 162016 Oct 21

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9948 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other23rd International Conference on Neural Information Processing, ICONIP 2016
CountryJapan
CityKyoto
Period16/10/1616/10/21

Keywords

  • AUC
  • Classification
  • H-measure
  • Noise
  • Precision-recall curve
  • Ranking
  • Robustness
  • ROC curve
  • TaKS

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'On the noise resilience of ranking measures'. Together they form a unique fingerprint.

  • Cite this

    Berrar, D. (2016). On the noise resilience of ranking measures. In Neural Information Processing - 23rd International Conference, ICONIP 2016, Proceedings (Vol. 9948 LNCS, pp. 47-55). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9948 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-46672-9_6