Avoiding model selection bias in small-sample genomic datasets

Daniel Berrar, Ian Bradbury, Werner Dubitzky

Research output: Contribution to journalArticle

37 Citations (Scopus)

Abstract

Motivation: Genomic datasets generated by high-throughput technologies are typically characterized by a moderate number of samples and a large number of measurements per sample. As a consequence, classification models are commonly compared based on resampling techniques. This investigation discusses the conceptual difficulties involved in comparative classification studies. Conclusions derived from such studies are often optimistically biased, because the apparent differences in performance are usually not controlled in a statistically stringent framework taking into account the adopted sampling strategy. We investigate this problem by means of a comparison of various classifiers in the context of multiclass microarray data. Results: Commonly used accuracy-based performance values, with or without confidence intervals, are inadequate for comparing classifiers for small-sample data. We present a statistical methodology that avoids bias in cross-validated model selection in the context of small-sample scenarios. This methodology is valid for both k-fold cross-validation and repeated random sampling.

Original languageEnglish
Pages (from-to)1245-1250
Number of pages6
JournalBioinformatics
Volume22
Issue number10
DOIs
Publication statusPublished - 2006 May 15
Externally publishedYes

Fingerprint

Classifiers
Sampling
Microarrays
Throughput

ASJC Scopus subject areas

  • Clinical Biochemistry
  • Computer Science Applications
  • Computational Theory and Mathematics

Cite this

Avoiding model selection bias in small-sample genomic datasets. / Berrar, Daniel; Bradbury, Ian; Dubitzky, Werner.

In: Bioinformatics, Vol. 22, No. 10, 15.05.2006, p. 1245-1250.

Research output: Contribution to journalArticle

Berrar, D, Bradbury, I & Dubitzky, W 2006, 'Avoiding model selection bias in small-sample genomic datasets', Bioinformatics, vol. 22, no. 10, pp. 1245-1250. https://doi.org/10.1093/bioinformatics/btl066
Berrar, Daniel ; Bradbury, Ian ; Dubitzky, Werner. / Avoiding model selection bias in small-sample genomic datasets. In: Bioinformatics. 2006 ; Vol. 22, No. 10. pp. 1245-1250.
@article{478dd786b5124a868f94e43fe1d51732,
title = "Avoiding model selection bias in small-sample genomic datasets",
abstract = "Motivation: Genomic datasets generated by high-throughput technologies are typically characterized by a moderate number of samples and a large number of measurements per sample. As a consequence, classification models are commonly compared based on resampling techniques. This investigation discusses the conceptual difficulties involved in comparative classification studies. Conclusions derived from such studies are often optimistically biased, because the apparent differences in performance are usually not controlled in a statistically stringent framework taking into account the adopted sampling strategy. We investigate this problem by means of a comparison of various classifiers in the context of multiclass microarray data. Results: Commonly used accuracy-based performance values, with or without confidence intervals, are inadequate for comparing classifiers for small-sample data. We present a statistical methodology that avoids bias in cross-validated model selection in the context of small-sample scenarios. This methodology is valid for both k-fold cross-validation and repeated random sampling.",
author = "Daniel Berrar and Ian Bradbury and Werner Dubitzky",
year = "2006",
month = "5",
day = "15",
doi = "10.1093/bioinformatics/btl066",
language = "English",
volume = "22",
pages = "1245--1250",
journal = "Bioinformatics",
issn = "1367-4803",
publisher = "Oxford University Press",
number = "10",

}

TY - JOUR

T1 - Avoiding model selection bias in small-sample genomic datasets

AU - Berrar, Daniel

AU - Bradbury, Ian

AU - Dubitzky, Werner

PY - 2006/5/15

Y1 - 2006/5/15

N2 - Motivation: Genomic datasets generated by high-throughput technologies are typically characterized by a moderate number of samples and a large number of measurements per sample. As a consequence, classification models are commonly compared based on resampling techniques. This investigation discusses the conceptual difficulties involved in comparative classification studies. Conclusions derived from such studies are often optimistically biased, because the apparent differences in performance are usually not controlled in a statistically stringent framework taking into account the adopted sampling strategy. We investigate this problem by means of a comparison of various classifiers in the context of multiclass microarray data. Results: Commonly used accuracy-based performance values, with or without confidence intervals, are inadequate for comparing classifiers for small-sample data. We present a statistical methodology that avoids bias in cross-validated model selection in the context of small-sample scenarios. This methodology is valid for both k-fold cross-validation and repeated random sampling.

AB - Motivation: Genomic datasets generated by high-throughput technologies are typically characterized by a moderate number of samples and a large number of measurements per sample. As a consequence, classification models are commonly compared based on resampling techniques. This investigation discusses the conceptual difficulties involved in comparative classification studies. Conclusions derived from such studies are often optimistically biased, because the apparent differences in performance are usually not controlled in a statistically stringent framework taking into account the adopted sampling strategy. We investigate this problem by means of a comparison of various classifiers in the context of multiclass microarray data. Results: Commonly used accuracy-based performance values, with or without confidence intervals, are inadequate for comparing classifiers for small-sample data. We present a statistical methodology that avoids bias in cross-validated model selection in the context of small-sample scenarios. This methodology is valid for both k-fold cross-validation and repeated random sampling.

UR - http://www.scopus.com/inward/record.url?scp=33646856024&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33646856024&partnerID=8YFLogxK

U2 - 10.1093/bioinformatics/btl066

DO - 10.1093/bioinformatics/btl066

M3 - Article

VL - 22

SP - 1245

EP - 1250

JO - Bioinformatics

JF - Bioinformatics

SN - 1367-4803

IS - 10

ER -