A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats

Yo In'nami, Rie Koizumi

Research output: Contribution to journalArticle

35 Citations (Scopus)

Abstract

A meta-analysis was conducted on the effects of multiple-choice and openended formats on L1 reading, L2 reading, and L2 listening test performance. Fifty-six data sources located in an extensive search of the literature were the basis for the estimates of the mean effect sizes of test format effects. The results using the mixed effects model of meta-analysis indicate that multiple-choice formats are easier than open-ended formats in L1 reading and L2 listening, with the degree of format effect ranging from small to large in L1 reading and medium to large in L2 listening. Overall, format effects in L2 reading are not found, although multiple-choice formats are found to be easier than open-ended formats when any one of the following four conditions is met: the studies involve between-subjects designs, random assignment, stem-equivalent items, or learners with a high L2 proficiency level. Format effects favoring multiple-choice formats across the three domains are consistently observed when studies employ between-subjects designs, random assignment, or stem-equivalent items.

Original languageEnglish
Pages (from-to)219-244
Number of pages26
JournalLanguage Testing
Volume26
Issue number2
DOIs
Publication statusPublished - 2009 Apr

Fingerprint

performance
Multiple Choice
Performance Test
Meta-analysis
L2 Listening
Assignment
L2 Reading
L2 Proficiency
Effect Size
Mixed Effects Model

Keywords

  • Listening test
  • Meta-analysis
  • Multiple-choice
  • Open-ended
  • Reading test
  • Test format
  • Test method

ASJC Scopus subject areas

  • Linguistics and Language
  • Social Sciences (miscellaneous)

Cite this

A meta-analysis of test format effects on reading and listening test performance : Focus on multiple-choice and open-ended formats. / In'nami, Yo; Koizumi, Rie.

In: Language Testing, Vol. 26, No. 2, 04.2009, p. 219-244.

Research output: Contribution to journalArticle

@article{906bf41bbcc94d9898fe890c8d769c50,
title = "A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats",
abstract = "A meta-analysis was conducted on the effects of multiple-choice and openended formats on L1 reading, L2 reading, and L2 listening test performance. Fifty-six data sources located in an extensive search of the literature were the basis for the estimates of the mean effect sizes of test format effects. The results using the mixed effects model of meta-analysis indicate that multiple-choice formats are easier than open-ended formats in L1 reading and L2 listening, with the degree of format effect ranging from small to large in L1 reading and medium to large in L2 listening. Overall, format effects in L2 reading are not found, although multiple-choice formats are found to be easier than open-ended formats when any one of the following four conditions is met: the studies involve between-subjects designs, random assignment, stem-equivalent items, or learners with a high L2 proficiency level. Format effects favoring multiple-choice formats across the three domains are consistently observed when studies employ between-subjects designs, random assignment, or stem-equivalent items.",
keywords = "Listening test, Meta-analysis, Multiple-choice, Open-ended, Reading test, Test format, Test method",
author = "Yo In'nami and Rie Koizumi",
year = "2009",
month = "4",
doi = "10.1177/0265532208101006",
language = "English",
volume = "26",
pages = "219--244",
journal = "Language Testing",
issn = "0265-5322",
publisher = "SAGE Publications Ltd",
number = "2",

}

TY - JOUR

T1 - A meta-analysis of test format effects on reading and listening test performance

T2 - Focus on multiple-choice and open-ended formats

AU - In'nami, Yo

AU - Koizumi, Rie

PY - 2009/4

Y1 - 2009/4

N2 - A meta-analysis was conducted on the effects of multiple-choice and openended formats on L1 reading, L2 reading, and L2 listening test performance. Fifty-six data sources located in an extensive search of the literature were the basis for the estimates of the mean effect sizes of test format effects. The results using the mixed effects model of meta-analysis indicate that multiple-choice formats are easier than open-ended formats in L1 reading and L2 listening, with the degree of format effect ranging from small to large in L1 reading and medium to large in L2 listening. Overall, format effects in L2 reading are not found, although multiple-choice formats are found to be easier than open-ended formats when any one of the following four conditions is met: the studies involve between-subjects designs, random assignment, stem-equivalent items, or learners with a high L2 proficiency level. Format effects favoring multiple-choice formats across the three domains are consistently observed when studies employ between-subjects designs, random assignment, or stem-equivalent items.

AB - A meta-analysis was conducted on the effects of multiple-choice and openended formats on L1 reading, L2 reading, and L2 listening test performance. Fifty-six data sources located in an extensive search of the literature were the basis for the estimates of the mean effect sizes of test format effects. The results using the mixed effects model of meta-analysis indicate that multiple-choice formats are easier than open-ended formats in L1 reading and L2 listening, with the degree of format effect ranging from small to large in L1 reading and medium to large in L2 listening. Overall, format effects in L2 reading are not found, although multiple-choice formats are found to be easier than open-ended formats when any one of the following four conditions is met: the studies involve between-subjects designs, random assignment, stem-equivalent items, or learners with a high L2 proficiency level. Format effects favoring multiple-choice formats across the three domains are consistently observed when studies employ between-subjects designs, random assignment, or stem-equivalent items.

KW - Listening test

KW - Meta-analysis

KW - Multiple-choice

KW - Open-ended

KW - Reading test

KW - Test format

KW - Test method

UR - http://www.scopus.com/inward/record.url?scp=68149161356&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=68149161356&partnerID=8YFLogxK

U2 - 10.1177/0265532208101006

DO - 10.1177/0265532208101006

M3 - Article

AN - SCOPUS:68149161356

VL - 26

SP - 219

EP - 244

JO - Language Testing

JF - Language Testing

SN - 0265-5322

IS - 2

ER -