Standard

Setting standards to a scientific literacy test for adults using the item-descriptor (ID) matching method. / Haschke, Linda; Kampa, Nele; Hahn, Inga; Köller, Olaf.

Standard setting in education: The nordic countries in an international perspective. Hrsg. / Sigrid Blömeke; Jan-Eric Gustafsson. Berlin : Springer, 2017. S. 319-339 (Methodology of Educational Measurement and Assessment).

Publikation: Forschung - BegutachtungBeiträge in Sammelwerken

Harvard

Haschke, L, Kampa, N, Hahn, I & Köller, O 2017, Setting standards to a scientific literacy test for adults using the item-descriptor (ID) matching method. in S Blömeke & J-E Gustafsson (Hrsg.), Standard setting in education: The nordic countries in an international perspective. Methodology of Educational Measurement and Assessment, Springer, Berlin, S. 319-339. DOI: 10.1007/978-3-319-50856-6_18

APA

Haschke, L., Kampa, N., Hahn, I., & Köller, O. (2017). Setting standards to a scientific literacy test for adults using the item-descriptor (ID) matching method. in S. Blömeke, & J-E. Gustafsson (Hrsg.), Standard setting in education: The nordic countries in an international perspective (S. 319-339). (Methodology of Educational Measurement and Assessment). Berlin: Springer. DOI: 10.1007/978-3-319-50856-6_18

Vancouver

Haschke L, Kampa N, Hahn I, Köller O. Setting standards to a scientific literacy test for adults using the item-descriptor (ID) matching method. in Blömeke S, Gustafsson J-E, Hrsg., Standard setting in education: The nordic countries in an international perspective. Berlin: Springer. 2017. S. 319-339. (Methodology of Educational Measurement and Assessment). Erhältlich von, DOI: 10.1007/978-3-319-50856-6_18

BibTeX

@inbook{2cb7850aa3a94d9381286f92be9e9589,
title = "Setting standards to a scientific literacy test for adults using the item-descriptor (ID) matching method",
abstract = "Common standard setting methods such as the Angoff or the Bookmark method require panellists to imagine minimally competent persons or to estimate response probabilities, in order to define cut scores. Imagining these persons and how they would perform is criticised as cognitively demanding. These already challenging judgemental tasks become even more difficult, when experts have to deal with very heterogeneous or insufficiently studied populations, such as adults. The Item-Descriptor (ID) Matching method can reduce the arbitrariness of such subjective evaluations by focusing on rather objective judgements about the content of tests. At our standard setting workshop, seven experts had to match the item demands of 22 items of a scientific literacy test for adults with abilities described by performance level descriptions (PLDs) of the two proficiency levels Basic and Advanced. Since the ID Matching method has hardly been used in European standard settings, the method has not been evaluated comprehensively. In order to evaluate the appropriateness and correct interpretation of cut scores, information about the validity of standard setting methods is essential. In this chapter, we aim to provide procedural and internal evidence for the use and interpretation of the derived cut scores and PLDs using the ID Matching method. With regard to procedural validity, we report high and consensual agreement of the experts regarding explicitness, practicability, implementation, and feedback, which we assessed by detailed questionnaires. The inter-rater reliability for the panellists’ classification of items was low, but increased during subsequent rounds (κ = .38 to κ = .63). The values are consistent with findings of earlier studies which support internal validity. We argue that the cut scores and PLDs derived from the application of the ID Matching method are appropriate to categorise adults as scientifically illiterate, literate, and advanced literate.",
keywords = "Methodological research and development, Item-Descriptor Matching method, Internal validity, External validity, Science abilities, Standard setting",
author = "Linda Haschke and Nele Kampa and Inga Hahn and Olaf Köller",
year = "2017",
month = "2",
doi = "10.1007/978-3-319-50856-6_18",
isbn = "978-3-319-50855-9",
series = "Methodology of Educational Measurement and Assessment",
publisher = "Springer",
pages = "319--339",
editor = "Sigrid Blömeke and Jan-Eric Gustafsson",
booktitle = "Standard setting in education",
address = "Germany",

}

RIS

TY - CHAP

T1 - Setting standards to a scientific literacy test for adults using the item-descriptor (ID) matching method

AU - Haschke,Linda

AU - Kampa,Nele

AU - Hahn,Inga

AU - Köller,Olaf

PY - 2017/2/28

Y1 - 2017/2/28

N2 - Common standard setting methods such as the Angoff or the Bookmark method require panellists to imagine minimally competent persons or to estimate response probabilities, in order to define cut scores. Imagining these persons and how they would perform is criticised as cognitively demanding. These already challenging judgemental tasks become even more difficult, when experts have to deal with very heterogeneous or insufficiently studied populations, such as adults. The Item-Descriptor (ID) Matching method can reduce the arbitrariness of such subjective evaluations by focusing on rather objective judgements about the content of tests. At our standard setting workshop, seven experts had to match the item demands of 22 items of a scientific literacy test for adults with abilities described by performance level descriptions (PLDs) of the two proficiency levels Basic and Advanced. Since the ID Matching method has hardly been used in European standard settings, the method has not been evaluated comprehensively. In order to evaluate the appropriateness and correct interpretation of cut scores, information about the validity of standard setting methods is essential. In this chapter, we aim to provide procedural and internal evidence for the use and interpretation of the derived cut scores and PLDs using the ID Matching method. With regard to procedural validity, we report high and consensual agreement of the experts regarding explicitness, practicability, implementation, and feedback, which we assessed by detailed questionnaires. The inter-rater reliability for the panellists’ classification of items was low, but increased during subsequent rounds (κ = .38 to κ = .63). The values are consistent with findings of earlier studies which support internal validity. We argue that the cut scores and PLDs derived from the application of the ID Matching method are appropriate to categorise adults as scientifically illiterate, literate, and advanced literate.

AB - Common standard setting methods such as the Angoff or the Bookmark method require panellists to imagine minimally competent persons or to estimate response probabilities, in order to define cut scores. Imagining these persons and how they would perform is criticised as cognitively demanding. These already challenging judgemental tasks become even more difficult, when experts have to deal with very heterogeneous or insufficiently studied populations, such as adults. The Item-Descriptor (ID) Matching method can reduce the arbitrariness of such subjective evaluations by focusing on rather objective judgements about the content of tests. At our standard setting workshop, seven experts had to match the item demands of 22 items of a scientific literacy test for adults with abilities described by performance level descriptions (PLDs) of the two proficiency levels Basic and Advanced. Since the ID Matching method has hardly been used in European standard settings, the method has not been evaluated comprehensively. In order to evaluate the appropriateness and correct interpretation of cut scores, information about the validity of standard setting methods is essential. In this chapter, we aim to provide procedural and internal evidence for the use and interpretation of the derived cut scores and PLDs using the ID Matching method. With regard to procedural validity, we report high and consensual agreement of the experts regarding explicitness, practicability, implementation, and feedback, which we assessed by detailed questionnaires. The inter-rater reliability for the panellists’ classification of items was low, but increased during subsequent rounds (κ = .38 to κ = .63). The values are consistent with findings of earlier studies which support internal validity. We argue that the cut scores and PLDs derived from the application of the ID Matching method are appropriate to categorise adults as scientifically illiterate, literate, and advanced literate.

KW - Methodological research and development

KW - Item-Descriptor Matching method

KW - Internal validity

KW - External validity

KW - Science abilities

KW - Standard setting

U2 - 10.1007/978-3-319-50856-6_18

DO - 10.1007/978-3-319-50856-6_18

M3 - Contributions to collected editions/anthologies

SN - 978-3-319-50855-9

T3 - Methodology of Educational Measurement and Assessment

SP - 319

EP - 339

BT - Standard setting in education

PB - Springer

ER -

ID: 658746