Expected Agreement Coefficient for Norm-Referenced Tests With Classical Test Theory
DOI:
https://doi.org/10.17140/PCSOJ-2-110Keywords:
Norm-referenced, Agreement coefficient, Coefficient alpha, Classical test theoryAbstract
There are two types of agreement coefficients for psychological test scores: norm-referenced and criterion-referenced agreement coefficients. These coefficients were derived within the framework of generalizability theory that is known for its theoretical and practical complexities. Under the framework of classical test theory, the paper derived the norm-referenced agreement coefficient. This derivation was based on the assumption of randomly equivalent test forms. The resulted expected agreement coefficient was different from its counterpart in generalizability theory. However, the estimators of this norm-referenced agreement coefficient were equal under the two frameworks to coefficient alpha reliability.
References
Brennan RL. Generalizability theory and classical test theo-ry. Applied Measurement in Education. 2010; 24(1): 1-21. doi: 10.1080/08957347.2011.532417
Brennan RL, Kane MT. An index of dependability for mastery tests. J Educ Meas. 1977; 14(3): 277-289. doi: 10.1111/j.1745- 3984.1977.tb00045.x
Brennan RL, Kane MT. Signal/noise ratios for domain-refer-enced tests. Psychometrika. 1977; 42(4): 609-625. doi: 10.1007/ BF02295983
Gao X, Brennan R, Guo F. Modeling measurement facets and assessing generalizability in a large-scale writing assessment. GMAC Research Report. 2015.
Glaser R, Nitko AJ. Measurement in learning and instruction. In: Thorndike RL, ed. Educational measurement. Washington DC, USA: American Council on Education; 1971.
Popham WJ, Husek TR. Implications of criterion-referenced measurement. J Educ Meas. 1969; 6(1): 1-9. doi: 10.1111/j.1745- 3984.1969.tb00654.x
Kuder GF, Richardson MW. The theory of the estimation of test reliability. Psychometrika. 1937; 2(3): 151-160. doi: 10.1007/BF02288391
Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951; 16(3): 297-334. doi: 10.1007/ BF02310555
Almehrizi RS. Coefficient alpha and reliability of scale scores. Applied Psychological Measurement. 2013; 37(6): 438-459. doi:
1177/0146621613484983 10. Cronbach LJ, Shavelson RJ. My Current thoughts on coeffi-cient alpha and successor procedures. Educ Psychol Meas. 2004; 64(3): 391-418. doi: 10.1177/0013164404266386
Kane MT, Brennan RL. Agreement coefficients as in-dices of dependability for criterion-referenced tests. Ap-plied Psychological Measurement. 1980; 4(1): 105-126. doi: 10.1177/014662168000400111
Almehrizi R. Normalization of mean squared differences to measure agreement for continuous data. Stat Methods Med Res. 2013. doi: 10.1177/0962280213507506
AlKharusi H. Generalizability theory: An analysis of vari-ance approach to measurement problems in educational assess-ment. J Studies Educ. 2012; 2(1): 184-196. doi: 10.5296/jse. v2i1.1227̂̂̂̂̂̂̂
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
All content published in Psychology and Cognitive Sciences (PCS) is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license allows anyone to copy, distribute, transmit, and adapt the work — including for commercial purposes — provided appropriate credit is given to the original author(s) and source.
