Prediction of the difficulty level in a standardized reading comprehension test: contributions from cognitive psychology and psychometrics

Authors

  • Armel Brizuela Universidad de Costa Rica
  • Eiliana Montero-Rojas Universidad de Costa Rica

DOI:

https://doi.org/10.7203/relieve.19.2.3143

Keywords:

Cognitive psychology, language processing, reading comprehension, Item Response Theory, Linear Logistic Test Model, Task Analysis, Item difficulty level.

Abstract

This research seeks to identify possible predictors of the difficulty level of reading comprehension items used in a standardized psychometric test for university admission. Several potential predictors of difficulty were proposed, namely, propositional density, negations, grammatical structure, vocabulary difficulty, presence of enhancement elements (words highlighted typographically), item abstraction level and degree of similarity between correct option and relevant text to resolve the item. By Linear Logistic Test Model (Fisher, 1973) it was found that the number of propositions, the syntactic structure, and fundamentally, the presence of difficult words contributed to the prediction of the item difficulty level.

Author Biographies

Armel Brizuela, Universidad de Costa Rica

Filólogo y especialista en métodos de investigación cuantitativa en el campo de la psicología. Labora como investigador en el Programa Permanente de la Prueba de Aptitud Académica y en el Programa de Pruebas Específicas para Ingreso a Carrera (adscritos al Instituto de Investigaciones Psicológicas de la Universidad de Costa Rica) .

Eiliana Montero-Rojas, Universidad de Costa Rica

Doctora en Medición y Evaluación Educativa de la Universidad Estatal de Florida, USA, y bachiller en Estadística de la Universidad de Costa Rica (UCR). Catedrática de la UCR desde 1993, se desempeña como docente e investigadora en la Escuela de Estadística, en el Instituto de Investigaciones Psicológicas y en diversos programas de posgrado. Autora o coautora de más de 30 publicaciones científicas. Sus áreas de trabajo incluyen modelos mixtos (multinivel) y modelos de ecuaciones estructurales, evaluación de impacto, modelos de medición, especialmente IRT y Rasch, y validación de instrumentos e indicadores. Su experiencia en consultorías incluye a las siguientes organizaciones: ETS (Educational Testing Service) de Estados Unidos, Fundación Jacobs (Suiza), INCAE Business School, UNAIDS, Universidad de Chile, Escuela Andaluza de Economía Social, Programa Estado de la Educación de Costa Rica, Ministerio de Educación de Costa Rica y Ministerio de Educación de Nicaragua. La American Statistical Association de Estados Unidos le otorgó la distinción “Educational Ambassador 2010-2011”

References

Attoresi, H.F., Picón, J., Abal, F., Aguerri, M. & Galibert, M.S. (2009). Aplicación del modelo LLTM de Fisher al análisis de las fuentes de dificultad de ítemes de razonamiento deductivo. Interdisciplinaria, 26(1), 77-93.

Baker, F. (1993). Sensitivity of the Linear Logistic Test Model to Misspecification of the Weight Matrix. Applied Psychological Measurement, 17(3), 201-210. doi: 10.1177/014662169301700301

http://dx.doi.org/10.1177/014662169301700301

Belinchón, M., Igoa, J. & Rivière, A. (1998). Psicología del lenguaje: Investigación y teoría. Espa-a: Editorial Trotta.

Bond, T. & Fox, C. (2001). Applying the Rasch Model: Fundamental Measurement in the Human Sciences. Estados Unidos: Lawrence Erlbaum Associates, Inc.

Borsboom, D., Mellenbergh, G.J. & Heerden, J. (2004). The Concept of Validity. Psychological Review, 111(4), 1061–1071. doi: 10.1037/0033-295X.111.4.1061

http://dx.doi.org/10.1037/0033-295X.111.4.1061

Chen, Y., MacDonald, G. & Leu, Y. (2011). Validating Cognitive Sources of Mathematics Item Difficulty: Application of the LLTM to Fraction Conceptual Items. The International Journal of Educational and Psychological Assessment, 7(2), 74-93. Recuperado de https://sites.google.com/site/tijepa2012/home

Clark-Carter, D. (2004). Quantitative Psychological Research. Estados Unidos: Psychology Press.

Cortada de Kohan, N. (2003). Posibilidad de integración de las teorías cognitivas y la psicometría moderna. Revista Argentina de Neuropsicología, 1, 8-23.

Cronbach, L.J. & Meehl, P.E. (1955). Construct Validity in Psychological Test. Psychological Bulletin, 52(2), 281-302.

http://dx.doi.org/10.1037/h0040957

Daneman, M. & Hannon, B. (2001). Using Working Memory Theory to Investigate the Construct Validity of Multiple-Choice Reading Comprehension Tests Such as the SAT. Journal of Experimental Psychology: General, 130(2), 208-233.

http://dx.doi.org/10.1037/0096-3445.130.2.208

Davey, B. (1988). Factors Affecting the Difficulty of Reading Comprehension Items for Successful and Unsuccessful Readers. The Journal of Experimental Education, 56(2), 67-76.

http://dx.doi.org/10.1080/00220973.1988.10806468

De Boeck, P. & Wilson, M. (2004). Explanatory Item Response Models: A Generalized Linear and Nonlinear Approach. NY: Springer.

http://dx.doi.org/10.1007/978-1-4757-3990-9

DeMauro, G., Merritt, A. & Adams, R. (1994). Delimiting the Verbal Domain (Research Report RR-94-34). Princeton, NJ: Educational Testing Service.

Drum, P.A., Calfee, R.C. & Cook, L.K. (1981). The Effects of Surface Structure Variables on Performance in Reading Comprehension. Reading Research Quarterly, 16(4), 486-514.

http://dx.doi.org/10.2307/747313

Embretson, S. & Daniel, R. (2008). Understanding and quantifying cognitive complexity level in mathematical problem solving items. Psychology Science Quarterly, 50(3), 328-344.

Embretson, S. E. & Wetzel, D. (1987). Component Latent Trait Models for Paragraph Comprehension Tests. Applied Psychological Measurement, 11(2), 175-193. doi: 10.1177/014662168701100207

http://dx.doi.org/10.1177/014662168701100207

Embretson, S. E. (1996). The New Rules of Measurement. Psychological Assessment, 8(4), 341-349.

http://dx.doi.org/10.1037/1040-3590.8.4.341

Embretson, S. E. (2010). Cognitive Design Systems: A Structural Modeling Approach Applied to Developing a Spatial Ability Test. En S. Embretson (Ed.), Measuring Psychological Constructs: Advances in Model-Based Approaches (pp. 247-271). Washington, DC: American Psychological Association.

Embretson, S.E. & Gorin, J. (2001). Improving Construct Validity with Cognitive Psychology Principles. Journal of Educational Measurement, 38(4), 343-368. doi: 10.1111/j.1745-3984.2001.tb01131.x

http://dx.doi.org/10.1111/j.1745-3984.2001.tb01131.x

Embretson, S.E. (1989). Latent Trait Models as an Information-Processing Approach to Testing. International Journal of Educational Research, 13(2), 189-203. doi:10.1016/0883-0355(89)90006-2

http://dx.doi.org/10.1016/0883-0355(89)90006-2

Escudero, I. (2010). Las inferencias en la comprensión lectora: una ventana hacia los procesos cognitivos en segundas lenguas. Revista Nebrija de Lingüística Aplicada, 7(4), 1-32. Recuperado de http://www.nebrija.com/revista-linguistica/

Farr, R., Pritchard, R. & Smitten, B. (1990). A Description of What Happens When an Examinee Takes a Multiple-Choice Reading Comprehension Test. Journal of Educational Measurement, 27(3), 209-226. doi: 10.1111/j.1745-3984.1990.tb00744.x

http://dx.doi.org/10.1111/j.1745-3984.1990.tb00744.x

Fischer, G. (1997). Unidimensional Linear Logistic Rasch Model. En W. Van Der Linden y R. Hambleton, Handbook of Modern Item Response Theory. New York: Springer-Verlag.

Fisher, G. H. (1973). The Linear Logistic Test Model as an Instrument in Educational Research. Acta Psychologica, 37, 359-374. doi: 10.1016/0001-6918(73)90003-6

http://dx.doi.org/10.1016/0001-6918(73)90003-6

Freedle, R. & Kostin, I. (1991). The Prediction of SAT Reading Comprehension Item Difficulty for Expository Prose Passages (Research Report RR-91-29). Princeton, NJ: Educational Testing Service.

Freedle, R. & Kostin, I. (1992). The Prediction of GRE Reading Comprehension Item Difficulty for Expository Prose Passages for each of Three Item Types: Main Ideas, Inferences and Explicit Statements (GRE Board Report No. 87-1OP). Princeton, NJ: Educational Testing Service.

Gilbert, M.S., Picón, J.C., Lozzia, G.S., Aguerri, M.E. & Attorresi, H.F. (2010). Componentes de dificultad de ítems para la evaluación de operaciones lógicas / Una aplicación del modelo LLTM. SUMMA Psicológica UST, 7(1), 3-14.

Gilhooly, K. (2004). Working Memory and Reasoning. En J. Leighton y R. Sternberg (Eds.), The Nature of Reasoning (pp. 49-77). Estados Unidos: Cambridge University Press.

Gorin, J. S. (2007). Reconsidering Issues in Validity Theory. Educational Researcher, 36(8), 456-462. doi: 10.3102/0013189X07311607

http://dx.doi.org/10.3102/0013189X07311607

Gorin, J.S. & Embretson, S.E. (2006). Item Difficulty Modeling of Paragraph Comprehension Items. Applied Psychological Measurement, 30(5), 394-411. doi: 10.1177/0146621606288554

http://dx.doi.org/10.1177/0146621606288554

Gorin, J.S. (2005). Manipulating Processing Difficulty of Reading Comprehension Questions: The Feasibility of Verbal Item Generation. Journal of Educational Measurement, 42(4), 351-373. doi: 10.1111/j.1745-3984.2005.00020.x

http://dx.doi.org/10.1111/j.1745-3984.2005.00020.x

Graesser, A. C., Millis, K. & Zwaan, R.A. (1997). Discourse Comprehension. Annual Review of Psychology, 48, 163-189. doi: 10.1146/annurev.psych.48.1.163

http://dx.doi.org/10.1146/annurev.psych.48.1.163

Green, K. & Smith, R. (1987). A Comparison of Two Methods of Decomposing Item Difficulties. Journal of Educational and Behavioral Statistics, 12(4), 369-381. doi: 10.3102/10769986012004369

http://dx.doi.org/10.3102/10769986012004369

Kintsch, W. & van Dijk, T. (1978). Toward a Model of Text Comprehension and Production. Psychological Review, 85(5), 363-394. doi: 10.1037/0033-295X.85.5.363

http://dx.doi.org/10.1037/0033-295X.85.5.363

Hunt, E. (2011). Human Intelligence. Estados Unidos: Cambridge University Press

Iza, M. & Ezquerro, J. (2000). Elaborative Inferences. Anales de Psicología, 16(2), 227-249.

Kintsch, W. (1998). Comprehension: A Paradigm for Cognition. NY: Cambridge University Press.

Kirsch, I. & Mosenthal. (1990). Exploring Document Literacy Underlying the Performance of Young Adults. Reading Research Quarterly, 25(1), 5-30.

http://dx.doi.org/10.2307/747985

Leroy, G., Helmreich, S. y Cowie, J.R. (2010). The influence of text characteristics on perceived and actual difficulty of health information. International Journal of Medical Informatics. Recuperado de http://web.cgu.edu/faculty/leroy/content/Papers/2010-Leroy-IJMedInform-Perceived-and-Actual-Text-Difficulty.pdf. doi:10.1016/j.ijmedinf.2010.02.002

http://dx.doi.org/10.1016/j.ijmedinf.2010.02.002

López, I. & Pita, S. (1999). Medidas de concordancia: el índice de Kappa. Recuperado de http://www.fisterra.com/mbe/investiga/kappa/kappa2.pdf

Mair, P. & Hatzinger, R. (2012). Extended Rasch Modeling: The R Package eRm [Vignette]. Recuperado de http://cran.r-project.org/web/packages/eRm/eRm.pdf

Martínez, M., Hernández, M. & Hernández, M. (2006). Psicometría. Madrid: Alianza Editorial.

Messick, S. (1995). Validity of Psychological Assessment. American Psychologist, 50(9), 741-749. doi: 10.1037/0003-066X.50.9.741

http://dx.doi.org/10.1037/0003-066X.50.9.741

Ozuru, Y., Rowe, M., O'Reilly, T. & McNamara, D. (2008). Where's the difficulty in standardized reading tests: The passage or the question? Behavior Research Methods, 40(4), 1001-1015. doi: 10.3758/BRM.40.4.1001

http://dx.doi.org/10.3758/BRM.40.4.1001

Poinstingl, H. (2009). The Linear Logistic Test Model (LLTM) as the methodological foundation of item generating rules for a new verbal reasoning test. Psychology Science Quarterly, 51(2), 123-134.

Powers, D.E. & Wilson, S. (1995). Answering the New SAT Reading Comprehension Questions without the Passages. Journal of Educational Measurement, 32(2), 105-129. doi: 10.1111/j.1745-3984.1995.tb00458.x

http://dx.doi.org/10.1111/j.1745-3984.1995.tb00458.x

Real, E., Olea, J., Ponsoda, V., Revuelta, J. & Abad, F. (1999). Análisis de la dificultad de un test de matemáticas mediante un modelo componencial. Psicológica, 20, 121-134.

Revuelta, J. & Ponsoda, V. (1998). Un test adaptativo informatizado de análisis lógico basado en la generación automática de ítems. Psicothema, 10(3), 709-716.

Ríos, G. (2011). Características del lenguaje de los jóvenes costarricenses desde la disponibilidad léxica. Espa-a: Ediciones Universidad de Salamanca.

Romero, S., Ponsoda, V. & Ximénez, C. (2008). Análisis de un test de aritmética mediante el modelo logístico lineal de rasgo latente. Revista Latinoamericana de Psicología, 40(1), 85-95.

Rupp, A., Ferne, T. & Choi, H. (2006). How Assessing Reading Comprehension with Multiple-Choice Questions Shapes the Construct: A Cognitive Processing Perspective. Language Testing, 23(4), 441-474. doi: 10.1191/0265532206lt337oa

http://dx.doi.org/10.1191/0265532206lt337oa

Sánchez, V. (s.f.). Corpus de frecuencia de uso de vocabulario en estudiantes de secundaria [datos no procesados]. Universidad de Costa Rica: Estudios de Lexicografía Hispano-Costarricense.

Sheehan, K. & Mislevy, R. (1990). Integrating Cognitive and Psychometric Models to Measure Document Literacy. Journal of Educational Measurement, 27(3), 255-272. doi: 10.1111/j.1745-3984.1990.tb00747.x

http://dx.doi.org/10.1111/j.1745-3984.1990.tb00747.x

Sheehan, K. M., & Ginther, A. (2001). What do Passage-Based Multiple-Choice Verbal Reasoning Items Really Measure? An Analysis of the Cognitive Skills Underlying Performance on the Current TOEFL Reading Section. Paper presented at the 2000 Annual Meeting of the National Council of Measurement in Education, New Orleans, LA.

Sonnleitner, P. (2008). Using the LLTM to Evaluate an Item-Generating System for Reading Comprehension. Psychology Science Quarterly, 50(3), 345-362.

Stahl, S.A. & Jacobson, M.J. (1986). Vocabulary Difficulty, Prior Knowledge, and Text Comprehension. Journal of Reading Behavior, 28(4), 309-323. doi: 10.1080/10862968609547578

http://dx.doi.org/10.1080/10862968609547578

Tabachnick, B. & Fidell, L. (2007). Using Multivariate Statistics. New York: Pearson Education.

Downloads

Published

2013-12-24

Issue

Vol. 19 No. 2 (2013)

Section

Research Articles

License

The authors grant non-exclusive rights of exploitation of works published to RELIEVE and consent to be distributed under the Creative Commons Attribution-Noncommercial Use 4.0 International License (CC-BY-NC 4.0), which allows third parties to use the published material whenever the authorship of the work and the source of publication is mentioned, and it is used for non-commercial purposes. 

    The authors can reach other additional and independent contractual agreements, for the non-exclusive distribution of the version of the work published in this journal (for example, by including it in an institutional repository or publishing it in a book), as long as it is clearly stated that the Original source of publication is this magazine.  

    Authors are encouraged to disseminate their work after it has been published, through the internet (for example, in institutional archives online or on its website) which can generate interesting exchanges and increase work appointments. 

   The fact of sending your paper to RELIEVE implies that you accept these conditions.