Intended and unintended interpretations and uses of PISA results: A consequential validity perspective
DOI:
https://doi.org/10.7203/relieve.22.1.8294Keywords:
PISA, validity, uses of test scores, validation, developing tests
Abstract
This paper explores the relevance of considering the consequences of testing as part of discussions about the validity, and validation research, in the context of the OECD Programme for International Student Asssessment, PISA. We first describe the modern conception of validity as a core aspect of quality of tests and testing systems, evolving around the proposed interpretations and uses of test scores: “Validity refers to the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests. Validity is, therefore, the most fundamental consideration in developing tests and evaluating tests.” (AERA, APA & NCME, 2014, p. 11). In particular, we focus on the role that consequences have played in the literature on test validity and validation. We then introduce PISA and its intended interpretations and uses as the basis for examining its validity. This is followed by summarizing existing empirical studies on the uses and consequences of PISA. Finally, the paper presents missing pieces in the validity evidence related to consequences and discusses the importance of a pro-active agenda on these topics by the PISA stakeholders at international and national levels.
Author Biographies
Sandy Taut, Pontificia Universidad Católica de Chile
Diego Palacios, Pontificia Universidad Católica de Chile
References
American Educational Research Association, American Psychological Association & National Council for Measurement in Education [AERA, APA & NCME] (2014). The Standards for Educational and Psychological Testing. Washington, D.C.: AERA.
Amoroso, J. M., Moreno, J. M., Gortazar, L., Herrera Sosa, K. M., Kutner, D., & Bodewig, C. (2015). Poland - Skilling up the next generation : an analysis of Poland’s performance in the program for international student assessment, 1–21. Retrieved from http://documents.worldbank.org/curated/en/2015/12/25518729/poland-skilling-up-next-generation-analysis-poland%E2%80%99s-performance-program-international-student-assessment
Baird, J.-A., Isaacs, T., Johnson, S., Stobart, G., Yu, G., Sprague, T., & Daugherty, R. (2011). Policy effects of PISA. Retrieved from http://research-information.bristol.ac.uk/en/publications/policy-effects-of-pisa(833739c4-7e0a-4c18-b249-a3f12120065f).html
Bonal, X., & Tarabini, A. (2013). The role of PISA in shaping hegemonic educational discourses, policies and practices: The case of spain. Research in Comparative and International Education, 8(3), 335–341. http://dx.doi.org/10.2304/rcie.2013.8.3.335
Breakspear, S. (2012). The policy impact of PISA: An Exploration of the Normative Effects of International Benchmarking in School System Performance. OECD Journals, (71), 1–32. DOI: http://dx.doi.org/10.1787/19939019
Brennan, R. (2006). Perspectives on the Evolution and Future of Educational Measurement. In R. Brennan (ed.), Educational Measurement, 4th ed., pp. 1-16. Westport, CT: Praeger.
Carabaña, J. (2008). Las diferencias entre pa´ises y regiones en las pruebas PISA. Retrieved from papers3://publication/uuid/2C1CA410-1F9E-49FE-8384-85AC6D3F3C84
Choi, Á., & Jerrim, J. (2015). The Use (and Misuse) of PISA in Guiding Policy Reform: The Case of Spain. SSRN Electronic Journal, 1–16. Doi: http://dx.doi.org/10.2139/ssrn.2580141
Chung, J. (2016). The (mis)use of the Finnish teacher education model: “policy-based evidence-making”? Educational Research. Retrieved from http://www.tandfonline.com/doi/full/10.1080/00131881.2016.1167485
Cronbach, L. (1988). Five perspectives on the validity argument. In H. Wainer & H. Braun (eds.), Test validity, pp. 3-17. Hillsdale, NJ: Lawrence Erlbaum.
Elstad, E. (2010). Pisa Debates and Blame Management Among the Norwegian Educational Authorities : Press Coverage and, 48, 10–22.
Ertl, H. (2006). Educational standards and the changing discourse on education: the reception and consequences of the PISA study in Germany. Oxford Review of Education, 32(5), 619–634. DOI: http://dx.doi.org/10.1080/03054980600976320
Gür, B. S., Çelik, Z., & Özoğlu, M. (2012). Policy options for Turkey: a critique of the interpretation and utilization of PISA results in Turkey. Journal of Education Policy, 27(1), 1–21. DOI: http://dx.doi.org/10.1080/02680939.2011.595509
Instituto Nacional de Evaluación Educativa. (2014). PISA 2012 Resolución de problemas de la vida real. Resultados de matemáticas y lectura por ordenador. Informe Español. Versión preliminar. Instituto Nacional de Evaluación Educativa. Retrieved from http://www.mecd.gob.es/dctm/inee/internacional/pisa2012-resolucionproblemas/pisaresoluciondeproblemas.pdf?documentId=0901e72b8198bee8
Instituto Nacional para la Evaluación de la Educación. (2013). México en 2012. Mexico: Instituto Nacional parala Evaluación de la Educación. Retrieved from http://www.inee.edu.mx/images/stories/2013/principal/PISA2013/PISA_2012041213web1.pdf
Jakubowski, M. (2015). Opening up opportunities: education reforms in Poland, (January).
Jornet, J. (2013, January 30). Cuestionados los supuestos malos datos españoles del informe Pisa. Comunidad Valenciana. Valencia. Retrieved from http://ccaa.elpais.com/ccaa/2013/01/30/valencia/1359572336_318312.html
Jornet, J. (2016a). España en PISA. Valencia: Ateneo Mercantil de Valencia.
Jornet, J. (2016b, January 26). "La educación no está tan mal; el informe PISA. Levante. El Mercantil Valenciano, p. 10. Valencia.
Jornet, J. (2016c, January 26). Cómo desmontar el informe PISA. Las Provincias. Valencia. Retrieved from http://www.lasprovincias.es/comunitat/201601/26/como-desmontar-informe-pisa-20160126001834-v.html
Kane, M. (2006). Validity. In Brennan, R. (ed.), Educational Measurement, 4th ed., pp. 17-64. Westport, CT: Praeger.
Kane, M. (2013). The argument-based approach to validation. School Psychology Review, 42(4), 448-457.
Linn, R. (1998). Partitioning responsibility for the evaluation of the consequences of use. Educational Measurement: Issues and Practice, 17(2), 28-30.
Martens, D. K., Nagel, A.-K., Windzio, M., & Weymann, A. (2010). Transformation of Education Policy. Basingstoke: Palgrave.
Martinez Rizo, F. et al. (2015). Las pruebas ENLACE y Excale. Un estudio de validación. Cuaderno de Investigación No. 40. México. DF: Instituto Nacional para la Evaluación de la Educación.
Mehrens, W. (1997). The consequences of consequential validity. Educational Measurement: Issues and Practice, 16(2), 16-18.
Messick, S. (1989). Validity. In R. Linn (ed.), Educational measurement (3rd ed.), pp. 13-103. Washington, D.C.: American Council on Education.
Meyer, H.-D. & Zahedi, K. (2014, May 4). Open Letter to Andreas Schleicher, OECD, Paris. Retrieved from http://www.globalpolicyjournal.com/blog/05/05/2014/open-letter-andreas-schleicher-oecd-paris
Ministerio de Educación de Chile. (2012). Evidencias para Políticas Públicas en Educación: Selección de Investigaciones Concurso Extraordinario FONIDE-PISA. Santiago de Chile: Ministerio de Educación de Chile. Retrieved from https://s3.amazonaws.com/archivos.agenciaeducacion.cl/documentos-web/Estudios+Internacionales/PISA/Evidencias_para_Politicas_Publicas_en_Educacion_FONIDE_PISA.pdf
Ministerio de Educación de Chile. (2014). Informe Nacional Resultados Chile Pisa 2012. Santiago de Chile: MINEDUC. Retrieved from https://s3.amazonaws.com/archivos.agenciaeducacion.cl/documentos-web/Estudios+Internacionales/PISA/Informe_Nacional_Resultados_Chile_PISA_2012.pdf
Neumann, K., Fischer, H. E., & Kauertz, A. (2010). FROM PISA TO EDUCATIONAL STANDARDS: THE IMPACT OF LARGE-SCALE ASSESSMENTS ON SCIENCE EDUCATION IN GERMANY. International Journal of Science and Mathematics Education, 8(3), 545–563. http://doi.org/10.1007/s10763-010-9206-7
Organisation for Economic Co-Operation and Development (2008). External evaluation of the policy impact of PISA, (November), 3–5. Retrieved from http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=EDU/PISA/GB(2008)35/REV1&docLanguage=En
Organisation for Economic Co-Operation and Development (2010). PISA 2009 Results: Executive Summary. Executive Summary, 1–21. Retrieved from http://www.oecd.org/pisa/pisaproducts/46619703.pdf
Organisation for Economic Co-Operation and Development (2013). General guidelines for the availability and uses of the PISA-based test for schools. Retrieved on May 15, 2016 from https://www.oecd.org/pisa/aboutpisa/PISA-based-test-for-schools-guidelines.pdf
Organisation for Economic Co-Operation and Development (2014, April). Education Policy Outlook Spain. Retrieved May 1, 2016, from http://www.oecd.org/edu/EDUCATION%20POLICY%20OUTLOOK%20SPAIN_EN.pdf
Organisation for Economic Co-Operation and Development (2014). PISA 2012 Technical Report. Retrieved on May 1, 2016, from https://www.oecd.org/pisa/pisaproducts/PISA-2012-technical-report-final.pdf
Popham, W. (1997). Consequential validity: Right concern - wrong concept. Educational Measurement: Issues and Practice, 16(2), 9-13.
Prenzel, M., Sälzer, C., Klieme, E. & Köller, O. (eds.) (2013). PISA 2012. Fortschritte und Herausforderungen in Deutschland (PISA 2012. Improvements and challenges in Germany). Münster: Waxmann.
Ravela, P. (2011). ¿ Qué hacer con los resultados de PISA en América Latina ? PREAL. Programa de Promoción de La Reforma Educativa En América Latina Y El Caribe, 58.
Schafer, W., Wang; J. & Wang, V. (2009). Validity in action: State assessment validity evidence for compliance with NCLB. In R. Lissitz (ed.), The concept of validity, pp. 173-193. Charlotte, NC: Information Age Publishing.
Shepard, L. (1997). The centrality of test use and consequences for test validity. Educational Measurement: Issues and Practice, 16(2), 5-8.
Stobart, G., & Eggen, T. (2012). High-stakes testing – value, fairness and consequences. Assessment in Education: Principles, Policy & Practice, 19(1), 1–6. DOI: http://dx.doi.org/10.1080/0969594X.2012.639191
Takayama, K. (2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education, 44(4), 387–407. DOI: http://dx.doi.org/10.1080/03050060802481413
Taut, S., Santelices, V. & Stecher, B. (2012). Validation of a national teacher assessment and improvement system. Educational Assessment Journal, 17(4), 163-199. DOI: DOI: http://dx.doi.org/10.1080/10627197.2012.735913
Taut, S., Santelices, V., Araya, C. & Manzi, J. (2010). Theory unterlying a national teacher evaluation program. Evaluation and Program Planning, 33, 477-489. DOI: DOI: http://dx.doi.org/10.1016/j.evalprogplan.2010.01.002
Tveit, S. (2013). Educational assessment in Norway. Assessment in Education: Principles, Policy & Practice, 21(2), 221–237. DOI: http://dx.doi.org/10.1080/0969594X.2013.830079
Urteaga, E. (2010). Los resultados del estudio PISA en Francia. Revista Complutense de Educación, 21, 231–244.
Downloads
Published
Issue
Section
License
The authors grant non-exclusive rights of exploitation of works published to RELIEVE and consent to be distributed under the Creative Commons Attribution-Noncommercial Use 4.0 International License (CC-BY-NC 4.0), which allows third parties to use the published material whenever the authorship of the work and the source of publication is mentioned, and it is used for non-commercial purposes.
The authors can reach other additional and independent contractual agreements, for the non-exclusive distribution of the version of the work published in this journal (for example, by including it in an institutional repository or publishing it in a book), as long as it is clearly stated that the Original source of publication is this magazine.
Authors are encouraged to disseminate their work after it has been published, through the internet (for example, in institutional archives online or on its website) which can generate interesting exchanges and increase work appointments.
The fact of sending your paper to RELIEVE implies that you accept these conditions.
Language
Information