Intended and unintended interpretations and uses of PISA results: A consequential validity perspective

作者

  • Sandy Taut Pontificia Universidad Católica de Chile
  • Diego Palacios Pontificia Universidad Católica de Chile

##plugins.pubIds.doi.readerDisplayName##:

https://doi.org/10.7203/relieve.22.1.8294

关键词:

PISA, validity, uses of test scores, validation, developing tests

摘要

This paper explores the relevance of considering the consequences of testing as part of discussions about the validity, and validation research, in the context of the OECD Programme for International Student Asssessment, PISA. We first describe the modern conception of validity as a core aspect of quality of tests and testing systems, evolving around the proposed interpretations and uses of test scores: “Validity refers to the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests. Validity is, therefore, the most fundamental consideration in developing tests and evaluating tests.” (AERA, APA & NCME, 2014, p. 11). In particular, we focus on the role that consequences have played in the literature on test validity and validation. We then introduce PISA and its intended interpretations and uses as the basis for examining its validity. This is followed by summarizing existing empirical studies on the uses and consequences of PISA. Finally, the paper presents missing pieces in the validity evidence related to consequences and discusses the importance of a pro-active agenda on these topics by the PISA stakeholders at international and national levels.

##submission.authorBiographies##

##submission.authorWithAffiliation##

Degree in Psychology from the University of Cologne (Germany) and her Ph.D. in Education from the University of California Los Angeles (UCLA, USA). She is an assistant professor at the School of Psychology at Pontificia Universidad Católica de Chile and an associate researcher at the Measurement Center MIDE UC.  She is de Corresponding author. Her address is: Pontificia Universidad Católica de Chile, Escuela de Psicología, Centro de Medición MIDE UC, Avda Vicuña Mackenna 4860, Macul, Santiago (Chile).

##submission.authorWithAffiliation##

Associate researcher at the Measurement Center MIDE UC. at Pontificia Universidad Católica de Chile. His postal address is: Pontificia Universidad Católica de Chile, Escuela de Psicología, Centro de Medición MIDE UC, Avda Vicuña Mackenna 4860, Macul, Santiago (Chile)

参考

American Educational Research Association, American Psychological Association & National Council for Measurement in Education [AERA, APA & NCME] (2014). The Standards for Educational and Psychological Testing. Washington, D.C.: AERA.

Amoroso, J. M., Moreno, J. M., Gortazar, L., Herrera Sosa, K. M., Kutner, D., & Bodewig, C. (2015). Poland - Skilling up the next generation : an analysis of Poland’s performance in the program for international student assessment, 1–21. Retrieved from http://documents.worldbank.org/curated/en/2015/12/25518729/poland-skilling-up-next-generation-analysis-poland%E2%80%99s-performance-program-international-student-assessment

Baird, J.-A., Isaacs, T., Johnson, S., Stobart, G., Yu, G., Sprague, T., & Daugherty, R. (2011). Policy effects of PISA. Retrieved from http://research-information.bristol.ac.uk/en/publications/policy-effects-of-pisa(833739c4-7e0a-4c18-b249-a3f12120065f).html

Bonal, X., & Tarabini, A. (2013). The role of PISA in shaping hegemonic educational discourses, policies and practices: The case of spain. Research in Comparative and International Education, 8(3), 335–341. http://dx.doi.org/10.2304/rcie.2013.8.3.335

Breakspear, S. (2012). The policy impact of PISA: An Exploration of the Normative Effects of International Benchmarking in School System Performance. OECD Journals, (71), 1–32. DOI: http://dx.doi.org/10.1787/19939019

Brennan, R. (2006). Perspectives on the Evolution and Future of Educational Measurement. In R. Brennan (ed.), Educational Measurement, 4th ed., pp. 1-16. Westport, CT: Praeger.

Carabaña, J. (2008). Las diferencias entre pa´ises y regiones en las pruebas PISA. Retrieved from papers3://publication/uuid/2C1CA410-1F9E-49FE-8384-85AC6D3F3C84

Choi, Á., & Jerrim, J. (2015). The Use (and Misuse) of PISA in Guiding Policy Reform: The Case of Spain. SSRN Electronic Journal, 1–16. Doi: http://dx.doi.org/10.2139/ssrn.2580141

Chung, J. (2016). The (mis)use of the Finnish teacher education model: “policy-based evidence-making”? Educational Research. Retrieved from http://www.tandfonline.com/doi/full/10.1080/00131881.2016.1167485

Cronbach, L. (1988). Five perspectives on the validity argument. In H. Wainer & H. Braun (eds.), Test validity, pp. 3-17. Hillsdale, NJ: Lawrence Erlbaum.

Elstad, E. (2010). Pisa Debates and Blame Management Among the Norwegian Educational Authorities : Press Coverage and, 48, 10–22.

Ertl, H. (2006). Educational standards and the changing discourse on education: the reception and consequences of the PISA study in Germany. Oxford Review of Education, 32(5), 619–634. DOI: http://dx.doi.org/10.1080/03054980600976320

Gür, B. S., Çelik, Z., & Özoğlu, M. (2012). Policy options for Turkey: a critique of the interpretation and utilization of PISA results in Turkey. Journal of Education Policy, 27(1), 1–21. DOI: http://dx.doi.org/10.1080/02680939.2011.595509

Instituto Nacional de Evaluación Educativa. (2014). PISA 2012 Resolución de problemas de la vida real. Resultados de matemáticas y lectura por ordenador. Informe Español. Versión preliminar. Instituto Nacional de Evaluación Educativa. Retrieved from http://www.mecd.gob.es/dctm/inee/internacional/pisa2012-resolucionproblemas/pisaresoluciondeproblemas.pdf?documentId=0901e72b8198bee8

Instituto Nacional para la Evaluación de la Educación. (2013). México en 2012. Mexico: Instituto Nacional parala Evaluación de la Educación. Retrieved from http://www.inee.edu.mx/images/stories/2013/principal/PISA2013/PISA_2012041213web1.pdf

Jakubowski, M. (2015). Opening up opportunities: education reforms in Poland, (January).

Jornet, J. (2013, January 30). Cuestionados los supuestos malos datos españoles del informe Pisa. Comunidad Valenciana. Valencia. Retrieved from http://ccaa.elpais.com/ccaa/2013/01/30/valencia/1359572336_318312.html

Jornet, J. (2016a). España en PISA. Valencia: Ateneo Mercantil de Valencia.

Jornet, J. (2016b, January 26). "La educación no está tan mal; el informe PISA. Levante. El Mercantil Valenciano, p. 10. Valencia.

Jornet, J. (2016c, January 26). Cómo desmontar el informe PISA. Las Provincias. Valencia. Retrieved from http://www.lasprovincias.es/comunitat/201601/26/como-desmontar-informe-pisa-20160126001834-v.html

Kane, M. (2006). Validity. In Brennan, R. (ed.), Educational Measurement, 4th ed., pp. 17-64. Westport, CT: Praeger.

Kane, M. (2013). The argument-based approach to validation. School Psychology Review, 42(4), 448-457.

Linn, R. (1998). Partitioning responsibility for the evaluation of the consequences of use. Educational Measurement: Issues and Practice, 17(2), 28-30.

Martens, D. K., Nagel, A.-K., Windzio, M., & Weymann, A. (2010). Transformation of Education Policy. Basingstoke: Palgrave.

Martinez Rizo, F. et al. (2015). Las pruebas ENLACE y Excale. Un estudio de validación. Cuaderno de Investigación No. 40. México. DF: Instituto Nacional para la Evaluación de la Educación.

Mehrens, W. (1997). The consequences of consequential validity. Educational Measurement: Issues and Practice, 16(2), 16-18.

Messick, S. (1989). Validity. In R. Linn (ed.), Educational measurement (3rd ed.), pp. 13-103. Washington, D.C.: American Council on Education.

Meyer, H.-D. & Zahedi, K. (2014, May 4). Open Letter to Andreas Schleicher, OECD, Paris. Retrieved from http://www.globalpolicyjournal.com/blog/05/05/2014/open-letter-andreas-schleicher-oecd-paris

Ministerio de Educación de Chile. (2012). Evidencias para Políticas Públicas en Educación: Selección de Investigaciones Concurso Extraordinario FONIDE-PISA. Santiago de Chile: Ministerio de Educación de Chile. Retrieved from https://s3.amazonaws.com/archivos.agenciaeducacion.cl/documentos-web/Estudios+Internacionales/PISA/Evidencias_para_Politicas_Publicas_en_Educacion_FONIDE_PISA.pdf

Ministerio de Educación de Chile. (2014). Informe Nacional Resultados Chile Pisa 2012. Santiago de Chile: MINEDUC. Retrieved from https://s3.amazonaws.com/archivos.agenciaeducacion.cl/documentos-web/Estudios+Internacionales/PISA/Informe_Nacional_Resultados_Chile_PISA_2012.pdf

Neumann, K., Fischer, H. E., & Kauertz, A. (2010). FROM PISA TO EDUCATIONAL STANDARDS: THE IMPACT OF LARGE-SCALE ASSESSMENTS ON SCIENCE EDUCATION IN GERMANY. International Journal of Science and Mathematics Education, 8(3), 545–563. http://doi.org/10.1007/s10763-010-9206-7

Organisation for Economic Co-Operation and Development (2008). External evaluation of the policy impact of PISA, (November), 3–5. Retrieved from http://www.oecd.org/officialdocuments/publicdisplaydocumentpdf/?cote=EDU/PISA/GB(2008)35/REV1&docLanguage=En

Organisation for Economic Co-Operation and Development (2010). PISA 2009 Results: Executive Summary. Executive Summary, 1–21. Retrieved from http://www.oecd.org/pisa/pisaproducts/46619703.pdf

Organisation for Economic Co-Operation and Development (2013). General guidelines for the availability and uses of the PISA-based test for schools. Retrieved on May 15, 2016 from https://www.oecd.org/pisa/aboutpisa/PISA-based-test-for-schools-guidelines.pdf

Organisation for Economic Co-Operation and Development (2014, April). Education Policy Outlook Spain. Retrieved May 1, 2016, from http://www.oecd.org/edu/EDUCATION%20POLICY%20OUTLOOK%20SPAIN_EN.pdf

Organisation for Economic Co-Operation and Development (2014). PISA 2012 Technical Report. Retrieved on May 1, 2016, from https://www.oecd.org/pisa/pisaproducts/PISA-2012-technical-report-final.pdf

Popham, W. (1997). Consequential validity: Right concern - wrong concept. Educational Measurement: Issues and Practice, 16(2), 9-13.

Prenzel, M., Sälzer, C., Klieme, E. & Köller, O. (eds.) (2013). PISA 2012. Fortschritte und Herausforderungen in Deutschland (PISA 2012. Improvements and challenges in Germany). Münster: Waxmann.

Ravela, P. (2011). ¿ Qué hacer con los resultados de PISA en América Latina ? PREAL. Programa de Promoción de La Reforma Educativa En América Latina Y El Caribe, 58.

Schafer, W., Wang; J. & Wang, V. (2009). Validity in action: State assessment validity evidence for compliance with NCLB. In R. Lissitz (ed.), The concept of validity, pp. 173-193. Charlotte, NC: Information Age Publishing.

Shepard, L. (1997). The centrality of test use and consequences for test validity. Educational Measurement: Issues and Practice, 16(2), 5-8.

Stobart, G., & Eggen, T. (2012). High-stakes testing – value, fairness and consequences. Assessment in Education: Principles, Policy & Practice, 19(1), 1–6. DOI: http://dx.doi.org/10.1080/0969594X.2012.639191

Takayama, K. (2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education, 44(4), 387–407. DOI: http://dx.doi.org/10.1080/03050060802481413

Taut, S., Santelices, V. & Stecher, B. (2012). Validation of a national teacher assessment and improvement system. Educational Assessment Journal, 17(4), 163-199. DOI: DOI: http://dx.doi.org/10.1080/10627197.2012.735913

Taut, S., Santelices, V., Araya, C. & Manzi, J. (2010). Theory unterlying a national teacher evaluation program. Evaluation and Program Planning, 33, 477-489. DOI: DOI: http://dx.doi.org/10.1016/j.evalprogplan.2010.01.002

Tveit, S. (2013). Educational assessment in Norway. Assessment in Education: Principles, Policy & Practice, 21(2), 221–237. DOI: http://dx.doi.org/10.1080/0969594X.2013.830079

Urteaga, E. (2010). Los resultados del estudio PISA en Francia. Revista Complutense de Educación, 21, 231–244.

##submission.downloads##

已出版

2016-07-08

期

卷 22 编号 1 (2016)

栏目

Monográfico

##submission.license##

Los autores ceden de forma no exclusiva los derechos de explotación de los trabajos publicados a RELIEVE (a los solos efectos de favorecer la difusión de los artículos publicados:firmar contratos de difusión, de integración en bases de datos, etc.) y consienten que se distribuyan bajo la licencia de Creative Commons Reconocimiento-Uso No Comercial 4.0 International (CC-BY-NC 4.0), que permite a terceros el uso de lo publicado siempre que se mencione la autoría de la obra y la fuente de publicación, y se haga uso sin fines comerciales.

Los autores pueden llegar a otros acuerdos contractuales adicionales e independientes, para la distribución no exclusiva de la versión del trabajo publicado en esta revista (por ejemplo, incluyéndolo en un repositorio institucional o publicándolo en un libro), siempre y cuando se cite claramente que la fuente original de publicación es esta revista.

A los autores se les anima a difundir su trabajo después de publicado, a través de internet (por ejemplo, en archivos institucionales en línea o en su página web) lo que puede generar intercambios interesantes y aumentar las citas del trabajo.

La mera remisión del artículo a RELIEVE supone la aceptación de estas condiciones.