PISA and TALIS, congruence or discrepancy?

Authors

  • Mª José Fernández-Díaz Universidad Complutense de Madrid
  • Jesús Miguel Rodríguez-Mantilla Universidad Complutense de Madrid
  • Angélica Martínez-Zarzuelo Universidad Complutense de Madrid

DOI:

https://doi.org/10.7203/relieve.22.1.8247

Keywords:

TALIS, PISA, OECD, educational assessment, international evaluations, academic achievement, teaching practice, teaching staff, research, compulsory secondary education.

Abstract

International educational evaluations on a grand scale provide information of great interest to the educational, scientific and political community. Many studies have derived from them, both nationally and internationally. On the other hand, it is well known from the beginning of these macro-assessments the diversity of opinions they have generated. In this context, the aim of the study presented here is to analyse and go deeply into the relationships between the results from PISA 2012 and those relating to teaching practice of secondary TALIS 2013, trying to find out the consistencies and discrepancies between the results of both, considering that these two evaluations should show certain coherence between the results obtained by students and teacher ratings. The descriptive analysis carried out taking the overall scores on both evaluations show obvious discrepancies, as countries occupying the top positions in the results of students in PISA are located in the last places in teaching practice of teachers. To go deeper into this aim data from TALIS-PISA link have been used, where it is assumed that more intense relationships should be produced as there is a direct link between students and teachers evaluated. The analysis also show the lack of the expected coherence. These results lead to reflect on the possible underlying causes and so it is proposed the need for a coordinated design of both evaluation processes.

Author Biographies

Mª José Fernández-Díaz, Universidad Complutense de Madrid

Decana de la Facultad de Educación de la Universidad Complutense de Madrid y Profesora del Departamento de Métodos de Investigación y Diagnóstico en Educación, acreditada para catedrática. Sus principales áreas de especialización son: metodología de investigación, evaluación y medida en educación, dirección y liderazgo, formación de profesores  y calidad y evaluación de centros, profesores y programas educativos. Ha dirigido varios proyectos I+D+i y participado en la evaluación de otros para el Ministerio de Economía y Competitividad y varias agencias, entre ellas la ANEP. Ha sido evaluadora externa de la Agencia de Calidad, Acreditación y Prospectiva de las Universidades de Madrid y Presidenta del Comité de Calidad de la Educación de la Asociación Española de la Calidad (AEC). Colabora con universidades extranjeras y otros organismos internacionales.

Jesús Miguel Rodríguez-Mantilla, Universidad Complutense de Madrid

Doctor en Educación, Licenciado en Psicopedagogía y Diplomado en Magisterio de Educación Primaria por la Universidad Complutense de Madrid, es profesor de Métodos de Investigación en la Universidad Complutense de Madrid, en la Universidad Camilo José Cela . Sus últimos trabajos y publicaciones se enmarcan en el estudio del síndrome de Burnout, Evaluación de Competencias docentes, Clima escolar y en el análisis de la Calidad de las instituciones educativas.

Angélica Martínez-Zarzuelo, Universidad Complutense de Madrid

Doctora en Educación por la Universidad Complutense de Madrid. Licenciada en Matemáticas por la Universidad de Valladolid, con Máster Universitario en Investigación Matemática por la Universidad Complutense de Madrid. Con experiencia en evaluaciones externas nacionales e internacionales del sistema educativo en sus diferentes fases: muestreo, aplicación, seguimiento, codificación, análisis de datos y realización de informes, en el marco del Instituto Nacional de Evaluación Educativa.

References

Albergaria-Almeida, P., da Silva Lopes, B., & Martinho, M. (2015). Student assessment strategies in Portugal: an analysis of TALIS 2013. Procedia-Social and Behavioral Sciences, 186, 841-846. doi:10.1016/j.sbspro.2015.04.143

Calero, J., & Escardíbul, J. O. (2012). El rendimiento del alumnado de origen inmigrante en PISA 2012. PISA, 4-31.

Carabaña, J. (2015). La inutilidad de PISA para las escuelas. Madrid: La Catarata.

Clotfelte, C.T., Ladd, H. F. y Vigdor, J. L. (2007). Teacher Credentials and Student Achievement in High School: A Cross Subject Analysis with Fixed Effects. Journal of Human Resources, 45, 655-681. DOI: 10.3386/w13617

Dohn, N. B. (2007). Knowledge and skills for PISA-Assessing the assessment. Journal of Philosophy of Education, 41(1), 1-16. DOI: 10.1111/j.1467-9752.2007.00542.x

Eveleigh, F., & Freeman, C. (2012). An exploratory analysis of the TALIS and PISA link data: an investigation of the possible relationships. The Need for Educational Research to Champion Freedom, Education and Development for All, ECER 2012.

Fernandez-Díaz, M. J., Rodríguez-Mantilla, J. M., & Martínez-Zarzuelo, A. (2015). Teaching Practice of Secondary Education Teachers in Spain according to TALIS 2013. Revista Española de Pedagogía, 73(261), 225-244.

Ferrer, F. (2012). PISA: Aportaciones e incidencia sobre las políticas educativas nacionales. Revista española de educación comparada, (19), 11-16. DOI: http://dx.doi.org/10.5944/reec.19.2012.7941

Font, C. M., Badia, M. C., Alemany, I. G., Besora, M. C., Gisbert, D. D., Arce, R. L., Alonso, P. C., Seuba, M. C., Castilla, M. N., Lamo, S. S., Valdivia, I. M. Á., Villanueva, L. V. y Boekaerts, M. (2009). Pisa como excusa.: Repensar la evaluación para cambiar la enseñanza (Vol. 24). Grao.

Grek, S. (2009). Governing by numbers: The PISA ‘effect’in Europe. Journal of education policy, 24(1), 23-37. DOI:10.1080/02680930802412669

Gumus, E., & Bellibas, M. S. (2016). The effects of professional development activities on principals’ perceived instructional leadership practices: multi-country data analysis using TALIS 2013. Educational Studies, 1-15. DOI:10.1080/03055698.2016.1172958

Gustafsson, J. E. (2003). What Do We Know About Effects of School Resources on Educational Results?. Swedish Economic Policy Review, 10, 77-110.

Hair, J., Anderson, R., Tathan, R. y Black, W. (2009). Análisis multivariante (Madrid, Pearson).

Hernández, F. H. (2006). El informe PISA: una oportunidad para replantear el sentido de aprender en la escuela secundaria. Revista de educación, (1), 357-379.

Jensen, B., Sandoval, A., Knoll, S y González, E.J. (2012). The Experience of New Teachers: Results from TALIS 2008. OECD Publishing.

Kaplan, D., & Turner, A. (2012). Statistical Matching of PISA 2009 and TALIS 2008 Data in Iceland, OECD Education Working Papers, 78. Paris: OECD Publishing.

Kelly, D., Nord, C. W., Jenkins, F., Chan, J. Y., & Kastberg, D. (2013). Performance of US 15-Year-Old Students in Mathematics, Science, and Reading Literacy in an International Context. First Look at PISA 2012. NCES 2014-024. National Center for Education Statistics.

Lizasoain, L., Tourón, J., & Sobrino, Á. (2015). La evaluación del profesorado español y el impacto del feedback en las prácticas docentes. Análisis de TALIS 2013. Revista Española de Pedagogía, 73(262), 465-481.

Martín, E., & Rizo, F. M. (2009). Avances y desafíos en la evaluación educativa. Organización de Estados Iberoamericanos para la Educación, la Ciencia y la Cultura.

ME, Ministerio de Educación. (2009). TALIS (OCDE) Estudio Internacional sobre la Enseñanza y Aprendizaje. Informe Español. Madrid: Ministerio de Educación, Subdirección General de Documentación y Publicaciones.

MECD, Ministerio de Educación, Cultura y Deporte. (2014a). PISA 2012 Programa para la evaluación internacional de los alumnos. Informe español. Resultados y contexto. Madrid: Ministerio de Educación, Cultura y Deporte.

MECD, Ministerio de Educación, Cultura y Deporte. (2014b). PISA 2012 Competencia Financiera. Informe español. Madrid: Ministerio de Educación, Cultura y Deporte.

MECD, Ministerio de Educación, Cultura y Deporte. (2014c). PISA 2012 Resolución de problemas de la vida real. Resultados de matemáticas y lectura por ordenador. Informe español. Madrid: Ministerio de Educación, Cultura y Deporte.

MECD, Ministerio de Educación, Cultura y Deporte. (2014d). TALIS 2013. Estudio internacional de la enseñanza y el aprendizaje. Informe español. Madrid: Ministerio de Educación, Cultura y Deporte.

Méndez, I. (2015). Prácticas Docentes y Rendimiento Estudiantil: Evidencia a partir de PISA 2012 y TALIS 2013. Conserjería de Educación, Cultura y Turismo del Gobierno de la Rioja, Fundación Santillana e Instituto Nacional de Evaluación Educativa.

Mendizábal, Á. C. (2016). Explorando los límites de PISA. RASE: Revista de la Asociación de Sociología de la Educación, 9(1), 163-165.

Ming-ren, Z. H. A. O. (2015). The Status and Implications of Teachers' Professional Development in An International Perspective: Based on the Results of TALIS 2013. Teacher Education Research, 3, 016.

Mortimore, P. (2009). Alternative models for analysing and representing countries’ performance in PISA. Paper commissioned by Education International Research Institute, Brussels, Belgium.

Neves, C. (2008). Las organizaciones internacionales y la evaluación de los sistemas de educación y formación: Análisis crítico y comparativo. Revista Europea de Formación Profesional, (45), 78-98.

OECD, Organisation for Economic Co-operation and Development. (2009). Creating effective teaching and learning environments: First results from TALIS. Paris: OECD Publishing.

OECD, Organisation for Economic Co-operation and Development (2013a). PISA 2012 Results: Excellence through Equity. Giving every student the chance to succeed. Vol II. OECD Publishing.

OECD, Organisation for Economic Co-operation and Development (2013b). PISA 2012 Results: Ready to Learn. Students’engagement, drive and self-beliefs. Vol III. OECD Publishing.

OECD, Organisation for Economic Co-operation and Development (2013c). PISA 2012 Results: What Makes Schools Successful? Resources, policies and practices. Vol IV. OECD Publishing.

OECD, Organisation for Economic Co-operation and Development (2013d). PISA 2012. Technical Report. OECD Publishing.

OECD, Organisation for Economic Co-operation and Development (2014a). PISA 2012 Results: What Students Know and Can Do. Student performance in mathematics, reading and science. Vol I. OECD Publishing.

OECD, Organisation for Economic Co-operation and Development (2014b). PISA 2012 Results: Creative Problem Solving. Students’ skilla in tackling real-life problems. Vol V. OECD Publishing.

OECD, Organisation for Economic Co-operation and Development (2014c). PISA 2012 Results: Students and Money. Financial Literacy skills for the 21st century. Vol VI. OECD Publishing.

OECD, Organisation for Economic Co-operation and Development. (2014d). TALIS 2013 Results: An International Perspective on Teaching and Learning. Paris: OECD Publishing.

Peña-López, I. (2012). PISA 2012 Assessment and Analytical Framework. Mathematics, Reading, Science, Problem Solving and Financial Literacy.

Pérez, A. I., & Soto, E. (2011). Luces y sombras de PISA. Sentido educativo de las evaluaciones externas. Cultura y Educación, 23(2), 171-182. DOI: 10.1174/113564011795944758

Perry, S. M., Sealy, K. M., & Hawkins, R. J. (2016). Opportunity to Learn and Students from Economically Disadvantages Homes: Implications for School Leaders Through an Examination of TALIS 2013. Available at SSRN (Social Science Research Network) 2717054.

Ruiz, J. B., Gil, M. G., Navas, M. F., Ramos, R. Y., Ruiz, M. P. S., & Núñez, M. J. S. (2011). Todos queremos ser Finlandia”. Los efectos secundarios de PISA. Education in the Knowledge Society (EKS), 12(1), 320-339.

Sánchez, C. C., & Delgado, F. J. H. (2013). Informe PISA en España. Un análisis al detalle. Profesorado. Revista de Currículum y Formación del Profesorado, 17(2), 20.

Sánchez, E. & García-Rodicio, H. (2006). Re-lectura del estudio PISA: qué y cómo se evalúa e interpreta el rendimiento de los alumnos en la lectura. Revista de educación, (1), 195-226.

Sealy, K. M., Perry, S. M., & DeNicola, T. C. (2016). Relationships and Predictors of Principal Job Satisfaction Across Multiple Countries: A Study Using Talis 2013 and PISA 2012. Available at SSRN 2717056.

Sedghi, A., Arnett, G., & Chalabi, M. (2013). Pisa 2012 results: which country does best at reading, maths and science. The Guardian. Retrieved October, 21, 2015.

Sjøberg, S. (2015). PISA and global educational governance–A critique of the project, its uses and implications. Eurasia Journal of Mathematics, Science & Technology Education, 11(1), 111-127. DOI: 10.12973/eurasia.2015.1310a

Stacey, K. (2015). The international assessment of mathematical literacy: PISA 2012 framework and items. In Selected Regular Lectures from the 12th International Congress on Mathematical Education (pp. 771-790). Springer International Publishing.

Thomson, S., De Bortoli, L., & Buckley, S. (2013). PISA 2012: How Australia measures up. Melbourne: Australian Council for Educational Research.

Vieluf, S., Kaplan, D., Klieme, E. y Bayer, S. (2012). Teaching Practices and Pedagogical Innovation: Evidence from TALIS. OECD Publishing.

Villar, A. (2013). Rendimiento, esfuerzo y productividad: análisis de los resultados en matemáticas de los estudiantes españoles según PISA (2012). PISA 2012: Programa para la evaluación internacional de los alumnos.

Wayne, A. J. y Youngs, P. (2003). Teacher characteristics and student achievement gains: A review. Review of Educational Research, 73, 89-122.

Wenglinsky, H. (2002). How schools matter: The link between teacher classroom practices and student academic performance. Education Policy Analysis Archives, 11(12).

Wheater, R. (2013). Achievement of 15 year olds in England: PISA 2012 national report (OECD Programme for International Student Assessment), December 2013.

Downloads

Published

2016-07-08

Issue

Vol. 22 No. 1 (2016)

Section

Special Section

License

The authors grant non-exclusive rights of exploitation of works published to RELIEVE and consent to be distributed under the Creative Commons Attribution-Noncommercial Use 4.0 International License (CC-BY-NC 4.0), which allows third parties to use the published material whenever the authorship of the work and the source of publication is mentioned, and it is used for non-commercial purposes. 

    The authors can reach other additional and independent contractual agreements, for the non-exclusive distribution of the version of the work published in this journal (for example, by including it in an institutional repository or publishing it in a book), as long as it is clearly stated that the Original source of publication is this magazine.  

    Authors are encouraged to disseminate their work after it has been published, through the internet (for example, in institutional archives online or on its website) which can generate interesting exchanges and increase work appointments. 

   The fact of sending your paper to RELIEVE implies that you accept these conditions.