Correcting for Scale Usage Differences among Latin American Countries, Portugal, and Spain in PISA

Authors

  • Jia He German Institute for International Educational Research
  • Fons Van de Vijver Tilburg University

DOI:

https://doi.org/10.7203/relieve.22.1.8282

Keywords:

Extreme response style, overclaiming, anchoring vignettes, comparability, validity.

Abstract

This paper investigated the effects of corrections for scale usage preference in seven Latin American countries, Portugal and Spain in student self-reports in the 2012 Programme for International Student Assessment (PISA). These targeted countries tend to show a strong tendency of expressing strong opinions and self-enhancement, which can pose serious validity threats in cross-cultural comparisons of self-reports. We examined to what extent score corrections, that have been proposed, would change the patterning of the cross-cultural differences. We corrected for the scale usage preferences in a measure of teacher support among 39,045 students in nine countries, based on extreme response style, overclaiming, and anchoring vignettes, respectively. These measures showed different effects:  (1) All correction methods helped to improve measurement invariance, although the correction based on anchoring was less effective in reaching scalar invariance compared with the correction of extreme response style and overclaiming; (2) controlling for extreme response style and overclaiming changed the mean score of Spain to a greater extent than other countries, which seems to present a more realistic patterning, whereas the changes on correlations with other measures were rather limited. The use of anchored scores led to drastic changes both in means and correlations. A firm conclusion about which method is to be preferred cannot be given as there is no evidence which method enhances the validity of scores in these countries more. We discuss the necessity and practicability of correction methods.

Author Biographies

Jia He, German Institute for International Educational Research

Ph.D Cross-Cultural Psychology , Tilburg University , Tilburg , The Netherlands , 2011-2015. Master of Arts , Intercultural Communication , Shanghai International Studies University , Shanghai , China, 2007-2009 . Bachelor of Arts , Marketing, Dongbei University of Finance and Economics , Dalian, China, 2003-2007. Research Deutsches Institut für Internationale Pädagogische Forschung (DIPF). Post Doc Deutsches Institut für Internationale Pädagogische Forschung (DIPF) since 2015. Research on self-report data comparability in large-scale international surveys , using advanced psychometric methods and new item formats , etc. Research World Bank Group Consultant since 2015. Assistance to the National Educational Assessment and Examination Agency on the Ethiopian national educational assessment proficiency scaling and standard setting . Analyst OECD since 2015.

Fons Van de Vijver, Tilburg University

Professor of Cross-Cultural Psychology at the Tilburg University, theNorth-West University and the University of Queensland, known for his work on cross-cultural research and on "methods and data analysis of comparative research". Van de Vijver received both his MA and in 1991 his PhD in Psychology at the Tilburg University. He is appointed Professor cross-cultural psychology at Tilburg University, and is also Professor at the North-West University in South Africa and the University of Queensland in Australia. In 2013 he and Maria Cristina Richaudreceived the APA Award for Distinguished Contributions to the International Advancement of Psychology.

References

Arbuckle, J. L. (2006). AMOS user's guide. Chicago, IL: SPSS.

Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71, 460-502. doi: http://dx.doi.org/ http://dx.doi.org/10.1177/0013164410375112

Chen, C., Lee, S.-y., & Stevenson, H. W. (1995). Response style and cross-cultural comparisons of rating scales among East Asian and North American students. Psychological Science, 6, 170-175. doi: http://dx.doi.org/10.1111/j.1467-9280.1995.tb00327.x

Cheung, G. W., & Rensvold, R. B. (2000). Assessing extreme and acquiescence response sets in cross-cultural research using structural equations modeling. Journal of Cross-Cultural Psychology, 31, 187-212. doi: http://dx.doi.org/10.1177/0022022100031002003

Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255. doi: http://dx.doi.org/10.1207/s15328007sem0902_5

Diamantopoulos, A., Raeynolds, N. L., & Simintiras, A. C. (2006). The impact of response styles on the stability of cross-national comparisons. Journal of Business Research, 59, 925-935. doi: http://dx.doi.org/10.1016/j.jbusres.2006.03.001

Goldstein, H. (2004). International comparisons of student attainment: Some issues arising from the PISA study. Assessment in Education Principles Policy and Practice, 11, 319-330. doi: http://dx.doi.org/10.1080/0969594042000304618

Gorur, R. (2014). Towards a sociology of measurement in education policy. European Educational Research Journal, 13, 58-72. doi: http://dx.doi.org/10.2304/eerj.2014.13.1.58

Harzing, A.-W. (2006). Response styles in cross-national survey research: A 26-country study. International Journal of Cross Cultural Management, 6, 243-266. doi: http://dx.doi.org/10.1177/1470595806066332

He, J., van de Vijver, F., J. R., Domínguez, A. d. C., & Mui, P. H. C. (2014). Toward a unification of acquiescent, extreme, and midpoint response styles: A multilevel study. International Journal of Cross-Cultural Management, 14, 306-322. doi: http://dx.doi.org/10.1177/1470595814541424

He, J., & van de Vijver, F. J. R. (2015a). Effects of a general response style on cross-cultural comparisons: Evidence from the Teaching and Learning International Survey. Public Opinion Quarterly, 79, 267-290. doi: http://dx.doi.org/10.1093/poq/nfv006

He, J., & Van de Vijver, F. J. R. (2015b). The motivation-achievement paradox in international educational achievement tests: Toward a better understanding. In R. B. King & A. B. I. Bernardo (Eds.), The psychology of Asian learners: A festschrift in honor of David Watkins (pp. 253-268). Singapore: Springer.

He, J., & van de Vijver, F. J. R. (2015c). Self-presentation styles in self-reports: Linking the general factors of response styles, personality traits, and values in a longitudinal study. Personality and Individual Differences, 31, 129-134. doi: http://dx.doi.org/10.1016/j.paid.2014.09.009

Hofstede, G. (1980). Culture's consequences: Comparing values, behaviors, institutions and organizations across nations. Beverly Hills, CA: Sage.

Hofstede, G. (2009). Dimension data matrix. Dimension Data Matrix Retrieved 03/02/2001 http://www.geerthofstede.eu/dimension-data-matrix

King, G., Murray, C. J. L., Salomon, J. A., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98, 191-207. doi: http://dx.doi.org/ doi: http://dx.doi.org/10.1017/S000305540400108X

King, G., & Wand, J. (2007). Comparing incomparable survey responses: Evaluating and selecting anchoring vignettes. Political Analysis, 15, 46-66. doi: http://dx.doi.org/10.1093/pan/mpl011

Kyllonen, P. C., & Bertling, J. P. (2014). Innovative questionnaire assessment methods to increase cross-country comparability. In L. Rutkowski, M. v. Davier & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 277-286). Boca Raton, FL: CRC Press.

OECD. (2013a). PISA 2012 Assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. Paris, France: OECD Publishing.

OECD. (2013b). PISA 2012 technical report. Paris, France: OECD Publishing.

Paulhus, D. L. (1991). Measurement and control of response biases. In J. Robinson, P. Shaver & L. Wrightsman (Eds.), Measures of personality and social psychological attitudes (Vol. 1, pp. 17-59). San Diego, CA: Academic Press.

Paulhus, D. L., Harms, P. D., Bruce, M. N., & Lysy, D. C. (2003). The over-claiming technique: Measuring self-enhancement independent of ability. Journal of Personality and Social Psychology, 84, 890-904. doi: http://dx.doi.org/10.1037/0022-3514.84.4.890

Rutkowski, L., von Davier, M., & Rutkowski, D. (Eds.). (2014). Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis. Boca Raton, FL: CRC Press.

Smith, P. B. (2004). Acquiescent response bias as an aspect of cultural communication style. Journal of Cross-Cultural Psychology, 35, 50-61. doi: http://dx.doi.org/10.1177/0022022103260380

Smith, P. B. (2011). Communication styles as dimensions of national culture. Journal of Cross-Cultural Psychology, 42, 216-233. doi: http://dx.doi.org/10.1177/0022022110396866

Uskul, A. K., Oyserman, D., & Schwarz, N. (2010). Cultural emphasis on honor, modesty or self-enhancement: Implications for the survey response process. In J. A. Harkness, M. Broun, B. Edwards, T. P. Johnson, L. Lyberg, P. P. Mohler, B.-E. Pennell & T. W. Smith (Eds.), Survey methods in multinational, multiregional and multicultural contexts (pp. 191-201). New York, NY: Wiley.

van de Vijver, F. J. R., & Leung, K. (1997). Methods and data analysis of comparative research. Thousand Oaks, CA: Sage.

Wand, J., & King, G. (2007). Anchoring vignetttes in R: A (different kind of) vignette. Retrieved from http://wand.stanford.edu/anchors/doc/anchors.pdf

Downloads

Published

2016-07-08

Issue

Vol. 22 No. 1 (2016)

Section

Special Section

License

The authors grant non-exclusive rights of exploitation of works published to RELIEVE and consent to be distributed under the Creative Commons Attribution-Noncommercial Use 4.0 International License (CC-BY-NC 4.0), which allows third parties to use the published material whenever the authorship of the work and the source of publication is mentioned, and it is used for non-commercial purposes. 

    The authors can reach other additional and independent contractual agreements, for the non-exclusive distribution of the version of the work published in this journal (for example, by including it in an institutional repository or publishing it in a book), as long as it is clearly stated that the Original source of publication is this magazine.  

    Authors are encouraged to disseminate their work after it has been published, through the internet (for example, in institutional archives online or on its website) which can generate interesting exchanges and increase work appointments. 

   The fact of sending your paper to RELIEVE implies that you accept these conditions.