Abstract
Open classroom discussion (OPD) is a recognized school practice, that promotes civic knowledge on students. However, the study of its effectiveness includes various methodological challenges. OPD items are reference-shift items, and if their rater-response nature is ignored, researchers may specify a compositional model leading to underestimation. Moreover, OPD scores of schools are subject to students inter rater variability. Common advice in this regard is to exclude groups with low inter-rater agreement. Nevertheless, this recommendation can result into a considerable loss of sample. In this paper, we argue that a within-between model specification is needed to address the first problem. For the second problem, it is proposed to use a dispersion effect model. This later model studies OPD relations to civic knowledge, at conditional levels of students’ lack of agreement on OPD ratings. Caveats on the use of students as raters of school practices are discussed.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bell, A., Jones, K., & Fairbrother, M. (2018). Understanding and misunderstanding group mean centering: A commentary on Kelley et al.’s dangerous practice. Quality and Quantity, 52(5), 2031–2036. https://doi.org/10.1007/s11135-017-0593-5
Caro, D. H., & Lenkeit, J. (2012). An analytical approach to study educational inequalities: 10 hypothesis tests in PIRLS 2006. International Journal of Research & Method in Education, 35(1), 3–30. https://doi.org/10.1080/1743727X.2012.666718
Carrasco, D., Banerjee, R., Treviño, E., & Villalobos, C. (2020). Civic knowledge and open classroom discussion: Explaining tolerance of corruption among 8th-grade students in Latin America. Educational Psychology, 40(2), 186–206. https://doi.org/10.1080/01443410.2019.1699907
Carrasco, D., & Torres Irribarra, D. (2018). The role of classroom discussion. In A. Sandoval-Hernández, M. M. Isac, & D. Miranda (Eds.), Teaching tolerance in a globalized world (pp. 87–101). https://doi.org/10.1007/978-3-319-78692-6
Castellano, K. E., Rabe-Hesketh, S., & Skrondal, A. (2014). Composition, context, and endogeneity in school and teacher comparisons. Journal of Educational and Behavioral Statistics, 39(5), 333–367. https://doi.org/10.3102/1076998614547576
Engelhard, G. J., & Wind, S. A. (2018). Invariant measurement with raters and rating scales. Routledge.
Hahn, C. L., & Tocci, C. M. (1990). Classroom climate and controversial issues discussions: A five nation study. Theory and Research in Social Education, 18(4), 344–362. https://doi.org/10.1080/00933104.1990.10505621
Isac, M. M., Maslowski, R., Creemers, B., & van der Werf, G. (2014). The contribution of schooling to secondary-school students’ citizenship outcomes across countries. School Effectiveness & School Improvement, 25(January 2015), 29–63. https://doi.org/10.1080/09243453.2012.751035
Kamata, A., & Vaughn, B. K. (2011). Multilevel IRT modeling. In J. J. Hox & J. K. Roberts (Eds.), Handbook of advanced multilevel analysis (pp. 41–57). https://doi.org/10.4324/9780203848852.ch3
Knowles, R. T., & McCafferty-Wright, J. (2015). Connecting an open classroom climate to social movement citizenship: A study of 8th graders in Europe using IEA ICCS data. Journal of Social Studies Research, 39(4), 255–269. https://doi.org/10.1016/j.jssr.2015.03.002
Lang, J. W. B., Bliese, P. D., & de Voogt, A. (2018). Modeling consensus emergence in groups using longitudinal multilevel methods. Personnel Psychology, 71(2), 255–281. https://doi.org/10.1111/peps.12260
Lüdtke, O., & Robitzsch, A. (2009). Assessing within-group agreement: A critical examination of a random-group resampling approach. Organizational Research Methods, 12(3), 461–487. https://doi.org/10.1177/1094428108317406
Lüdtke, O., Robitzsch, A., Trautwein, U., & Kunter, M. (2009). Assessing the impact of learning environments: How to use student ratings of classroom or school characteristics in multilevel modeling. Contemporary Educational Psychology, 34(2), 120–131. https://doi.org/10.1016/j.cedpsych.2008.12.001
Lüdtke, O., Trautwein, U., Kunter, M., & Baumert, J. (2006). Reliability and agreement of student ratings of the classroom environment: A reanalysis of TIMSS data. Learning Environments Research, 9(3), 215–230. https://doi.org/10.1007/s10984-006-9014-8
Masters, G. N. (2016). Partial credit model. In W. J. van der Linden (Ed.), Handbook of item response theory. Volume One. Models (pp. 109–126). CRC Press.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Nielsen & Lydiche.
Rights, J. D., Preacher, K. J., & Cole, D. A. (2019). The danger of conflating level-specific effects of control variables when primary interest lies in level-2 effects. British Journal of Mathematical and Statistical Psychology, 4, bmsp.12194. https://doi.org/10.1111/bmsp.12194
Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142–151. https://doi.org/10.3102/0013189X10363170
Schneider, B., Salvaggio, A. N., & Subirats, M. (2002). Climate strength: A new direction for climate research. Journal of Applied Psychology, 87(2), 220–229. https://doi.org/10.1037/0021-9010.87.2.220
Schulz, W., Carstens, R., Losito, B., & Fraillon, J. (2018). ICCS 2016 technical report (W. Schulz, R. Carstens, B. Losito, & J. Fraillon, Eds.). International Association for the Evaluation of Educational Achievement (IEA).
Schweig, J. D. (2016). Moving beyond means: Revealing features of the learning environment by investigating the consensus among student ratings. Learning Environments Research, 19(3), 441–462. https://doi.org/10.1007/s10984-016-9216-7
Torney-Purta, J. (2009). Matters for policy and practice. American Psychologist, 64(4), 825–837.
Woehr, D. J., Loignon, A. C., & Schmidt, P. (2015). Aggregation aggravation: The fallacy of the wrong level revisited. In C. E. Lance & R. J. Vandenberg (Eds.), More statistical and methodological myths and urban legends (pp. 311–326). Routledge.
Acknowledgements
Research funded by the Ministerio de Educación, Gobierno de Chile and Comisión Nacional de Investigación Científica y Tecnológica CONICYT (PIA 160007), Centro de Estudios Avanzados (CJE); and Fondo Nacional de Desarrollo Científico y Tecnológico FONDECYT N° 1180667.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Carrasco, D., Treviño, E., López Hornickel, N., Castillo, C. (2021). Students Ratings Their Open Classroom Discussion. In: Wiberg, M., Molenaar, D., González, J., Böckenholt, U., Kim, JS. (eds) Quantitative Psychology. IMPS 2020. Springer Proceedings in Mathematics & Statistics, vol 353. Springer, Cham. https://doi.org/10.1007/978-3-030-74772-5_41
Download citation
DOI: https://doi.org/10.1007/978-3-030-74772-5_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-74771-8
Online ISBN: 978-3-030-74772-5
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)