More confusion and frustration, better learning: The impact of erroneous examples
ARTICLE
J. Elizabeth Richey, Carnegie Mellon University, United States ; Juan Miguel L. Andres-Bray, University of Pennsylvania, United States ; Michael Mogessie, Carnegie Mellon University, United States ; Richard Scruggs, Juliana M.A.L. Andres, University of Pennsylvania, United States ; Jon R. Star, Harvard University, United States ; Ryan S. Baker, University of Pennsylvania, United States ; Bruce M. McLaren, Carnegie Mellon University, United States
Computers & Education Volume 139, Number 1, ISSN 0360-1315 Publisher: Elsevier Ltd
Abstract
Prior research suggests students can sometimes learn more effectively by explaining and correcting example problems that have been solved incorrectly, compared to problem-solving practice or studying correct solutions. It remains unclear, however, what role students' affect might play in the process of learning from erroneous examples. Specifically, it may be that students experience greater confusion and frustration while studying erroneous examples, but that their confusion and frustration lead to greater learning. We analyzed student log data from previously published research comparing erroneous example instruction of decimal number mathematics to problem-solving instruction in a computer-based intelligent tutoring system. We created and applied affect detectors for a combination of confusion and frustration (“confrustion”) and compared the role of confrustion across conditions. As predicted, students in the erroneous example condition experienced greater confrustion while working through the instructional materials. However, contrary to predictions, confrustion was negatively correlated with posttest and delayed posttest performance across conditions, though less so for the erroneous example condition. Given that students in the erroneous example condition performed better on the delayed posttest than students in the problem-solving condition, it appears they learned more despite also experiencing greater confrustion rather than because of it. Results suggest that learning from erroneous examples may be an inherently more confusing and frustrating process than traditional problem solving. More generally, this research demonstrates that logging student actions at a step-by-step problem-solving level and analyzing those logs to infer affect can be a powerful way to investigate learning.
Citation
Richey, J.E., Andres-Bray, J.M.L., Mogessie, M., Scruggs, R., Andres, J.M.A.L., Star, J.R., Baker, R.S. & McLaren, B.M. (2019). More confusion and frustration, better learning: The impact of erroneous examples. Computers & Education, 139(1), 173-190. Elsevier Ltd. Retrieved August 14, 2024 from https://www.learntechlib.org/p/209945/.
This record was imported from Computers & Education on June 3, 2019. Computers & Education is a publication of Elsevier.
Full text is availabe on Science Direct: http://dx.doi.org/10.1016/j.compedu.2019.05.012Keywords
References
View References & Citations Map- Adams, D., McLaren, B.M., Durkin, K., Mayer, R.E., Rittle-Johnson, B., & Isotani, S. (2014). Using erroneous examples to improve mathematics learning with a web-based tutoring system. Computers in Human Behavior, 36, pp. 401-411.
- Aleven, V., McLaren, B.M., & Sewall, J. (2009). Scaling up programming by demonstration for intelligent tutoring systems development: An open-access website for middle school mathematics learning. IEEE Transactions on Learning Technologies, 2(2), pp. 64-78.
- Aleven, V., McLaren, B.M., Sewall, J., van Velsen, M., Popescu, O., & Demi, S. (2016). Example-tracing tutors: Intelligent tutor development for non-programmers. International Journal of Artificial Intelligence in Education, 26(1), pp. 224-269.
- Andres, J.M.L., & Rodrigo, M.M.T. (2014). The Incidence and persistence of affective states while playing Newton's playground. 7th IEEE international conference on humanoid, nanotechnology, information technology, communication and control, environment, and management.
- Atkinson, R.K., Derry, S.J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), pp. 181-214.
- Atkinson, R.K., Renkl, A., & Merrill, M.M. (2003). Transitioning from studying examples to solving problems: Effects of self-explanation prompts and fading worked-out steps. Journal of Educational Psychology, 95(4), pp. 774-783.
- Baker, R.S.J.d., Corbett, A.T., & Wagner, A.Z. (2006). Human classification of low-fidelity replays of student actions. Proceedings of the educational data mining workshop at the 8th international conference on intelligent tutoring systems, pp. 29-36.
- Baker, R.S.J.d., D'Mello, S.K., Rodrigo, M.M.T., & Graesser, A.C. (2010). Better to Be frustrated than bored: The incidence, persistence, and impact of learners' cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), pp. 223-241.
- Baker, R.S.J.d., Gowda, S.M., Wixon, M., Kalka, J., Wagner, A.Z., & Salvi, A. (2012). Sensor-free automated detection of affect in a cognitive tutor for algebra. Proceedings of the 5th international conference on educational data mining, pp. 126-133.
- Baker, R.S.J.d., & Inventado, P.S. (2014). Educational data mining and learning analytics. Learning analytics: From research to practice Berlin, Germany: Springer.
- Barnett, S.M., & Ceci, S.J. (2002). When and where do we apply what we learn?: A taxonomy for far transfer. Psychological Bulletin, 128(4), p. 612.
- Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57, pp. 289-300.
- Bjork, E.L., & Bjork, R.A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society, pp. 56-64. New York: Worth Publishers.
- Booth, J.L., Lange, K.E., Koedinger, K.R., & Newton, K.J. (2013). Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction, 25, pp. 24-34.
- Borasi, R. (1987). Exploring mathematics through the analysis of errors. For the Learning of Mathematics, 7(3), pp. 2-8.
- Borasi, R. (1994). Capitalizing on errors as “springboards for inquiry”: A teaching experiment. Journal for Research in Mathematics Education, pp. 166-208.
- Botelho, A.F., Baker, R., Ocumpaugh, J., & Heffernan, N. (2018). Studying affect dynamics and chronometry using sensor-free detectors. Proceedings of the 11th international conference on educational data mining, pp. 157-166.
- Brown, D.E. (1992). Using examples and analogies to remediate misconceptions in physics: Factors influencing conceptual change. Journal of Research in Science Teaching, 29(1), pp. 17-34.
- Calvo, R.A., & D'Mello, S. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), pp. 18-37.
- Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794.
- Chi, M.T.H. (2008). Three types of conceptual change: Belief revision, mental model transformation, and categorical shift. Handbook of research on conceptual change, pp. 61-82. Hillsdale, NJ: Erlbaum.
- Chi, M.T. (2009). Active‐constructive‐interactive: A conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1(1), pp. 73-105.
- Chi, M.T., Bassok, M., Lewis, M.W., Reimann, P., & Glaser, R. (1989). Self‐explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), pp. 145-182.
- Chi, M.T.H., de Leeuw, N., Chiu, M., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18, pp. 439-477.
- DeFalco, J.A., Rowe, J.P., Paquette, L., Georgoulas-Sherry, V., Brawner, K., & Mott, B.W. (2018). Detecting and addressing frustration in a serious game for military training. International Journal of Artificial Intelligence in Education, 28(2), pp. 152-193.
- Desmet, L., Gregoire, J., & Mussolin, C. (2010). Developmental changes in the comparison of decimal fractions. Learning and Instruction, 20, pp. 521-532.
- Durkin, K., & Rittle-Johnson, B. (2012). The effectiveness of using incorrect examples to support learning about decimal magnitude. Learning and Instruction, 22(3), pp. 206-214.
- D'Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), pp. 1082-1099.
- D'Mello, S.K., Craig, S.D., Witherspoon, A.W., McDaniel, B.T., & Graesser, A.C. (2008). Automatic detection of learner's affect from conversational cues. User Modeling and User-Adapted Interaction, 18(1 – 2), pp. 45-80.
- D'Mello, S.K., & Graesser, A. (2010). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-Adapted Interaction, 20(2), pp. 147-187.
- D'Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), pp. 145-157.
- D'Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, pp. 153-170.
- Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46(1), pp. 6-25.
- Gadgil, S., Nokes-Malach, T.J., & Chi, M.T.H. (2012). Effectiveness of holistic mental model confrontation in driving conceptual change. Learning and Instruction, 22(1), pp. 47-61.
- Grafsgaard, J.F., Wiggins, J.B., Boyer, K.E., Wiebe, E.N., & Lester, J.C. (2013). Automatically recognizing facial expression: Predicting engagement and frustration. Proceedings of the international conference on educational data mining (EDM), pp. 43-50.
- Große, C.S., & Renkl, A. (2007). Finding and fixing errors in worked examples: Can this foster learning outcomes?. Learning and Instruction, 17(6), pp. 612-634.
- Hayes, A.F. (2013). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. New York, NY, US: Guilford Press.
- Heitzmann, N., Fischer, F., & Fischer, M.R. (2018). Worked examples with errors: When self-explanation prompts hinder learning of teachers diagnostic competences on problem-based learning. Instructional Science, 46(2), pp. 245-271.
- Hiebert, J., & Wearne, D. (1985). A model of students' decimal computation procedures. Cognition and Instruction, 2, pp. 175-205.
- Isotani, S., Adams, D., Mayer, R.E., Durkin, K., Rittle-Johnson, B., & McLaren, B.M. (2011). Can erroneous examples help middle-school students learn decimals?. The Proceedings of the sixth European Conference on technology enhanced learning: Towards Ubiquitous learning (EC-TEL 2011), pp. 181-195.
- Johnson, C.I., & Mayer, R.E. (2010). Adding the self-explanation principle to multimedia learning in a computer-based game-like environment. Computers in Human Behavior, 26, pp. 1246-1252.
- Kai, S., Almeda, M.V., Baker, R., Heffernan, C., & Heffernan, N. (2018). Decision tree modeling of wheel-spinning and productive persistence in skill builders. JEDM | Journal of Educational Data Mining, 10(1), pp. 36-71. Available online: https://jedm.educationaldatamining.org/index.php/JEDM/article/view/210.
- Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93, pp. 579-588.
- Kapur, M. (2014). Productive failure in learning math. Cognitive Science, 38(5), pp. 1008-1022.
- Kapur, M. (2016). Examining productive failure, productive success, unproductive failure, and unproductive success in learning. Educational Psychologist, 51(2), pp. 289-299.
- Kapur, M., & Bielaczyc, K. (2012). Designing for productive failure. The Journal of the Learning Sciences, 21(1), pp. 45-83.
- Koedinger, K.R., Baker, R.S.J.d., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. Handbook of educational data mining Boca Raton, FL: CRC Press.
- Koedinger, K.R., Stamper, J.C., Leber, B., & Skogsholm, A. (2013). LearnLab's DataShop: A data repository and analytics tool set for cognitive science. Topics in Cognitive Science, 5(3), pp. 668-669.
- Kostyuk, V., Almeda, M.V., & Baker, R.S. (2018). Correlating affect and behavior in reasoning mind with state test achievement. Proceedings of the international conference on learning analytics and knowledge, pp. 26-30.
- Lee, D.M.C., Rodrigo, M.M.T., Baker, R.S.J.d., Sugay, J.O., & Coronel, A. (2011). Exploring the relationship between novice programmer confusion and achievement. Proceeds of the 4th bi-annual Conference on affective Computing and intelligent interaction, 2011.
- Lehman, B., D'Mello, S., Strain, A., Mills, C., Gross, M., Dobbins, A., (2013). Inducing and tracking confusion with contradictions during complex learning. International Journal of Artificial Intelligence in Education, 22(1–2), pp. 85-105.
- Limón, M. (2001). On the cognitive conflict as an instructional strategy for conceptual change: A critical appraisal. Learning and Instruction, 11(4–5), pp. 357-380.
- Liu, Z., Pataranutaporn, V., Ocumpaugh, J., & Baker, R.S.J.d. (2013). Sequences of frustration and confusion, and learning. Proceedings of the 6th international conference on educational data mining, pp. 114-120.
- Mayer, R.E., & Johnson, C.I. (2010). Adding instructional features that promote learning in a game-like environment. Journal of Educational Computing Research, 42, pp. 241-265.
- McLaren, B.M., Adams, D.M., & Mayer, R.E. (2015). Delayed learning effects with erroneous examples: A study of learning decimals with a web-based tutor. International Journal of Artificial Intelligence in Education, 25(4), pp. 520-542.
- McLaren, B.M., Lim, S., & Koedinger, K.R. (2008). When and how often should worked examples be given to students? New results and a summary of the current state of research. Proceedings of the 30th annual conference of the cognitive science society, pp. 2176-2181. Austin, TX: Cognitive Science Society.
- McLaren, B.M., van Gog, T., Ganoe, C., Karabinos, M., & Yaron, D. (2016). The efficiency of worked examples compared to erroneous examples, tutored problem solving, and problem solving in classroom experiments. Computers in Human Behavior, 55, pp. 87-99.
- McNamara, D.S. (2004). SERT: Self-explanation reading training. Discourse Processes, 38(1), pp. 1-30.
- Melis, E. (2004). Erroneous examples as a source of learning in mathematics. CELDA, 2004, pp. 311-318.
- Muldner, K., Burleson, B., & VanLehn, K. (2010). “Yes!”: Using tutor and sensor data to predict moments of delight during instructional activities. Proceedings of the international conference on user modeling and adaptive presentation (UMAP’10), pp. 159-170.
- Ocumpaugh, J., Baker, R.S., & Rodrigo, M.M.T. (2015). Baker Rodrigo Ocumpaugh monitoring protocol (BROMP) 2.0 technical and training manual. New York, NY: Teachers College, Columbia University. Manila, Philippines: Ateneo Laboratory for the Learning Sciences.
- Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), pp. 1-4.
- Paquette, L., Baker, R.S., Sao Pedro, M.A., Gobert, J.D., Rossi, L., & Nakama, A. (2014). Sensor-free affect detection for a simulation-based science inquiry learning environment. International conference on intelligent tutoring systems, pp. 1-10. Cham: Springer.
- Paquette, L., de Carvalho, A.M.J.A., & Baker, R.S. (2014). Towards understanding expert coding of student disengagement in online learning. Proceedings of the 36th annual cognitive science conference, pp. 1126-1131.
- Pardos, Z.A., Baker, R.S., San Pedro, M., Gowda, S.M., & Gowda, S.M. (2014). Affective states and state tests: Investigating how affect and engagement during the school year predict end-of-year learning outcomes. Journal of Learning Analytics, 1(1), pp. 107-128.
- Pekrun, R., Goetz, T., Frenzel, A.C., Barchfeld, P., & Perry, R.P. (2011). Measuring emotions in students' learning and performance: The Achievement Emotions Questionnaire (AEQ). Contemporary Educational Psychology, 36(1), pp. 36-48.
- Pekrun, R., Goetz, T., Titz, W., & Perry, R. (2002). Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37(2), pp. 91-105.
- Putt, I.J. (1995). Preservice teachers ordering of decimal numbers: When more is smaller and less is larger!. Focus on Learning Problems in Mathematics, 17(3), pp. 1-15.
- Renkl, A. (1997). Learning from worked‐out examples: A study on individual differences. Cognitive Science, 21(1), pp. 1-29.
- Renkl, A. (2002). Worked-out examples: Instructional explanations support learning by self-explanations. Learning and Instruction, 12(5), pp. 529-556.
- Renkl, A., & Atkinson, R.K. (2002). Learning from examples: Fostering self-explanations in computer-based learning environments. Interactive Learning Environments, 10(2), pp. 105-119.
- Renkl, A., & Atkinson, R.K. (2003). Structuring the transition from example study to problem solving in cognitive skill acquisition: A cognitive load perspective. Educational Psychologist, 38(1), pp. 15-22.
- Resnick, L.B., Nesher, P., Leonard, F., Magone, M., Omanson, S., & Peled, I. (1989). Conceptual bases of arithmetic errors: The case of decimal fractions. Journal for Research in Mathematics Education, pp. 8-27.
- Rittle-Johnson, B., & Star, J.R. (2009). Compared with what? The effects of different comparisons on conceptual knowledge and procedural flexibility for equation solving. Journal of Educational Psychology, 101(3), p. 529.
- Rodrigo, M.M.T., Baker, R.S., Jadud, M.C., Amarra, A.C.M., Dy, T., Espejo-Lahoz, M.B.V., (2009). Affective and behavioral predictors of novice programmer achievement. Proceedings of the ACM SIGCSE annual conference on innovation and technology in computer science education, Vol. 41, pp. 156-160. New York, NY: ACM Press.
- Rowe, J.P., Shores, L.R., Mott, B.W., & Lester, J.C. (2011). Integrating learning, problem solving, and engagement in narrative-centered learning environments. International Journal of Artificial Intelligence in Education, pp. 115-133.
- Rushton, S.J. (2018). Teaching and learning mathematics through error analysis. Fields Mathematics Education Journal, 3(1), p. 4.
- Sackur-Grisvard, C., & Léonard, F. (1985). Intermediate cognitive organizations in the process of learning a mathematical concept: The order of positive decimal numbers. Cognition and Instruction, 2, pp. 157-174.
- San Pedro, M.O.Z., Baker, R.S.J.d., Bowers, A.J., & Heffernan, N.T. (2013). Predicting college enrollment from student interaction with an intelligent tutoring system in middle school. Proceedings of the 6th international conference on educational data mining, pp. 177-184.
- Sao Pedro, M.A., de Baker, R.S., Gobert, J.D., Montalvo, O., & Nakama, A. (2013). Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill. User Modeling and User-Adapted Interaction, 23(1), pp. 1-39.
- Schmidt, R.A., & Bjork, R.A. (1992). New conceptualization of practice: Common principles in three paradigms suggest new concepts for training. Psychological Science, 3(4), pp. 207-217.
- Schneider, B., Krajcik, J., Lavonen, J., Salmela-Aro, K., Broda, M., & Spicer, J. (2015). Investigating optimal learning moments in U.S. and Finnish science classes. Journal of Research in Science Teaching, 53(3), pp. 400-421.
- Shute, V.J., D'Mello, S., Baker, R., Cho, K., Bosch, N., & Ocumpaugh, J. (2015). Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game. Computers & Education, 86, pp. 224-235. Available online: https://doi.org/10.1016/j.compedu.2015.08.001.
- Siegler, R.S. (2002). Microgenetic studies of self-explanation. Microdevelopment: Transition processes in development and learning.
- Siegler, R.S., & Chen, Z. (2008). Differentiation and integration: Guiding principles for analyzing cognitive change. Developmental Science, 11(4), pp. 433-448.
- Sinha, T., Bai, Z., & Cassell, J. (2017). A new theoretical framework for curiosity for learning in social contexts. Proceedings of 12th European Conference on technology enhanced learning (EC-TEL ’17), pp. 254-269. Cham: Springer.
- Smith, J.P., DiSessa, A.A., & Roschelle, J. (1994). Misconceptions reconceived: A constructivist analysis of knowledge in transition. The Journal of the Learning Sciences, 3(2), pp. 115-163.
- Soderstrom, N.C., & Bjork, R.A. (2015). Learning versus performance: An integrative review. Perspectives on Psychological Science, 10(2), pp. 176-199.
- Stacey, K. (2005). Travelling the road to expertise: A longitudinal study of learning. Proceedings of the 29th conference of the international group for the psychology of mathematics education, Vol. 1, pp. 19-36. Melbourne: PME.
- Stacey, K., Helme, S., & Steinle, V. (2001). Confusions between decimals, fractions and negative numbers: A consequence of the mirror as a conceptual metaphor in three different ways. Proceedings of the 25th conference of the international group for the psychology of mathematics education, Vol. 4, pp. 217-224. Utrecht: PME.
- Stacey, K., & Steinle, V. (1998). Refining the classification of students' interpretations of decimal notation. Hiroshima Journal of Mathematics Education, 6, pp. 49-69.
- Steinle, V. (2004). Changes with age in students' misconceptions of decimal numbers.
- Steinle, V., & Stacey, K. (2004). Persistence of decimal misconceptions and readiness to move to expertise. Proceedings of the 28th conference of the international group for the psychology of mathematics education, Vol. 1, pp. 225-232. Bergen-Norway: Bergen University College.
- Trafton, J.G., & Reiser, B.J. (1993). The contribution of studying examples and solving problems to skill acquisition. Proceedings of the 15th annual conference of the cognitive science society, pp. 1017-1022. Hillsdale, NJ: Erlbaum.
- Tsovaltzi, D., Melis, E., & McLaren, B.M. (2012). Erroneous examples: Effects on learning fractions in a web-based setting. International Journal of Technology Enhanced Learning, 4(3–4), pp. 191-230.
- Van Gog, T., Kester, L., & Paas, F. (2011). Effects of worked examples, example-problem, and problem-example pairs on novices' learning. Contemporary Educational Psychology, 36(3), pp. 212-218.
- Vosniadou, S. (2012). Reframing the classical approach to conceptual change: Preconceptions, misconceptions and synthetic models. Second international handbook of science education Dordrecht: Springer.
- Ward, M., & Sweller, J. (1990). Structuring effective worked examples. Cognition and Instruction, 7(1), pp. 1-39.
These references have been extracted automatically and may have some errors. Signed in users can suggest corrections to these mistakes.
Suggest Corrections to References