You are here:

Employing game analytics techniques in the psychometric measurement of game-based assessments with dynamic content

, , Universitt Kassel

Journal of e-Learning and Knowledge Society Volume 11, Number 3, ISSN 1826-6223 e-ISSN 1826-6223 Publisher: Italian e-Learning Association


The adaptation Game-Based Assessment (GBA) (Mislevy et al., 2014) has been growing in the last years backed by video games’ capability of offering a task model to assess learners’ complex knowledge. Since the variables generated from such performances are not directly interpretable, assessment frameworks such as Evidence-Centred Design (ECD) came into play (Mislevy & Almond, 2003). In this work we show our initial findings when using game analytics techniques (Saif El-Nasr et al., 2013) such as play metrics to analyse players’ performance in an open world 3D game for traffic education. The results show that play metrics can be used in cases where game has a dynamic user-generated content of unknown structure. Additionally, we discuss how these metrics can form the basis of measuring psychometric principles that ECD uses to evaluate assessments, which are validity, reliability, comparability and fairness (Mislevy & Wilson, 2003).


Jaffal, Y. & Wloka, D. (2015). Employing game analytics techniques in the psychometric measurement of game-based assessments with dynamic content. Journal of e-Learning and Knowledge Society, 11(3),. Italian e-Learning Association. Retrieved March 24, 2019 from .


View References & Citations Map


  1. Buckley B., Janice G., Horwitz P. & O’Dwyer L. (2010), Looking inside the black box: assessing model-based learning and inquiry in BioLogica™, International Journal of Learning Technology, 5 (2), 166-190.
  2. Buschang R., Kerr D. & Chung G. (2012), Examining feedback in an instructional videogame using process data and error analysis, in: National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Report 817.
  3. Canossa A. & Drachen A. (2009), Patterns of play: Play-personas in user-centred game development, in: Breaking New Ground: Innovation in Games, Play, Practice and Theory, London, Brunel University.
  4. Clarke-Midura J. & Dede C. (2010), Assessment, technology, and change, Journal of Research on Technology in Education, 42 (3), 309-328.
  5. Cronbach L. (1951), Coefficient alpha and the internal structure of tests, psychometrika, 16 (3), 297-334.
  6. Csapó B., Ainley J., Bennett R.E., Latour T. & Law, N. (2012), Technological issues for computer-based assessment, Assessment and teaching of 21st century skills, 143-230.
  7. De Klerk S., Bernard V. & Eggen T. (2015), Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a Bayesian network example, Computers& Education, 85, 23-34.
  8. Drachen A. & Canossa A. (2009), Analyzing spatial user behavior in computer games using geographic information systems, in: Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous Era. 182-189, ACM.
  9. Gobert J., Sao Pedro M., Baker R., Toto E. & Montalvo O. (2012), Leveraging educational data mining for real-time performance assessment of scientific inquiry skills within microworlds, JEDM-Journal of Educational Data Mining, 4 (1), 11143.
  10. Halverson R. & Owen E. (2014), Game-based assessment: an integrated model for capturing evidence of learning in play, International Journal of Learning Technology, 9 (2), 111-138.
  11. Isbister K. & Schaffer N., eds (2008), Game usability: Advancing the player experience, CRC Press.
  12. Iseli M., Koenig A., Lee J. & Wainess R. (2010), Automated Assessment of Complex Task Performance in Games and Simulations, in: Proceedings of the Interservice/ Industry Training, Simulation and Education Conference, Orlando, FL.
  13. Kerr D. (2014), Into the Black Box: Using Data Mining of In-Game Actions to Draw Inferences from Educational Technology about Students’ Math Knowledge, Ph. D. Thesis, University of California.
  14. Kim J. & Chung G. (2012), Use of a Survival Analysis Technique in Understanding Game Performance in Instructional Games, in: National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Report 812.
  15. Kim H., Gunn D., Schuh E., Phillips B., Pagulayan R. & Wixon D. (2008), Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems, in: Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 443-452, ACM.
  16. Klein G., Pfaff M. & Drury J. (2009), Supporting a Robust Decision Space, in: AAAI Spring Symposium: Technosocial Predictive Analytics. 66-71, Stanford University.
  17. Klinkenberg, S., Straatemeier M. & Vander Maas H. (2011), Computer adaptive practice of maths ability using a new item response model for on the fly ability and difficulty estimation, Computers& Education, 57 (2), 1813-1824.
  18. Lamb L., Annetta L., Vallett D. & Sadler T. (2014), Cognitive diagnostic like approaches using neural-network analysis of serious educational videogames, Computers& Education, 70, 92-104.
  19. Levy R. (2014), Dynamic Bayesian network modeling of game based diagnostic assessments, in: CRESST Conference: Warp Speed, Mr. Sulu: Integrating Games, Technology, and Assessment to Accelerate Learning in the 21st Century, Redondo Beach, CA.
  20. Levy R. (2013), Psychometric and evidentiary advances, opportunities, and challenges for simulation-based assessment, Educational Assessment, 18 (3), 182-207.
  21. Margolis M. & Clauser B. (2006), A regression-based procedure for automated scoring of a complex medical performance assessment. Mislevy R.,Willamson D. And Bejar, I. Eds (2006), Automated Scoring of Complex Tasks in Computer Based Testing, Lawrence Erlbaum, Mahwah, NJ.
  22. Mislevy R., Wilson M., Ercikan K. & Chudowsky N. Eds (2003), Psychometric principles in student assessment, Springer Netherlands. Mislevy R., Oranje A., Bauer M., von Davier A., Hao J., Corrigan S., Hoffman E.,
  23. DiCerbo K. & John M. (2014), Psychometric considerations in game-based assessment, in: GlassLab Report.
  24. Mislevy R., Steinberg L., Breyer J., Almond R. & Johnson L. (2002), Making sense of data from complex assessments, Applied Measurement in Education, 15 (4), 363-389.
  25. Mislevy R., Almond R. & Lukas J. (2003), A brief introduction to evidence-centered design, ETS Research Report Series, 3, 1-29.
  26. Quellmalz E., Davenport J., Timms M., DeBoer G., Jordan K., Huang C. & Buckley B. (2013), Next-generation environments for assessing and promoting complex science learning, Journal of Educational Psychology, 105 (4), 1100.
  27. Quellmalz E., Timms M. & Buckley B. (2010), The promise of simulation-based science assessment: The Calipers project, International Journal of Learning Technology, 5 (3), 243-263.
  28. Quellmalz E., Timms M., Silberglitt M. & Buckley B. (2012), Science assessments for all: Integrating science simulations into balanced state science assessment systems, Journal of Research in Science Teaching, 49 (3), 363-393.
  29. Saif El-Nasr, M., Drachen A. & Canossa A., eds (2013), Game analytics: Maximizing the value of player data, Springer Science& Business Media.
  30. Scardamalia M., Bransford J., Kozma B. & Quellmalz E. (2012), New assessments and environments for knowledge building, in: Assessment and teaching of 21st century skills. 231-300, Springer Netherlands.
  31. Shute V. (2011), Stealth assessment in computer-based games to support learning, Computer games and instruction, 55 (2), 503-524.
  32. Shute V. & Kim Y.J. (2011), Does playing the World of Goo facilitate learning. Yun
  33. Dai D. Eds (2012), Design research on learning and thinking in educational settings: Enhancing intellectual growth and functioning, Routledge, New York.
  34. Shute V., Ventura M. & Kim Y.J. (2013), Assessment and learning of qualitative physics in newton’s playground, The Journal of Educational Research, 106 (6), 423-430.
  35. Shute V., Ventura M., Bauer M. & Zapata-Rivera D. (2009), Melding the power of serious games and embedded assessment to monitor and foster learning. Ritterfeld U., Cody M. & Vorderer P. Eds (2009), Serious games: Mechanisms and effects, Routledge, New York.
  36. Stevens, R. & Casillas A. (2006), Artificial neural networks. Mislevy R.,Willamson D. And Bejar, I. Eds (2006), Automated Scoring of Complex Tasks in Computer Based Testing, Lawrence Erlbaum, Mahwah, NJ.
  37. Stevens R., Soller A., Cooper M. & Sprang, M. (2004), Modeling the development of problem solving skills in chemistry with a web-based tutor, in: Intelligent tutoring systems. 580-591, Springer Berlin Heidelberg.

These references have been extracted automatically and may have some errors. If you see a mistake in the references above, please contact