You are here:

Evaluating and Comparing the Usability of Web-based Course Management Systems
ARTICLE

, University of South Florida, United States ; , Turkey

JITE-Research Volume 10, Number 1, ISSN 1539-3585 Publisher: Informing Science Institute

Full Text

Abstract

Course Management Systems (CMS) are an increasingly important part of academic systems in higher education. When choosing a Course Management System for an educational institution, the usability of the system is the key to the effectiveness and efficiency of the online courses that are to be implemented. The goal of this paper is to report the results of a comparative usability study conducted in 2008-2009 on two different course management systems: BlackBoard and Moodle. 135 students enrolled in the Fall 2008 and Spring 2009 sections of Introduction to Educational Technology participated in the study (72 and 63 respectively). At the beginning of each semester, participants were randomly divided into two groups to experience different CMSs at different times. It can be concluded from this study that in almost every module or function comparison that was made, Moodle was favored by course participants over Blackboard with the exception of the Discussion Board module where scores were not significantly different. At the end of the study, the researchers concluded that use of Moodle in online courses can be a suitable alternative to the current CMS system (BlackBoard). In fact, now that the pilot has showed that Moodle is as effective as BlackBoard, the researchers have already shared their experiences with other faculty members and expanded their investigations by involving numerous other online courses, instructors, and students, because the product showed significant potential for further examination. This study adds to the growing body of studies that are carried out to see if an open source CMS (Moodle) warrants consideration as an alternative to the institution’s current course management system. In addition to comparing the students’ feedback quantitatively, this study also tried to explain in detail what specific component / function of each CMS students found useful or better than in the other. Rather than focusing only on student satisfaction scores, this study further investigated what aspect of each module for each CMS course participants particularly liked or disliked.

Citation

Unal, Z. & Unal, A. (2011). Evaluating and Comparing the Usability of Web-based Course Management Systems. Journal of Information Technology Education: Research, 10(1), 19-38. Informing Science Institute. Retrieved August 11, 2024 from .

Keywords

References

View References & Citations Map
  1. Agarwal, R., & Prasad, J. (1999). Are individual differences germane to the acceptance of new information technologies? Decision Sciences, 30(2), 361-391.
  2. BlackBoard. (2010). BlackBoard Incorporation. Retrieved May 25, 2010 from www.blackboard.com
  3. Brown, B.L. (2000). Web-based training. ERIC Clearinghouse on Adult, Career, and Vocational Education. Retrieved on May 3, 2010 from http://www.ericacve.org/docgen.asp?tbl=digests & ID=103
  4. Brush, A.J.B, Ames, M., & Davis, J. (2004). A comparison of synchronous remote and local usability studies for an expert interface. Proceedings of Computer Human Interaction 2004, 1179-1182
  5. Burrell-Ihlow, M. (2009). An investigation of teaching and learning: Using course management software (CMS) in a typically face to face course. International Journal on E-Learning, 8(1), 5-16. Chesapeake, VA: AACE.
  6. Chen, D-T., Wang, Y-M., & Hung, D. (2009). A journey on refining rules for online discussion: Implications for the design of learning management systems. Journal of Interactive Learning Research, 20(2), 157-173. Chesapeake, VA: AACE.
  7. Colace, F., Santo, M.D., & Vento, M. (2002). Evaluating on-line learning platforms: A case study. Paper presented at the 36th Hawaii International Conference on System Sciences, Big Island, Hawaii.
  8. Davis, F.D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319-340.
  9. Davis, F.D., Bagozzi, R.P., & Warshaw, P.R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982-1003.
  10. De Lera, E., Fernandez, C., & Valverde, L. (2010). The emotional gap in virtual online environments. In Z. Abas et al. (Eds.), Proceedings of Global Learn AsiaPacific 2010 (pp. 67-70). AACE.
  11. Dray, S., & Siegel, D. (2004). Remote possibilities? International usability testing at a distance. Interactions, 11(2), 10-17.
  12. Gilbert, L., & Moore, D. (1998). Building interactivity in Web courses: Tools for social and instructional interaction. Educational Technology, 38, 29–35.
  13. Graham, M., & Scarborough, H. (2001). Enhancing the learning environment for distance education students. Distance Education, 22(2), 232-244.
  14. Heo, M. (2009). Design considerations for today’s online learners: A study of personalized, relationshipbased social awareness information. International Journal on E-Learning, 8(3), 293-311. Chesapeake, VA: AACE.
  15. Hsu, M.H., & Chiu, C.M. (2004). Internet self-efficacy and electronic service acceptance. Decision Support Systems, 38(3), 369-381.
  16. Iding, M.K., Auernheimer, B., Crosby, M.E., & Klemm, E.B. (2002). Guidelines for designing evaluations of web based instructional materials. Paper presented at the 36th Hawaii International Conference on System Sciences, Big Island, Hawaii.
  17. Igbaria, M., Zinatelli, N., Cragg, P., & Cavaye, A. (1997). Personal computing acceptance factors in small firms: A structural equation model. MIS Quarterly, 21(3), 279-305.
  18. InsideHigherEd. (2010). Blackboard buys Angel. Inside higher education organization, Retrieved May 25, 2010 from http://www.insidehighered.com/news/2009/05/07/bb Inversini, A., Botturi, L., & Triacca, L. (2006). Evaluating LMS usability for enhanced elearning experience. Paper presented at the EDMEDIA Conference, Orlando, Florida.
  19. Johnson, S.D., Suriya, C., Yoon, S.W., Berrett, J.V., & La Fleur, J. (2002). Team development and group processes of virtual learning teams. Computers and Education, 39(4), 379-393.
  20. Kakasevski, G., Mihajlov, M., Arsenovski, S., & Chungurski S. (2008). Evaluating usability in learning management system Moodle. Proceedings of the 30th International Conference ITI Cavtat/Dubrovnik, Croatia, June 23-26, 2008, P.613-618
  21. Kantner, L., Sova, H.D., & Rosenbaum, S. (2003). Alternative methods for field usability research. Proceedings of Special Interest Group for Design of Communication Conference, 2003, 68-72
  22. Kearsley, G. (1997). A guide to online education. Retrieved May 1, 2010, from http://gwis.circ.gwu.edu/etl/online.html
  23. Kibaru, F., & Dickson-Deane, C. (2010). Model for training usability evaluators of e-learning. In J. Sanchez& K. Zhang (Eds.), Proceedings of World Conference on E-Learning inCorporate, Government, Healthcare, and Higher Education 2010 (pp. 517-522). Chesapeake, VA: AACE.
  24. Kidney, G., & Puckett, E. (2003). Rediscovering the first principles through online learning. Quarterly Review of Distance Education, 4, 203–212.
  25. Lansari, A., Tubaishat, A., & Al-Rawi, A. (2010). Using a learning management system to foster independent learning in an outcome-based university: A gulf perspective. Proceedings of Issues in Informing Science and Information Technology (pp. 73-87). Retrieved from http://iisit.org/Vol7/IISITv7p073087Lansari733.pdf
  26. Lucas, H.C., & Spitler, V.K. (1999). Technology use and performance: A field study of broker workstations. Decision Sciences, 30(2), 291–311.
  27. Machado, M., & Tao, E. (2007). Blackboard vs. Moodle: Comparing user experience of learning management systems. Paper presented at the 37th ASEE/IEEE Frontiers in Education Conference, Milwaukee, WI.
  28. Mathieson, K. (1991). Predicting user intentions: Comparing the technology acceptance model with the theory of planned behavior. Information Systems Research, 2(3), 173-191.
  29. Merrill, M. (1997). Instructional strategies that teach. Computer Based Training Solutions, 1–11.
  30. Moodle. (2010). Moodle organization. Retrieved May 25, 2010 from www.moodle.org Neal, L. (N.D.). Q&A with Don Norman. ELearn Magazine. Retrieved December 21, 2010 from http://www.elearnmag.org/subpage.cfm?section=articles&article=16-1
  31. Nielsen, J. (1993). Usability engineering. Massachusetts: Academic Press
  32. Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen& R.L. Mack (Eds.), Usability inspection methods (pp. 25-62). New York: John Wiley& Sons.
  33. Nielsen, J. (2003). Usability 101: Introduction to usability. Retrieved May 25, 2010 from http://www.useit.com/alertbox/20030825.html
  34. Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Educational Technology& Society, 9(2), 178-197
  35. Nguyen, T., Chang, V., Chang, E., Jacob, C., & Turk, A. (2008). A contingent method for usability evaluation of web-based learning systems. In K. McFerrin et al. (Eds.), Proceedings of Society for Information Technology& Teacher Education International Conference 2008 (pp. 579-585). Chesapeake, VA:
  36. Petersen, D. (2007). Usability theory, practice and evaluation for learning objects. In K. Harman& A. Koohang (Eds.), Learning objects: Applications, implications, & Future directions (pp.337-370). Santa
  37. Ramos, K., MacLean, A., Bates, P., Wylie, P., & Brempah, A. (2010). Using Moodle lessons for the development of an e-learning programme in women’s heath. In J. Sanchez& K. Zhang (Eds.), Proceedings of World Conference on E-Learning inCorporate, Government, Healthcare, and Higher Education 2010 (pp. 764-769). Chesapeake, VA: AACE.
  38. Rivard, S., & Huff, S. (1988). Factors of success for end-user computing. Communications of the ACM, 31(5), 552-561.
  39. Rovai, A.P., Ponton, M.K., & Baker, J.D. (2008). Distance learning in higher education. A programmatic approach to planning, design, instruction, evaluation, and accreditation. New York, NY: Teachers College Press.
  40. Rowley, D.E. (1994). Usability testing in the field: Bringing the laboratory to the user. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence, 252257.
  41. Rozanski, E.P., & Haake, A.R. (2003). Curriculum and content: The many facets of HCI. Paper presented at the 4th Conference on Information Technology Curriculum on Information Technology Education, Lafayette, IN.
  42. Simbulan, M.S. (2007). Learning objects’ user interface. In K. Harman& A. Koohang (Eds.), Learning objects: Applications, implications, & Future directions. (pp. 259-336). Santa Rosa, CA: Informing
  43. Taylor, S., & Todd, P.A. (1995). Assessing IT usage: The role of prior experience. MIS Quarterly, 19(4), 561–570.
  44. Tennyson, R.D., & Cocchiarella, M.J. (1986). An empirically based instructional design theory for teaching concepts. Review of Educational Research, 56(1), 40-71.
  45. Tsang, P., Kwan, R., & Fox, R. (2007). Enhancing learning through technology. World Scientific Publishing Co.
  46. Ullman, C., & Rabinowitz, M. (2004). Course management systems and the reinvention of instruction. THE Journal. Retrieved May 25, 2010 from www.thejournal.com/articles/17014
  47. Venkatesh, V., & Davis. F.D. (1996). A model of the antecedents of perceived ease of use: Development and test. Decision Sciences, 27(3), 451–481.
  48. Venkatesh, V., & Davis, F.D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management Science, 46(2), 186-204.
  49. Wang, H. (2010). Towards a real virtual classroom. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 2010 (pp. 1123-1129). Chesapeake, VA: AACE.

These references have been extracted automatically and may have some errors. Signed in users can suggest corrections to these mistakes.

Suggest Corrections to References

Links