The Delphi Method for Graduate Research ARTICLE
Gregory J. Skulmoski, Zayed University, United Arab Emirates ; Francis T. Hartman, Jennifer Krahn, University of Calgary, Canada
JITE-Research Volume 6, ISSN 1539-3585 Publisher: Informing Science Institute
The Delphi method is an attractive method for graduate students completing masters and PhD level research. It is a flexible research technique that has been successfully used in our program at the University of Calgary to explore new concepts within and outside of the information systems body of knowledge. The Delphi method is an iterative process to collect and distill the anonymous judgments of experts using a series of data collection and analysis techniques interspersed with feedback. The Delphi method is well suited as a research instrument when there is incomplete knowledge about a problem or phenomenon; however it is not a method for all types of IS research questions. The Delphi method works especially well when the goal is to improve our understanding of problems, opportunities, solutions, or to develop forecasts. In this paper, we provide a brief background of the Classical Delphi followed by a presentation of how it has evolved into a flexible research method appropriate for a wide variety of IS research projects, such as determining the criteria for IS prototyping decisions, ranking technology management issues in new product development projects, and developing a descriptive framework of knowledge manipulation activities. To illustrate the method’s flexibility, we summarize distinctive non-IS, IS, and graduate studies Delphi research projects. We end by discussing what we have learned from using the Delphi method in our own research regarding this method's design factors and how it may be applied to those conducting graduate studies research: i) methodological choices such as a qualitative, quantitative or mixed methods approach; ii) initial question degree of focus whether it be broad or narrowly focused; iii) expertise criteria such as technical knowledge and experience, capacity and willingness to participate, sufficient time, and communication skills; vi) number of participants in the heterogeneous or homogeneous sample, v) number of Delphi rounds varying from one to 6, vi) mode of interaction such as through email, online surveys or groupware, vii) methodological rigor and a research audit trail, viii) results analysis, ix) further verification through triangulation or with another sample, and x) publishing of the results. We include an extensive bibliography and an appendix with a wide-ranging list of dissertations that have used the Delphi method (including brief research description, number of rounds and sample size). The Delphi method is a flexible, effective and efficient research method that can be successful used by IS graduate students to answer research questions in information systems and to rigorously advance the IS body of knowledge.
Skulmoski, G.J., Hartman, F.T. & Krahn, J. (2007). The Delphi Method for Graduate Research. Journal of Information Technology Education: Research, 6, 1-21. Informing Science Institute. Retrieved November 15, 2018 from https://www.learntechlib.org/p/111405/.
- Adler, M. & Ziglio. E. (1996). Gazing into the oracle: The Delphi Method and its application to social policy and public health. London: Jessica Kingsley Publishers.
- Alexander, D.C. (2004). A Delphi study of the trends or events that will influence the future of California charter schools. Digital Abstracts International, 65 (10), 3629. (UMI No. 3150304).
- Anantatmula, V.S.P. (2004). Criteria for Measuring Knowledge Management Efforts in Organizations. Digital Abstracts International, 65 (02), 597. (UMI No. 3123064). Skulmoski, Hartman, & Krahn
- Ashton, R. (1986). Combining the judgments of experts: How many and which ones? Organizational Behavior and Human Decision Processes, 38(3), 405-415.
- Ayers, R.W. (1985). Perceptions of the future roles of public school administrators as viewed by selected authors in educational futures, professors of administration and chief school administrative officers: A Delphi study. Digital Abstracts International, 46 (08), 2137. (UMI No. 8522642).
- Birdsall, I.A. (2004). It seemed like a good idea at the time: The forces affecting implementation of strategies for an information technology project in the Department of Defense. Digital Abstracts International, 65 (07), 2756. (UMI No. 3142229).
- Bolger, F. & Wright, G. (1994). Assessing the quality of expert judgment: Issues and analysis. Decision Support Systems, 11(1), 1-24.
- Braguglia, K.H. (1994). A national Delphi study of the fashion industry for curriculum development in collegiate programs of fashion merchandising. Digital Abstracts International, 55 (02), 225. (UMI No. 9416458).
- Branch, J.W. (2000). Environmental education programming for the Louisiana Cooperative Extension Service. Digital Abstracts International, 61 (12), 4646. (UMI No. 9998660).
- Brancheau, J.C., Janz, B.D., & Wetherbe, J.C. (1996). Key issues in information systems management: 1994-95 SIM Delphi results. MIS Quarterly, 20(2), 225-243.
- Brown, S.M. (1988). An assessment of ethical dilemmas experienced by university or college counseling center directors: A Delphi study. Digital Abstracts International, 49 (07), 1700. (UMI No. 8815244).
- Brungs, A. & Jamieson, R. (2005). Identification of legal issues for computer forensics. Information Systems Management, 22(2), 57-66.
- Cabaniss, K. (2001). Counseling and computer technology in the new millennium: An Internet Delphi study. Digital Abstracts International, 62 (01), 87. (UMI No. 3000284).
- Carman, W.H. (1999). An application of the Delphi method of forecasting to the future of technology infrastructure in West Virginia high schools. Digital Abstracts International, 60 (06), 1839. (UMI No. 9926681).
- Carson, W.A., III (2005). Successful implementation of enterprise resource planning software: A Delphi study. Digital Abstracts International, 66 (01), 242. (UMI No. 3161750).
- Chapman, C.N. (1992). Issues in photography that will confront photography education programs at California State University campuses by the year 2000. Digital Abstracts International, 53 (07), 2212.
- Christian, L.R. (2003). Essential characteristics of accreditation site visit team members: A Delphi study. Digital Abstracts International, 64 (03), 1158. (UMI No. 3084176).
- Costa, C.A. (2000). Future of sport management research: A Delphi study. Digital Abstracts International, 61 (11), 4332. (UMI No. 9994851).
- Cramer, R.H. (1990). Issues related to the education of gifted children in the United States: A Delphi study. Digital Abstracts International, 51 (05), 1574. (UMI No. 9028025).
- Creswell, J. (1994). Research design: Qualitative& Quantitative approaches. Thousand Oaks, USA: Sage Publications.
- Creswell, J. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, USA: Sage Publications.
- Czinkota, M. & Ronkainen (1997). International business and trade in the next decade: Report from a Delphi study. Journal of International Business Studies, 28(4), 827-844.
- Dahlby, G.K. (2004). Identification of emerging information technology issues of the 21st Century affecting public school board policies. Digital Abstracts International, 65 (06), 2033. (UMI No. 3136302). The Delphi Method for Graduate Research
- Dalkey, N.C. & Helmer, O. (1963). An experimental application of the Delphi Method to the use of experts. Management Science, 9(3), 458-468.
- Delbeq, A., Van de Ven, A., & Gustafson, D.H. (1975). Group techniques for program planning: A guide to nominal group and Delphi processes. Glenview, USA: Scott, Foresman and Company.
- Dietz, T. (1987). Methods for analyzing data from Delphi panels: Some evidence from a forecasting study. Technological Forecasting and Social Change, 31(1), 79-85.
- Doke, E.R. & Swanson, N.E. (1995). Decision variables for selecting prototyping in information systems development: A Delphi study of MIS managers. Information& Management, 29(4), 173-183.
- Duncan, N.B. (1995). Capturing flexibility of information technology infrastructure: A study of resource characteristics and their measure. Journal of Management Information Systems, 12(2), 37-58.
- Fink, A. & Kosecoff, J. (1985). How to conduct surveys: A step-by-step guide. London, UK, Sage Publications.
- Ford, R.T. (1989). A descriptive study of health experts' attitudes about an innovative medical self-care model. Digital Abstracts International, 28 (01), 120. (UMI No. 1337728). Fowler Jr., F. (1993). Survey research methods. Thousand Oaks, USA: Sage Publications.
- Friend, J.G. (2001). A Delphi study to identify the essential tasks and functions for ADA coordinators in public higher education. Digital Abstracts International, 62 (04), 1339. (UMI No. 3012967).
- Gerdsri, N. (2005). An analytical approach on building a technology development envelope (TDE) for roadmapping of emerging technologies. Digital Abstracts International, 66 (03), 1697. (UMI No. 3169411).
- Glass, R.I. (1997). Telling good numbers from bad ones. IEEE Software, 14(4), 15-19.
- Good, J.J. (1998). Recommendations for change in physical education: A survey of selected physical education professionals. Digital Abstracts International, 59 (07), 2417. (UMI No. 9839207).
- Gustafson, D.H., Shukla, R.K., Delbecq, A., & Walster, G.W. (1973). A comparison study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups and nominal groups. Organizational Behavior and Human Performance, 9(2), 280-291.
- Hartman, F. & Baldwin, A. (1995). Using technology to improve the Delphi method. Journal of Computing in Civil Engineering, 9, 244-249.
- Hartman, F. & Jugdev, K. (1998). Leadership undertow: Project manager fears and frustrations. Proceedings of the 29th Annual Project Management Institute Seminars and Symposium, Long Beach, USA.
- Holmes, W.M. (2005). Emerging practice in occupational therapy: An exploratory study of its nature and competencies for practice. Digital Abstracts International, 66 (11). (UMI No. 31955778).
- Holsapple, C.W. & Joshi, K.D. (2002). Knowledge manipulation activities: Results of a Delphi study. Information& Management, 39(6), 477-492.
- Keil, M., Tiwana, A. & Bush, A. (2002). Reconciling user and project manager perceptions of IT project risk: A Delphi study. Information Systems Journal, 12(2), 103-119.
- Kincaid, S.O. (2003). Web-based courses in human services: A comparison of student and faculty perceptions of factors that facilitate or hinder learning. Digital Abstracts International, 64 (07), 2403. (UMI No. 3098041).
- Koch, T. (1994). Establishing rigor in qualitative research: The decision trail. Journal of Advanced Nursing, 19(11), 5.
- Krahn, J. & Hartman, F. (2004). Important leadership competencies for project managers: The fit between competencies and project characteristics. Proceedings of the 2004 PMI Research Conference, London, UK. Skulmoski, Hartman, & Krahn
- Krebsbach, S.J.G. (1998). A set of learning outcomes for the learner in the two-year institution of higher education in order to function in work, community, and family at the beginning of the twenty-first century. Digital Abstracts International, 59 (09), 3329. (UMI No. 9907508).
- Kuo, N.-W. & Yu, Y.-H. (1999). An evaluation system for national park selection in Taiwan. Journal of Environmental Planning and Management, 42(5), 735-743.
- Lam, S.S.Y., Petri, K.L., & Smith, A.E. (2000). Prediction and optimization of a ceramic casting process using a hierarchical hybrid system of neural networks and fuzzy logic. IIE Transactions, 32(1), 83-92.
- Laxton, R.E. (2002). Quality indicators of effective pupil transportation programs. Digital Abstracts International, 63 (11), 3803. (UMI No. 3071361).
- Lecklitner, G.L. (1984). Protecting the rights of mental patients: A view of the future. Digital Abstracts International, 46 (01), 306. (UMI No. 8504044).
- Levinson, J.M. (2005). To gain consensus on a definition of multicultural children's literature: A Delphi study. Digital Abstracts International, 66 (08), 2869. (UMI No. 3184291).
- Linstone, H. & Turloff, M. (1975). The Delphi method: Techniques and applications. London, UK: Addison-Wesley.
- Mason, J. (1996). Qualitative researching. Thousand Oaks, USA: Sage Publications.
- Menix, K.D. (1997). Validation of change management concepts by nurse managers and educators: Baccalaureate curricular implications. Digital Abstracts International, 58 (10), 5329. (UMI No. 9812039).
- Mullen, J.A. (1993). Superintendents predict the future of education: A modified Delphi. Digital Abstracts International, 32 (02), 389. (UMI No. MM82110).
- Nambisan, S., Agarwal, R., & Tanniru, M. (1999). Organisational mechanisms for enhancing user innovation in information technology. MIS Quarterly, 23(8), 365-395.
- Niederman, F., Brancheau, J.C., & Wetherbe, J.C. (1991). Information systems management issues for the 1990s. MIS Quarterly, 15(4), 475-500.
- Nolan, L.N. (1994). Forward to the future: A Delphi study of the future of education. Digital Abstracts International, 55 (07), 1747. (UMI No. 9430911).
- Parente, F.J., Anderson, J.K., Myers, P., & O’Brien, T. (1994). An examination of factors contributing to Delphi accuracy. Journal of Forecasting, 3(1), 173-183.
- Peffers, K. & Tuunanen, T. (2005). Planning for IS applications: A practical, information theoretical method and case study in mobile financial services. Information& Management, 42(3), 483-492.
- Perez, V.L. & Schueler, R. (1982). The Delphi method as a tool for information requirements specification. Information& Management, 5(3), 157-168.
- Pieko, E. (2005). Improving the quality of information technology (IT) security audits for federal agencies. Digital Abstracts International, 61 (09), 3401. (UMI No. 9987367).
- Prescott, P. & Soeken, K. (1989). The potential uses of pilot work. Nursing Research, 30, 60-62.
- Prestamo, A.M. (2000). A comprehensive inventory of technology and computer skills for academic reference librarians. Digital Abstracts International, 61 (09), 3401. (UMI No. 9987367).
- Richards, J.E. (2000). Public health informatics: A consensus on core competencies. Digital Abstracts International, 61 (08), 2964. (UMI No. 9983325).
- Roberson, Q.M., Collins, C.J., & Oreg, S. (2005). The effects of recruitment message specificity on applicant attraction to organizations. Journal of Business& Psychology, 19(3), 319-340.
- Rosenbaum, J. (1985). A College and University Curriculum Designed to Prepare Students For Careers in Non-Broadcast Private Telecommunications: A Delphi Method Survey of Professional Video Communicators. Digital Abstracts International, 46 (09), 2548. (UMI No. 8525512).
- Rowe, G. & Wright, G. (1999). The Delphi technique as a forecasting tool: Issues and analysis. International Journal of Forecasting, 15(4), 353-375.
- Sadleowski, M. (1986). The problem of rigor in qualitative research. Advances in Nursing Science, 8(3), 27-37.
- Sawyer, S. (1997). Share your results and your methods. IEEE Software, 10.
- Schmidt, R. (1997). Managing Delphi surveys using nonparametric statistical techniques. Decision Sciences, 28(3), 763-774.
- Schmidt, R., Lyytinen, K., Keil, M., & Cule, P. (2001). Identifying software project risks: An international Delphi study. Journal of Management Information Systems, 17(4), 5-36.
- Schmidt, V.V. (1995). Awakening intuition: A Delphi study. Digital Abstracts International, 56 (09), 3498. (UMI No. 9543808).
- Scott, G. (2000). Critical technology management issues of new product development in high-tech companies. Journal of Product Innovation Management, 17(1), 57-77.
- Shook, S.A. (1994). The identification of key change agents and techniques related to the change from an industrial arts program to a technology education program. Digital Abstracts International, 55 (10), 3113. (UMI No. 9506289).
- Shuman, D. (2000). Implementation theory and determinants for success: A case study of televised distance learning implementation in an urban university. Digital Abstracts International, 61 (04), 1371. (UMI No. 9970453).
- Silverman, A.S. (1981). Development of content areas and objectives for a curriculum in death and dying education for junior high school students. Digital Abstracts International, 42 (05), 1975. (UMI No. 8122977).
- Skulmoski, G. & Hartman, F. (2002). The Delphi method: Researching what does not exist (yet). Proceedings of the International Research Network on Organization by Projects, IRNOP V Conference, Renesse, The Netherlands.
- Thomson, B.R. (1985). Appropriate and inappropriate uses of humor in psychotherapy as perceived by certified reality therapists: A Delphi study (Delphi Method). Digital Abstracts International, 47 (01), 90. (UMI No. 8606095).
- Topper, W.W. (2006). Leadership change in privately controlled businesses: A Delphi study of succession planning best practices. Digital Abstracts International, 67 (01). (UMI No. 3206379).
- Vazquez-Ramos, R.A. (2003). A Delphi study to assess a potential set of items to evaluate participatory ethics in rehabilitation counseling. Digital Abstracts International, 64 (04), 1231. (UMI No. 3087663).
- Warner, M. (1990). Recreational Foodservice management: A Delphi study of needed competencies. Digital Abstracts International, 51 (02), 430. (UMI No. 9019144).
- Watson, A.C. (1982). A Delphi Study of Paradox in Therapy. Digital Abstracts International, 43 (10), 3380. (UMI No. 8304627).
- Wei, W.-J.M. (2000). Development of possible cognitive competencies for use in developing a criterionreferenced performance test to license kindergarten teachers in Taiwan. Digital Abstracts International, 61 (09), 3462. (UMI No. 9987282). Skulmoski, Hartman, & Krahn
- Whittinghill, W.D. (2000). Identification of the initial curriculum components for the preparation of graduate-level substance abuse counselors. Digital Abstracts International, 61 (08), 3072. (UMI No. 9984510).
- Wilke, B.J. (1982). The Future of the General Instruction Physical Education Program in Higher Education: A Delphi Study. Digital Abstracts International, 43 (07), 2278. (UMI No. 8226951).
- Wynekoop, J.L. & Walz, D.B. (2000). Investigating traits of top performing software developers. Information Technology& People, 13(3), 186-197.
- Yang, Y.N. (1998). Integrating the WorldWide Web into art education: Guidelines for designing a Webbased art teacher education curriculum. Digital Abstracts International, 60 (03), 622. (UMI No. 9921650).
- Zaghloul, R. & Hartman, F. (2003). Construction contracts: The cost of mistrust. International Journal of Project Management, 21(6), 419-424.
- Zanetell, B.A. (2003). Stakeholder collaboration and discourse: Delphi-generated global and local visions for water resources management. Digital Abstracts International, 63 (12), 5742. (UMI No. 3075864).
These references have been extracted automatically and may have some errors. If you see a mistake in the references above, please contact firstname.lastname@example.org.