Evaluating a Complex Program: Where to Start and How to Finish
Data on the intrinsic characteristics of an educational program are essential in pilot testing new programs in order to determine how successfully design components have achieved their intended purpose. At the outset of an evaluation of such data, it is necessary to define the evaluator's role and determine what intrinsic information will be required, when it will be needed, who can provide it, and how it will be collected. The Adjusted Agreement Index (AAI) was used in a case study to summarize data from a complex program developed by the Institute of Canadian Bankers, in which eight university-level distance education courses were simultaneously pilot-tested for 26 weeks with over 1,000 Canadian students. The AAI score depicts relative agreement by respondents and is calculated by subtracting the percentage of respondents who disagree with a statement from the percentage who agree. Results indicate that the AAI is a helpful tool in summarizing results clearly so that decision-makers will know which intrinsic characteristics require immediate attention for revision purposes. The AAI is easy to calculate and use with subjective response questionnaires. Two references are listed, and four figures illustrate evaluation design, AAI calculation, score distribution for various AAI results, and student perceptions of assignments by course. (LMM)
Kemerer, R.W. & Schmid, R.F. Evaluating a Complex Program: Where to Start and How to Finish.