You are here:

Evaluating a Complex Program: Where to Start and How to Finish



Data on the intrinsic characteristics of an educational program are essential in pilot testing new programs in order to determine how successfully design components have achieved their intended purpose. At the outset of an evaluation of such data, it is necessary to define the evaluator's role and determine what intrinsic information will be required, when it will be needed, who can provide it, and how it will be collected. The Adjusted Agreement Index (AAI) was used in a case study to summarize data from a complex program developed by the Institute of Canadian Bankers, in which eight university-level distance education courses were simultaneously pilot-tested for 26 weeks with over 1,000 Canadian students. The AAI score depicts relative agreement by respondents and is calculated by subtracting the percentage of respondents who disagree with a statement from the percentage who agree. Results indicate that the AAI is a helpful tool in summarizing results clearly so that decision-makers will know which intrinsic characteristics require immediate attention for revision purposes. The AAI is easy to calculate and use with subjective response questionnaires. Two references are listed, and four figures illustrate evaluation design, AAI calculation, score distribution for various AAI results, and student perceptions of assignments by course. (LMM)


Kemerer, R.W. & Schmid, R.F. Evaluating a Complex Program: Where to Start and How to Finish. Retrieved April 19, 2019 from .

This record was imported from ERIC on March 21, 2014. [Original Record]

ERIC is sponsored by the Institute of Education Sciences (IES) of the U.S. Department of Education.

Copyright for this record is held by the content creator. For more details see ERIC's copyright policy.