Dimensions of evaluation

by David Jaques

Types of evaluation are not primarily distinguished by the techniques involved (for example the difference between questionnaires and interviews) but by their purpose: by what evaluation is perceived to be for, who it is for, and what sort of outcome is expected. These pages describe a number of dimensions of evaluation. Find out about what evaluation of teaching and courses has gone on recently in your department and then read through the pages, identifying where this evaluation is placed on each of the dimensions.

Quantitative vs Qualitative Evaluation

Quantitative evaluation attempts to measure or obtain a quantitative fix on what is going on. It counts instances, students, or frequencies of ratings and uses the numbers which emerge to provide a picture of what is happening. These numbers are often compared with numbers derived from evaluations of other courses, other teachers, or other occasions on which a course was taught in order to obtain a quantitative picture of whether a course is better or worse than some other course. For example in 53 interesting ways to appraise your teaching item 1 represents an attempt to summarise the performance of a lecturer in six scores for the purpose of comparing that lecturer with others. Qualitative evaluation attempts to describe what is going on. The descriptions can consist of observations, transcripts of interviews, photographs, videotapes or other kinds of information which convey the quality of what is being evaluated. For example in 53 interesting ways to appraise your teaching item 22 involves a method for portraying students' overall reactions to a course through very brief, randomly collected video interviews. Often quantitative evaluation is associated with the use of questionnaires, and qualitative evaluation with the use of interviews. However questionnaires can ask open-ended questions and collect qualitative data, and interviews can ask standard closed questions and count how often different responses are elicited.

What type of evaluation goes on in your department, quantitative or qualitative? And does the type of evaluation used provide the type of information which is required?

Hypothesis testing vs Goal Free Evaluation

Related to the distinction between quantitative and qualitative evaluation is the distinction between evaluation which is intended to test specific hypotheses and evaluation which is open-minded, or 'goal free' about what is going on in a course. In 53 interesting ways to appraise your teaching item 12 illustrates a questionnaire which is designed to check whether students believe the same things as the lecturer about a course. The lecturer's hunches, or hypotheses, are the questionnaire items, and students are invited to state whether they agree with these hunches or not. The use of the questionnaire gives a quantitative indication of the level of support for the lecturer's hunches. This is in marked contrast to item 19 which describes a group discussion technique in which there is no constant upon what is examined. Whatever students come up with is what is explored. There isn't a simple relationship between quantitative methods and hypothesis testing, however. Item 18 describes a goal-free approach to devising a quantitative questionnaire, and interviews (see item 15) can be used to check out hypotheses.

What type of evaluation goes on in your department, hypothesis testing, or goal free? And does the type of evaluation used provide the type of information which is required?

Teacher-centred vs learner-centred

An extraordinary proportion of evaluation questions, hypotheses which are tested, and questionnaire items, are concerned with teaching: with how the lecturer performs, particularly in lectures. In 53 interesting ways to appraise your teaching items 1, 2, 7, 8, 10 and 14 contain teacher-centred questionnaires. It is unusual for evaluations to ask about how students performed on a course: how many hours they put in compared with other courses, how hard they tried, how much they read, whether they took useful notes in lectures, and so on. Items 3 and 4 contain questionnaires which are, in the main, about the student's perception of learning rather than of teaching. As students in higher education spend far more time learning independently than in being taught, the common emphasis of evaluation on teaching seems egocentric or even perverse. Item 11 demonstrates how to distinguish between teacher-centred and learner-centred approaches, by asking students to rate teacher-centred issues, but then also asking them to rate how important these issues are. It may be that a teacher is rated as lacking a sense of humour (a teacher-centred concern) but also that a sense of humour is considered of no importance.

Is the evaluation undertaken in your department teacher-centred or learner-centred? And does the type of evaluation used provide the type of information which is required? What would you like to know about your courses from the learner's perspective?

Back to top

Own perspective vs others' perspective

This issue goes beyond the difference between the teacher's perspective and the learner's perspective. In evaluating a field course the teacher may be concerned about whether course objectives were met and the students may be concerned about whether they enjoyed themselves or whether it was a good use of their time. The Finance Officer may be concerned about whether the course objectives could have been achieved in a less expensive way, and employers may be concerned about whether the field experience provided a relevant experience of contemporary working practices. The teacher's own perspective is only one of several interested parties. Sometimes the perspective of others can be useful to illuminate a course in unexpected ways. In educational evaluation one strategy is based on the approach of anthropology. Some anthropologists assume that it is trained, neutral outsiders who are in the best position to observe and understand how a culture operates. Similarly there can be special advantages in using the perspective of outsiders in course evaluation. Outsiders might include colleagues from another department or even another institution, educational researchers or educational development consultants from an educational methods unit. Educational practices which we take for granted can look pretty bizarre to someone from industry or local government. Sometimes the more distant the other person's perspective, the more interesting is the view.

Has the evaluation undertaken in your department been all from your own perspective, or have others' views been involved? If you wanted another person's perspective, who would you choose, and why?

Monitoring fixed courses vs exploring responsive courses

Many courses are perceived as relatively fixed. They may have been running for a long time without giving cause for alarm, or be constrained by external professional bodies, resource or staffing problems, or requirements of other courses for which the course is a prerequisite. Evaluating such a course is more like a monitoring process. Fine-grain details are examined and broad assumptions are not questioned. It is assumed that the broad framework and major decisions about course content, teaching methods and assessment are either already correct or that there is no scope for change and so no point in evaluating them. Sometimes evaluations are carried out which collect data about aspects of a course which cannot easily be changed. When this happens the evaluation data is often considered to be flawed in some way if it poses too great a threat to entrenched views about the course. The sample is claimed to be too small or biased, or questions are considered to be ambiguous. These criticisms of the evaluation might not have been raised if the data had not challenged basic assumptions about the course. It may not be worthwhile carrying out anything more than routine monitoring if there is little chance of responding to evidence suggesting fundamental change. On the other hand it may not be worthwhile undertaking routine monitoring if large scale change is possible and you are prepared to be very flexible. There is little point collecting detailed ratings of specific lecturing skills if you have not yet made up your mind whether or not to stick with a lecture-based course. A second issue about the difference between fixed and responsive courses is that some courses only change after they have finished, in time for the next group of students, whereas others can change while the course is running, in response to student feedback, in a way which benefits the current group of students. Often it is the case that some design features cannot change during the operation of the course (such as the assessment scheme) whereas others can easily be modified (such as the operation of seminars or even the timetable). It may be worth evaluating some aspects of a course as you go along and others after it has finished. One good reason for attempting to be responsive during a course is that few students experience the consequences of evaluation directly. Only the next group of students sees the benefit. Over time this can make students disillusioned about filling in questionnaires or giving their time to evaluations.

In your department, is the evaluation more like the monitoring of fixed courses, or the exploration of responsive courses? What further scope is there for evaluating aspects of courses while they are in operation, with a view to implementing change straight away?

Back to top

Collective responsibility vs Individual responsibility

In many North American Universities it is common for most evaluation to be undertaken by the administration for the purposes of making decisions about the future of courses and of staff. In Britain most evaluation is still undertaken by individuals for their own purposes, or by small groups of lecturers. The data in America is often published and used by students to choose courses, and by the administration to make employment decisions. In Britain the data is usually owned by individuals and small groups and is highly confidential. A few years ago it looked as though the CNAA might pass course evaluation data in its possession, collected for course validation purposes, over to the funding body of the time, for the purpose of making differential funding decisions. The quantity of course evaluation temporarily plummeted. Who takes responsibility for evaluation, and who owns the data, are crucial questions determining the nature of evaluation. Faculty-wide or institution-wide evaluations are nearly always crudely quantitative in nature, to allow easy comparisons between courses or between lecturers. Individually initiated evaluations tend to be more qualitative in nature, more varied, and address issues of local concern. Sometimes there are subtle conventions about what department-wide evaluations can legitimately examine and what they cannot. For example at Oxford Polytechnic several departments undertook comparative reviews of the workloads across all the courses in the department. The lecturers responsible for those courses identified as having abnormal workloads were encouraged to undertake individual evaluations in order to diagnose and cure these abnormalities. But the departments concerned did not feel that they could legitimately evaluate details of the courses themselves. The mood is shifting on this issue, and several departments at Oxford now have standard questionnaires which all courses are required to use, and questionnaire results are required to be reported annually and publicly. Responsibility for evaluation is becoming more collective. The danger here is that individual responsibility may wane as a result.

In your department, what is the balance between collective and individual responsibility for different types of evaluation, who owns the data, and what is reported publicly?

General issues vs specific issues

A problem which can emerge if most of the responsibility for evaluation is individual or located in small groups, is that wider educational issues tend not to get evaluated. Sometimes there are large-scale changes in the way an institution operates, and somebody should be responsible for evaluating these changes to see if they have been worthwhile. For example at Oxford Polytechnic such large-scale changes have included reducing the number of courses which students need to take and pass in order to obtain a degree, and reorganising the timetable so that all teaching takes place in three-hour instead of one-hour blocks. You need good evidence to have the courage to make such decisions, and good evidence about their consequences. If all evaluation is undertaken by individual lecturers for their own purposes, no such general issues are ever explored. Issues of wider concern could include the consequences of larger classes, the effects of library loan policies, the desirability of extending the teaching day into the evening, the effect of shortening or lengthening the term and the size and furnishing of teaching rooms. These are issues which affect everybody.

List the general issues which are of concern to you in teaching your courses which would need to be evaluated by someone else on your behalf. Who might undertake such evaluation and how would you make your need for evaluation data felt?

Back to top

Outcome vs process

It is common for evaluation studies to seek evidence of outcomes of courses or features of courses, rather than to seek evidence about the process of the course. For example a Geology field course evaluation at Oxford focussed on students' views at the end of the course. The students rated the field element as invaluable for consolidating and applying knowledge. The students felt that they would like more fieldwork and courses containing fieldwork were considered especially rewarding. Employers rated the fieldwork courses highly and graduates are successful in getting jobs in which fieldwork skills are applied. This looks like an impressively positive measure of the outcomes of the course. However there is almost no data which might guide the future design and operation of fieldwork. If different lecturers ran the course, if funding problems shortened the duration of fieldwork, or if the fieldwork locations became inaccessible, would this make any difference? Could the Geologists do almost anything they liked with the fieldwork and still get positive outcomes? Their evaluation did not inform them about this. At the same time Geographers were evaluating their fieldwork. They were already convinced about the overall value of fieldwork, about the outcomes of fieldwork, but they were unsure what were the crucial components of the process of fieldwork which made it especially effective. Their evaluation consisted of student-made videos during the fieldwork, and student diaries. This provided a rich source of insights into the processes of learning during fieldwork which guided its subsequent design. However they did not end up with evidence, and certainly not quantitative evidence, about the outcomes of fieldwork. Lecturers often tell me that they are disappointed with their evaluation data because it does not tell them what to do about the problems uncovered. The reason is that they didn't seek evidence about process, but only about outcome. Items 22, 23, 33, and 34 in 53 interesting ways to appraise your teaching are concerned with process rather than outcome.

Does the evaluation in your department tend to seek evidence about outcomes, or does it also seek evidence about process? List examples of evidence which could be collected about process.

Political (demonstration) vs Personal (learning)

The reason for inquiring about outcome rather than process is very often political. Evaluation is often used as a device to back up arguments for funds, to obtain course approval, to defend against cuts, and so on. I have been asked to undertake evaluations by Heads of Department into situations where there is already a strongly formed view of what is wrong and what needs changing. Evaluation is dragged in in an attempt to provide credibility and spurious objectivity to arguments formed on the basis of subjective impressions. The desire to demonstrate a point clearly involves political use of evaluation. In contrast the Geographers were genuinely interested in learning about what made their field work effective. The sort of evidence they collected highlighted as many problems as successes, but they moved forwards in their understanding of how to design effective field work.

In your department, is evaluation used for mainly political purposes, or is it part of a genuine attempt to understand and learn about how to teach and run courses?

This paper was first published by OCSLD in 1989