(The following information is the result of a review of evaluation procedures for online courses carried out by Harvey Mellar at the Institute of Education in 2003)

The evaluation of on-line courses presents some additional issues for evaluation that do not apply to face-to-face courses operating according to well understood and widely agreed proceedures.

1.Major review

To be carried out at the end of the first run of the course, and every 3 to 5 years at periodic review. These criteria are additional to those used for face-to-face courses. Expert review of:

1.1. Course design	Does the course take advantage of the unique possibilities of on-line delivery? (Suggested criteria provided in 4.1)
1.2. Materials	Examination of the quality of course web site, and teaching materials. (Suggested criteria provided in 4.2)
1.3. On-line interactions	Examination of activities used, student performance, participation and interactions and the role of the tutor(s). (Suggested guidelines provided in 2.3)
1.4. Management and support of on-line tutors	Are tutors new to on-line teaching given appropriate support? Procedures for the management of the course team. (Suggested guidelines provided in 2.4)
1.5. Technology	Is just-in-time technical assistance available to students and tutors? Are procedures in place to guarantee security of student work?
1.6. Access	Students are advised/surveyed about the programme to determine that they possess the self-motivation and commitment to learn on-line (Possible provision of an on-line pre-course taster pack for prospective students.). Provision is made for students with disabilities and special needs. Integrated student support services are available on-line to learners. Equal access to necessary course materials is available to all learners.

2. Annual review

2.1. Administrative data	Does the administrative and technical infrastructure provides access to all prospective and enrolled learners? Reliability of technical infrastructure – hardware, software, staffing and technical assistance. Accounts for using networks,web sites, and virtual classroom software are set up quickly. Timely web content updating. Delivery of learning materials within # days of registration. Reliable, timely information of any changes. Response to email messages within # hours. Feedback to drafts within # days. Grade reporting within # days. Final results available within # days. Comparisons with similar face-to-face courses on attainment, retention and completion rates, reasons for withdrawal and costs.
2.2. Evaluation Questionnaire data	Processing of data from evaluation questionnaires, delivered on-line and completed anonymously by students and tutors (Sample questions provided in 5.1, 5.2).
2.3. On-line activities, environment and reference material	Course team review of design of on-line activities and of on-line environment, and of the relevance and appropriateness of reference and reading materials.

3. Mid-course review

A short on-line evaluation form should be created to allow students to feedback on their experience of the course to date, at a point about halfway through the course. This is an additional support to alert staff to potential problems before the course finishes.

4. Guidelines and Criteria

4.1. Course design	What is the justification for the course being delivered on-line? Is the design adapted to take account of the progressive nature of on-line course delivery (e.g. Salmon’s 5-stage model of on-line courses)? How is the role of the tutor (as facilitator and subject expert) represented and realised through the design? How does the course design contribute to the effective management and flow of information – to achieve critical mass in both respects? Do the on-line activities make appropriate use of e-resources and e-facilities, and are opportunities provided for active, joint, problem-solving on-line? Are opportunities provided for joint, knowledge construction on-line, that goes beyond a summary of individual contributions? Are the on-line activities appropriate to the learning objectives at the different stages of the course? Is the structure of the course- in terms of temporal structure, pacing of input material, and range of activity types- appropriate to the needs of the participants and to the objectives of the course? Is the assessment mode appropriate to the course design and objectives? Does the course design allow for flexibility in the design of the on-line environment in response to the requirements of individual cohorts of students?
4.2. Course materials	Does theWeb-site design promote ease of readability, accessibility, navigability with respect to the course materials and shared workspace areas? Is the user interface designed to allow ease of access even with the use of low bandwidth service providers? Good integration of multi-media and text? Do multi-media and graphic elements appropriately support and enhance the conveying of information? Is there effective use of outline frames, and embedded structuring of information through use of hyperlinks, to (a) provide a conceptual framework for the subject material and (b) avoid information density and overload? Is the provision of navigation markers, advance organisers, unit summaries and self-assessment tasks adequate to promote deep-level engagement with the subject content? Is there provision of low-cost access to material banks,which are regularly updated, and from which journal articles etc can be downloaded? Are references to URL’s regularly updated?
4.3. On-line interaction	An analysis of on-line interaction includes examination of the tutor’s role, online activities, student performance, interaction and participation: Tutor role Tutor models standards for accountability and appropriate on-line interaction. Tutor addresses and supports individual student needs, skills, and knowledge. Tutor provides timely, specific, and authentic feedback. Tutor monitors student interaction on-line and makes appropriate interventions to promote participation, effective learning-group management, and conceptual understanding of the subject material. Tutor ensures the pacing of student progression through the course. Tutor provides criteria for grading and weighting for each assignment. Tutor provides private communication to students when appropriate. On-line activities Do the activities make full use of the interactive facilities and the information resources of the on-line environment? Is the learning group provided with a suitably structured on-line environment, group management skills and procedures, and appropriate resources for each activity type undertaken? Are the activities appropriate to the learning task and the educational philosophy of the course and the subject? Student performance, interaction and participation There are now a number of frameworks for content analysis of CMC transcripts available, although few have been developed on the basis of empirical research. These frameworks are designed to provide information on: The rate, frequency and distribution of messaging across the group and across the time-frame of an activity. and/or The direction and density of messaging-thus also identifying: topic threads, length, complexity and organisation of the threads, degree to which contributions of individual participants are taken up for discussion and further development. and/or measures of critical thinking
4.4.Management and support of on-line tutors	Is there a pre-course training/induction programme for tutors who are new to on-line teaching? Are there procedures in place for the support of on-line tutors during the course? (For example: a mentor for each tutor, a buddy (teaching partner), a tutor’s on-line discussion site, feedback from peer review of their on-line teaching.) What procedures are in place to promote a sense of inclusion and cohesion among the course team? (For example: regular,mandatory, face-to-face meetings for all members of the course team, team teaching, openness on the part of course leaders to take suggestions from the teaching team regarding the management and design of the course, a course newsletter, efficient and accurate communication of course information to all course team members.) Do the schedules for on-line tutors acknowledge the intensity of on-line teaching? (e.g. 4 weeks on-line tutoring followed by 4 weeks other duties) Is the distribution of course information the task of a centralised administrative system, in order to limit the workload of the tutor? What opportunities are available for on-line tutors,who are non-IOE staff, to contribute to material development and course design for the courses they are involved with? What opportunities for professional development are available to on-line tutors? What procedures are in place to monitor the quality of the tutoring during the course? For example: random sampling of the transcript, sampling of feedback,mid-course questionnaire (1.3.), virtual teaching visits. Statistics on staff retention.

5. Feedback from Students and Tutors

5.1. Student feedback

Student feedback form needs to be on-line and to be anonymous.A number of additional questions are required over and above those required for face-toface courses. The following suggests the areas that need to be considered and makes some suggestions about possible questions.

Strategic intent of e-learning programme: Identify whether the objectives you had in taking the course on-line are being met, e.g.

I was able to schedule my learning around my professional and personal commitments.
Taking the course electronically saved me time
Taking the course on-line saved me money.
I would not have been able to take this course if I had had to come into college for 10 evening seminars.
I prefer the on-line mode of delivery of the course to a face-to-face mode.

Content: Incorporation of additional questions, e.g.

The resources (on-line materials, printed materials, digitised readings) were easily accessible but also sufficiently challenging
The course pack (including readings, course information and contact details) were sent out in good time.
The participation expectations were made clear at the outset.
The timing of activities was clear.
The sequencing of activities was clear.
What needed to be done in each activity was made clear in the instructions.
I was encouraged to use several resources (e.g.Web links, textbooks, email) to construct knowledge.

Interface design: Were the learning activities facilitated by the technology used, e.g.

I had all the necessary technical skills for taking part in this course
The cost of taking this on-line course (including internet connection time, printing, etc.) was no more than I had expected.
The technology encouraged meaningful interaction with other students
It was easy for me to navigate my way through the resources and activities.
The combination of text and interaction was appropriate and enhanced my learning.

Interactivity: Level of interaction with faculty and other students e.g.

I was able to interact with my tutor effectively.
I was able to interact with other students effectively.
I was able to interact with the administrative staff effectively.

Student support: Adequacy of technological support and appropriateness of use of technologies for the course, and the role of the tutor, e.g.

The tutor facilitated collaborative working
The tutor was accessible (by email, phone and in person)
The tutor used questions effectively to facilitate my learning
Technical support was readily accessible and timely
I was clear about the support available and where to access it
The tutor gave timely and informative feedback on the outcomes of the activities.
The library contributed positively to my learning.

5.2. Tutor feedback

Feedback from tutors (and other staff working on the course) about their experience of the course. This is particularly important where a range of tutors work on the course, not just full time IoE academic staff. Particular features that should be included:

Staff satisfaction with the course
Workload
Professional development activities carried out in the area of on-line learning
Staff satisfaction with the management of the course team
Recommendations for changes to the course or course team management procedures
Self–evaluation of tutoring on the course to be undertaken as part of the peer review process.

References

Paulsen M.F. (1995) The on-line report on pedagogical techniques for computer-mediated communication. Available at http://nettskolen.nki.no/forskning/19/cmcped.html (last accessed 24 April 2008)
Steeples C. and Jones C. (eds.) (2002) Networked learning: Perspectives and issues. London. Springer

Browser not supported