Appendix: Evaluation of Online Courses

1.4. Management and support of on-line tutors

• Does the administrative and technical infrastructure provides access to all prospective and enrolled learners?
• Reliability of technical infrastructure – hardware, software, staffing and technical assistance.
• Accounts for using networks,web sites, and virtual classroom software are set up quickly.
• Timely web content updating.
• Delivery of learning materials within # days of registration.
• Reliable, timely information of any changes.
• Response to email messages within # hours.
• Feedback to drafts within # days.
• Grade reporting within # days.
• Final results available within # days.
• Comparisons with similar face-to-face courses on attainment, retention and completion rates, reasons for withdrawal and costs.

2.2. Evaluation Questionnaire data

• What is the justification for the course being delivered on-line?
• Is the design adapted to take account of the progressive nature of on-line course delivery (e.g. Salmon’s 5-stage model of on-line courses)?
• How is the role of the tutor (as facilitator and subject expert) represented and realised through the design?
• How does the course design contribute to the effective management and flow of information – to achieve critical mass in both respects?
• Do the on-line activities make appropriate use of e-resources and e-facilities, and are opportunities provided for active, joint, problem-solving on-line?
• Are opportunities provided for joint, knowledge construction on-line, that goes beyond a summary of individual contributions?
• Are the on-line activities appropriate to the learning objectives at the different stages of the course?
• Is the structure of the course- in terms of temporal structure, pacing of input material, and range of activity types- appropriate to the needs of the participants and to the objectives of the course?
• Is the assessment mode appropriate to the course design and objectives?
• Does the course design allow for flexibility in the design of the on-line environment in response to the requirements of individual cohorts of students?

• Does theWeb-site design promote ease of readability, accessibility, navigability with respect to the course materials and shared workspace areas?
• Is the user interface designed to allow ease of access even with the use of low bandwidth service providers?
• Good integration of multi-media and text?
• Do multi-media and graphic elements appropriately support and enhance the conveying of information?
• Is there effective use of outline frames, and embedded structuring of information through use of hyperlinks, to (a) provide a conceptual framework
• for the subject material and (b) avoid information density and overload?
• Is the provision of navigation markers, advance organisers, unit summaries and self-assessment tasks adequate to promote deep-level engagement with the subject content?
• Is there provision of low-cost access to material banks,which are regularly updated, and from which journal articles etc can be downloaded?
• Are references to URL’s regularly updated?

An analysis of on-line interaction includes examination of the tutor’s role, online activities, student performance, interaction and participation:

Tutor role

• Tutor models standards for accountability and appropriate on-line interaction.
• Tutor addresses and supports individual student needs, skills, and knowledge.
• Tutor provides timely, specific, and authentic feedback.
• Tutor monitors student interaction on-line and makes appropriate interventions to promote participation, effective learning-group management, and conceptual understanding of the subject material.
• Tutor ensures the pacing of student progression through the course.
• Tutor provides criteria for grading and weighting for each assignment.
• Tutor provides private communication to students when appropriate.

On-line activities

• Do the activities make full use of the interactive facilities and the information resources of the on-line environment?
• Is the learning group provided with a suitably structured on-line environment, group management skills and procedures, and appropriate resources for each activity type undertaken?
• Are the activities appropriate to the learning task and the educational philosophy of the course and the subject?

Student performance, interaction and participation

There are now a number of frameworks for content analysis of CMC transcripts available, although few have been developed on the basis of empirical

research. These frameworks are designed to provide information on:

• The rate, frequency and distribution of messaging across the group and across the time-frame of an activity.

and/or

• The direction and density of messaging-thus also identifying: topic threads, length, complexity and organisation of the threads, degree to which contributions of individual participants are taken up for discussion and further development.

and/or

• measures of critical thinking

4.4.Management and support of on-line tutors

• Is there a pre-course training/induction programme for tutors who are new to on-line teaching?
• Are there procedures in place for the support of on-line tutors during the course? (For example: a mentor for each tutor, a buddy (teaching partner), a tutor’s on-line discussion site, feedback from peer review of their on-line teaching.)
• What procedures are in place to promote a sense of inclusion and cohesion among the course team? (For example: regular,mandatory, face-to-face meetings for all members of the course team, team teaching, openness on the part of course leaders to take suggestions from the teaching team regarding the management and design of the course, a course newsletter, efficient and accurate communication of course information to all course team members.)
• Do the schedules for on-line tutors acknowledge the intensity of on-line teaching? (e.g. 4 weeks on-line tutoring followed by 4 weeks other duties)
• Is the distribution of course information the task of a centralised administrative system, in order to limit the workload of the tutor?
• What opportunities are available for on-line tutors,who are non-IOE staff, to contribute to material development and course design for the courses they are involved with?
• What opportunities for professional development are available to on-line tutors?
• What procedures are in place to monitor the quality of the tutoring during the course? For example: random sampling of the transcript, sampling of feedback,mid-course questionnaire (1.3.), virtual teaching visits.
• Statistics on staff retention.

Student feedback form needs to be on-line and to be anonymous.A number of additional questions are required over and above those required for face-toface courses. The following suggests the areas that need to be considered and makes some suggestions about possible questions.

Strategic intent of e-learning programme: Identify whether the objectives you had in taking the course on-line are being met, e.g.

• I was able to schedule my learning around my professional and personal commitments.
• Taking the course electronically saved me time
• Taking the course on-line saved me money.
• I would not have been able to take this course if I had had to come into college for 10 evening seminars.
• I prefer the on-line mode of delivery of the course to a face-to-face mode.

Content: Incorporation of additional questions, e.g.

• The resources (on-line materials, printed materials, digitised readings) were easily accessible but also sufficiently challenging
• The course pack (including readings, course information and contact details) were sent out in good time.
• The participation expectations were made clear at the outset.
• The timing of activities was clear.
• The sequencing of activities was clear.
• What needed to be done in each activity was made clear in the instructions.
• I was encouraged to use several resources (e.g.Web links, textbooks, email) to construct knowledge.

Interface design: Were the learning activities facilitated by the technology used, e.g.

• I had all the necessary technical skills for taking part in this course
• The cost of taking this on-line course (including internet connection time, printing, etc.) was no more than I had expected.
• The technology encouraged meaningful interaction with other students
• It was easy for me to navigate my way through the resources and activities.
• The combination of text and interaction was appropriate and enhanced my learning.

Interactivity: Level of interaction with faculty and other students e.g.

• I was able to interact with my tutor effectively.
• I was able to interact with other students effectively.
• I was able to interact with the administrative staff effectively.

Student support: Adequacy of technological support and appropriateness of use of technologies for the course, and the role of the tutor, e.g.

• The tutor facilitated collaborative working
• The tutor was accessible (by email, phone and in person)
• The tutor used questions effectively to facilitate my learning
• Technical support was readily accessible and timely
• I was clear about the support available and where to access it
• The tutor gave timely and informative feedback on the outcomes of the activities.
• The library contributed positively to my learning.

Feedback from tutors (and other staff working on the course) about their experience of the course. This is particularly important where a range of tutors work on the course, not just full time IoE academic staff. Particular features that should be included:

• Staff satisfaction with the course
• Workload
• Professional development activities carried out in the area of on-line learning
• Staff satisfaction with the management of the course team
• Recommendations for changes to the course or course team management procedures
• Self–evaluation of tutoring on the course to be undertaken as part of the peer review process.