Appendix: UCL Quality Framework

Course materials will support the overall programme aims, provide clear learning objectives and promote active learning.

1.1 Course structure. All study modules will contain the following, presented in consistent house style:

• A clear statement of aims;
• Explicit and measurable learning objectives;
• Supporting materials for self study (such as reflection exercises and practical activities);
• Exercises for group interaction;
• An estimation of learner workload, expressed in terms of expected study hours per week.

1.2 Reading materials. These will be

• Intellectually rigorous (i.e. methodologically robust, logically consistent, and can be placed in a coherent theoretical framework);

• Up to date (i.e. not superseded by more recent work), accurate and relevant;
• Clearly written, professionally presented and accessible;
• Transferable to an international context;
• Classified by level of difficulty (e.g. introductfory, standard and supplementary);
• Copyright cleared.

1.3 Multimedia materials. These will be

• Appropriate to the goals and the learning outcomes of the course;
• Technically reliable;
• Compatible with minimum user specifications;
• Flexible – i.e. can be used in different ways by students with different learning styles;
• As far as possible, upgradable at marginal cost.

1.4 Accessibility. At least 95 percent of required course materials will be available either on line via the programme website or as a paper or CD included within the MSc fee.

Course materials will be written according to standard templates. The tutors’ manual will contain guidelines for writing open learning resources and agreed rules for house style.

All course content will be peer reviewed against the standards listed left, and modified accordingly, at four separate stages: scoping (draft content for module); writing (detailed content of each study unit); interim review (annual updating of content before each new intake); and major review (5-yearly overview of scope and content). A full draft of each module will be sent to the External Assessor for approval before being made available to students.

Student generated material from one year (exemplary essays, collaborative projects) will be peer reviewed by the standard mechanisms before being added to content for future years.

Peer review meetings will be attended by a multidisciplinary team including one author, a senior academic with an overview of the programme, an educationalist, and a technical adviser.

The tutor responsible for writing/updating the module will liaise with administrative and technical staff to ensure materials are available on-line in agreed house style or have been sent to students before the module begins.

Suggestions for updating and revising of material by both students and staff will be systematically collected in a revision file, reviewed and fed into the peer review process.

Examples of study resources, examined for evidence of

• Aims and objectives;
• Exercises and activities;
• Consistency of style and presentation;

Examples of reading lists, checked against on-line reading materials and study packs.

Minutes and reports from the internal peer review process:

• Peer review meetings;
• Tutors’ annual reports on
• module;
• Quality Monitoring Group;
• Departmental Teaching Committee;
• Faculty Graduate Teaching Committee;
• College Academic Committee.

Minutes and reports from the external peer review process:

• External Assessors;
• Visiting Examiners.

Student feedback

• Summary of evaluation questionnaires;
• Transcript of Feedback Forum postings;
• Correspondence and complaints.

Quality failure should be detected via

• Internal peer review meetings;
• Tutors’ annual
• reports;
• External Assessors’ reports;
• Visiting Examiners’ reports;
• Student evaluations;
• Critical incidents.

Formal on-line discussions on key topics (virtual seminars) will support the overall programme goals through high quality, focussed academic discourse, collaboration and lateral support.

2.1 Framework. Virtual seminars will be:

• Regular: one per study unit;
• Time limited: held between specified dates;
• Asynchronous: not held in real time ‘chat’;
• Compulsory: participation will be a course requirement;
• Small: 15 students or fewer per seminar group;
• Archived: stored in an accessible, indexed and searchable on-line repository.

2.2 Format. The format of the virtual seminar will be

• Structured: will follow an explicit and standard format based around specific group tasks;
• Focussed: will address a topic identified in advance and based on the unit learning objectives;
• Threaded: comments on particular themes will be linked via declarative subject titles.

2.3 Process. The process of the virtual seminar will be

• Moderated: introduced, co-ordinated and summarised by a named student;
• Facilitated: overseen and supported by the module tutor, who will monitor content and depth of discussion;
• Contained: follow explicit and agreed ground rules for on-line behaviour in groups.

2.4 Content. The discourse of the virtual seminar will be:

• Inclusive: will seek and address a range of different inputs and diversity of perspectives;
• Challenging: will promote and support critical reflection on course materials and on-line contributions;
• Constructive: will promote and support the creation and sharing of new knowledge;
• Collaborative: will promote and support the sharing of materials and production of joint outputs by students;
• Grounded: will draw on students’ own experiences and backgrounds as well as course materials.

The module tutor will

• Create a discussion forum for each virtual seminar by the specified date;
• Ensure all students have access to the appropriate fora;
• Close the forum on the specified date and archive the contents.

Course materials will list the following details for the virtual seminar:

• Start and end date;
• Aims;
• Preparatory work;
• Outline structure (e.g. whether students should work in pairs or as a whole group);
• A link to the overall learning objectives of the study unit and assignment.

The induction module for students will include training in ‘threading’ messages, ground rules for on line group work, and e-moderating skills. Module tutors will reinforce and continue this training in subsequent modules.

Academic staff will review the process of virtual seminars at regular Quality Monitoring Group meetings to identify and share learning points.

See also Section 3: Tutor Performance and Development.

Transcripts of virtual seminars, examined for evidence of

• Timing, group size, and input from all students;
• Consistent and logical structure (e.g. threading);
• Focussed discussion  related to unit objectives;
• Moderation by student;
• Facilitation by tutor;
• Adherence to ground rules and prompt and appropriate management of individuals who deviate (e.g. on-line ‘bullying’);
• Interdisciplinary and interprofessional sharing of knowledge and experience;
• Interactive discourse leading to knowledge construction and higher order learning (such as analysis, synthesis, evaluation);
• Exploration beyond the course;
• Archived transcript plus student-generated summary.

Student feedback

• Summary of evaluation questionnaires;
• Transcript of Feedback Forum postings;
• Correspondence and complaints.

Quality failure should be detected via

• Internal peer review meetings;
• Tutors’ annual reports;
• External Assessors’ reports;
• Visiting Examiners’ reports;
• Student evaluations;
• Critical incidents.

Module tutors will be appropriately qualified, trained and supported to deliver high quality learner support in the on-line environment.

3.1 Tutor competence. All tutors must be judged suitably qualified and experienced in all relevant content (subject) areas through formal approval by the Programme Director, Department Teaching Committee, and Faculty Tutor

3.2 Tutor induction. All tutors must demonstrate satisfactory performance in the Induction Course for Tutors, including:

• Navigation of course software;
• Use of email and bulletin board;
• Use of html editor to prepare web pages;
• Liaison with administrative and technical staff.

3.3 Tutor performance. All tutors must demonstrate the following in the on-line environment, to the satisfaction of the Programme Director, Faculty Tutor, and Visiting Examiners:

• Technical competence (ease of navigation, use of full functionality of interactive software);
• Content expertise (accuracy and relevance of information and support provided);
• Effective communication (e.g. questioning, challenging, seeking clarification);
• Group facilitation (e.g. establishing and implementing ground rules, maintaining a focus on the group task, promoting collaborative working, managing dysfunctional behaviour).

3.4 Tutor behaviour on line. All tutors must behave professionally in the on-line environment, and be receptive to feedback from students and staff.

3.5 Tutor development. All tutors must actively embrace the principle of continuing professional development. They will be required to participate in an annual appraisal of performance and review of training needs.

Job descriptions for tutors will contain explicit standards for experience and qualifications in both subject content and technical skills. A full curriculum vitae and/or personal learning log will be reviewed by the Programme Director before a tutor is appointed to a new subject area.

The Induction Course for Tutors will define and address explicit performance standards in technical competence.

Tutors will receive training and support to develop their online tutoring skills. Novice tutors will be supported by experienced tutors through a lead and support tutor system during a probationary period. Achieving and maintaining competence in on-line tutoring is a prerequisite for a definitive contract.

Codes for behaviour in on-line groups will be available in the student manual and the tutors’ manual.

The Programme Director will undertake and document an annual appraisal of all academic staff. They will be encouraged to work towards specific goals such as:

• Short courses to meet identified needs;
• Membership of ILT or other professional bodies;
• Higher degrees;
• Presentations at conferences;
• Publications in peer reviewed academic journals.

Human resource file, with Job descriptions for tutor roles;

• Curricula vitae of tutors;
• Certificates of satisfactory completion of Induction Course for Tutors.

Transcripts of virtual seminars, examined for evidence of

• Technical competence;
• Content expertise;
• Effective communication;
• Group facilitation.

Student feedback, especially via explicit question about tutor input in on-line evaluation questionnaire for each module.

Academic staff personal learning log or equivalent

Quality failure of tutors should be detected via

• Failure to pass Induction Course;
• Internal peer review of virtual seminar;
• Visiting Examiners’ reports;
• Student feedback, especially via on-line evaluation questionnaires;
• Critical incidents.

NB Student performance is not a very sensitive indicator of overall tutor competence. As a general rule, poor performance by a majority of students on a course should raise questions about course design or tutor competence, whereas poor performance by a minority of students is usually attributable to other factors.

Assessment will be

Valid: will give an accurate and complete reflection of the student’s ability and performance;

Reliable: will give consistent and coherent results between markers;

Fair: will guard against cheating and take a broad and sensitive view of what is ‘correct’ or ‘incorrect’;

Appropriate to the educational objectives of the overall programme and to the particular objective being assessed;

Efficient: will use tutor time appropriately and avoid double handling;

Timely: will provide students with feedback at an appropriate stage in their learning;

Formative as well as summative.

4.1 Setting assignments. All course assignments will be reviewed as part of the peer review process to ensure that

• The form and content is appropriate to the workload and matches the unit objectives;
• A structured schedule stating how marks will be allocated is included;
• An agreed indicative answer is placed on file for markers, giving examples of the standard required for pass and distinction.

4.2 Submission of assignments. The facility for electronic submission of assignments will be

• Simple, clear, and user friendly;
• Technically secure and reliable;
• Anonymised.

4.3 Marking and feedback.

• 90 percent of assignments will be returned to students within 21 days; All students will receive written formative feedback that is clear, constructive, relevant, thorough, and inoffensive.

4.4 Examination procedure.

• Examination papers will follow the general standards for assignments given in 4.1;
• The External Programme will notify students of examination dates.
• A mock examination with an indicative marking schedule will be posted at least one month before the date of the examination.

Assignments and examination questions will be peer reviewed internally and externally as part of the course content (see Section 1). Papers will be marked on the basis of agreed marking schedules and indicative answers.

The MSc administrator will maintain the electronic assignment submission procedure and provide prompt help to students who have difficulty using it.

Assignments will be double marked and a formal arbitration procedure used if there is a discrepancy of  more than 10% between markers (see full regulations for details). Tutors will use an agreed template for providing feedback to students. Novice markers will shadow a senior until they demonstrate the ability to mark consistently and provide appropriate feedback.

The module tutor will ensure that a mock examination is developed, peer reviewed, and posted in good time.

Assignments as supplied in course materials, and examination papers, including marking schedules, indicative answers and feedback templates.

Visiting Examiner reports, with particular reference to scrutiny of assignments and examination papers against unit objectives.

TMA returns spreadsheet, with particular reference to timing of submissions and consistency between markers.

Examples of tutor feedback forms to individual students.

Copies of administrative notices about examinations.

Student feedback

• Summary of evaluation questionnaires;
• Transcript of Feedback Forum postings;
• Correspondence and complaints.

Quality failure should be detected via

• Internal peer review meetings;
• Tutors’ annual reports;
• TMA returns statistics;
• Co-marker evaluation of feedback sheet to students;
• External Assessors’ reports;
• Visiting Examiners’ reports;
• Student feedback and evaluations;
• Critical incidents.

Our programme will be supported by accessible, accurate, and up to date documentation. Support and advice will be tailored to the needs of individual students. There will be an effective system of student representation

5.1 Documentation. Students and prospective students will have timely access to details of the programme, which will indicate clearly what is expected of learners in each module and give the expected time commitment and credit value. This documentation will support students in making informed choices to meet their personal learning needs.

5.2 Attention to individual support needs

• Before starting the programme, all students will be required to assess and declare their motivation and commitment to it;
• All students will have access to a personal progress log giving a summary of modules taken, marks awarded and credits gained;
• All students will be allocated a named member of academic staff whom they may approach for advice on module choices and in times of crisis;
• Academic staff will regularly review the needs of students identified by the module tutor as academically weaker or otherwise having particular needs.

5.3 Library support. Adequate journals and other resources will be available on-line for students to explore beyond the course materials. Students will be able to obtain assistance to help them seek and use electronically stored information successfully.

5.4 Social support. Students will have access to an online social forum for informal discussion and support.

5.5 Student views will be actively sought via a range of approaches including on-line feedback questionnaires for each module, an anonymous feedback forum, and an independent, structured complaints procedure. We will respond promptly and sensitively to their expressed concerns.

A named member of academic staff will check and update the on-line student manual and publicly available programme details against the standards listed left at least annually. A named librarian or informaticist will check and update library resources at least annually.

The Programme Director will ensure that all students are allocated a named adviser from the academic staff, who will respond promptly to enquiries about study options and overall progress. In the event of a serious student crisis, their personal adviser will submit a confidential report to the Programme Director.

A member of the course team will ensure that all students have access to the on-line student manual, to an accurate personal progress log, to the online library, and to the relevant social forum.

All students will be asked to complete an online feedback questionnaire for every study module. The MSc Administrator will collate responses for the module tutor, who will review them against the standards listed left before preparing an annual report on the module.

All students will be informed of the independent complaints procedure.

The Quality Monitoring Group of academic staff will meet once per term and include a regular review of student feedback and of named students with particular needs.

Student manual, website and marketing materials, examined for evidence of clarity, accuracy and completeness of information on programme content and format.

On-line summary of progress on an individual student basis, generally accessed via the on-line ‘Check your progress’ tool.

Administrative data, such as: List of all students and personal advisers;

Membership lists of modules and social for a. Student feedback

• Summary of evaluation questionnaires;
• Transcript of Feedback Forum postings;
• Correspondence and complaints.

Anonymised confidential reports on student crises.

Minutes of Quality Monitoring Group meetings.

Quality failure should be detected via Tutors’ annual reports

• External Assessors’ reports;
• Visiting Examiners’ reports;
• Student feedback and evaluations;
• Feedback on publicly available information from prospective students;
• Critical incidents, especially confidential reports on student crises

Administrative and technical systems will support the programme goals through high quality service delivery, multidisciplinary teamwork, effective communication, and robust technological infrastructure. Administrative and technical staff will have clear roles and responsibilities and be adequately supported in their work.

6.1 Customer care. Students and prospective students will receive prompt, courteous and accurate responses from the UCL course team to enquiries and correspondence.

6.2 Multidisciplinarity. Administrative and technical input will be an integral aspect of the presentation and marketing of the programme and of the development and delivery of course materials. Staff offering this support will work as part of the multidisciplinary course team.

6.3 Development of support staff. All administrative and technical staff will be suitably qualified for the work required of them and have

• An accurate and up to date job description
• A structured Induction Course for Support Staff on appointment
• A confidential annual appraisal
• Training and support appropriate to their role

6.4 Technology strategy. A documented strategy will be in place to define and monitor technical standards. This will be based on established industry quality standards and include issues of system design, hardware and software specification, technical support for staff and students, and confidentiality and data protection.

The Programme Director will oversee the appointment, induction, supervision and appraisal of administrative and technical staff against the standards listed left.

The Technical Director will prepare and maintain a comprehensive technology strategy against the standards listed left.

The Technical Director will will deliver full support for staff involved in on-line course development. Academic staff will acknowledge and incorporate the input of administrative and technical staff when developing course materials and systems.

The Programme Director will identify a dedicated training budget and allocate funds judiciously for both in-house training and external courses for all staff.

All staff will be required to keep a personal training log documenting professional goals, courses attended, and new training needs identified.

Human resource file, with

• Job descriptions foradministrative and technical roles;
• Curricula vitae of these staff;
• Certificates of satisfactory completion of Induction Course for Support Staff.

Student feedback, especially via explicit questions about administrative and technical support in on-line evaluation questionnaires.

Copies of particular items of correspondence or communication.

Staff personal learning log or equivalent.

Quality failure should be detected via

• Annual appraisal returns;
• External Assessors’ reports;
• Visiting Examiners’ reports;
• Student feedback and evaluations;
• Critical incidents.