Guidance on anonymous marking in Turnitin - including second marking, moderating and external examining

On this page:

Introduction

The guidance below takes you through several different anonymous marking scenarios with Turnitin afforded by UCL's Marking Policy.  Elsewhere Digi Ed provides instructions about the particulars of setting up and operating a Turnitin Assignment.

| Top of page |

Terms

• Anonymous marking: see below for a definition and explanation.
• Bulk-download: downloading the contents of a submission inbox for marking outside of the Turnitin assignment.
• Candidate number: a unique identifier (four letters and one digit) which students receive annually.
• Combinations of blind/open, full/sampled second marking: refer to the UCL Marking Policy.
• External examining: refer to the UCL Marking Policy.
• GradeMark: Turnitin's own marking environment.
• Intermediary: somebody without marking responsibilities, perhaps in an administrative or technical role.
• Markings: all notes, specific and general comments, and numeric marks made by markers.
• Moderating: refer to the UCL Marking Policy.
• Moodle Assignment: Moodle offers a separate online assessment environment. Its marking and feedback possibilities currently differ from Turnitin's. Only Turnitin offers a Similarity Report, and Turnitin's suite of online annotation tools is slightly more developed.
• Paper ID: a numeric identifier unique to each paper, generated by Turnitin, but currently irrelevant outside of the Turnitin assignment.
• Post Date: point at which marks and feedback are released to students (at this point student anonymity is lifted).
• Proxy identifier: see below for a definition and explanation.
• Student Record Number (SRN): this appears on the front of every student's ID card.
• Submission filename: the name of the file(s) submitted; this becomes important in cases where students' work is downloaded for marking outside Turnitin.
• Submission title: the title students type into Turnitin when submitting their work. This displays in the Turnitin inbox and can be used to allocate submissions to subject specialist markers.
• TurnitinUK.com: allows access (where there are existing permissions) to a Turnitin submission inbox independently of Moodle; affords downloading of Grademarked work, and original unmarked work - though only after the Post Date. Please note Digital Education is unable to offer technical support for your activities on turnitinuk.com,  however, Turnitin does offer support.
• Window of opportunity: a phrase used in this guidance to refer to a period just after Grademarking is complete (if used) and just before the Post Date de-anonymises students. During this window the still-anonymous, Grademarked submissions, complete with markings, can be downloaded from turnitinUK.com for distribution to other markers outside of Turnitin after the Post Date. The window of opportunity is negotiated by course leaders, markers and intermediaries.

| Top of page |

Definition of anonymous marking for online assessment

After considering a range of possible definitions, the Board of Examiners has settled on the following definition of anonymous marking on Moodle, Turnitin or other online environment: student names are absent from their submissions at the point of marking.

Instead of their name, students are instructed to use a unique proxy identifier.

There is no expectation that student work remains permanently anonymous to a marker after they have finished marking.

| Top of page |

What proxy identifier should be used instead of the student name?

Why is a proxy identifier needed at all? Because staff, sometimes working offline with downloaded files, need a robust, convenient way to refer to any given student, reconcile their markings with those of other markers, and update the student's record in the absence of a name.

Until a technical solution becomes available, students should be instructed to include this proxy identifier in both the title and file name of their submission.

The recommended proxy identifier is the Candidate Number. The Student Record Number (SRN) may be used instead, if preferred.

The proxy identifier isn't generated automatically, so students need be instructed to title their submission and file name so as to include it.

| Top of page |

Open second marking

An overview of marking in Turnitin can be found in UCL miniguide M09b.

Give or take some variation in settings, the inbox will look something like this (click for full size):

Full open second marking

Since second markers need to view the markings of first markers, the order of marking is important.

First markers access the submission inbox and can carry out marking using Turnitin's Grademark (or according to departmental guidelines). At an agreed date and time (before the Post Date), the second marker accesses the same inbox. Depending on local policy, second markers can use Grademark to add their markings alongside the first marker's. Turnitin will not currently distinguish the two sets of comments, so the second marker can initial or otherwise identify theirs.

One consideration here is whether, on the Post Date, students are expected to see the second marker's comments as well as the first marker's. If only the first marker's, then the second marker accesses Turnitin to view the submission and first marking, but records all their own markings outside Turnitin on a marksheet provided (this enables record keeping, which the alternative practice of deleting second marker comments in advance of the Post Date does not). Turnitin plans to accommodate multiple markers with 'multiple grading spaces' though they have not been specified in detail.

Another consideration is whether there is more than a single first and single second marker. Where a group of markers is involved, there is a need to divide the work between them. The anonymised inbox allows sorting by submission title, so if students are instructed to title their submission with their Candidate Number, it should be possible to sort by these (inevitably a few students will make errors, but Moodle provides ways to reconcile work with a mistyped Candidate Number). Students should also include their Candidate Number in the name of their uploaded file.

Where individual markers are allocated marking based on subject specialism, students may be instructed to include a very short identifying phrase – perhaps the question number - in the submission filename and submission title, so that markers can view the contents of the inbox and identify the work they need to mark.

| Top of page |

Sampled open second marking

Where Grademark has been used and second marking precedes the Post Date, the submission inbox allows the second marker to apply the sampling criteria by applying the sort function to the numeric mark column so as to easily find firsts, fails, upper borderlines, etc. In addition their attention could be directed to any particular submissions with reference to the PaperID.

Where Grademark has been used and second marking needs to happen after the Post Date, then an intermediary needs to access the inbox via TurnitinUK.com and sort it to identify a sample, which is then downloaded and securely passed outside Turnitin to the second marker. If this can be arranged after marking is complete but before the Post Date de-anonymises student work, this both potentially saves a great deal of time for whomever carries out this task, and also enables a marker to do it (because the work is still anonymous). However, if the Post Date has passed then an intermediary needs to do this, manually re-anonymising all submissions by removing or redacting the student name Turnitin appends after the Post Date.

Please note that Digital Education is unable to offer technical support for your turnitinuk.com activity - Turnitin does offer support though.

| Top of page |

Open second marking after the Post Date

In Turnitin, the Post Date permanently removes student anonymity.

As for sampled open second marking, where Grademark has been used, it saves time to download the still-anonymous inbox after marking is complete but before the Post Date de-anonymises student work. However, if the Post Date has passed, an intermediary needs to carry out the task, re-anonymising all submissions manually by removing or redacting the student name Turnitin appends after the Post Date.

Where marking has taken place outside Turnitin, the process is more convoluted. If this happens before the Post Date then the original files may be downloaded from Moodle.  Before securely passing to the second marker, it is important to check that the Candidate Number is present in the filename. If after the Post Date then an intermediary downloads the original files, re-anonymises these by removing the name Turnitin appends to the submission filename, ensures the Candidate Number is present in the filename, and securely passes these to the second marker along with the first marker's markings.

Please note that Digital Education is unable to offer technical support for your turnitinuk.com activity - Turnitin do offer their own support though.

| Top of page |

Blind second marking

Full blind second marking

Since the order of marking is irrelevant here, markers can take advantage of the online environment by both accessing the work they need to mark at the same time; the first need not wait for the second and both can use the entire marking period. However, they cannot use Grademark, since their markings would be visible to each other i.e. they need to record marks and feedback outside Turnitin.

It is now possible to bulk-download the Turnitin inbox of anonymised work, so one marker can give feedback via Turnitin, while the other looks at the work outside Turnitin.

| Top of page |

Sampled blind second marking

The guidance is similar to that for 'Full blind second marking', except that it is on selected submissions. So markers would either need to be instructed about how to sort the inbox to take the sample, or an intermediary would need to download and re-anonymise the original submissions and send these securely to the blind second marker outside Turnitin, along with any marksheets required.

| Top of page |

Blind second marking after the Post Date

See the section on 'Open second marking after the Post Date' – the difference would be that the intermediary would pass anonymised original submissions (not the marked ones) to the second marker with a blank marksheet (not a completed one).

| Top of page |

Moderation

The moderator is tasked with checking the assignation of marks on a full set of submissions. If Grademark has been used and moderation happens before the Post Date, moderators can view the contents of the submission inbox on Turnitin, including the markings. If markings have been made outside Turnitin, then these need to be sent to the moderator either by the markers or via an intermediary.

| Top of page |

Sampled moderation

External examiners sometimes undertake sampled moderation. Unless a window of opportunity could be made before the Post Date lifts anonymity, an intermediary would need to pass the examiner(s) a downloaded, re-anonymised sample of marked work as set out in 'Open second marking after the Post Date' or 'Blind second marking after the Post Date', above.

| Top of page |

Moderation after the Post Date

The guidance here is the same as for 'Open second marking after the Post Date'.

| Top of page |

External examining

Since external examiners are routinely provided with an UCL User ID, they could be given a role in a Moodle course area which allows them access to its respective submission inboxes.

| Top of page |

Sampled external examining

The guidance here is the same as for 'Open second marking after the Post Date'. The submission inbox allows sorting by various columns including numeric mark, allowing examiners to easily find firsts, fails, upper borderlines, and also allows submissions to be randomly selected. In addition their attention could be directed to particular submissions with reference to the PaperID.

| Top of page |

External examining after the Post Date

The guidance here is the same as for 'Open second marking after the Post Date'.

| Top of page |

Questions and Answers

Isn't this too complicated to offer any advantages over paper processes?

UCL Digital Education understand if staff feel these processes with Turnitin are overly complicated, and we encourage you to consider the relative merits of the alternative  Moodle Assignment activity.

We recommend feeding back your experiences with Turnitin.  Meanwhile however, many departments are eager to surmount the complications in order to exploit the benefits of online assessment, namely: convenient submission for students without queuing or traveling; relief for administrative staff who used to receive the paper submissions; quick, easy transfer between markers and other staff; relative portability for markers; access for markers from anywhere with a connection; no marking days lost to postage; legibility and potential contextualisation of feedback; future reference to feedback; and freeing up of expensive central London storage space.  Digital Education continue to pursue improvements to the platforms we offer.

What do markers need to know about avoiding allegations of intentional, rather than unconscious, bias?

Along with UCL Equality and Diversity initiatives and other governance structures, this guidance is intended to help staff avoid allegations of unconscious bias. In the case of an allegation of intentional bias, the student would need to provide supporting evidence according to the Student Complaints Procedure. Where there are sufficient grounds to warrant further investigation, Moodle may be able to provide some records which inform the case, since it keeps track of page views for all its users. The implication here is that if, during the time they are marking, markers avoid consulting the Participation Report, or Logs for the assignment being marked or previous de-anonymised assignments, they will not have been able (via Moodle) to associate names with work.

Can't staff who are supposed to be anonymously marking after the Post Date simply look up students' names on turnitinuk.com?
This is technically possible where a marker has a Tutor role on the Moodle course area. However, it would require a purposeful decision, on the part of a marker, to flout the anonymous marking policy. Similar opportunities for malpractice exist in many areas of academia; as a consequence of the high degree of workplace autonomy academia requires. To design an entire digital assessment approach round an assumption of deliberate malpractice on the part of academic markers would be extreme and bring many, and negative, side effects.

Is it possible to re-anonymise student submissions after the Post Date?

The short answer is no, even though Moodle's Turnitin integration has unintentionally allowed this in the past. As such if this capability reappears, it is vulnerable to being removed by Turnitin without notice, and cannot be relied upon. Moodle Assignment prevents re-anonymisation but it allows download of a currently-anonymous submission inbox, which permits anonymous copies to be retained.

Can staff bulk-download an entire inbox of anonymous submissions directly from turnitinuk.com?

Yes, this became possible in 2016.

Do markers need to be excluded from a Moodle area containing an assignment after that assignment's Post Date?

No, there is no expectation that student work remains anonymous after marking because, without reference to work and markings, markers are likely to experience more obstacles to discussing the feedback with individual students, and may be hampered in considering students' development over time. According to the definition of anonymous marking in use here, only markers who will be marking after the Post Date would need to be temporarily excluded from a Moodle area containing an assignment (because students names will have been revealed by the time those markers access the submission inbox). Otherwise markers may remain in their usual roles in those Moodle areas.

Doesn't reuse of the Candidate Number compromise anonymous marking? Doesn't the appearance of the Candidate Number alongside the student name, after the Post Date of the first assignment, compromise anonymous marking? 

Not according to the definition of anonymous marking in use here - as long as the student name does not appear on any piece of anonymous work yet to be marked.

Technical limitations oblige us to balance competing benefits of anonymous marking and assessment feedback. A more defensive view of anonymity would seek a technical solution which treated assessors as if they were deliberately intending discrimination, rather than simply subject to unconscious bias. Since there is a grievance process for students who wish to allege intentional discrimination, it is considered sufficient here to take measures to disrupt unconscious bias. Since the Candidate Number changes each year, it serves the purpose of disrupting unconscious bias. There is a recognition on the part of the Board of Examiners that the Candidate Number may be used on multiple occasions over the year since (because of technicalities of updating each student's record) its repeated use may be the only feasible means of communicating assessment feedback to students in circumstances of anonymous marking.

Doesn't Tutor access to the Participation Report compromise anonymous marking?

Not according to the definition of anonymous marking in use here. By default, Moodle course areas allow staff in a Tutor role access to a Participation Report which exists to allow teaching staff to make pastoral interventions, send selective reminders to students, and keep records. A defensive view of anonymity might seek to exclude these since they could allow staff to anecdotally associate student names with late work. It is possible for staff in a Moodle course Administrator role to remove access to these reports from Tutors – however, this is a blunt and potentially resource-intensive measure which would incidentally prevent teaching staff from viewing reports on any other activity, or being able to send out selective reminders. Reports being an important aspect of student feedback to staff, the benefits for anonymity need to be weighed against the educationally undesirable side effects.

Wouldn't it be better to gather all assignments into a single Moodle area and exclude markers after the Post Date?

Digital Education advise against this for reasons set out at M09b7 - Staff Turnitin FAQs.

Wouldn't duplicate submission points help with blind marking?

We considered the duplicate submission points, one for each marker. However, this increased workload and introduced complexity further along the process. For example, there is a need to set the assignments up to avoid 100% match in the Similarity Report. Arrangements would need to be made for external examiners to view both while retaining student anonymity. The process also makes further demands on students with further opportunities for error. None of these is very difficult in itself, but taken together they could seem considerable. Consequently, we only suggest this as an option for blind second marking, and in any case recommend considering Moodle Assignment as an alternative.

Should we set the Post Date a year in advance and amend it when we are ready, to ensure that names are not revealed before all students' work has been marked?

Digital Education recommend that the Post Date is the same as a the date on which students have been told to expect feedback. If anonymous marking needs to happen after that, the original unmarked submissions can be downloaded by an intermediary, re-anonymised and sent to the marker. Students can be apprised that their mark is provisional, and Moodle's Turnitin Integration allows marks to be changed after the Post Date.

| Top of page |

Footnotes

* Currently draft so awaiting link.

| Top of page |

Guidance on anonymous marking in Turnitin - including second marking, moderating and external examining by UCL Digital Education is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Based on a work at https://wiki.ucl.ac.uk/display/MoodleResourceCentre/.