M09b9 - Using PeerMark

There is an issue with PeerMark that if a student starts a review but does not submit it before the due date it will appear as 'submitted' after due date.  This is NOT reflected in the downloadable spreadsheet of scaled responses, as the attempt was never officially submitted, therefore the number of submissions in the spreadsheet will not reflect the number displayed in PeerMark. This can be identified by seeing if the students review submission date was the same as the due date. PeerMark is still usable, please just be aware of this possible discrepancy if you use the downloadable spreadsheet.

An alternative to the Turnitin PeerMark tool is the Moodle Workshop tool.  More information about this is available here - https://wiki.ucl.ac.uk/x/MpUpAg

Guidance for staff

Contents

What is PeerMark?

PeerMark is one of a number of ways Moodle editors can set up student peer review (and optionally self review) in Turnitin assignments. PeerMark can accommodate a range of different media beyond the essay and can happen before or alongside the standard tutor assessment. PeerMark entails feedback only from the student reviewers i.e. no grade or numeric mark. However, reviewers can be assigned numeric marks for their review.

The basic stages of a PeerMark activity are:

  1. Staff set up a Turnitin assignment to which students submit their work.
  2. Staff set up an associated PeerMark, including review questions, setting how many pieces of work each student reviews, by when, how allocated, whether anonymous &c.
  3. Students submit their work to the Turnitin assignment.
  4. Students review others' work - formative feedback only, without a numeric mark.
  5. Students receive the feedback given by their peers, along with a mark for their own review(s).
  6. Depending on the timing, students can then incorporate this as formative feedback into a final, perhaps credit-bearing, submission.
  7. Students can receive a mark from a staff assessor for their review.

What are the benefits of student peer assessment and peer feedback?

A well-conceived peer assessment activity can advance:

  • Students' ability to understand and work with assessment criteria.
  • Students' ability in the authentic academic practice of peer review.
  • Insights, through articulating judgements and producing constructive feedback, about how students can go about critiquing and improving their own work.
  • The possibility of feedback that is quicker, more individualised, and more plentiful than tutors are able to provide. 
  • The possibility of feedback on students' draft work, with sufficient time for amendments before its deadline.
  • Avoiding 'learned dependence' (Yorke, 2003) - students' over-reliance on tutor opinions and over-humility about the importance of their own understandings.
  • Triangulation - the original submission, peer reviews and tutor assessment (not to mention self assessment where used) can be compared, giving students new perspectives on their submission, the criteria, and the reviews they have written.
  • Relatedly, insights into subjectivity and governance in the assessment process.
  • Also relatedly, a departure from monologic, transmissive feedback as students weigh up the differences in the reviews. This in turn promises a desirable change in the way feedback is received from simple certainties to more sophisticated, evaluative thinking (Schommer, 1990).
  • An occasion for dialogue with tutors and peers about assessment.

Keith Topping (2009) suggests explaining to students,

'...that peer assessment involves students directly in learning, and should promote a sense of ownership, personal responsibility, and motivation. Teachers can also point out that peer assessment can increase variety and interest, activity and interactivity, identification and bonding, self-confidence, and empathy with others.'

Considerations

Won't students take each others' ideas? A widely-held concern - but Richard Milne (UCL Centre for Virology) comments on his own experience of setting up peer review activities, 'I wasn't worried about students stealing each others' ideas ... you discuss a subject with somebody else and then formulate your own way of thinking about it based on the conversation you’ve had'. Students can be encouraged to credit each others' ideas (and a convention can be agreed upon for circumstances of anonymity).

Can students at any level of knowledge carry out good peer reviews? Potentially yes - a meta-analysis by Falchikov and Goldfinch (2009) found that peer assessment at introductory levels was as valid as at higher levels. They attribute this to good preparation. McConlogue (2014) suggests beginning early with low stakes assessment - perhaps the introduction of a draft essay - and building up student expertise over the duration of a programme.


Can peer assessment work in every subject area? Yes, it seems so - Falchikov and Goldfinch (2009) found that subject area has little effect on validity or reliability of peer assessment. They also report that peer assessment of academic products (e.g. essays, posters) or processes (e.g. oral presentation skills, groupwork participation) tend to have more validity than those in the context of professional practice (e.g. internships). This may relate to students' greater familiarity with academic products and processes. Their research also suggests students may struggle with peer-assessment in multiple unfamiliar disciplines - so stick to one new discipline.
 
Will students be willing to do it? That depends. Read on.

Setting up a new PeerMark Assignment

First, set up your Turnitin assignment as normal.


In the Submission inbox for your assignment, click on the 'Launch Peermark Manager' icon

If this is your first PeerMark assignment you can click on the 'Create PeerMark Assignment' (Plus - icon).
Otherwise, you can use the green 'plus' icon at the top left of the window; a settings screen displays.


PeerMark Assignment settings

In the 'Peermark Assignment' tab of the PeerMark Manager you enter basic information about the activity.

Title - displays for students and should be distinctive and descriptive.

Point value (required) - Turnitin PeerMark requires students to give feedback only i.e. no numeric mark from students, so Point Value refers to the number of marks available for the review itself.

Instructions to students - brief guidance about what students should do and why.

Start date,

Due date,

Feedback release date

Then click 'Additional Settings'.

Advice

Although there is no linking between the Peermark dates and the 'parent' Turnitin assignment dates, DE recommends that you set the start date of the Peermark Assignment AFTER the due date of the Turnitin assignment.

This avoids the situation where a student can re-submit a paper that has already received a peer review.

Considerations corresponding to the settings above

Instructions: At first students do tend to prefer tutor marking. They may assume that there is a correct mark for their work which is not open to interpretation (McConlogue, 2012). For this reason, avoid relying on textual instructions alone. Most researchers (including Bloxham and West, 2007; McConologue, 2014; Nicol, 2010; Orsmond, 2004; Topping, 2009) agree about the need to involve students in discussing and ideally negotiating the yardstick against which they will measure others and themselves. It is important to discuss the rationale, criteria and expectations for peer and/or self review before, during and after the activity. Discussing or negotiating expectations clarifies how much time students are expected to spend on each review and how much feedback should be given. This helps to even out the quality and quantity of peer feedback, which in turn helps to avoid perceptions of unfairness (Cartney, 2010) and increase acceptance of peer-assessment. Less time is needed for preparation once students are familiar with the process and ethos.

Dates: That PeerMark asks students to give feedback only (i.e. not a numeric mark) raises possibilities for students reviewing draft work. This can happen at an early, relatively unpolished stage which remains open to rewriting on the basis of feedback (Colvill, 2010). In this case, set the Feedback Release Date to allow time for students to make changes in advance of their final credit-bearing submission. The time allowance for the PeerMark activity (i.e. between Start Date and Due Date) should reflect the time students are expected to spend and allow for their other commitments.

PeerMark Assignment - Additional Settings

On the 'Peermark Assignment' tab there is a link for additional settings. Here's some explanation for the less obvious ones.

'Award full points if the review is written' 

If ticked this means tutors will not be able to mark the reviews and a student will need to meet set requirements for every part of the review in order to get the available marks, on an all-or-nothing basis. If unticked, tutors can assign and differentiate marks for each student's review.  

'Allow students to view author and reviewer names'

If left unticked, you probably need to remind students not to put any identifying information in the title, filename, or body of their work.

'Paper(s) automatically distributed by Peermark'

This sets the number of randomly allocated papers each student has to review.

'Papers(s) selected by the student'

This sets the number of papers a student can choose to review. Students can review a combination of allocated and selected papers.

'Require self-review'

If checked, a student has to review their own paper. It isn't currently possible to select self review only - the number allocated by PeerMark has to be at least one.

Considerations

Award full points if the review is written. This means that students  get full marks if they fill in all parts of the peer review form, and no marks if they don't. As an all-or-nothing setting it might be an incentive to participate, but do keep in mind the importance of dialogue at all stages.

Allow students to view author and reviewer names. Students are likely to experience some social discomfort about commenting on each others' work. Enabling anonymity helps to avoid the peer review being determined by friendship, enmity or power processes, and disrupts any collusion among students. Clear criteria and an ethos which encourages mutual constructive criticism. It may be necessary work out with students a convention for referencing each others' work in the absence of names, should they want to do so.

Allow submitters to read all papers after the Start Date. This allows students to view all the pieces of work as they are submitted.

Allow students to read ALL papers and ALL reviews after the Feedback Release Date. This allows students to view all submissions and reviews after the peer review process has ended. Enabling this allows students to benchmark both their submissions and their reviews, and could open up the possibility of conversations which outlast the PeerMark activity.

Distribution of papers. Keep in mind boredom, tiredness and time pressures when deciding how many submissions each student should review. Falchikov and Goldfinch (2009) didn't find that larger numbers of reviewers brought any validity gains. In fact they reported that large numbers might reduce reliability due to the 'diffusion of responsibility effect' whereby students are less likely to perceive their own review as mattering. (Falchikov and Goldfinch were comparing peer and tutor marks rather than feedback, though).

Require self-review. Since one of the aims of peer-assessment is to help students use the criteria in their own work, self-review would be helpful. However, because it may pose a distinct idiosyncratic or cultural set of complications related to self-esteem, self- confidence, modesty, and how students habitually estimate their own ability (Saito and Fujita, 2004) , it is a good idea to ask students to carry it out after they have completed their peer review(s). For this reason PeerMark requires at least one peer review, even if there's a self review.

Adding Questions

Make sure you click the 'Save and Continue' button to proceed; this returns you to the top of the Settings page with a confirmation message.

Proceed to the next tab in the Settings.

The 'PeerMark Questions' tab of the PeerMark Manager allows you to create the questions you want the peer reviewers to answer.

To add a question, click 'Add question'.

Enter your question text, the question type. There are two types of question you can use;
a 'Free Response' question - for example "What is the thesis of the paper?" and a 'Scale' question – for example "How well does the introduction pull you in as a reader? Scale, Not very well to Really well".


For a 'Free response' question, enter the minimum answer length (this counts words).

For a 'Scale' question, enter the scale size and the lowest and highest values.

You can also use Libraries to manage your PeerMark questions. To create a Library, click 'Save to Library' > 'Add Library', and name your Library - when you save the Library, the questions you have created are saved to that Library; to retrieve questions from a Library, click 'Add From Library'. A question may exist in more than one Library.

Note that if you change a question in an Assignment, you need to re-save to any Libraries you've already saved it to.

There is also a 'Sample Library' from which you can add existing questions.

Considerations

Who decides the questions? Peer assessment researchers strongly recommend involving students in developing and clarifying the assessment criteria - even if they arrive at similar criteria to the tutors (and of course the criteria are likely to be aligned to the module's intended learning outcomes) . Involving students in this way increases a sense of ownership, reduces anxiety, and also promotes a shared understanding about the meaning of the criteria. As well as improving the reliability and validity of the assessment, it also builds confidence in the process. Orsmond (2004, Section 2) discusses alternative techniques for introducing assessment criteria to students, including giving opportunities to practice applying the criteria.

What kinds of questions? Falchikov and Goldfinch (2009) found that asking students to make a global judgement (i.e. of the submission as a whole) based on distinct criteria was more effective than either a judgement without criteria, or separate judgements of separate dimensions of the submission. McConlogue (2014) points out that as well as value judgements, reviewed students also expect feedback that signposts how they can improve; PeerMark's open question type allows for these.

Order of questions. Topping (2009) recommends asking students to give positive feedback first,  since this improves subsequent acceptance of negative feedback.

Opportunities for practice. Again, practice is a clear recommendation from the peer assessment literature and could be incorporated into the aforementioned recommended discussion of the criteria.

Distribution

Please note that after reviewing has started you won't be able to pair students - so do make any allocations in advance.

In the 'Distribution' tab of the PeerMark Assignment manager you can see all the student accounts associated with this assignment and how they will be allocated reviews. If you want to, this is where you can get involved with who reviews whose work.

If you can't see all the accounts you are expecting, click outside of the Peermark Manager to return to your Turnitin assignment page; then click its 'Turnitin Students' tab. From there you can click 'Enrol all students', which will bring in all students 'enrolled' in that Moodle course space.


If you need to exempt students from the PeerMark activity, you can exclude them by clicking their adjacent red Minus icon; their name displays greyed-out and they gain a green Plus icon, which you can click if you need to reinstate them.

If you want to pair students (so that a particular student is allocated the work of another particular student to review, overriding any other distribution settings) you can do so by clicking the blue Plus icon and then selecting a student to pair with from the dropdown list. Paired students are then required to review the work they are allocated. Note that PeerMark displays this as a 'Forced' allocation.

Considerations.

Does it matter which students review which other students' work? By default Turnitin makes random allocations, but if tutors want students to connect e.g. on the basis of interest, there are alternative ways to do this. Tutors can manually pair individual students through PeerMark. Or thirdly, PeerMark has the option of letting students choose the work they review (note though that this introduces the possibility that some students will receive feedback from more sources than other students - which may be contentious).

Accessing Peermark reviews

In the 'Submission Inbox' you can see details of all the PeerMark assignments set up for that Turnitin assignment.

Click on the 'Launch Peermark Reviews' icon

If there is more than one Peermark assignment set up for this Turnitin assignment you can select the one you want.

The 'Reviews' tab shows you a list of the students.

Students that have submitted a paper will have an icon next to them under the 'Review' column. The 'Received' column shows how many reviews a student's submission has received. The 'Submitted' column shows how many reviews a student has submitted. Clicking on the numbers with a grey background takes you to either the 'Received Reviews' or the 'Submitted Reviews' tab, and from there, clicking on the blue 'tick' icon launches the document viewer.

Tutors can also write reviews. Clicking on the blue 'Write instructor review' icon in the 'Review' column allows you to write an additional review as the assignment tutor. Can reviewed students distinguish the tutor review?

If you have left a review for a submission it will display a green 'Edit instructor review' icon.

The document viewer

At the top of the document you will see details of which paper you are viewing, and which student has reviewed it.

On the right hand column you will see a column with two tabs.
The questions tab shows you all the questions you set up for this PeerMark assignment, plus the reviewers answers.
The comments tab will show any comments if the reviewer has added them.

Marking reviews

Once the due date has passed, you can go into the document viewer for a particular review and enter a grade in the in the top right of the screen.

HOW are grades released????


Tips

Saving – Apart from the 'Save & Continue' button on the Peermark Assessment tab, there are no other save buttons within the manager. To exit the manager you click out of the manager somewhere on the greyed out portion of the screen.

Bugs

Sometimes, when switching between tabs, you will see an 'Error, you are not authorised to access this resource' message. Click away from the window and reopen it vis the Peermark manage icon.


References

  • Bloxham, S., & West, A. (2007). Learning to write in higher education: students’ perceptions of an intervention in developing understanding of assessment criteria. Teaching in Higher Education, 12(1), 77–89.
  • Cartney, P. (2010). Exploring the use of peer assessment as a vehicle for closing the gap between feedback given and feedback used. Assessment & Evaluation in Higher Education, 35(5), 551–564. 
  • Covill, A. (2010). Comparing Peer Review and Self-Review as Ways to Improve College Students’ Writing. Journal of Literacy Research, 42(2), 199–226.
  • Falchikov, N., & Goldfinch, J. (2000). Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks. Review of Educational Research, 70(3), 287–322.
  • McConlogue, T. (2012). But is it fair? Developing students’ understanding of grading complex written work through peer assessment. Assessment & Evaluation in Higher Education, 37(1), 113–123.
  • M.cConlogue, T. (2014). Making judgements: investigating the process of composing and receiving peer feedback. Studies in Higher Education, 1–12. 
  • Milne, R., (2013). Peer review of virology essays. 
  • Nicol, D., (2007). Peer Evaluation in Assessment Review project. Available from http://www.reap.ac.uk/PEER.aspx
  • Nicol, D., (2010). From monologue to dialogue: improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education, 35(5), 501–517.
  • Orsmond, P. (2004). Self- and peer-assessment: guidance on practice in the biosciences. Leeds: Centre for Bioscience, Higher Education Academy.
  • Sadler, D. (2010) Beyond feedback: developing student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35(5), 535-550.

  • Saito, H., & Fujita, T. (2004). Characteristics and user acceptance of peer rating in EFL writing classrooms. Language Teaching Research, 8(1), 31–54..
  • Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82(3), 498–504.
  • Sorensen, E. (2013). Experiences of using peer assessment in a 4th year design module. 
  • Topping, K. J. (2009). Peer Assessment. Theory Into Practice, 48(1), 20–27.
  • Yorke, M. (2003). Formative assessment in higher education: moves towards theory and the enhancement of pedagogic practice. Higher Education, 45, 477–501.