Considerations

Won't students take each others' ideas? A widely-held concern - but Richard Milne (UCL Centre for Virology) comments on his own experience of setting up peer review activities, 'I wasn't worried about students stealing each others' ideas ... you discuss a subject with somebody else and then formulate your own way of thinking about it based on the conversation you’ve had'. Students can be encouraged to credit each others' ideas (and a convention can be agreed upon for circumstances of anonymity).

Can students at any level of knowledge carry out good peer reviews? Potentially yes - a meta-analysis by Falchikov and Goldfinch (2009) found that peer assessment at introductory levels was as valid as at higher levels. They attribute this to good preparation. McConlogue (2014) suggests beginning early with low stakes assessment - perhaps the introduction of a draft essay - and building up student expertise over the duration of a programme.

Considerations corresponding to the settings above

Instructions: At first students do tend to prefer tutor marking. They may assume that there is a correct mark for their work which is not open to interpretation (McConlogue, 2012). For this reason, avoid relying on textual instructions alone. Most researchers (including Bloxham and West, 2007; McConologue, 2014; Nicol, 2010; Orsmond, 2004; Topping, 2009) agree about the need to involve students in discussing and ideally negotiating the yardstick against which they will measure others and themselves. It is important to discuss the rationale, criteria and expectations for peer and/or self review before, during and after the activity. Discussing or negotiating expectations clarifies how much time students are expected to spend on each review and how much feedback should be given. This helps to even out the quality and quantity of peer feedback, which in turn helps to avoid perceptions of unfairness (Cartney, 2010) and increase acceptance of peer-assessment. Less time is needed for preparation once students are familiar with the process and ethos.

Dates: That PeerMark asks students to give feedback only (i.e. not a numeric mark) raises possibilities for students reviewing draft work. This can happen at an early, relatively unpolished stage which remains open to rewriting on the basis of feedback (Colvill, 2010). In this case, set the Feedback Release Date to allow time for students to make changes in advance of their final credit-bearing submission. The time allowance for the PeerMark activity (i.e. between Start Date and Due Date) should reflect the time students are expected to spend and allow for their other commitments.

Considerations

Award full points if the review is written. This means that students  get full marks if they fill in all parts of the peer review form, and no marks if they don't. As an all-or-nothing setting it might be an incentive to participate, but do keep in mind the importance of dialogue at all stages.

Allow students to view author and reviewer names. Students are likely to experience some social discomfort about commenting on each others' work. Enabling anonymity helps to avoid the peer review being determined by friendship, enmity or power processes, and disrupts any collusion among students. Clear criteria and an ethos which encourages mutual constructive criticism. It may be necessary work out with students a convention for referencing each others' work in the absence of names, should they want to do so.

Allow submitters to read all papers after the Start Date. This allows students to view all the pieces of work as they are submitted.

Allow students to read ALL papers and ALL reviews after the Feedback Release Date. This allows students to view all submissions and reviews after the peer review process has ended. Enabling this allows students to benchmark both their submissions and their reviews, and could open up the possibility of conversations which outlast the PeerMark activity.

Distribution of papers. Keep in mind boredom, tiredness and time pressures when deciding how many submissions each student should review. Falchikov and Goldfinch (2009) didn't find that larger numbers of reviewers brought any validity gains. In fact they reported that large numbers might reduce reliability due to the 'diffusion of responsibility effect' whereby students are less likely to perceive their own review as mattering. (Falchikov and Goldfinch were comparing peer and tutor marks rather than feedback, though).

Require self-review. Since one of the aims of peer-assessment is to help students use the criteria in their own work, self-review would be helpful. However, because it may pose a distinct idiosyncratic or cultural set of complications related to self-esteem, self- confidence, modesty, and how students habitually estimate their own ability (Saito and Fujita, 2004) , it is a good idea to ask students to carry it out after they have completed their peer review(s). For this reason PeerMark requires at least one peer review, even if there's a self review.

Considerations

Who decides the questions? Peer assessment researchers strongly recommend involving students in developing and clarifying the assessment criteria - even if they arrive at similar criteria to the tutors (and of course the criteria are likely to be aligned to the module's intended learning outcomes) . Involving students in this way increases a sense of ownership, reduces anxiety, and also promotes a shared understanding about the meaning of the criteria. As well as improving the reliability and validity of the assessment, it also builds confidence in the process. Orsmond (2004, Section 2) discusses alternative techniques for introducing assessment criteria to students, including giving opportunities to practice applying the criteria.

What kinds of questions? Falchikov and Goldfinch (2009) found that asking students to make a global judgement (i.e. of the submission as a whole) based on distinct criteria was more effective than either a judgement without criteria, or separate judgements of separate dimensions of the submission. McConlogue (2014) points out that as well as value judgements, reviewed students also expect feedback that signposts how they can improve; PeerMark's open question type allows for these.

Order of questions. Topping (2009) recommends asking students to give positive feedback first,  since this improves subsequent acceptance of negative feedback.

Opportunities for practice. Again, practice is a clear recommendation from the peer assessment literature and could be incorporated into the aforementioned recommended discussion of the criteria.

Considerations.

Does it matter which students review which other students' work? By default Turnitin makes random allocations, but if tutors want students to connect e.g. on the basis of interest, there are alternative ways to do this. Tutors can manually pair individual students through PeerMark. Or thirdly, PeerMark has the option of letting students choose the work they review (note though that this introduces the possibility that some students will receive feedback from more sources than other students - which may be contentious).