Setting up Item-Based Marking (FLOWmulti only)
- Claudia Cox
Note: Item-based marking is a specific, advanced marking workflow which is used to address complex assessment scenarios. For more information, the following guidance may be useful:
Getting Started | |
|---|---|
Initial Setup | |
During and after marking |
Overview
Item-based marking allows markers to be allocated to a specific part of the assessment, rather than individual students. This allows them to see only the relevant sections of the exam for marking. All markers' scores, along with any automatic scoring, can then be collated and added together to produce final grades for student submissions at Manager level.
This makes the marking process significantly easier to complete in WISEflow in scenarios where:
Markers with different specialisms need to assess different parts of the submission, and
All questions in the exam must be answered (i.e. there are no optional/ “X out of Y” questions)
This workflow is not compatible with the use of rubrics.
Adjusting the flow settings
Note: The FLOWmulti assignment/ exam paper will need to be attached to the flow before you can follow the steps below. You can’t allocate markers to different sections of the exam paper if it isn’t there!
Access the flow in the Managing role. Under the Marking and review panel, click on Manage settings:
Click the pencil button and enable Item-based marking:
Return to the main flow page. Under the Marking and review panel, click the Manage matrix button that has appeared:
The Item allocation matrix will display a table containing all items (pages) of the exam and any markers that have been added to the flow. Items containing questions that require manual marking will need to be allocated to the appropriate marker(s) using the cells in the Marker column(s). Items that contain only automatically scored questions will automatically be ticked in the Auto-scoring column.
The maximum achievable points for each item in the exam will be set based on the points set for each question in the exam paper. You should not need to adjust these, but it is possible if the item is allocated to a marker.
Note: In the example screenshot above, one marker has been allocated to items 2 and 4 of the exam and two markers have been allocated to item 3. When multiple markers are allocated to an item, WISEflow will divide the workload between the markers equally.
It is not possible to allocate uneven workloads in item based marking (i.e. marker 1 marks 60% of item 3, marker 2 marks 40%), however as a workaround you could transfer the remainder of the unmarked submissions to somebody else.
If you wish to have multiple markers looking at all of the submissions in an item (i.e. full, independent second marking) you will need to set up Assessor (marker) groups to accommodate this.
Once all the items in the exam have been allocated, either to a marker or auto-scoring, check the grade scaler. The maximum score will be set based on the total of all the items that are being scored. If the format of the final grade for the assessment is pass/fail or letter grades, you will be able to adjust the grade boundaries here.
If the final grades will be displayed as percentages, there is no action needed. In either case, click Save setup to finish.
Pre-moderation Preparation
Once all scores have been recorded by the markers, you will need to provide information on the scores and calculated grades to the moderator.
Access the flow in the Managing role. Under the Marking and review panel, click on Manage matrix:
Click Go to marking overview:
This displays a table of all of the students and their submitted scores, along with the calculated final grade. Use the keyboard shortcut CTRL+A to select all of the text on this page and then copy it.
Open a new spreadsheet in MS Excel. Select cell A1 in the spreadsheet and paste this data into Excel. Click on Paste options (or press CTRL on your keyboard) and select Match destination formatting (or press the M key):
You should now have a table of plain text data which can be cleaned up further by following these steps:
a) Delete all rows before the student names (approx. rows 1 - 29).
b) Right click on the header of column A and click Insert to add a new column to the spreadsheet. Select all of the student names and cut and paste them into the newly created column so that they align with the score data:
c) Delete any rows above your data as needed to tidy it up. You will now need to anonymise the students. Students are listed in the order of their flow ID (a number that is assigned to them when they are first added to the flow).
You can obtain a list of student names and their corresponding flow IDs in WISEflow by scrolling down to the list of participants on the main flow page and clicking Tools > Export grades:
From here, click the burger button and select the option to Export all data as csv:
Open the downloaded .csv file and use the data here to replace the student names with candidate numbers in your score data spreadsheet. You could use a VLOOKUP function for this or you could sort the csv file from smallest flow ID to largest and then copy/paste the candidate numbers in. You can now give column A the header title “Candidate number”:
e) Delete any rows underneath your score data that contain unnecessary information (i.e. any copyright information that may have been copied from the score data screen in WISEflow during an earlier step). Select all of the data in the spreadsheet and add a Filter. The data should now be organised and ready to share with other staff members.
Unlocking/ Changing Item Scores
If a marker asks you to unlock an item for them, follow the steps below:
On the main flow page, navigate to the Marking and review panel and click the Manage matrix button. From here, click Go to marking overview:
In the marking overview, you can either unlock all submitted scores for a specific item by selecting this option from the corresponding Pencil button at the bottom of the score table, or you can click Unlock submitted scores to do this for a specific marker:
Successfully unlocked items or scores will be shown by a blue tick in the Score table cells:
Excluding an item or question from scoring
Tip: If only minor errors were made (e.g. the wrong question score or wrong correct answer was set), you can probably just rescore the question(s) in the item! The following guidance may be useful:
Excluding an item from scoring
Important: Once marking has started, DO NOT try to make this change via the item allocation matrix. If you change the item allocation matrix when marking is underway, it will unlock all submitted scores and markers will have to re-lock and re-submit all of their marking.
If an entire section of an exam needs to be excluded from scoring (e.g. due to significant errors or the item no longer being in use), access the Marking overview and click the Pencil button for the relevant item in the score table. Select the option to Deactivate item and click Confirm:
If the final grade will be displayed as passed/ not passed or a letter grade, make adjustments to the grade scaler via the Adjust grade scaler button and click Save setup to finish.
Excluding a question from scoring
If an exam question within an item needs to be excluded from scoring (e.g. due to a significant error in the content that would have prevented many students from answering correctly), the exam’s Author should follow the steps in the guidance linked in the tip at the top of this section, setting the score for the question to 0. You can then proceed to follow the steps outlined in the Manager guidance.
Changing Marker Allocations (Marker Transfer)
If a marker’s work needs to be re-allocated during the marking process (e.g. because they are no longer able to complete this), you can transfer this to another user following the steps below:
Access the flow in the Managing role. Under the Marking and review panel, click on Manage matrix:
Click the Go to marking overview button:
Underneath the score table, click the Transfer items button:
Use the transfer menu to select the Original assessor (marker) whose work is to be tranferred, and the Receiving assessor (marker) who will be given the work:
Click Transfer. Once the process has been completed, click Close to finish. The change in marker allocation will be reflected in the Item allocation matrix and any feedback (comments or annotations) that were delivered by the original marker will now be editable on the receiving marker’s end.
Finalising the Grades
These steps should be followed once moderation is complete and the calculated grades have been agreed.
Access the flow in the Managing role. Under the Marking and review panel, click on Manage matrix:
Click the Go to marking overview button:
Calculated final grades are displayed in the Scaled grade column in the score table. Click Set final grades and then Approve:
The calculated grades will now be populated in the Final Grade column and can be released to students and/ or sent back to Portico. Parts of this page will now become read-only and you will no longer be able to change any item allocations or settings.
Unlocking Finalised Grades (Item-Based Marking)
Unlike other marking workflows, it is not possible to unlock a finalised grade for an individual student where item-based marking is in use. Instead, click the burger button in the upper right of the main flow page, select the option to Reset grades and click Proceed:
This will remove the finalised grades, allowing you to unlock specific items or a specific marker’s scores as described in the section of this guidance on Unlocking/ Changing Item Scores.