Setting up Rubric-Based Marking
- Claudia Cox
Note: Rubric-based marking is a specific, advanced marking workflow which is used to address complex assessment scenarios. It is not the same as just attaching a rubric to a flow and using a standard marker allocation. For more information, the following guidance may be useful:
Getting Started | |
|---|---|
Initial Setup | |
Immediately after submission | |
During and after marking |
Overview
Rubric-based marking allows markers to use rubrics to submit scores for part of the assessment, rather than a grade for the entire assessment. All markers' scores can then be collated and either averaged or added together to produce final grades for student submissions at Manager level.
This makes the marking process significantly easier to complete in WISEflow in scenarios where:
Markers with different specialisms need to assess different parts of the submission and the assessment features X out of Y questions (e.g. “Answer THREE of the following FIVE questions”).
Markers are assessing different aspects or parts of a whole submission.
Adjusting the flow settings
Access the flow in the Managing role. Under the Marking and review panel, click on Manage rubrics:
Click on the Manual marking setting located in the upper right corner, which will reveal an option to switch to rubric-based marking. Click confirm:
Determine how the final grades will be calculated by clicking the Wrench button available underneath Calculation methods:
The calculation methods can be mixed and matched, depending on how complex the marking process is. Both calculation methods are set to “Sum” by default. This table shows the expected behaviour for the different calculation settings available:
| Calculation Setting | Behaviour/ Example |
|---|---|---|
Score between assessors | Sum | The scores submitted by markers using the same rubric will be added together. E.g. Marker 1 and Marker 2 use Rubric A to mark a student’s work. Marker 1 awards 6 points, Marker 2 awards 8 points. The total score awarded to the student is 14. |
Average | The scores submitted by markers using the same rubric will be averaged. E.g. Marker 1 and Marker 2 use Rubric A to mark a student’s work. Marker 1 awards 6 points, Marker 2 awards 8 points. The total score awarded to the student is 7. | |
Score between rubrics | Sum | The scores submitted by markers using different rubrics will be added together. E.g. Marker 1 uses Rubric A to mark a student’s work and awards 6 points. Marker 2 uses Rubric B to assess the same student and awards 8 points. The total score awarded to the student is 14. |
Average | The scores submitted by markers using different rubrics will be averaged. E.g. Marker 1 uses Rubric A to mark a student’s work and awards 6 points. Marker 2 uses Rubric B to assess the same student and awards 8 points. The total score awarded to the student is 7. |
Next, add the rubric(s) to the flow, either by searching for them or entering a reference code and clicking Attach. In this example, two different rubrics have been added to mark section A and section B of an exam:
Use the Rubric allocation tab below this section to allocate the rubrics to the relevant markers. Rubrics can be allocated either by clicking the Pencil button in the Rubrics column of the Assessor table, or by clicking Open rubric allocation matrix (useful for large marking teams):
You will now need to add a grade scaler to allow WISEflow to use the rubric scores to calculate the final, overall grades. Scroll back up the page to the Settings panel and click the Wrench button underneath the Grade scaler:
Check that the maximum score of the rubric(s) is correct based on the calculation methods that were set earlier in step 3. If the format of the final grade is pass/ fail or letter grades, you will be able to adjust the grade boundaries here. If the final grades will be displayed as percentages, click Add to finish.
This completes the intial setup. If the assessment requires markers to be allocated to students based on which questions they have answered, you should follow the Pre-marking Preparation steps below. Otherwise, proceed directly to Allocating markers to student submissions.
Pre-marking Preparation (FLOWmulti only)
This process will help determine which students have answered which questions, so that the marker allocation to students can be set up correctly. If you need to allocate markers according to which questions were answered on any other flow type, you will need to add yourself to the flow as a Reviewer and check each individual student submission.
WARNING: These steps require you to temporarily switch the flow to the legacy marking tool and must be followed once the submission deadline has passed but before marking has begun.
Switching between the new and legacy marking tool while marking is actively underway will result in the loss of marking progress (i.e. grades and feedback) which cannot be recovered.
Access the flow in the Managing role. Under the Marking and review panel, click on Manage settings:
Untick Enable new marker journey:
Add yourself to the flow as a Reviewer and then access the flow via the Reviewing role. Click on Statistics (note that it may take a while for statistics to load for the first time, especially for larger cohorts):
On the statistics page, click the Selected responses tab. For questions that require manual marking, this will show you a table of which questions were answered, indicated by a + symbol. Click on the burger button to the right and select Export all data to a spreadsheet:
Access the flow in the Managing role again. Under the Marking and review panel, click on Manage settings and tick Enable new marker journey to finish. You now have a spreadsheet containing all the anonymised students and information on which questions were answered.
Based on this information, you can now adjust the marker allocation so that markers only see submissions where they have questions to mark.
Pre-moderation Preparation
Once all scores have been recorded by the markers, you will need to provide this information to the moderator.
Access the flow in the Managing role. Under the Marking and review panel, click on Manage marking progression:
At the bottom of this page, click on the Marking overview tab:
This displays a table of all of the students and their scores submitted via the rubric(s), along with the calculated final grade.Use the columns button to do the following:
Switch off
First name
Last name
UCL user id
Email
Switch on
Candidate number
Open a new spreadsheet in MS Excel. Select column F (this assumes there are two columns containing rubric scores; adjust accordingly) and set the number format to Text. This will ensure that the scores are formatted correctly:
Back in WISEflow, use your cursor to highlight all of the data in the marking overview table and copy it. Switch to Excel, select cell A1 in the spreadsheet and paste this data into Excel. Click on Paste options (or press CTRL on your keyboard) and select Match destination formatting (or press the M key):
You should now have a table of plain text data which can be cleaned up further by following these steps:
a) Delete rows 1 and 2 (the header is now in the correct place at the top of the spreadsheet)
b) Press CTRL+F to open up the Find and Replace function. Select the Replace tab and enter “Filterable column is not filtered” (without quotes) in the Find what: field. Leave the Replace with: field blank.
Click Replace all to tidy up the headers.
c) In cells D1 and E1 (assuming there are two columns containing rubric scores; adjust accordingly), enter the titles of the rubrics and/ or the names of the markers the scores relate to.
d) In cell F1 (assuming two rubrics are in use; adjust accordingly), enter “Total score”.
e) Select all of the data in the spreadsheet and add a Filter. Click on the filtering option in cell A1 and select Sort from smallest to largest. The data should now be organised and ready to share with other staff members:
Tip: if scores need to be amended following moderation, it may suffice to share your screen via a Teams call to show the moderator the updated results rather than following these steps again to produce a new spreadsheet. Remember to hide/ display the columns specified in step 3 first so that the students remain anonymous!
Unlocking/ Changing Rubric Scores
Note: This process is only relevant to the specific rubric-based marking workflow described on this page. If you want to unlock grades that have been finalised (e.g. by a Marker or Reviewer, or by a Manager using the guidance on this page), you probably meant to look at our guidance on Unlocking a Finalised Grade.
If the scoring in any rubric needs to be amended following moderation, you will need to unlock it first for the marker.
Access the flow in the Managing role. Under the Marking and review panel, click on Manage marking progression:
At the bottom of this page, click on the Marking overview tab:
Select the relevant score that needs to be changed. This will open a tooltip that shows the details of the marker. Click Unlock score.
The score will now change colour and the rubric status will change to “Opened” to indicate that the marker can edit their score and resubmit the rubric.
Changing Marker or Rubric Allocations
Important: try to make any changes to marker or rubric allocations before marking has taken place wherever possible. Making these changes during the marking process is likely to lead to errors and loss of data is a real risk.
Changing Marker Allocations
See information on how to Change or remove marker allocations from specific students.
Changing Rubric Allocations
WARNING: If a marker has started using a rubric and their allocation to the rubric is removed or the rubric is deleted from the flow, they will lose any data stored via the rubric and this cannot be recovered!
Navigate to the rubrics (marking progression) page on the flow via the Assessment and review panel.
Use the rubric allocation tab to make changes to the existing rubric allocation. Rubrics can be allocated either by clicking the Pencil button in the Rubrics column of the Assessor table (recommended when making changes to an existing allocation), or by clicking Open rubric allocation matrix:
You will receive a warning message explaining the consequences of any changes, which you must acknowledge before you can change the allocation. Tick the box to confirm, make the necessary changes and click Save to finish.
Note: If you need to add a new rubric to the flow before following these steps and find that you are unable to, you may need to delete the grade scaler that was set up in step 6 of the initial setup. Make sure you add a new grade scaler in afterwards, otherwise you will not be able to calculate the final grades.
Finalising the Grades
These steps should be followed once moderation is complete and the calculated grades have been agreed.
Access the flow in the Managing role. Under the Marking and review panel, click on Manage marking progression:
At the bottom of this page, click on the Marking overview tab:
Click Submit all and then Confirm. The calculated grades will now be populated in the Final Grade column and can be released to students and/ or sent back to Portico. Parts of this page will now become read-only and you will no longer be able to change any rubrics or settings.
If any grades need to change at this stage, they will need to be unlocked in the usual way first. See Unlocking a Finalised Grade. You will then need to unlock the rubric(s) so that new scores can be submitted and the grade can be recalculated (Unlocking/ Changing Rubric Scores).