The College of Science and Engineering set to review inefficiencies in paper based marking and to evaluate digital assessment tools. There have been trials of different tools in some schools in the past which benefited specific departments but have so far not clarified the benefits for a more general use across the College. The Covid pandemic meant that a quick shift from on campus assessments to online assements was needed for session 20/21. In order to support the schools in making this move, the College purchased a 1 year license for the online marking platform GradeScope from Turnitin in December 2020.
GradeScope is currently run as a pilot within the College of Science and Engineering. So far six of the seven schools have trialled GradeScope: Maths, Engineering, Informatics, Chemistry, Physics and Astronomy, Geosciences. Biological Sciences continue to be content that ExamOnline from Intelligent Assessment Technologies (see SCE011 ExamOnline Trial ) fulfil their online assessment and marking needs.
This project evaluated the effectiveness of the Gradescope tool for grading digital and paper assessments. The experience of professional and academics staff who used GradeScope for either of both exam diets (January and May 2021) was captured and analysed. The evaluation report delivered by this project will be part of the business case to be put to the College Senior Leadership Team to advise on future provision needed to support online assessment and marking.
The project will engage with the users in the pilot to review and analyse their experience with GradeScope. This will include:
- A review of the current manual, paper based process
- Establishing a baseline estimate for use of Gradescope - how much administrative effort does it take schools who haven’t previously used Gradescope.
- Agreeing Gradescope Evaluation Critieria
- Evaluation will cover formative and summative assessment undertaken over a 6 month period between December 2020 and May 2021 exam diets.
- Recommendation paper
Out of scope
- Evaluation of other online assessment platforms in use in SCE, i.e. STACK, ExamOnline, or elsewhere in the University, i.e. QMP.
- User training or training design
- Run or decommission the pilot installation
Objectives and Deliverables
|O1||Review of current manual, paper based process required to set up assessments|
|O1.D1||Report documenting the AS-IS process||Must||
|O2||Review of GradeScope Pilot|
|O2.D1||Recommendation report including review of GradeScope Pilot||Must||
O1.D1 The documentation of the AS-IS process was partially descoped.
The project planned to document the experience of the schools (Physics and Astronomy and Geosciences) that did not use GradeScope during the January diet, and to compare this to the experience of the schools that use GradeScope in January (Maths, Engineering, Informatics and Chemistry). However, this could not be seen as a strict AS-IS process. Setting remote exams last January was a first for the schools and not representative of the standard on campus process. It was also a one off process. The additional pressure and workload created by the move to remote working meant some staff didn't have enough time to train and use GradeScope ahead of the January diet but would by May 2021.
A true AS-IS would require to look back to on campus examinations done in January 2019 before the pandemic. Time and effort to set these were not recorded. While we can capture staff memories of how much time and effort this required, it would not be an accurate measure. It was therefore agreed that there was little value in documenting this process, especially as it is widely assumed to be very unlikely that on campus exams will return in the same pre-pandemic setup. Experience of Learn Grade Centre, Gradex and manual processes was captured in a cut down AS-IS document linked above.
|B1||Document manual process, providing evidence and data||Descoped (as above)|
|B2||Document GradeScope pilot use and feedback||Achieved|
|B3||Inform and feed to a Business Case for the College to decide on future provision of online marking tool||Achieved|
Analysis of Resource Usage:
Staff Usage Estimate: 38 days
Staff Usage Actual: 30 days
Other Resource Estimate: £0
Other Resource Actual: £0
Other Resource Variance: 0%
Explanation for Variance
|Business Area||Role||Estimate||Actual||Explanation for variance|
|Project Service||PM||12 days||7 days||Lack of availability, see PICCL1|
|BA||26 days||23 days||AS-IS descoped as noted above|
The business analyst conducted interviews, workshops and a survey to capture the experience of using GradeScope within the College of Sciences and Engineering. Both academics and professionals support staff were included in order to capture the requirements for setting exams as well as for marking exams.
The pilot demonstrated that Gradescope offers significant advantages over the other tools deployed in the May 2020 diet and the majority of staff are keen to retain these benefits. Gradescope is not a perfect fit for the College’s needs, but its does present an opportunity to address significant gaps in digital marking for STEM assessments currently. Users reported a wide range of minor frustrations with Gradescope, and although there were some more significant issues when it was first implemented, such as problems with anonymization, these have now been resolved by the supplier.
The majority of staff found that Gradescope was easy to use and either saved time or did not affect the amount of time they spend on marking and marking-related administration. 72.5% of surveyed staff said Gradescope should be retained for use in 2021/22. Most also noted a marked preference to keep GradeScope for the long term to provide stability and allow it to get embedded in school processes. Withdrawing Gradescope at this stage would cause considerable frustration to staff who are stretched by the demands of online teaching and have already changed their exam processes twice since the start of the pandemic. Even if an alternative system with similar functionality to Gradescope could be found, staff would have to invest time in learning to use the new system and updating information for students and markers, retaining Gradescope gives the institution the opportunity to mature its approach to online exams and develop clear strategy to support future summative assessment practises.
The recommendation from the GradeScope Evaluation is to retain GradeScope for the next 12 months.
Several factors will have to be considered if GradeScope is to be retained for longer:
- The outcome of the TEL068 E-Assessment Procurement and Implementation project due later in the summer but expected to not completely fulfil the requirements from SCE.
- The outcome of the curriculum review which may impact on how assessments are conducted.
- The requirement for a procurement excercise after the end of current 1 year + 1 optional year license agreement.
- The possible merge of Turnitin with GradeScope, noting the University Turnitin license will come to an end next year, paving the way for a potential joint procurment.
Key Learning Points
- The pilot user group was extremely helpful in providing input and support for the project and the analysis.
- Informatics and Mathematics use Gradescope to mark coursework as well as exams. There is potential for other schools to extend the use of Gradescope by adopting a similar approach. Another alternative use would be to support the marking of exams taken on campus with papers scanned on campus.
- Some of the more negative comments were raised by the schools who used GradeScope in May 2021 (GeoSciences and Physics and Astronomy). This contrasts with the input from other schools which use GradeScope in both January and May 2021 indicating that additional experience and training may alleviate some the concerns raised.
- Some staff used opposition to Gradescope as a means of expressing their dissatisfaction with online exams more generally, which they view as a threat to academic integrity. It is noted that this sits outside the scope of this evaluation and concerns will be referred to Academic Affairs within the College Office to clarify the Institutions position on the long term use of online exams.
- The supplier engagement, training and support has been good. The ability to submit items to be considered for inclusion in the development roadmap and the ability to vote to prioritise items on this roadmap have been particularly well received, especially that some of the items submitted during the pilot have been addressed and deployed.
- SSO and user accounts management will be considered if the College decide to retain GradeScope.
Expected Next Steps
- The Sponsor and The Business Lead will be presenting the evaluation report to the SCE Learning and Teaching Committee (LTC) in September.
- If LTC provides their support then the Sponsor and Business lead will put a business case forward with the College to purchase Gradescope for 2022.
- No action from IS Applications is required until SCE informs us (probably by December) whether they intend to move forward with a procurement. ITC approval will be required.
- All relevant teams will be involved if SCE move forward with Gradescope.