Summary and Recommendations

Summary

As part of this trial, this small project was able to carry out a substantial amount of analysis into the use of ExamOnline and provides us with useful insight. The table below is a crude summary of the main benefits and drawbacks identified during the trial.

 

Benefits Drawbacks
  1. Support for hand drawn diagrams
  2. Cloud based application
  3. Easy client installation
  4. Built-in lock down exam mode
  5. User friendly administration interface
  6. Time adjustment can be made at the individual user level
  7. Exams can be created and scheduled months ahead of time
  8. Less paper used and no transport of exam papers needed
  9. Lower risk of papers going missing or being destroyed in transit
  10. Improved digital experience for digital students (inc. text editor functionality) and liked by students.
  11. More inclusive. Easier and seamless support for some for adjustment (inc. spell checking, font sizes). There is no need for students who need the support of a computer to be in a different room.
  12. Resilient to network outage during the exam
  13. No issue with deciphering handwriting
  14. All markers have access all papers simultaneously
  15. Significantly faster marking workflow
  16. Useful method/format for feedback
  17. Better, more specific feedback given to student
  18. Faster feedback than current process is possible
  1. The size of the computer labs will be a limitation for large classes, with the largest lab in KB having 99 SDs.
  2. Limited support for MAC
  3. Poorer environment conditions with noise and warmth from PCs
  4. Additional SDs required
  5. Requires larger rooms and space out desk
  6. Higher risk of exposing answers to other users
  7. Loss of networking can prevent the start of an exam
  8. Increase risk of cancellation of exam due to power outage
  9. Feedback not fast enough for students

Details of the analysis can be found in the Success Criteria Analysis and Evaluation Report provided by LTW EDE team, the Student Surveys and the Trial Observations.

It is clear that online exams are preferred by students. The benefits listed above are not exclusively specific to ExamOnline and would be shared by other online tools. The limited evaluation permitted by this trial indicates that ExamOnline does not meet enough of the essential criteria to justify a straightforward adoption as a centrally supported service at this stage. 

Recommendations

1 - Further trials outside the School of Biological Sciences to build a business case for central support

There is already several services in this space either supported centrally, in colleges and at individual school level. This diversity can be confusing, appears disjointed and inconsistent from the student perspective as pointed out by the “Headscape” review. While it is recognised that one single application cannot cover the breadth of requirements across all the subject areas, it seems sensible to consolidate as much as practically possible.

During the trial exam, the PM advised Tom Mitchell from Intelligent Assessment Technologies of the then upcoming open procurement for an online assessment application for the College of Medicine and Veterinary Medicine, see project MVM107. It is regrettable that Intelligent Assessment Technologies did not participate in this procurement. This would have indicated a clear interest in providing a service to the wider university community and would have enabled the University to formally evaluate the software. This is a competitive and busy market place with 29 notes of interest received for that procurement.

A directory of applications currently in use across the University would help:

  • to provide an overview of the existing provision
  • to indicate how a school can subscribe to a service
  • to give school access to local applications for testing applications or getting support from other users
  • to identify potential gaps in the existing provision

Further trials are needed to assess whether the use of ExamOnline has a general appeal, outside of the School of Biological Sciences. These should target different groups of students and involve schools across the University, preferably in a different Colleges. This is required in order to assess how it could complement the current central services and avoid compartmentalisation of applications within each school.

We note that the Sponsor used ExamOnline on the 4 April for a summative assessment of his MSc in Biotechnology class where students answered five questions out of eight. Although this was out of scope of this project, this enabled the Sponsor to test a SAQ type test, how the application completes when not interrupted by a power cut and to get feedback from a second cohort of students, this time at postgraduate level, and with high overseas student numbers.

2 - Pilot within the School of Biological Sciences

The differentiating feature of ExamOnline is its ability to support hand drawn diagrams. That functionality itself may be enough for the school or the college to justify them purchasing the application.

Intelligent Assessment Technologies licensing is based on the number of student users, with an annual starting price of £1,600 for 100 students, to £12,200 for 5,000 students. The procurement requirements will depend on the size and cost of the license considered, please refer to the procurement for more details. We note that even the smallest license, 1 year for 100, would require quotes from three different suppliers.

Since support for hand drawn diagrams is not a commonly supported feature in online assessment applications, it may not be feasible to get appropriate quotes from many alternative suppliers. While this trial of ExamOnline demonstrated benefits in the use of this tool, it also raised some concerns and identified gap in the functionality. A carefully planned and controlled pilot providing a small scale, time bound implementation of ExamOnline would be a useful approach to minimise the risk of adopting the tool at local level. A clear roadmap and governance has to be defined ahead of the pilot with targets to test the functionality that was not available in the trial version and to give Intelligent Assessment Technologies the opportunity to address the functionality gaps. At the end of the period, an objective review of the pilot should be carried out to confirm whether the expected benefits have been achieved, and whether the risks associated with a full deployment have been mitigated. This review will inform the decision to adopt or not ExamOnline.

 

Project Info

Project
ExamOnline Trial
Code
SCE011
Programme
SCE Portfolio Projects (OTHSCE)
Management Office
ISG PMO
Project Manager
Muriel Mewissen
Project Sponsor
David Cavanagh
Current Stage
Close
Status
Closed
Start Date
20-Oct-2016
Planning Date
n/a
Delivery Date
n/a
Close Date
28-Aug-2017
Programme Priority
1
Overall Priority
Normal
Category
Discretionary