Summary and Recommendations
Summary
As part of this trial, this small project was able to carry out a substantial amount of analysis into the use of ExamOnline and provides us with useful insight. The table below is a crude summary of the main benefits and drawbacks identified during the trial.
| Benefits | Drawbacks |
|---|---|
|
|
Details of the analysis can be found in the Success Criteria Analysis and Evaluation Report provided by LTW EDE team, the Student Surveys and the Trial Observations.
It is clear that online exams are preferred by students. The benefits listed above are not exclusively specific to ExamOnline and would be shared by other online tools. The limited evaluation permitted by this trial indicates that ExamOnline does not meet enough of the essential criteria to justify a straightforward adoption as a centrally supported service at this stage.
Recommendations
1 - Further trials outside the School of Biological Sciences to build a business case for central support
There is already several services in this space either supported centrally, in colleges and at individual school level. This diversity can be confusing, appears disjointed and inconsistent from the student perspective as pointed out by the “Headscape” review. While it is recognised that one single application cannot cover the breadth of requirements across all the subject areas, it seems sensible to consolidate as much as practically possible.
During the trial exam, the PM advised Tom Mitchell from Intelligent Assessment Technologies of the then upcoming open procurement for an online assessment application for the College of Medicine and Veterinary Medicine, see project MVM107. It is regrettable that Intelligent Assessment Technologies did not participate in this procurement. This would have indicated a clear interest in providing a service to the wider university community and would have enabled the University to formally evaluate the software. This is a competitive and busy market place with 29 notes of interest received for that procurement.
A directory of applications currently in use across the University would help:
- to provide an overview of the existing provision
- to indicate how a school can subscribe to a service
- to give school access to local applications for testing applications or getting support from other users
- to identify potential gaps in the existing provision
Further trials are needed to assess whether the use of ExamOnline has a general appeal, outside of the School of Biological Sciences. These should target different groups of students and involve schools across the University, preferably in a different Colleges. This is required in order to assess how it could complement the current central services and avoid compartmentalisation of applications within each school.
We note that the Sponsor used ExamOnline on the 4 April for a summative assessment of his MSc in Biotechnology class where students answered five questions out of eight. Although this was out of scope of this project, this enabled the Sponsor to test a SAQ type test, how the application completes when not interrupted by a power cut and to get feedback from a second cohort of students, this time at postgraduate level, and with high overseas student numbers.
2 - Pilot within the School of Biological Sciences
The differentiating feature of ExamOnline is its ability to support hand drawn diagrams. That functionality itself may be enough for the school or the college to justify them purchasing the application.
Intelligent Assessment Technologies licensing is based on the number of student users, with an annual starting price of £1,600 for 100 students, to £12,200 for 5,000 students. The procurement requirements will depend on the size and cost of the license considered, please refer to the procurement for more details. We note that even the smallest license, 1 year for 100, would require quotes from three different suppliers.
Since support for hand drawn diagrams is not a commonly supported feature in online assessment applications, it may not be feasible to get appropriate quotes from many alternative suppliers. While this trial of ExamOnline demonstrated benefits in the use of this tool, it also raised some concerns and identified gap in the functionality. A carefully planned and controlled pilot providing a small scale, time bound implementation of ExamOnline would be a useful approach to minimise the risk of adopting the tool at local level. A clear roadmap and governance has to be defined ahead of the pilot with targets to test the functionality that was not available in the trial version and to give Intelligent Assessment Technologies the opportunity to address the functionality gaps. At the end of the period, an objective review of the pilot should be carried out to confirm whether the expected benefits have been achieved, and whether the risks associated with a full deployment have been mitigated. This review will inform the decision to adopt or not ExamOnline.