Trial Observations

SCE011 was initiated at the request of the Sponsor seeking for support from Information Services to evaluate the potential of the ExamOnline application from Intelligent Assessment Technologies through a trial run. The School of Biological Sciences selected ExamOnline because it supports the inclusion of hand drawn diagrams. This feature is of critical importance for them because all essay exams in that field are expected to include or allow the inclusion of hand drawn diagrams.

1.    Trial Preparation

The preparations for the trial was well underway when the project started, with the Sponsor having already secured access to a free trial license for ExamOnline, and recruited student volunteers from his Year 3 Biology class.

Setup

Date Wednesday 16 November 2017, 2-3pm
Location Computer lab, room JCMB 4325D
License Standard 30 day free trial license negotiated to open ended, see http://www.intelligentassessment.com/examonline/licensing/
IT ExamOnline is a cloud based application and only a client is required to be installed locally. This was done manually on all the Supported Desktops (SDs), approximatively 65, in the computer lab. The process was trivial and only took 15 minutes. The client was removed from the desktops at the next automated monthly rebuilt of the lab SDs. These steps would not be required if the application was run as a service, instead Desktop Services would add the application to the list of packages included as standard in the SDs. The desktop gets lock down in exam mode when ExamOnline is opened. A secure folder on Datastore was created to store the scanned diagrams. Local computing support was book to assist during the trial.
Users 21 student volunteers from Year 3 Biology course. Students were given access to ExamOnline during a tutorial session to familiarise themselves with the application ahead of the trial. The mock exam was marked and the student provided extensive feedback.
Administration

The mock exam consisted of one essay question and included a compulsory hand drawn diagram. The exam was created and scheduled by academic staff using a local ExamOnline client. This can be done by support staff ahead of time. We note that currently there is no client for MAC.Temporary anonymised accounts were set up (i.e. student1, student2…). Any adjustment to the length of exam can be set for each user. If the application was run as a service then integration with EASE would be required.

Evaluation

An online survey was designed to capture the feedback from the students on their experience of using ExamOnline for taking the mock exam. A link to the survey was added to the SDs announcement screen for that day to allow the student to fill in the survey right after the exam. A second survey was created to capture the experience of students on receiving feedback after the exam in ExamOnline. This survey would be circulated after the exams have been marked and feedback provided.

Success criteria Efforts were made to gather a set of requirements and success criteria for an online application for the School of Biological Sciences against which the student experience and software would be evaluated. Time was very limited as the project started less than a month before the trial. This was noted as risk 2. A suitable draft list was produced but it would have benefited from further refinements, and from the input of a larger group of staff to better represent the varied, and potential different, needs with the school and college. The success criteria list is available at  sce011-_success_criteria_-_0.2.pdf

The use of ExamOnline demonstrated the following:

Benefits Drawbacks
  • Support for hand drawn diagrams
  • Cloud based application
  • Easy client installation
  • Built-in lock down exam mode
  • User friendly administration interface
  • Time adjustment can be made at the individual user level
  • Exams can be created and scheduled months ahead of time
  • Less paper used and no transport of exam papers needed
  • Lower risk of papers going missing or being destroyed in transit
  • The size of the computer labs will be a limitation for large classes, with the largest lab in KB having 99 SDs.
  • Limited support for MAC

2.    Trial

The mock exam took place as planned. In attendance were the Sponsor, another academic staff member, project manager, local computer support representative and Tom Mitchell from Intelligent Assessment Technologies.

Observations before the exam

  • Computer labs provide a different environment to traditional exam rooms. In this instance, the room was very warm and had a level of background noises due to the number of desktops.
  • Extra time should be allocated before the exam to ensure all students have the correct setup. Some SDs were switched off, and had to be powered on. Some had to be re-started to troubleshoot issues with user roaming profiles and keyboard mapping. Functionality in ExamOnline allows the time to be adjusted at the user level, even once the exam has started, and up to 10 minutes of the end the exam. This help in granting extra time to individuals affected by technical issues. Once logged in, all students were able to easily navigate to the application and start the exam.
  • More computers than students are required to provide an alternative machine in case of breakdown or technical issues during the exam.
  • The number of students was roughly a third of the lab capacity. However, natural distribution would have them grouped mainly in one block at the back of the room. They were quite close together, and students could easily see the screens of people sitting next to them. It is hard to tell whether they could or would want to read what is displayed on the next screen. Exam rooms specifically set up with more spaced out desktops then computer would be worth considering to reduce the risk of students checking each other notes.
  • Special bar coded paper was distributed to the students for the hand drawn diagrams. The barcodes enable the scanned copies to be linked to the student online exams.

Observations during the exam

  • Once logged in, all students were able to navigate to the application and start the exam. The link to the exam becomes activate at the exact start time. This created some confusion with students being unsure as to what link to select or whether the link worked after they clicked on the link right before it was activated. A countdown or clear wait/ready status would improve the interface.
  • All students cope with the online aspect of the exam very well. Typing was not an issue with most of them touch typing. They quickly took advantage of the online and the text editor features like enlarging the exam window to full screen, spell checker, increasing the font size, adding bold face titles, first planning their answer with bullet points then editing sections in a non-sequential order, moving or deleting text, or adding text within existing paragraphs. The spell checking was provided by the browser, student should be made aware of this ahead of the exam, and check that this functionality is switch on or off in their profile according to their choice.
  • Invigilator with administrator access to ExamOnline can monitor the activity during the exam. This includes the last time the draft was saved for each users, as well as the network connection to each desktop. As a test, the network cable was temporarily removed from one SD. The event was tracked on the interface. The student experience was unaffected. The exam was saved locally during the network outage and then saved remotely on the server when the network returned.
  • The exam terminated abruptly shortly before 3pm due to a general power cut on the KB site (http://reports.is.ed.ac.uk/alerts/index.cfm?fuseaction=view_alert&alert_id=6313) which took down the network and all the SDs. Little information was lost, only the last 29 seconds of typing but the students were unable to finish the exam. In this case, it wasn’t possible to change the end time of the exam because it happened less than 10 minutes before the end of the exam which is the cut off time after which adjustments are no longer possible. The power cut lasted close to two hours and it wasn’t feasible either to keep the students in the exam room. A small power cut, earlier the exam, could be accommodated by granting extra time to all students once the power is back, or from a external location with network access, and the exam could be resumed.
  • Online exams like any exams can be affected by unusual circumstances like fire alarms or power cuts. Procedures are in place to deal with this. However, they are likely to be more affected than paper based exams in case of power cut with SDs and network down as well as the lights. We note that laptops with suitable battery life would not be a workaround because the wifi routers were also down during the power cut. Loss of networking is an additional factor that doesn’t affect paper based exams, and a procedure on how to deal with this risk is needed. In this case, network access is needed to access the cloud based application and start the online exam. A network outage at the time when an exam is due to start will delay the exam, or could potentially cancel it depending on the length of the outage. Similarly, a network outage lasting past the end of the exam will require the local copies of the papers to be downloaded on USB drives and uploaded to the cloud server later.

Observations after the exam

The students were generally extremely positive about the experience.

They are digital learners and this is what they expect.

It’s cool!

Why can’t all exams be like that?

  • A few minor points were raised like the keyboard shortcuts didn’t work, or the spell checker wasn’t on. Most of these could be addressed with training and standardised setup.
  • The hand drawn diagrams were gathered. They were later
  • Marking can start once the student close the application.
  • Students were not able to fill in the online survey to provide feedback after the exam due to the power cut. A link to the survey was emailed to them at a later date.

The use of ExamOnline demonstrated the following:

Benefits Drawbacks
  • Improved digital experience for digital students (inc. text editor functionality) and liked by students.
  • More inclusive. Easier and seamless support for some for adjustment (inc. spell checking, font sizes). There is no need for students who need the support of a computer to be in a different room.
  • Resilient to network outage during the exam
  • Poorer environment conditions with noise and warmth from PCs
  • Additional SDs required
  • Requires larger rooms and space out desk
  • Higher risk of exposing answers to other users
  • Loss of networking can prevent the start of an exam
  • Increase risk of cancellation of exam due to power outage

3.    Post Trial

  • All students provided feedback through an online survey on their experience of using ExamOnline to take a mock exam. The results of the surveys are at: examonline_student_survey_part_1.pdf.
  • LTW EDE analysed this feedback and a comprehensive report is available at: ltw_ede_examonline_evaluation.pdf 
  • In addition, LTW EDE were granted access to the trial software and evaluated its functionality against the success criteria. Tom Mitchell from Intelligent Assessment Technologies was very responsive at answering questions raised about the software.
  • It was not possible to assess all the success criteria during this short evaluation. Unlike a formal procurement, we did not have access to all staff or information required. For example, affordability could not be measured against a budget as no fund is currently available for this. Similarly, assessing the SLA information would have required a support team to be involved but ownership of the service is not determined. In some cases, the evidence is only anecdotal and would benefit from a more extensive and formal evaluation.
  • The result of the evaluation shows that ExamOnline meets fully only less than half of the must have requirements. Some of the gap are serious, and potential show stoppers, such as error messages present during testing, lack of support for accessibility and poor support for EASE or group management. The full review of success criteria is available at: sce011-_success_criteria_-_ltw_ede_evaluation_-_1.0.pdf

 

Requirement Priority   Fully met Partially met Not met Not assessed
Must Have 21 9 4 7 1
Should Have 16 11 2 1 2
Could Have 2 2 0 0 0
  • Marking of the exams was done by the Sponsor and a colleague. While the individual marking time would be similar, both staff had access to all the essays and mark them simultaneously, providing a time saving and avoiding the second marker being delayed by the first marker. The follow time estimates were provided by the Sponsor:

  • The Sponsor indicates that the process could fit in with existing tools, i.e. the distribution of feedback could be integrated into our VLE and a spreadsheet of marks can be uploaded to EUCLID.
  • Providing access to external markers and moderation of the marking process was not tested during the trial.
  • Feedback was emailed to the students as an annotated PDF and provided within the 15 days limit.
  • ExamOnline can show any or all of the mark, comments and markers name in the feedback report depending on the options selected. Under service conditions the feedback would be provided within the application, but because the student trial accounts were only valid for the exam, the PDF was emailed to the students directly instead.
  • A second online survey was designed to capture the experience of getting feedback on the online exam, and emailed to students as per the first survey. In this case, only 13 out of 21 students filled in the survey. The results of this surveys are available at: PDF icon examonline_student_survey_part_2.pdf
  • Student found this method of getting feedback unanimously useful. The quality and details of the feedback was appreciated. The ability to comment or request further clarification on the feedback provided within ExamOnline would be welcomed. This could not be tested during the trial as mentioned before.
  • Students would have like to receive the feedback sooner. Releasing the feedback as soon as the annotations are made without the mark is supported by ExamOnline and would speed up the process. However, a majority of 77% would not like earlier feedback if it was without marks or marker names. The speed of feedback should still be faster than the current business process which precludes

 

The use of ExamOnline demonstrated the following:

Benefits Drawbacks
  • No issue with deciphering handwriting
  • All markers have access all papers simultaneously
  • Significantly faster marking workflow
  • Useful method/format for feedback
  • Better, more specific feedback given to student
  • Faster feedback than current process is possible
  • Feedback not fast enough for students

Project Info

Project
ExamOnline Trial
Code
SCE011
Programme
SCE Portfolio Projects (OTHSCE)
Management Office
ISG PMO
Project Manager
Muriel Mewissen
Project Sponsor
David Cavanagh
Current Stage
Close
Status
Closed
Start Date
20-Oct-2016
Planning Date
n/a
Delivery Date
n/a
Close Date
28-Aug-2017
Programme Priority
1
Overall Priority
Normal
Category
Discretionary