Closure Report

Project Summary

The stated objective for this project were:

In the project brief, the following deliverables were identified:

All of the deliverables and the objective have either been achieved or removed from scope as N/A - see below for explanation on D7 N/A.

D7 explanation - The UoE server is actually hosted by Amazon web servers, who maintain the availability across multiple data centres and sites. Therefore, it was deemed unnecessary to test. The UoE business continuity was included as part of this deliverable, but again, as the project developed and we more clearly understood how MBChB schedule and execute exams, it became clear that this process is already extremely robust. Local IT at Hugh Robson have a schedule which MBChB use to book labs abd additional rooms. They use existing methods to migrate all the kit from standard UoE build into an exam assessment mode (locked down using the project delivered configuration file) which is then tested before MBChB staff arrive. MBChB also overbook to leave themselves with some flexibility to absorb several local machine failures. If there was a catastrophic power failure, MBChB are likely to cancel the exam and reschedule. Note:- in these circumstances Practique would hold the exam status, such that once the power was restored, students could pick up from where they were interrupted. The project have supplied input into MBChB processes and will continue to do so.

Summary:- In a 2016 National Student Survey, the Medical School established that improving both the assessment and feedback of their student exams was their number one priority. The student experience is greatly enhanced when feedback can be provided at the question or topic level. Currently this is difficult without compromising the question bank. Hence, a new online assessment application is required to introduce a faster and much improved student experience, whilst enhancing MBChB administration efforts to support it.

The procurement phase was implemented with MVM107, the implementation and on-going project support of MBChB was implemented as MVM118, the data content was delivered by MVM119 and a new project has been initiated to expand the use of Practique with the implementation of OSCE (Objective Structured Clinical Examination ) MVM129.

Analysis of Resource Usage:

Staff Usage Estimate: 90 days

Staff Usage Actual: 116 days

Staff Resource Variance: 25%

Outcome

The Project introduced two piccls:-

1) scope and budget increase plus associated major milestone adjustment

2) scope and budget increase

Both were fully approved.

Explanation for variance

The project has been successful principally because of the efforts of the core project team, who have had to undertake significant extra work. The scope of the PM was much wider than usual, so extra work was required in several areas, not least around contracts, establishing the requirements, maintaining a visible project presence for the whole year and high level planning for MVM129 in particular.

Key Learning Points

The core Project team (academics, admin and PM) have continued to collaborate very positively in progressing both the implementation of a new exam assessment tool, Practique, whilst embedding and expanding the support and engagement processes surrounding it.

However, this has come at quite a cost to a small number of dedicated individuals! The continued goodwill of the core team (David Kluth, Jennifer Hill, Alan Gilchrist, Alan Jaap and Paul De'Ath) to work extended hours to develop, support and enhance the use of Practique throughout all medical years needs to be recognised. Additionally, the decision to embed the PM into the core team after the assessment tool had been delivered, whilst resulting in both a piccl (2) as well as a small budget over run, has ensured that both this project and the tool are being viewed a a success.  Therefore, this project would not have succeeded without that effort, much of which is still ongoing and that we need to be mindful of as we commence a new project MVM129 to deliver the OSCE assessment via Practique in 2018-19. The college were fully supportive of this and committed to it as their number 1 priority project.

See below two links to closure reports for two associated projects.

https://www.projects.ed.ac.uk/project/mvm107/completion-report

https://www.projects.ed.ac.uk/project/mvm119/closure-report

The question item bank now has over 5000 items, the school has executed in the region of 80+ exams (one technical issue on the very first exam resulting in a 20 minute delay to the start only - zero subsequently), has rolled out formative assessments to students on their own devices and in mid June 2018 will be hosting an assessment in Canada (Alberta University).

The core team have also trained the vast majority of academic and administrative staff who are all now using Practique.

Additional delivery:-

• defining and creating Practique user roles - access permission rights for Live and Test
• managing the single sign on via Ease - worked with supplier and IS development, tested, deployed and user id's all updated
• adding a time out - functionality to lock the user out of Practique, particularly required for NHS users
• creating and supporting roll-out of formative assessments on student own devices (ahead of supplier providing it - approx 10+ formative assessments executed) - we trialled it on a number of devices, wrote a technical support paper which the supplier reviewed and then rolled out successfully with almost zero technical issues. We were the first client to perform this.
• created training for LTW and assessment team for exam day support - the supplier have excellent support procedures for exam events, however, there was a need to quickly establish some checks for the assessment team and LTW (just this year) before the supplier were to be contacted
• managed engagement and academic/technical support for an exam to be held in Canada for an MBChB programme - the supplier has not rolled out to Canada yet but were prepared to support us. However, having established contact with local academic and IT support, the tool was assessed, pilot exams created and tested from UoE and locally in Alberta, through to formal approval that they would support the use of Practique
• handled all the GDPR and contract discussions surrounding Canada with Legal Services  - again the supplier had not yet rolled out to Canada, though no change was ultimately required within the contract, there were several meetings with Legal Services and the supplier before all parties signed off the approach
• delivered training to year coordinators and supported MBChB training to module organisers
• managed the relationship for the project with the supplier - extended this to monthly meetings, brought in assessment team - this will continue on into MVM129 and during that project departure from this role will be executed
• ran a strategy and planning meeting with the assessment team to consider the next three years planning requirements - this will continue to be developed as part of MVM129
• part of the University Practique user group - this group meet twice a year consisting of representatives from all of the supplier clients within the UK, such as Imperial, Leeds, Brighton etc
• submission to and presentations for UoE learning and teaching conference on June 20 https://www.ed.ac.uk/institute-academic-development/learning-teaching/cpd/workshops/learning-teaching-conference
• D6 - saves money, from both the servers and also that LTW are no longer required from 2018-19 from a support and development perspective
• Tested and deployed several release upgrades to test and live
• managed the procurement change for MVM129 - contract update which will ultimately be signed off as part of that project, all the preparation work has already taken place within this project

Outstanding Issues

There are no outstanding project risks or issues. This is partly due to the creation of MVM129 (OSCE delivery) which ensures that the existing core team (academic, administrative, MVM, IS) remain to support one another and the continued expanse of staff and students using the system.

https://www.projects.ed.ac.uk/unpublished/project/mvm118/risks/7 This risk articulates the issue of MBChB executing the type of structural changes that are required, but which the introduction of Practique has accelerated. I have closed it as part of this project, acknowledging that MBChB have made huge strides in this space. However, challenges do remain which the team are aware of, but note that it's mainly out of the power of this project to influence.

Further significant changes are likely to be 'anthropological rather than technological’, so we may introduce technological change, but we need humans to execute it, hopefully, we will determine what we want the technology to do, not the other way round!

IS news item from May 2018 - https://www.edweb.ed.ac.uk/information-services/about/organisation/applications-directorate/apps-directorate-news-events/more-success-with-practique-in-the-medical-school