Closure Report

 

Project Summary

This project followed two previous years of work in implementing and embedding Practique's use in the College. At the start of the project this was viewed as a year of stabilisation, extending the practitioners/disciplines using the software and enabling the college to work towards the benefits of working from one consolidated platform. Overall this project was expected to:

  • have a positive impact on Student Experience given the feedback from students already taking some exams in Practique
  • provide a recurring benefit of reduction in administration time once the solutions are live
  • provide the College with more of its exams in one consolidated platform

This project built on the work already delivered over the last two years in:

  • MVM107 Procurement of online application
  • MVM118 Deployment of FRY-IT Practique for MBChB
  • MVM119 Data Creation and Migration into Practique for MBChB
  • MVM129 Support for OSCEs in Practique for MBChB

This project was originally due to deliver in April 2020 but with the Covid lockdowns and restrictions that took place from March 2020, the Vets OSCEs were not live until March 2021, although the Vets did run a pilot in March 2020.

At the end of 20/21, the University started an e-Assessment Procurement and Implementation project (TEL068). The BMTO team, having sufficiently piloted using Practique, and with already having to deal with all the additional work that Covid caused for the team, decided to de-scope using Practique for a live written exams and put their resources into working on the University wide solution being developed by TEL068 - see change control 6.

 

Scope

No Scope
1

MBChB Admissions

2

OSCEs for Vets

3

Written Exams for BMTO

The project largely remained in scope with the following caveats both noted in change control 6:

  • BMTO did not run a live written exam with students
  • Admissions Assessments for 20/21 were brought into scope of the project to provide project manager support to the Admissions team who had to establish a new approach the Assessment days to accommodate the covid restrictions and need to work remotely.  

 

Objectives and Deliverables

No Description Priority Achieved
O1 Gap analysis between current processes and Practique    
O1D1 Exam Fit analysis session for Admissions MUST Yes
O1D2 Exam Fit report for Admissions MUST Yes
O1D3 Exam Fit analysis session for Vets MUST Yes
O1D4 Exam Fit report for Vets MUST Yes
O1D5 Exam Fit analysis session for BMTO MUST Yes
O1D6 Exam Fit report for BMTO MUST Yes
O2 Implement new instances    
O2D1 Delivery of Admissions Instance MUST

Yes

O2D2 Delivery of Vets/BMTO Instance MUST Yes
O2D3 Training Day each for Admissions, Vets and BMTO MUST Yes
O2D4 Creation of content for Admissions, Vets and BMTO in Practique MUST Yes
O2D5 SSO to be implemented for new instances (as per current) SHOULD Yes
O2D6 Pilot for Admissions SHOULD Yes
O2D7 Pilot for Vets SHOULD Yes
O2D8 Support calls with supplier for Admissions, Vets and BMTO following respective training days to go Live MUST Yes
O2D9 Build of ipads for Admissions and Vets MUST Yes
O3 Delivery using Practique    
O3D1 Admissions Assessment Days for MBChB MUST Yes - this was completed for 19/20 as planned but also with project support for the 20/21 Admissions Assessment days being brought into scope as noted above and in change control 6
O3D2 OSCEs for Vets MUST Yes
O3D3 Written Exams for BMTO MUST Yes - however, this was as a pilot run by staff rather than as a live exam with students as noted above in Scope section and in change control 6

The project delivered the Objectives and Deliverables that remained in scope.

 

Benefits anticipated by supplier 

Our end-to-end e-assessment management service means you can have all your assessments managed from one integrated platform including Admissions/Structured Interviews, OSCEs, MCQs, EMQs, SAQs and VSAQs. Some key benefits are:  

Benefit for the student

Will these be achievable?

Give your students feedback more quickly and when it matters with  automatically generated feedback from exams

Considered achievable

Improve the outcomes of your students by having better quality feedback  that is automatically linked to their assessments

Considered achievable
For the faculty and leadership team  

Gain an overview of the assessment and student performance to drive  better learning

Considered achievable

Improve the quality and reliability of your assessments with inbuilt analytics  and reporting

Considered achievable

Access to new and innovative assessment types to ensure that you can trial  and introduce new assessment types as the medical assessment landscape  continues to change

Considered achievable

Remote access for all users allowing multi-site involvement, secure exam  bank management and remote question authoring

Considered achievable

Demonstrate to the GMC that assessments meet gold standard practice

Considered achievable
For the administration  

Reduce the need for administrative tasks and increase time spent on  activities that add value

Considered achievable

Remote access for all users allowing multi-site involvement, secure exam  bank management and remote question authoring

Considered achievable

Immediate results processing - Save hours of time with electronic results  capture and inbuilt analytics. Our customers have reported saving between  50 and 80 person hours per OSCE exam using Practique

Considered achievable

Benefit from the extensive experience of the Fry e-assessment team who  have been supporting thousands of candidates take OSCEs (over 50% of UK  medical graduates have sat a summative OSCE on Practique)

Considered achievable

Benefits anticipated by Sponsor and Senior User for Admissions

The introduction of a new admissions process, with the incorporation of face to face assessments, carries a risk in terms of credibility of Edinburgh medical school, and the ability to deliver the process.

The use of e-assessments is likely to confer significant benefits:

Benefit

Will these be achievable?

The lack of need for multiple paper copies of the assessment process improves security

Realised

The software is intuitive to use, and will help assessors focus on the assessment process itself, rather than concerns about the technology

Realised

It will allow rapid uptake of outcomes, and will speed up the process, allowing us to inform applicants of our decision earlier. We hope that this will increase conversion rates

Realised early informing of applicants

It will support assessors, giving them a positive experience, making them more likely to volunteer in the future

Realised

It will allow timely and uniform feedback to candidates, to a level agreed prior to commencement of the assessments.

Realised

It will liberate the admissions team from having to upload the scores from paper copies

Realised

It will reduce the risk of transcription errors.

Realised

Benefits were achieved for the Admissions team with the following note from the sponsor for the 19/20 Assessment Days: 

"This has been a positive experience for almost everyone involved, with excellent feedback from applicants and, in general, from selectors. The ipads worked well, and everyone I spoke to saw the major advantage of this method. So I think the opposite has happened in terms of credibility from what we feared."

The Admissions team is keen to re-introduce Practique for the admissions cycle 2022/3 entry as we were unable to use if for 2021/22 entry, for technical reasons. This will require further work with the team from Fry to ensure that all stations can be run online.

Benefits anticipated by Sponsor and Senior User for Vets

The R(D)SVS is aware of the many benefits experienced by MBChB colleagues having attended and discussed extensively prior to the project, and therefore anticipates:

Benefit Will these be achievable?

Significant time saving for the assessment team

Yes

Preparation activity – thousands (estimated 40k) of printed mark sheets previously used no longer need to be printed and stored.

Yes

Exam day activity - reduction in collecting papers and organising examiners

Yes
Post activity – will be automated therefore expect similar benefits to MBChB team including less stressful organisation, examiner experience benefits etc Yes

Due to immediate availability of data and reports after OSCEs take place, School will be able to spend more time reviewing the results before sending to students

Yes

Vets confimed that "it is working as we need it too" and "Practique is a great system and the support I’ve had has been fantastic ".

Benefits anticipated by Sponsor and Senior User for BMTO

Benefit Will these be achievable?

For 2018/19 there are approximately 4800 UG student course enrolments within BMS.  A new system would reduce risk and allow a more robust way of managing our current multiple choice question assessment and examination processes.

Would be achievable

There would be a time saving for both academic and professional services staff when setting and approving assessments as well as a reduction in time for returning feedback for some assessment types.

Would be achievable
Exam paper creation using data gathered within the system.  Currently not possible other than by manual keying in Word question banks – not possible at present as there is no resource within the BMTO to undertake this time consuming task. Would be achievable
Exam paper approval process within one system removing the use of email and password protected files Would be achievable
Remove risk of correct answer tags being left in exam papers. Would be achievable
A new online system would allow for standard setting which is not currently possible. Would be achievable
Delivery of timely, detailed and bespoke feedback to students on their performance in MCQs/EMQs Would be achievable
Potential to use one system to deliver online tests rather than using multiple different systems Would be achievable
Potential to move to online delivery of more exams/assessments rather than paper based Would be achievable
BMS UG students are a potential growth area within CMVM.  This investment would facilitate future growth Would be achievable

Benefit Realisation is noted as "Would be achievable". On completion of the BMTO work, the sponsor advised that "we are at a stage where we feel that Practique can meet our requirements for the administrative processes that lead to the production of the exam paper and ultimately the delivery of the exam. Nevertheless, our pilot with Practique was in the context of it being a CMVM embedded system that we could evaluate knowing that the University QMP system could not meet our needs. The emergence of the TEL068 project has changed this context somewhat."

 

Success Criteria

Success Criteria Achieved?

Admissions: Would use again - Successful use of Practique for MBChB's Admissions Assessment days. Given this is new for the Medical School, it is expected that there will be opportunities to learn for following years - what went well and should be maintained, what could have gone better and should be changed for next year.

Yes - Paused for logistic reasons for 2021/22 admissions. Keen to use again for 2022/23 admissions

Vets: Would use again - Successful use of Practique for OSCEs. Given this is new software for Vets, it is anticipated that the School will wish to expand its use

Yes

BMTO: Would use again - Successful production of August written exam using Practique. It is anticipated that if successful, the School will wish to expand its use.

Yes - As noted above, having piloted a written exam in Practique, the BMTO prioritised its resources on TEL068. If there was not a University-wide solution being procured and implemented in TEL068, or if TEL068's solution does not meet their requirements, then BMTO would consider expanding the use of Practique. 

Overall, the experience of the pilots in all three areas was successful both in terms of the software and in working with the supplier.

Analysis of Resource Usage:

Staff Usage Estimate: 35 days in the Project Brief, 40 days agreed budget at Closure

Staff Usage Actual: 35 days

Other Resource Estimate: N/A

Other Resource Actual: N/A

Other Resource Variance: N/a

 

Explanation for Variance

While it would appear from the staff figures that the project has delivered to budget, it is worth noting that when the project brought the 20/21 Admissions Assessment days into scope, the budget was increased from 35 to 40 days to accommodate this additional work (see change control 5). Having delivered in 35 days, the project is under-budget from what was estimated to be required to reach closure.

The College had agreed costs with the supplier for the work this project set out to complete. Following the changes that the College teams had to make to adhere to Covid restrictions, the project manager negotiated a slight reduction in licence costs for 20/21 with the supplier.

Outcome

Admissions - delivered successfully for 19/20 and was largely due to the preparedness of the Admissions team and the thoroughness of their requirements for the system and how the Assessment days would run. For the 20/21 Assessment days, the Admissions team created new processes and ways of running the days digitally and these worked successfully. The support of the Unitemps service providing Circuit Managers was important to the success and the IT staff in supporting the use of Collaborate.

BMTO - Sufficiently piloted the Practique software to understand what impact of its use would be

Vets - Practique working as needed and considered a great system

 

Key Learning Points

Ref Date Title Impact Theme Stage
1 20-May-2021 BMTO - Sharing actual examples of wrtten questions before Exam Fit days would have been beneficial Was detrimental Business Analysis Close

Outstanding Issues

No outstanding issues for this project but worth noting that at programme level, contractual work is being carried out for the use of Practique in 21/22 for those areas that will not have enough time to migrate, regardless of the outcome of TEL068.

 

 

 

 

Project Info

Project
Practique Roll-Out
Code
MVM136
Programme
MVM Learning and Teaching (MVMLAT)
Management Office
ISG PMO
Project Manager
Susan Ridder-Patrick
Project Sponsor
Susan Rhind
Current Stage
Close
Status
Closed
Project Classification
Grow
Start Date
20-Sep-2019
Planning Date
10-Oct-2019
Delivery Date
07-May-2021
Close Date
21-May-2021
Overall Priority
Normal
Category
Discretionary

Documentation

Close