Closure Report

Project Summary

This is the fourth project that we have undertaken with MBChB over the last 2.5 years to introduce a new assessment tool and then extend it's use within the School.  This project focused on delivering OSCEs (Objective Structured Clinical Examination) for the academic year 2018-19 extending MBChB’s use of Practique beyond written exams. Utilising the same team who deployed Practique seamlessly into MBChB  will ensure that the continued development of support for our student body and the MBChB assessment team, academics and administrative functions continues.

Previous completed projects that should be referenced are:

https://www.projects.ed.ac.uk/project/mvm107  Procurement

https://www.projects.ed.ac.uk/project/mvm118  Implementation, contract and support

https://www.projects.ed.ac.uk/project/mvm119  Data migration and creation

The stated objectives for this project were:

In the project brief, the following deliverables were identified:

Both of the objectives were met and all of the deliverables were either fully completed or in the cases of D6, D10 and D12 sufficiently complete to enable closure of the project.

D6 explanation:- We identified additional blueprint elements and successfully implemented them. However, the reason for partial is that the product currently does not have the capacity to host two separate blueprints. Therefore, this has been raised with the supplier and they are investigating it. This will be managed outside of this project by the assessment team and the supplier relationship manager.

D10 explanation:- Whilst the project have fully supported several releases during the last year, it has become clear from a recent bug being identified, that the scale and scope of testing within MBChB and Fry-IT needs to expand. CMVM IT are now assisting MBChB to develop test scripts to aid future release testing, whilst Fry-IT have refreshed their test model. This will continue to be managed outside of the project.

D12 explanation:- The project team have provided information that MBChB will use to develop a formal strategy. This will be developed further outside of this project by the sponsor and MBChB team.

Summary:- In 2017-18 Academic year, having procured Practique, MBChB ran all their written exams on the new tool. This project was designed to extend MBChB's use of Practique to running all OSCEs (Objective Structured Clinical Examination) and maintain all written exams in 2018-19 Academic year. In parallel there were a number of additional objectives and deliverables that meant the PM remained embedded within MBChB for the whole year.

As of June 18, 2019, MBChB have executed 22 days worth of OSCE's successfully via Practique, whilst maintaining a successful year of executing written exams, with zero failures. It is anticipated that there will be in the region of 3-4 further days required to support OSCE resits this year.

Analysis of Resource Usage:

Staff Usage Estimate: 62 days

Staff Usage Actual: 67 days

Staff Usage Variance:  8%

Please note, the only ISG resources on this project were that of the Programme and Project Manager.

Additionally the project had access to colleagues throughout CMVM, whose time was not estimated or tracked.

Outcome

The Project introduced two PICCLs:-

1) scope and budget increase https://secure.projects.ed.ac.uk/unpublished/project/mvm129/issues/1 In July 2018 the scope was extended during the planning phase, with an increase in the accompanying budget.

2) issue with closure milestone https://secure.projects.ed.ac.uk/unpublished/project/mvm129/issues/2 In February 2019 the project responded to ISG management question (from WIS) on the closure date following the successful conclusion of a major milestone. The response noted the concern, highlighting the remaining scope items that were still being developed. The sponsor had requested that the PM remain on the project for several further months.

Both were approved.

Explanation for variance

There is minimal variance in both budget and milestone timelines. This is principally due to the on-going efforts of the core project team, who have maintained their commitment to one another and the continued development of Practique within MBChB with focus and energy. The scope of the PM remained wider than usual, so extra work was required in several areas, many of which were captured in the project brief, based as it was on the success from the earlier projects identified above.

Key Learning Points

Having established a core Project team (academics, admin, CMVM IT and PM) in early 2018 on MVM107 we have continued to collaborate very positively in all aspects of Practique, whilst embedding and expanding the support and engagement processes surrounding it. This and the excellent relationship that we have generated with the supplier Fry-IT, is the main reason for the continued success of this series of projects. The sponsor, College and project team were keen to continue building upon the success of year 1. The remit of the project was configured to enable the ISG Project Manager (PM) to remain engaged with the team well beyond the implementation of the OSCE software (including training, day 1 support etc), to provide a more holistic level of support and team engagement.

1. The team (ISG PM, CMVM academics, MBChB assessment team, CMVM IT) quickly established a strong bond who supported one another and were prepared and dedicated to the strategy enabling work to be developed alongside and outwith traditional project deliveries. This approach requires clarity and honesty with the sponsor and key stakeholders to ensure a sense value is retained. Certainly, ISG being visible after an implementation of software/hardware was seen as very positive and perhaps can be considered in the future.
2. A similar engagement was sought with the supplier Fry-IT, given that they were a small company it was appropriate to quickly establish a strong engagement with them, using positive communication. The openness and high involvement of the PM has strengthened the commitment and understanding between both organisations. Note, Fry-IT continue to develop their software and have extended it's use outside of the UK, so do not assume that high quality responses and commitment just occur, the relationship needs to be worked at. MBChB are certainly stepping into this space, but the PM did take a lead and that was appreciated by both MBChB and Fry-IT (also legal services and procurement previously).

Closure reports for three previous projects:-

https://www.projects.ed.ac.uk/project/mvm107/completion-report

https://secure.projects.ed.ac.uk/project/mvm118/closure-report

https://www.projects.ed.ac.uk/project/mvm119/closure-report

The core of this project was to extend the use of Practique within MBChB by running all the OSCE (Objective Structured Clinical Examination) assessments alongside continuing the execution of all written exams in the academic year 2018-19.

During this year (as of June 18, 2019) MBChB have executed 22 day's worth of OSCE assessments with zero cancellations and no impact to any written exams. MBChB anticipate a further 3-4 days will be required to execute resits. In parallel MBChB have continued to expand contributors and support staff across both written and OSCE's.

Additional delivery:-     

1. Extended deployment of  Practique from FRY-IT - cloud based servers hosted on Amazon Web Servers - to execute all OSCE exams plus the existing written paper based assessments throughout MBChB, ensuring 100% of these exams were successfully executed, with zero cancelled or postponed.
2. Continued provision of training for MBChB users, examiners, assessment team and CMVM support staff. MBChB are fully capable and do plan and perform training across Practique for all new users of the system as well as existing users especially when new functionality is introduced.
3. Configuration and setup of UoE platforms that will support running the assessments. Over 50 iPads were purchased along with two charging trolleys (that are also used to safely secure the devices when not in use or when being transported to the clinical assessment centre at the Western from Little France). In parallel CMVM IT have configured a Mac to manage release upgrades to these devices. It is anticipated that this will be handed over to MBChB directly next academic year. 
4. As part of the new contract, the team managed a training session for the project core team on July 10, 2018 from Fry-IT for OSCE.
5. A pilot was successfully implemented to enable MBChB examiners to assess Practique prior to exams.
6. The project maintained clear communication with all stakeholders for the duration.
7. Exam day preparation and day one support was provided by the PM, CMVM IT and Fry-IT.
8. The team continued to support MBChB change - not just the technical elements surrounding the introduction of a new exam application, but the structural, process and people based change that emerged alongside the technical delivery. 
9. The Practique User Group was already established as part of the initial roll out of Practique, however attendance has been reviewed and updated to make it more effective for MBChB going forward - these colleagues will work with FRY-IT to shape future releases, MBChB requirements and review all change emerging across the platform,  this group meet approx twice a year consisting of representatives from all of the supplier clients within the UK, such as Imperial, Leeds, Brighton etc.
10. As more colleagues are using Practique, there is a need to support developing requirements for improvements, MBChB are fully engaging with their community to identify new functions etc.
11. The relationship for the project with the supplier has been extended to ensure the right MBChB colleagues are now in place rather than project team members. 
12. The project team hosted two sessions to identify content required to allow MBChB and the sponsor to develop a strategy and planning approach for the next three years use of Practique. This remains ongoing for MBChB.
13. The project team supported the sponsors submission to and presentation for UoE learning and teaching conference on June 20, 2018 https://www.ed.ac.uk/institute-academic-development/learning-teaching/cpd/workshops/learning-teaching-conference
14. Continued support of all testing for future releases (now on V6.2), as noted previously, CMVM IT are further developing test scripts to aid MBChB in the future. This will continue to be developed out side of the project.
15. The current blueprint, particularly within the definition of skills, was updated, but crucially the product is not yet able to support two distinct blueprints. This has been raised with the supplier who are investigating. 
16. In parallel the written exams introduced a new question type - Very Short Answer Question (VSAQ) - alongside MCQ's successfully. It is anticipated that MBChB will continue to explore options within Practique that match where the programme develops.
17. Supported the MBChB in constructing, testing and then creating the new guidelines for module organisers to create new questions.
18. Continue developing engagement within the college to review the possibility of using Practique - clinical brain science, Vet School, BioMedical Teaching and admissions all received various presentations and trials from the project team. As of June 2019, the last three (Vet, BMTO, admissions) are generating CMVM project requests to purchase and use Practique within the college.
19. Manage contract update with Procurement, Legal Services and suppliers for initially the addition of OSCE and then a further two year extension.
20. The decommissioned scanners were re-purposed by the Vet school.
21. Due to limited wifi availability at the clinical assessment centre, several apple extenders were tested and purchased to improve wifi capacity. Procedures were also developed to ensure MBChB staff were aware how to manage the iPads at the end of the day with regard to charging and being online to receive new exams on a daily basis.

The team have observed a number of reasonably measurable improvements as a consequence of this project:-

• Significant time saving for MBChB assessment team
  • Preparation activity - historically up to a week printing generating 15 boxes of material was reduced to minimal printing (hours) and less than 3 boxes required
  • Exam day activity - reduction in collecting papers and organising examiners - members of staff now have breaks and the staff numbers have been reduced who attend
  • Post activity - historically up to a week required to input scores/comments and generate reports; this is now automatic within the system
  • Overall, the environment is much less stressful for the team and that ripples out to both the students and the examiners
• Examiners have expressed very positive feedback for the user interface that they use to assess students
• External examiners have commented favourably on the advances and ease of use of Practique - indeed they wish to access more (we've initiated a change request to assess for a new role to enable that)

Outstanding Issues

There are no outstanding project risks or issues outside of the three deliverables only partially completed (explanations above).

However, any future engagement with Fry-IT should consider that they remain a relatively small company (in terms of capacity/head count), who are continuing to develop the tool (with frequent input from UoE and the wider community) whilst growing their user base beyond the UK. The high level of responsiveness we have had over the last couple of years requires work to manage. This has frequently been driven by the PM and the MBChB assessment team lead. That has worked but with the PM dropping out of the programme, thoughts on how best to maintain and grow this relationship, particularly if other areas of CMVM employ Practique, should be considered.

Archive:-

June 19, 2019 close report V1 mvm129_close_report_v1_june_19.pdf

Attachment	Size
June 19 close report	35.34 KB