Closure Report

Project Summary

The stated objective for this project were:

In the project brief, the following deliverables were identified:

All of the objectives and deliverables have been achieved.

In a 2016 National Student Survey, the Medical School established that improving both the assessment and feedback of their student exams was their number one priority. The student experience is greatly enhanced when feedback can be provided at the question or topic level. Currently this is difficult without compromising the question bank. Hence, a new online assessment application is required to introduce a faster and much improved student experience, whilst enhancing MBChB administration efforts to support it.

The procurement phase was implemented with MVM107, the implementation and on-going project support of MBChB is held within MVM118 and this project (MVM119) delivered the data content.

Analysis of Resource Usage:

Staff Usage Estimate: 30 days

Staff Usage Actual: 33 days

Staff Usage Variance: 10%

Other Resource Estimate: £0

Other Resource Actual: £0

Other Resource Variance: 0%

Outcome

The project introduced one PICCL for additional budget request of 10 days which was fully approved.

Explanation for variance

 N/A.

Key Learning Points

Feedback from the PM and agreed with by MBChB:

A)the risk associated with the delay in signing off the contract (MVM107), was additionally compounded by the loss of several members of the MBChB administration team, which led to the PM being utilised across a number of guises more fully than had been planned especially with regards to content import and checks. PM's are frequently found in difficult situations where they can become the 'expert'. In this case, sponsors, programme and project team were aware of the situation, but the project sensibly appeared to have no alternative. I do agree with that, it just needs to be noted, because there will also be a lot of goodwill credited to these actions from one section of MVM, whilst another sees delay (MVM522).

B)the contract delay resulted in a situation where the project were unable to engage with the implementation team within Fry-IT. Although we pushed as much as we felt we could at the time, it became obvious once engagement fully commenced that the task of the bulk import was hugely significant, with around 5,000 items (detailed medical questions, resources, images etc) needing to be imported with high levels of security associated with the sharing of the content. Whilst some MBChB year directors had made significant progress in reviewing content whilst it still resided within OSCA and OSCA+, others were less forthcoming. We had to make a regrettable decision to acknowledge that we would import content that would later be removed, simply to make progress. It's difficult to see how we could have navigated this any other way.

C)the project team made a clear statement to stay in control as long as possible with regards to system access. For these purposes the PM was unofficially regarded as a member of that inner circle, with the knowledge and access to assist in importing, creating content within the system (based on MBChB documents) and generally being a central member of the team. It was very difficult to plan that period, and whilst a good decision which no doubt has helped with the implementation activity, it should be noted that the cost may not have been signed off as a pre-cursor to the project as a planned activity. A future consideration perhaps?

Positive Learning Points:

MVM107 was initiated in late November 2016 to deliver a new online system to support MBChB exams and provide the ability to generate and deliver quality feedback to our students. In 2017 two further projects were created to better manage the delivery of this new system, Fry-IT's Practique having won the procurement, MVM118 broadly covering the implementation, support and eventual decommission of OSCA and OSCA+ and this project MVM119 focusing on the data content within the system.

Fry-IT were able to deliver clear requirements for the import - column names, special characters etc. This enable IS LTW teams to develop export scripts to retrieve the existing content from OSCA and OSCA+. The export phase worked very well.

Once we had obtained the existing content, UoE teams, mainly technical, reviewed the content to try and reduce the amount of duplication and generally clean up the files (carriage returns, apostrophes etc) so that the files were ready to import. This was quite a laborious task and with hindsight, we could have revisited the extract scripts to establish whether these could be enhanced.

The first significant gap emerged here, as access to MBChB teams who could review the quality of the questions and associated content was minimal. This led to the decision to import more than we had anticipated and to conduct more of the data cleansing phase within Practique.

MBChB generated a brand new blueprint framework, which was reviewed and approved. This was an early decision that has to be right, as the bulk imports will utilise this structure from the outset. It has been constructed to map with Medical School definitions.

Both the mass bulk import (conducted by Fry-IT) and the subsequent adhoc imports performed by the Project team worked very well. This phase was certainly longer than planned, mainly due to the adhoc nature with which MBChB team members responded to the Project and Sponsor, EG, we were more reactive than we had wanted to be.

We held a couple of pilots - genuine questions, exam like conditions, students and support staff on-hand to fully test both MBChB processes and the software. The software was a complete success. The processes all worked, but we recognised that there would be some tweaks, but also potential to positively influence some of them in the future.

We hit all the major milestones, ensuring data content was ready, approved and built into exams (review and approved) and as of November 10 there have thus far been seven successful formative exams held.

This really supports separating the deployment of the application (MVM118) from the data content (MVM119) as the Project can now be closed cleanly.

In parallel the sponsor and MBChB staff created, shared and agreed with Year Directors a template for generating new content, along-with a clear schedule of delivery so that the Project team could import them. Subsequently, the Project team have created a document and an excel template that could be used by MBChB administrators to import significant numbers of questions straight from the spreadsheet rather than manually adding.

All the data resides in the system for the remaining formatives and MBChB are on schedule for all the summative exams too.

Recommendations:

The Project team and MBChB already have access to and are using Fry-IT's service desk. This allows us to report both issues but also make recommendations for future releases. MVM118 as the implementation project will cover the vast majority in detail. For MVM119 there are minimal recommendations to make:-

• MBChB will continue to roll out training and provide support for staff access to generate, review and approve new data content.
• An on-going review of the quality of resources, mainly images. We can host improved quality images, so should seek to replace images as we proceed.

Outstanding Issues

At the time of preparing this report (November 10, 2017), there are no outstanding issues. However, although an OLA has been created and agreed with IS LTW teams, this deliverable should really be contained within MVM118 where it will be reviewed once again as part of the closure of that project.