Completion Report

Project Summary:

The stated objectives for this project were:

In the project brief, the following deliverables were identified:

All of the objectives and deliverables have been achieved.

In a 2016 National Student Survey, the Medical School established that improving both the assessment and feedback was the number one priority. The student experience is greatly enhanced when feedback can be provided at the question or topic level. Currently this is difficult without compromising the question bank. Hence, a new online assessment application is required to introduce a faster and much improved student experience.

The following activities, Training, Data Migration, Creating new questions, Implementation and Support of the new Exam and Student Feedback System, FRY-IT's Practique, will follow during 2017 via a number of additional projects, principally MVM118 and MVM119.

Analysis of Resource Usage:

Staff Usage Estimate: 70 days (original estimate before the estimate was revised up to 93 via two PICCL's - see below)

Staff Usage Actual: 96.7 days

Staff Usage Variance: 35%

Other Resource Estimate: 0 days

Other Resource Actual: 0 days

Other Resource Variance: 0%

Explanation for variance:

The original suggestion for resource commitment for a procurement project was 100 days. IS Project Services were challenged by MBChB to try and reduce that number. This request was supported by MBChB having created a set of business requirements already. Consequently, the early phase was rushed, without a considered review phase and the project consequently were locked into a substantive list of requirements that introduced delays further on in the project.

This project was on course to successfully deliver to the original schedule until the winning supplier was selected. The final deliverable, completing the contract signing, has taken far longer than planned. In part this was due to the following:-

• the initial terms and conditions were standard UoE
• the winning supplier offered a 'Software as a Service' solution
• a second terms and conditions was created by the UoE
• in conjunction with the supplier a third and final terms and conditions was created

These activities led to an unexpected increase in both the project duration and associated effort.

The original estimate for the project was 70 days, but the final figure is 97 days overall.

There have been two PICCL's for additional effort:-

1. 10 days in May - change#2. This brought the final approved total to 80 days.
2. 13 days - change~5. This brought the final approved total to 93 days.

The main differences between estimates and actual figures can be explained as follows:

• ITT reviews and submissions - The Project was ideally targeting three suppliers, but ended up with five. This, in conjunction with the high number of requirements, meant that to maintain the timeline for this phase, additional time was spent by all members of the core evaluation team.
• Terms and Conditions - The project faced a challenge here, as the current procurement process only makes the T&C's visible once the winning supplier has been awarded the bid. At this stage we encountered problems around additional payments which necessitated a full review of all bid costing's against  changed criteria. This was followed by lengthy issues between the supplier and their ready made SAAS contract and the University who had to create one.
• Contract - Additional payments were made by the UoE to facilitate the Legal teams from both parties engaging with each other over a period of several months.

Extended duration

See above.

Scope Changes:

There were no changes to scope recorded.

Key Learning Points:

Feedback from the project team at UoE, include the following, "the only concern was the resulting impact throughout the procurement process of having a high number of requirements" and "in hindsight we needed to review both the quality and number prior to publishing the ITT".

Feedback from the PM, consider the timing of when the T&C's can be viewed.  Currently, any issues with them will only reveal themselves once the winning supplier has been identified.

Feedback from the PM, consider the requirement for signing a contract. The project were not provided with an electronic option for signing the document. So, a physical contract has been signed by three different parties, with all of them requesting a digital copy at the end rather than a physical. The plan now, to create a pdf from the signed physical and then destroy the physical copies and maintain a e-copy only.

Positive Learning Points:

The project was initiated in late November 2016, with some significant milestones already reasonably 'set in stone'. This led to an approach with the PM taking a central role to expedite the project deliverables in as efficient manner as possible.  To support this approach a number of risks were identified and tracked, but all stakeholders responded positively to the approach.

• No regular team meetings were held
• Ad-hoc meetings were scheduled for the convenience of the MBChB staff with the PM acting as the conduit and repository of all project updates
  • these were held face to face to limit email communication - though all meetings were documented and hosted upon the project site
  • while access to the MBChB staff was a continuing risk, there was significant support to ensure the project hit major milestones EG supplier demos, procurement Award feedback, consensus meetings etc
• Procurement were able to respond to PM led questions with good detail in a timely fashion
• Floral and paisley attire were greatly appreciated and aided in the substantive engagement with MBChB
• Project communications were created and shared across the wider stakeholder community at each milestone
• PM took responsibility for all risks included in the Risk Log, ensuring speed of response across the project team
• Supplier led demonstrations were held very quickly after the ITT window closed
• IS teams supported the risk associated with limited engagement prior to the ITT and were all involved in the tender review process
• IS LTW maintained this 'hands off' approach by leveraging the PM to attend all consensus meetings 
• The suppliers were presented with a detailed overview of what we wanted them to cover during their demonstrations
  • the Project team were also able to supply an old exam, which enabled us to view both their import process and review quality data that was fully understood by the audience

Recommendations:

• Instigate a stakeholder review of the requirements with sufficient time to make changes before publishing the ITT
• This review should attempt to limit both the number of questions and ensure a clear separation with supplier feedback and demonstration assessment
  • Receiving a high (5) number of suppliers, with a high number of requirements (122) meant that it took longer than usual to populate Award and then provide feedback
• Peer review team should be confirmed earlier in the project timescales for ITT review
• Consider utilising AV facilities to support staff at alternate sites
• The project instigated an 80/20 weighting criteria technical/financial split. In light of a response from one supplier, it is believed that a previous engagement influenced their financial input. Although this did not impact the outcome of the winning bid, future projects should be aware of any such information being in the 'public' domain
• Review the point at which Project team can review T&C within the procurement phase
• Review options for generating electronic contracts for signing.

Outstanding issues:

At the time of preparing this report (August 28, 2017), there are no outstanding issues.