Closure Report

Project Summary

 

Background

This project was to procure a new Software-as-a-Service (SaaS) application for a Work and Study Away team (as part of WSA Programme and TOM project under SAS004 - https://secure.projects.ed.ac.uk/project/sas004)

This followed a market consultation exercise which informed the programme Business Case approved at March 2018 Student Administration and Support Programme Board.

Milestones

Delivered Date

Comments

Planning Completed

Yes

30/03/18

 

Procurement Strategy Finalised and Approved 

Yes 04/04/18  
Communications Strategy Approved (SAS Aligned) Yes 30/03/18  
Requirements, Cost Scoring and Weighting Finalised Yes 04/07/18 Most elements finalised at 27/6/18 workshop with evaluation group members, but some amendments made to technical requirements with various IS leads up to 04/07/18
Invitation to Tender (ITT) Published Yes 06/07/18 Published on PCS website.
ITT Notice Period Expires Yes 06/08/18 Automatic milestone driven by date ITT published. At expiry, 2 responses received for consideration. AWARD system updated to allow evaluation group members to score responses.
Scoring Validated - Consensus Meeting Yes 18/09/18 Validation session held 06/09/18 with evaluation group members. Scoring report updated with procurement lead and passed to Sponsor for approval. Procurement recommendation report on 24th September formally recommended preferred supplier to SAS Programme leads.
ITT Evaluation Period Ends Yes 28/09/18  
Contract Intention to Award Letters Published Yes 15/10/18 Intention to Award (and Regret) letters issued after standstill period.
Contract Signed Yes 03/01/19 Final signature applied 3rd January 2019.
Project Handover to Implementation Yes 07/01/19 Handover completed with Jamie Thin 04/12/18. As that was before contract signed, milestone only closed once that had happened, in case any further handover required.
Contract Start Date Yes 03/01/19 Determined by last signature date, as contract signing triggers first instalment in payment schedule.
Project Closure   17/01/19 Sign-off meeting scheduled 17/01/19

Analysis of Resource Usage:

IS Staff Usage Estimate: 98 days

IS Staff Usage Actual: 132 days (of which the largest elements were Senior BA, PM and Technical Lead in that order)

IS Resource Variance: 34%

Other Resource Estimate: £0 - Time committed by procurement lead(s), business lead, Sponsor and evaluation/scoring group members not included in project costs.

Other Resource Actual: £0

Other Resource Variance: 0%

Outcome

  • Following a successful evaluation, UoE awarded a contract to the preferred supplier to supply their software as a service.  Thanks to the entire project team for their effort in making this happen.

Explanation for variance

  • Technical Lead - More time was spent by IS Technical Lead than originally estimated as, although standard set of questions were used, these required review with a number of IS leads against the functional requirements. In addition, the volume of scored and mandatory questions meant that evaluation was a time-consuming exercise, even for a limited number of ITT responses. The standard question sets should be reviewed with Procurement, to give clearer guidance on the appropriate level of question for the estimated contract value, i.e. so that same question set is not used for systems orders of magnitude apart in cost. This will make it easier both for vendors to respond to, and for evaluation groups to asses, while still complying with Procurement regulations.
  • Business Analysis  - BA days were significantly more than estimated. In part, this was due to more analysis work being required to turn high-level requirements into ITT-appropriate requirements that could be reviewed by the evaluation group. However, a large number of BA days were attributed to this project when, in reality, the work related to the WSA TOM project, i.e. service review. As the BA role overlapped both projects, it was impractical to distinguish time spent on one alone, and all BA time was allocated to SAS012. As the overall programme BA costs would be the same, this was accepted by the programme manager.
  • Business Case System Cost - The WSA Programme Business Case had estimated lifetime (6yr) system supplier costs, based on implementation costs, Y1 licence costs, and then 5yr annual costs (i.e Y2-Y5 inclusive). These estimates assumed VAT included. Once ITT responses had been received, and contract agreed, the actual lifetime system supplier costs were finalised. Although year 2-5 revenue costs were slightly higher than projected andescrow p/a costs were also required, the bulk of the additional cost was attributable to supplier development costs in Y1 to meet the detailed requirements, including integration and workflow, in the UoE ITT. Y1 licence and implementation costs were largely in line with the Business Case, but it had not allowed any contingency for development work.

Key Learning Points

  • Technical Scored Questions - The functional questions were pitched at the right level, and the right number of questions. However, the volume of non-functional questions (particularly weighted/scored) meant that evaluation of those was more time-consuming than estimated. This was the feedback both from suppliers responding, and from members of the evaluation team.
  • Prior Information Notice (PIN) - Market research, and Prior Information Notice, indicated that likely response rate would be 3-4 suppliers. In reality, only 2 suppliers responded, and one of those was rejected for failing to comply with mandatory questions (for each of functional, technical and procurement categories). After PIN issued, a further follow up with registered interest suppliers could have been engaged as part of market research. If done  UoE might have understood there was only one supplier that could meet requirements, which would inform us with different procurement route so as to avoid full OJEU tender process..
  • IS Technical Question Set - Although existence of standard IS question set saved significant time in agreeing technical questions (and reduced related risk), there was not a clear process for amending the default questions for specific project.
  • Contingency - Greater amounts of cost contingency should have been built in to the Programme Business Case. Although the market research that informed the Business Case was not inaccurate for system costs in terms of licences and supplier implementation, it did not allow adequate contingency for development and configuration work required to get the preferred supplier system integrated with UoE systems (e.g. EASE for user authentication, EUCLID/SITS for student courses, or MyEd for student/staff access to system). It also did not allow sufficient contingency for internal UoE costs in implementing the system.
  • Contract Visibility - Although the procurement recommendation report was generated in September 2018, the finalised contract which reflected the same costs was not available until December. The project manager should have ensured that the senior managers who would be signing the contract were aware of the costs projected by the recommendation report, and the impact of those on the overall programme business case, rather than waiting until the contract draft had been finalised.
  • Engagement - In addition to IS and project team members, the procurement evaluation group included representatives from LLC, Edinburgh Global, SPS, Engineering, Business School, Student Reps, Chemistry, Geosciences, Education, MVM, and the Careers Service. These were selected to represent a broad section of schools and departments with whom students seeking to work or study away would engage. Beyond that direct engagement, the project communicated its aims as part of the larger programme communications strategy, so all schools should have been informed.
  • Governance - The project did not create a distinct Project Board, instead relying on governance through overall Programme Board. A more direct Project Board could have ensured broader engagement, and provided governance decisions more quickly.
  • Risks - The project should not need evidence that a risk exists before adding it to the risk register and status report.

Outstanding Issues and Risks

  • None noted - the implementation will be a separate project (SAS016 - https://secure.projects.ed.ac.uk/project/sas016)
  • All Risks closed with contract signing. Specific contractual obligations (e.g. provision of insurance certificates; evidence of compliance with accessibility criteria) transferred to SAS016.

Project Info

Project
Work and Study Away - Procurement
Code
SAS012
Programme
Service Excellence - Student Administration and Support (SAS)
Management Office
Service Excellence PMO
Project Manager
Ranald Swanson
Project Sponsor
Chris Yeomans
Current Stage
Close
Status
Closed
Start Date
13-Mar-2018
Planning Date
07-Apr-2018
Delivery Date
07-Dec-2018
Close Date
21-Jan-2019
Overall Priority
Normal
Category
Discretionary

Documentation

Close