Closure Report

 

Project Summary

The new Exam Timetabling service went live in June 2018 ahead of the start of scheduling of the August 2018 resit exams.

The new solution has enabled personalised exam timetables to be sent to all students via their Office365 calendars. That has been the big win , to improve the student experience and save students time creating their own exam timetable.

As part of the new end-to-end solution, the project delivered:

- a new solution for Exam Scheduling - Scientia Exam Scheduler v3.5.9 (hosted on ISG on-premises infrastructure) 

- a new data feed from Student Systems to Exam Scheduler (Scientia Connect) 

- a Business Objects BI suite universe and customised exam reports

- SSRS standard exam reports (from Scientia)

- a python web application to publish exam web timetables

- a feed to a new instance of Calsync to send personalised student exam timetables to Student's office365 calendar

- associated documentation and agreed business processes to manage the new solution

--

Students have also got a new exam notification service via the MyEd portal , as part of the notifications backbone project

--

The actual delivery schedule was a change from the original Service excellence outline business case which envisaged a 6-month implementation project.

For a project of this complexity and size, a more realistic timeline was 6 months to build the solution, 6 months to complete integration testing and UAT, and then an extended post-go-live period of support from the project team.

A critical success factor for this project was re-negotiating the go-live date with the SEP SAS board and other senior stakeholders and moving it from Jan 2018 to June 2018.

This provided additional time for testing and bug fixing to get a robust solution, and also greatly reduced the risk profile of the project , as it to moved from a high risk big bang go-live during the largest Exam diet of the year ( April / May 2018 exam diet) to a more manageable live pilot running for the smallest exam diet of the year ( August 2018 resits).

This reduced the risk of disruption to critical business activities and where workarounds were required , the volume of manual workarounds were manageable due to lower volumes of students on exams during the resits. Typically 10 to 20% of students have to resit exams each year.

Service Status at Closure on 4-Nov 2019:

The closure of SAS005 was approved by all stakeholders.

Note: SAS005 is closing with a known availability and performance issue for Scientia Exam Scheduler v3.5.11. (2 concurrent users)

The business lead – Alison Ramsay – has requested that further work is progressed to improve availability and performance beyond the closure of the SAS005 project. This has been accepted by IS Apps Production - and IS Apps Production are doing an on-going investigation , with a proposal to delete old history data using the Historical data removal tool provided by Scientia. Removing some historical data may improve performance, however some of the availability and performance issues may be constraints of the Scientia software , and so this issue has also been escalated to the Scientia account manager and product manager.

SAS005 is also closing with a known performance issue with BI Suite when running Exam reports. This may be an issue with the BI suite platform or a specific issue with the Exams BI suite universe.

--

Major changes in project scope from Business Case to Delivery included : 

1] Build of new work flows to support import and data mapping of the exam adjustments for students needing extra provision. This is a complex area which is an integral part of the end-to-end solution and impacts over 3,000 students - yet it was not identified and costed as part of the business case ( additional unplanned essential work). This added additional consultancy costs from Scientia , +10 days paid consultancy from the Supplier, +20 days testing effort from the UoE project team

2] Additional reporting business requirements were identified during planning (not costed in the Business case) and added as 50d effort during planning. The initial  estimate of 50d effort was light - real costs was over 100d. The estimate of 50d assumed that ISG would only build a couple of exemplar reports and the remaining reports would be built locally by a report writer in the Exams team. However there was no resources to either build or test the Exams Bi reports within the Exams business team - so this work was pushed back to ISG.

ISG reporting effort included the effort to build flattened Database views, a Bi Suite universe and 12 BI Suite reports - to support the end-to-end exams business process.

--

Summary of What components were  delivered and When

 

No.

Description

Priority

Owner

Delivered When Comments

ID number

Output / outcome / benefit / strategic objective

MoSCoW

 

     

O1

Provide a comprehensive examination/resit timetable managed in Exam Scheduler v3.5.4 (For those students whose exams are already currently centrally scheduled).

An upgrade of the exam scheduling software and associated reporting functionality and the move of these services  onto a resilient IT platform.

 

Must

Alison Ramsay

Yes July 2018  

D1

Install Scientia Exam Scheduler on DEV , TEST and LIVE environments

Must

Jamie Thin

Yes July 2018  

D2

Install Scientia Exam Scheduler standard reporting package (SSRS) , to provide equivalent reporting with no loss of service and to provide staff view of Exam Timetable ( possibly via BI Suite). Evaluation of the SSRS reporting functionality against other reporting options including BI Suite

Must

Jamie Thin

Yes July 2018  

D3

Essential enhancements to reports where gaps identified after analysis

Should

Jamie Thin

Yes in multiple releases to BI Suite reports from July 2018 to Feb 2019  

D4

Investigate BI Suite for delivery of reports  in addition , or as alternative to SSRS , if there prove to be limitations with the SSRS package.

Should

Jamie Thin

Yes in multiple releases to BI Suite reports from July 2018 to Feb 2019  

O2

Publish personalised exam timetable to each student via Office 365 calendar.

Must

Alison Ramsay

Yes from July 2018  

D5

Build and test feed to Office 365 via existing Calsync tool

Must

Jamie Thin

Yes July 2018  

O3

For those students whose exam type is not currently centrally scheduled ( Class exams) , an investigation into the barriers to the use of this service. – Transfered to SAS006

 

Won’t – Out of scope for SAS005 , transferred to SAS006

Joe Brown ( as part of SAS006)

No  Out of scope  

O4

Integration of the software with Student Systems ( EUCLID)  to stop the requirement for the manual duplication of the data.

 

Must

Jamie Thin

Yes July 2018 New EUGEX views created and then new workflows created by Scientia in the Scientia connect product to import the Student and Student on Exam data into Scientia Exam scheduler

D6

Install and Test Scientia Connect (with consultancy from Scientia Ltd)

Must

Scientia consultants

Yes July 2018  

O5

A change to business processes / organisational hierarchy by having exam timetabling carried out within the larger Timetabling Unit.

 

Must

Scott Rosie , Head of Timetabling

Yes Org changes within Student Systems completed before July 2018  

O6

A redefined process for exam scheduling which uses the software to the best of its capabilities using (where possible) the auto-scheduling functionality to reduce manual effort. This needs to include the special adjustments data.

 

Must

Alison Ramsay

Yes / No - partially from July 2018 - New process has been fine tuned through doing the scheduling for 3 exam diets New processes for data import , new processes for scheduling of special exam sittings. Wide-scale use of auto-scheduling has not been possible as the Scientia scheduling engine rules do not match the UoE scheduling rules, particularly for filling large locations first

D7

Use Scientia consultancy to extend use of Auto-scheduling

Must

Scientia Consultants

No  n/a Constraints with how the Scientia software work has limited use of auto-scheduling

O7

Communicate the changes to the User community , and consult with Staff and Student users to understand the impact of the changes on users

Must

Scott Rosie / Alison Ramsay

Yes from July 2018 Regular Exams operations group scheduled with the user community in schools 

D8

A comprehensive Comms Plan to inform and explain the changes to the User Community

Must

Jamie Thin

Yes from July 2018  

O8

Decommission redundant legacy systems

Must

Stefan Kaempf

No  tbc legacy cold-fusion exam web timetables disabled - but not deleted yet - as a backup for the new python exam web timetables

D9

Decommission legacy systems , once the new systems are fully delivered and the new service is embedded. It is expected that the legacy systems will be switched off when the new systems are LIVE – but they will not be decommissioned until  the first running of the May 2018 exam diet has been completed – so that there is fall-back contingency in the event of any major issue on switching to Exam Scheduler v3.5.4.

While this work is in scope of SAS005 , depending on timing – we may consider closing SAS005 after the May 2018 Deployment review, and then transferring some budget to a separate small project to decommission the redundant legacy systems at a later date , rather than keep the whole SAS005 project open

Legacy systems to be decommissioned at a later date include:

Desktop install of ES v2

Access database used for Exam reporting

VM with old DEV copy of ES v3 ( from SAC022 project in 2014)

Exam Timetable search from Study / Student Administration pages on University Website

Coldfusion applications that run the search and personal Timetable for Students who need extra provision.

**These coldfusion search tools offer staff in schools another view of the Exam Timetabling data , and it is expected that equivalent views should be provided as part of the new system**

**Some analysis required to understand the Staff use cases , and to prove the cost/ benefit case for building equivalent functionality**

None of these legacy systems will be plugged into the new Exam Timetable data in Exam Scheduler v3.5.4 – so they will become redundant after Jan 2018 – however typically business users may want to retain access to the legacy systems for a limited time period to allow comparison between old functionality  and new functionality – and as a fall-back in a disaster recovery scenario

Must

Stefan Kaempf

Partially Action on Jamie Thin as Timetabling Programme manager, as part of TTU programme

legacy cold-fusion exam web timetables disabled - but not deleted yet - as a backup for the new python exam web timetables

Action on Jamie Thin to request Paul Gorman to decommision this locally developed legacy CF application , which is hosted on an ISG VM

** check status of decommisioning tasks for Exams office** - delete old instances of ESv2 classic - unless needed for historic reporting

** MS Access is being kept in the Exam office for some operational reporting during each exam diet and to record data that can't be recorded in Exam Scheduler - incl when exam papers are received from schools

D10

Provide a full 12-months post-go-live support to the Exams office through a full academic cycle

New - added to project scope

Jamie Thin

Yes by June 2019 Post-go-live support through Aug 2018 resits, Dec 2018 exam diet, April / May 2019 exam diet

Analysis of Resource Usage ( IS Applications):

Staff Usage Estimate: 266 days

Staff Usage Actual: 530 days

Staff Usage Variance: + 264 days ( + 100%)

Scientia Supplier Costs

Other Resource Estimate: Year 1 implementation  tbc £xxx

Other Resource Actual: Year 1 implementation tbc  £xxx

Other Resource Variance: Year 1 implementation  + £15k  for extra tech consultancy days  (+ xx%)

 

Analysis of IS Applications actual spend

 

The effort for IS apps doubled , reflecting the project duration increased from a SEP business case estimate of 6 months to actual of 2 years.

Duration increased 4 times, IS Apps costs doubled.

The increase in duration had a similar  impact on business costs, external supplier costs and Student systems costs.

As business effort was not tracked in time recording - haven't got exact numbers - 1 FTE back-fill for 2 years across existing team of 4 FTE. This temp back-fill post has now been made a new permanent post within the Exams office, reflecting the extra effort to produce personalised exam timetables for all taught students, and increase in number of students with exam adjustments.

 

This is analysis is for implementation only ( no procurement effort in SAS005, Scientia was an existing UoE supplier , Exam timetabling purchase was variation to existing contract)

 

SAS005 Estimate at end of Planning (TOR) = 266d ( June 2017)

 

SAS005 Agreed budget now = 495d

 

Estimated out-turn = 530d ( at closure in June 2019)

 

--

 

Main sources of change / rough split:

 

Change of scope – add 50d for BI Suite reporting / wider set of standard reports

 

Change of scope – add 50d for extended post-go live support for full 12 month exam business cycle

 

Hidden complexity – add 50d for additional unplanned work for complex data mapping and data flows for Students with exam adjustments ( special exam sittings)

 

Managing external supplier - Bug fixes for third-party software product – add 50d to support testing and upgrade to Scientia Exam Scheduler v3.5.9 ( added to scope of SAS005 , when supplier could only fix issue with extra reading time calculation through release of new version of the product)

 

Balance - extra effort from project team to cover for business team / support business testing & UAT  – when business lead not available due to business-as-usual workload ( very small business team)

 

Stage

Start date

End date

Estimate at TOR

Actual spend

Notes

Project Management

18/04/17

21/12/2018

 

Thru planning / build / testing / initial go-live and scale up to Dec 2018 main exam diet

Approx. 50d

174d

PM effort typically 20% of project budget , for original estimate of 266 = 53d , at 20% of 495 = 100d

Actual PM effort is 74d higher.

Reasons for variation – PM was also managing a 3rd party supplier (Scientia) and outputs from the Scientia team / contract management

Tips – add extra PM effort for managing 3rd party supplier

Plus business team very small , and gap in resources for BI suite testing – Jamie was a hands-on PM and picked up some BA tasks / testing Connect integration / specifying BI suite reports / investigating issues when there were no other resources within project team or business team to do these tasks.

Initiation

03/04/17

26/04/17

 

4.5d

Getting project started and assembling project team

Planning

06/04/17

10/01/19

 

1.5d

Other IS Apps contribution to planning – attending mtgs

Business Analysis

03/03/17

19/04/19

 

11d

Most BA resource provided by SEP contractor 50% for 6 months = 45 days

Plus BA in Timetabling unit = 50% for 9 months = 70 days

Plus BA resource from Student Systems partnership = 20% for 12 months = 40 days

Total BA resource /  rough actual spend =

System Analysis

08/05/17

03/11/17

 

14d

Design for incoming data feed and downstream integration

WP1 - Build - IS Apps servers

15/05/17

10/05/18

 

34d

Build of DEV / TEST / LIVE db and app servers, then working with external supplier to install the Scientia applications

This effort will be reduced for a cloud SAAS solution. Supplier does more, IS Apps does less – IS focus on SSO and integration feeds / staging tables / sending & receiving files from the cloud

WP2 - Build - Student systems connector

26/06/17

03/04/18

 

25d

Of this 14d business analysis, 11d for SSP dev for new EUGEX views , changes to EUGEX views

+ 30d plus effort from Scientia developer to build Connect data mappings / build new workflows

WP3 - Build - Calsync

29/06/17

22/02/19

 

15d

5d business analysis / 10d dev build

WP4 - BI Suite Reporting

28/07/17

13/02/19

50d

64d

20d initial Universe build + 3 standard reports

44d to support testing / re-work from testing / build and test new reports / provide essential enhancements

WP4 - Exam Web timetables

11/07/17

19/07/18

 

25d

Build new python web application / support testing / re-work

Phase 2 - Resits & Fixed Date Exams

03/04/17

27/04/18

 

0.1d

Scope changed – IS Apps effort not required – low cost solution found from importing result status from APT.

Most of effort done by Student systems and business area on back of work from APT – not tracked on ASTA time recording

Integration

11/09/17

15/02/19

 

3.5d

IS production to support integration testing

Acceptance

10/05/18

18/02/19

 

9d

IS production and IS Apps S/W dev team to support UAT

Deployment

16/01/18

22/02/19

 

35d

Technical deployments for go-live and bug fixing / enhancements

Phase 3 - Upgrade and small enhancements

16/08/18

29/03/19

 

50d

Post-go live bug fixing ahead of scale up to main Dec 2018 exam diet – mostly BI suite work

Unplanned activity

10/04/17

27/03/19

 

38d

Resourcing / meeting with other IS Apps teams / issues raised during testing and post-go-live

Best allocate 10% of original estimate to unplanned activities , not in plan – as contingency

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Outcome

The new solution has enabled personalised exam timetables to be sent to all students via their Office365 calendars. That has been the big win , to improve the student experience and save students time creating their own exam timetable.

Students have also got a new exam notification service via the MyEd portal , as part of the notifications backbone project

--

Time savings delivered to Students:

Feedback from UG students is that it takes them about an hour each to work out their exam timetable by browsing the current exam timetable by course, and then they need to check for emails that could contain changes to their exam timetable.

Now students can view their personalised exam timetable in Office 365 they save this time - approx  1 hour  per student  per exam diet (factoring in that previously students had to create their own manual exam timetable and then keep it up to date with any changes )

For the 19,000 students who take exams , approx - 17,500 did not get a personalised timetable before SAS005 , this translates to time savings for students of approx  17,500  hours per exam diet

[in cash terms for students - if they could earn £6.50 an hour in part-time work - this translates to potential loss of earnings for the UoE whole student population of circa £100k per exam diet]

[Translating this time saving for students into FTE of staff time to get a comparison , if the project requires additional staff FTE effort to provide a better service to students - then the time saving in student time per exam diet  is roughly equivalent to 13  staff FTE  for one year ( using 1 FTE =  7 hours x 190 days a year  = 1,330 hours)  ]

--

SAS005 has delivered a big saving of student time across the University , and gives them more time to focus on studying and revision for exams rather than worrying about pulling together their own exam timetable 

--

Note - approx 1,500 students who need extra provision(special exam arrangements)  , were already delivered a personalised exam timetable prior to SAS005 through a legacy web application. Their exam timetable service has improved as they now get their exam timetable direct to their Office 365 calendar, without having to find and log-into a separate web application.

--

This change to deliver personalised exam timetables to students for all centrally managed exams has come at a cost of additional staff effort  for the Exams team to deliver this personalised service to students. The impact of the new service is that the Exams team have required an additional approx 1FTE to be added to their core team.

Explanation for variance

This project required over 6 months of elapsed time to complete testing , due to complexity of NEP exam adjustment data, limited availability of the exams team , and turnaround time for the UoE project team and Scientia team to deliver bug fixes.

This testing effort to deliver a robust, accurate new solution was under-estimated in the outline business case.

- Hidden complexity of business processes to support Special sittings for Students with exam adjustments

- Hidden variation / exceptions for Fixed-date exams  and Resit exams 

- Student Systems data connector - ETL load of student exam reference data from Student Systems to Exam Scheduler - large, complex task

 Build of new work flows to support import and data mapping of the exam adjustments for students needing extra provision. This was a complex area which is an integral part of the end-to-end solution and impacted over 3,000 students - yet it was not identified and costed as part of the business case ( additional unplanned essential work). This added additional consultancy costs from Scientia , +10 days paid consultancy from the Supplier, +20 days testing effort from the UoE project team

- BI Suite operational reporting 

Additional reporting business requirements were identified during planning (not costed in the Business case) and added as 50d effort during planning. The initial  estimate of 50d effort was light - real costs was over 100d. The estimate of 50d assumed that ISG would only build a couple of exemplar reports and the remaining reports would be built locally by a report writer in the Exams team. However there was no resources to either build or test the Exams Bi reports within the Exams business team - so this work was picked up by ISG.

The SAS005 project team also facilitated the pilot of MyEd notifications for Exams - this required some extra effort for testing , which piggy-backed on other testing already underway for Calsync.

Key Learning Points

Management of Project : Build enough time in the plan for many iterations of build and test for any complex integrations - especially for an area like exam adjustments and special exam sittings with a lot of hidden complexity and exceptions

  • Challenge any simplifications and unrealistic timescales and unrealistic costs which come over from a simplified business case 
  • At SEP outline business case stage - more analysis and engagement with the 3rd party supplier  was required to better size the implementation project time-scale and budget.
  • Use of contractors - at the start of the project - the business analysis resource was provided by a SEP contractor BA who was not familiar with University of Edinburgh Student systems operations or business processes. This caused some problems during planning and business analysis stages. The resourcing was improved when Sabrina Fijalkowski returned to work after maternity leave , and the project benefited from her knowledge and experience as she was embedded in the business team. For business analysis best to if there is a BA embedded in the business area who knows the people and processes in that area.
  • A 6-month implementation project as proposed in the outline business case was optimistic, and was a high risk approach with a big-bang go-live for the largest exam diet of the year ( April / May 2018) . Moving the project to go live with the smallest exam diet of the year ( Aug 2018)  , was a practical approach to reducing the risk of disruption to business-as-usual exams.
  • The business case did not estimate any effort to tidy up the source reference data. Real exams data was not accurate enough to publish direct to student's calendars without first tightening up some of the business processes to collect and update Exam reference data and student exam enrolments. Once the student exam data is published to a Student's calendar it has to be accurate - otherwise the student will have a missing exam enrolement or an extra enrolment for the wrong exam in their calendar. Previously these data errors were hidden from students, and each exam had a few extra exam papers and extra seats to cope with inaccurate source data. Now the student exam enrolment data has to be 100% accurate or students will alert their School teaching office or the Exams team to data errors. This required one-off extra effort to introduce more robust processes to collect student mode of study , and ongoing effort by schools, student systems and exams team to maintain source data with fewer errors. 

Management of Project : Continuity of project team is key to success - the small core team of Business lead, Business tester, Timetabling Business analyst , Student Systems Business Analyst , Supplier tech consultant , Supplier business consultant and Project Manager was consistent throughout the project through most of the implementation stage of the project from Build to Closure. This helped completeness and  consistency of testing and issue management. 

Reporting : BI suite as a legacy reporting tool has its limitations. It was difficult to resource the BI suite work across ISG - a lack of developers and BI analysts within ISG to build the BI Suite universe, build BI reports and understand and correct the data mapping from the Exam Scheduler database to the University of Edinburgh BI DB views. Feedback from the users is that the system performance of BI suite is poor ( even after the upgrade to new version of BI suite) and does not suit operational demands of Exam timetabling. Locally the Exams team continue to use a hybrid of both BI suite reporting and MS Access reporting overlayed on top of BI Suite data extracts. For future BI work for Timetabling it is recommended that the new BI Tools - PowerBI or Qliksense are used.

Architecture : Need to limit churn in design proposals and recognise that new cross-cutting initiatives like Digital Transformation can be blocker to making quick decisions and there is a trade-off between moving to new technologies v delivering a solution now with existing technology and tried and tested approaches.

  • Noise re. Student Systems API added delay to solution design

    Initial response from Student Systems was that new views couldn't be created in EUGEX, all new systems would pull data from Student API,

    Then it was confirmed Student API wasn't available ,

    switched to Staging tables populated by Bulk API

    switched to new set of Staging tables to simplify load into Scientia

    switched to using Existing views in EUGEX

    Switched to adding 2 new views in EUGEX to simplfy joins and pull of data to Connect and Exam Scheduler, plus other existing views for full student and Exam info

 

Developing Software : Constraints on access to UoE servers for Scientia technical consultant reduced speed of Scientia build - need to agree to provide more flexible solution that can be launched by Scientia tech consultant when required

 

Outstanding Issues

The known system performance issue has been transferred to the IS Production Support team.

There is a proposed solution from Scientia  which involves archiving / deleting old history data - this deletion routine needs business testing before it can be run in LIVE. Business testing resource not available now ,  testing of the deletion scripts and deployment to LIVE will be agreed and scheduled  by IS Production support and the Student Systems Exams team in line with business resources.

 

13  JIRA issues / tasks have been transferred from SAS005 to TTU019 -including a task to document the annual clear down processes at end of each exam cycle.

All other SAS005 JIRAS have been completed.

 

 

Exceptions - Exams that are not fully centrally managed by the Exams team in Student Systems and were out-of-scope of the SAS005 project

School of Medicine - Clinical Exams

School of Medicine - 'Written' online exams 

Vet School - Clinical Exams

Vet School  - 'Written' online exams 

Business School - Fixed date exams

School of Engineering - Fixed date exams

All schools - 'in class' exams which are scheduled as part of the Course teaching timetable