Closure Report

 

Project Summary

This project aimed to complete a double upgrade of the SITS system from version 9.9.0 to 10.0.0, then to 10.1.0 so that the University remained within the maintenance agreement with the software supplier, Tribal. There is always time-pressure on the project as there is only a very limited window in which the LIVE upgrade can be delivered, and this requires a deployment out of hours in order to minimise impact on the wide user-base.

The process is well-understood by an experienced team whose knowledge and commitment once more ensured that the project was delivered on-time, well within budget and with minimal impact to users and to other projects operating in the SITS environment.

The release from the supplier included a complete overhaul of the assignment files and the upgrade being applied in a different manner - this took time for staff to understand and work through the news processes, and there was limited assistance from the supplier.  Due to this the planned upgrade date in March 2021 was delayed to April 2021, then further issues with the first upgrade to the DUST environment led to the upgrade being delayed until October 2021. The Project Team worked through all these issues and their knowledge of specific areas was vital in being able to work through the upgrades in all the environments. This was a great collaboration across several teams from Student Systems (SSP, Operations), and IS Apps (Dev Tech, Production and Project Services).

The project also agreed to upgrade the Data Futures (FUT) environment to version to 10.2 as features of these release were required for the Data Futures project (SAC077). This gave the Project Team another non-LIVE environment to run the upgrade process in, as well as experience the upgrade to 10.2 which will be deployed to the other environments as part of the next upgrade project. The upgrade to 10.2 FUT uncovered that 10.2 only supported Oracle 19c. The server that the FUT environment was on had a LINX operating system which did not support Oracle 19c. As this was a necessity for the SAC077 project, the decision was taken to include the build of a new virtual server running a newer operating system, database migration, database upgrade and then the 10.2 upgrade within this project. This was done between the DEV and TEST upgrades and didn't require any changes to milestones, however this was a scope change. The upgraded FUT environment was completed in early September 2021.

The project team had to deal with a few showstopper issues with Tribal, which led to the initial delay of the planned release in March then the backup weekend in April, the team performed extremely well with excellent engagement of all team members and stakeholders. Solving showstopper issues with Tribal required high engagement, resilience and problem solving to identify a workaround, which still enabled the team to carry out the live upgrade over the rescheduled weekend of 8th, 9th and 10th October. This was a great collaboration across several teams from Student Systems (SSP, Operations), and IS Apps (Dev Tech, Production and Project Services)

Objectives

  1. Upgrade SITS Student Record to a supported version of SITS (10.0/10.1)
  2. Minimise downtime for the LIVE system during upgrade
  3. Refresh DEV, TEST, TRN, DUST from LIVE (also include FUT)
  4. Allow sufficient time for  testing to reduce chance of critical post-go live issues
  5. Minimise overall time duration of the upgrade project - so that there is minimal disruption to other project work (i.e. quick back-to-back deployments into DEV / TEST / LIVE)
  6. Additional Objective - Create new server and DB for FUT, and upgrade to 10.1 in line with other environments, then upgrade to 10.2

Deliverables

Deliverable Priority Completed

1.All active EUCLID STAR databases: DEV, TEST , LIVE, TRN & DUST, FUT are upgraded to Oracle v12.2 (from current v12.1)  

Must Yes

2. Refresh all non live environments data (DEV, TEST, TRN, DUST) from LIVE

Should         

Yes

3. Complete a v10.0 trial upgrade in DUST to assess the new upgrade progress introduced by Tribal in v10.0 (v10.1 changes are minor). The objective is to review the assignment  asn files to bring it inline with Tribal’s recommendation

Should Yes

4. Upgrade EUCLID DEV  to v10.1

Must

Yes

5. Complete technical test of DEV. Carry out any required re-work. Obtain downtime for STAR / EUGEX  and DUST as back up while DEV is down

Must

Yes

6. Upgrade EUCLID TEST to v10.1   

Must

Yes

7 Complete regression test of TEST and any required re-work. Obtain downtime for STAR / EUGEX

Must

Yes
8. Create new server for FUT Must Yes
9. Upgrade FUT to 10.2 Must Yes

10. Upgrade EUCLID TRN  to v10.1

Must

Yes

11. Upgrade EUCLID LIVE to v10.1. Complete  testing and sign off

Must

Yes

12. Refresh TEST data from LIVE after LIVE upgrade

Should

 

13. Clone EUCLID DUST, FUT from LIVE

Must

 

14. Review and extend automated performance and testing scripts

Should

Yes

15. Comparison of  performance of all environments before and after upgrade

Must

Yes
16. Review the old legacy SRL (SITS components) syntax as recommended by Tribal new version. Assess the legacy code, to inform size/future plan and priority of work before go live Must Yes

Student Record system remains within the standard support contract with SITS supplier Tribal Benefits

  • Access to new functionality in new version
  • Understanding of any impact that the upgrade has on performance
    • Good work has been done post go live. The Team was able to reuse the good work done on Direct Admissions: PI monitoring, as well as analytics to monitor performance across university sites and applications.

Success Criteria

  • Minimum disruption to LIVE service during the software upgrade. Aim to complete it over a week-end
  • Minimum disruption to LIVE service during the database upgrade (expected 0.5 day downtime)
  • Minimum impact to training in TRN
  • Successful liaison with project teams working on other projects within SITS
  • Few open issues, and quick resolution of any critical issues in DEV / TEST/ TRN and LIVE - as these could impact other project work and testing in DEV and TEST, and will impact business users in the LIVE environment.
  • Minimum impact on downstream systems
  • No long term degradation in the performance of the service

 

 

Analysis of Resource Usage:

Staff Usage Estimate: 343 days

Staff Usage Actual: 387 days

Staff Usage Variance: +12%

 

Outcome

Explanation for Variance

  • The original go live date was moved from March 2021 to October 2021 - managed via Change - 
  • The increase in budget during the project was required due to the increased time required to resolve the technical challenges, dealing with supplier support and guidance, and the addition of scope for the upgrade of FUT, managed through the following change and incident logs- 
  • The support contract with Tribal had to be extended beyond July 2021 due to the upgrade being delayed
  • The Live Upgrade weekend took more staff time than anticipated, with work going on until the Sunday evening

 

Key Learning Points

1.) No ITI Resource to cover Launch weekend

ITI resource have previously been on standby over the launch weekend in the event of the project team being unable to access servers. Whilst they have never been required to intervene, the backup has been there if required. This was arranged for the original launch weekend in March 2021, however it wasn't in place for the rescheduled weekend.

In future projects, ITI will be engaged early to ensure there is suitable resource in place should there be an issue relating to server access during the live upgrade

2.) Backup PC for Remote Access

As the Project Team are still working from home in the most part, the work was carried out via remote access. In the run up to the Live Upgrade weekend, there were a number of unscheduled power cuts (outside of the University control) that affected Argyle House. As this was the main route for the majority of the team to gain access to carry out upgrade work, an additional PC was set up in Old College to give an alternative access route should there be any issues over the Live Upgrade weekend.

Whilst the majority of work is carried out remotely, this should be a standard task each year to ensure the upgrade can continue if one route of access is unavailable.

3.) Uncompiled packages and removal of indexes within EUGEX causing jobs to fail post-upgrade

Following the downtime, a number of packages became uncompiled due to EUGEX being unable to see tables. EUGEX contains static data over the upgrade weekend as no refresh jobs are running, however when SITS was brought back online these refresh jobs failed due to the missing indexes and uncompiled packages.

Steps have been added to the upgrade process to ensure packages and indexes are checked to prevent the issue from reoccurring.

4.) Not all users seeing the downtime holding message

When users attempted to access the system from an email link (e.g. application email) were shown a 'page not available' message rather than the holding message informing them of the downtime. This could lead to an increase in support calls reporting the system as being down.

5.) Quick Address Search broken when used through Self Service

Quick Address Search had been tested, however when being using through self service it was not functioning correctly due to a missing licence file. This file was missing in all environment barring DEV

6.) Use of 'out of the box' functionality where possible

During the upgrade, a piece of custom code in PDF generation had been overwritten and required intervention to correct after the error was picked up. This highlighted our use of custom solutions over what Tribal provide as 'out of the box' functionality.

Where there are options to use functionality provided by Tribal, this should be used to ensure we can get support on issues where required from the supplier.

7.) Additional Entity Records created

Additional Student Entity records were created as part of the upgrade. These included entities that are not in use by the UoE, and these subsequently caused the Student Merge process to fail. Student Systems Operations ran a clean up script to address the issue, however this should now become part of the upgrade to clear up any additional records.

8.) Review Known Bugs ahead of upgrade

A known bug (CAS processing issue in VCR) is being addressed in all the environments. Future projects should regularly review the known bugs and mitigate these, if possible, in advance of go live.

9.) Setup of UCAS Imports in other Environments

UCAS Imports were not working in TEST or LIVE post upgrade and required input from Tribal to resolve the issue. The UCAS link has now been set up in TEST and TRN which will allow Operations to test this further in the future.

10.) Actions in non-LIVE to reduce TOIL

Long working days over the LIVE double led to 28 hours being expended for technical team members. Team members commitment over the weekend to keep the planned downtime is very much appreciated. For future upgrades with staff welfare in mind, could actions be taken in non-LIVE environments to bring effort expended to levels of previous years?

What went well

This upgrade had many challenges, however the team illustrated excellent collaboration between SSP, Student Systems Operations and IS Applications

  • Team communication worked well with the Microsoft Teams channel being heavily used whilst the team work remotely
  • Bi-weekly progress meetings ensured issues were addressed as required, and allowed service leads to keep up to date with progress
  • Engagement and expertise from Student Systems Operations ensured issues were identified and resolved in a timely manner
  • Production, Development Technology and SSP Development all worked collaboratively throughout the project and provided backup for each other where required
  • Backups and knowledge sharing for all key roles was achieved for the live upgrade
  • Regression Testing was carried out by the SSP Testers and SSP Operations. This ensured knowledge was shared between team members and no further involvement of SSP Business Analysts were required

Outstanding Issues

The following issues are kept open in the backlog to ensure they can be taken care of during  the next SITS upgrade in project in 21/22 - 

  • There have been batch processing issues occurring intermittently since the upgrade. These are being investigated, and may move into the next project if found to be directly related to the upgrade

 

Project Info

Project
SITS double software upgrade
Code
SAC087
Programme
Student Systems Partnership SSP
Management Office
ISG PMO
Project Manager
Alan Donald
Project Sponsor
Brandi Headon
Current Stage
Close
Status
Closed
Project Classification
Run
Start Date
05-Oct-2020
Planning Date
23-Oct-2020
Delivery Date
10-Oct-2021
Close Date
30-Nov-2021
Overall Priority
Highest
Category
Compliance

Documentation

Close