Closure Report
Project Summary
The physical servers that were used to host our Oracle databases were reaching end of life (the already extended hardware contracts were due to expire in April 2022); the solution identified was to move all Oracle databases to a virtual infrastructure, and it was agreed that Oracle 19c should be the Oracle database version on the VM. The first service we migrated (JIRA) was to be the pilot to test migration to Oracle 19c on virtual servers. This was not expected to be hugely different to Oracle 12c but to be a safe option.
A number of advantages were highlighted in moving to a virtual infrastructure:
- Oracle databases can be managed more efficiently as they are hosted on their own virtual server and not in a shared environment with other databases
- Centos 7 Linux operating system is a more up to date OS
- Patching of virtual server and related Oracle database can be managed individually and doesn't need to be done in one big-bang approach, as is the case when all databases reside on one physical server
- Capability to fail servers across sites.
Although there are numerous advantages to moving to a Virtual Infrastructure, one potential disadvantage was identified:
- Potential impact on performance on heavy transactional systems.
This project was a continuation of the great work carried on previous projects INF142 and INF129. A number of Oracle databases were migrated as part of INF142 and no issues were identified in terms of impact on performance. In saying that, INF149 had different services to migrate, some larger services, heavier transactional services (namely MyEd and Apps), so this needed to be monitored closely regarding performance.
Scope
The scope of this project included the following:
- Capturing details of all databases and services that needed to be migrated including DEV, TEST and LIVE (and any others) databases for each service
- Capturing all End of Life (EOL) dates on databases and related servers including Oracle version EOL, Linux Server EOL and Physical Server EOL
- Proposed Operating system was Centos 7, so checking with Technical teams that all databases captured above can be moved to a virtual machine running that OS version
- Evaluating EOL dates and letting that guide the order of the databases being migrated. i.e. Any servers/services/databases with a soon to be EOL date would be migrated first and be the priority
- Planning the migrations to the replacement VMs according to the lists made in the above, earlier steps; taking business diaries and schedules into consideration and checking on any other projects running concurrently that could be a dependency for this project
- Refining the VM solution process developed under INF129 and INF142 and adjusting, if necessary, any of the processes and documentation associated with this
- Working with ITI Infrastructure Unix team to request, build, configure and prepare the virtual database servers that are required for migration
- Fully testing all databases and services before they are migrated ensuring any known issues are recorded
- Carrying out the planned migrations of the databases identified above
- Fully testing and analysing the migrated databases; including measurement of 'before & after' speeds/response times
- Switching and "Go Live" to the new VM-based databases once build and testing is complete and signed-off
- Decommissioning the physical servers that are replaced with the introduction of virtual machines, under this project
Out of scope
- Any upgrades of Oracle databases from versions lower than 12c (already identified via other projects) would not be included in the scope of this project
- Any Oracle 12c databases that were identified as needing to be migrated under other areas of work (i.e. People & Money;SaaS applications) were not be included in this project
- Metro Stretch cluster was out of scope
Objectives and Deliverables
No. | Description | MoSCoW | Delivered |
O1 | To list all databases that need to migrate from their current servers because of the upcoming redundancy of the Hardware and Linux versions RHEL 6 & CentOS 6 | ||
D1 | A definitive listing of all Oracle databases running on servers running RHEL 6 and Centos 6. | Must | Yes |
O2 | To classify (by their required or mandatory OS) the databases that will need to be migrated to new VMs as part of this Project | ||
D2 | A listing of all Oracle databases that need to be migrated to new VMs from their current server and OS, and Analysis and agreement that they can move to Centos 7 | Must | Yes |
O3 | To plan the migrations to the replacement VMs, according to the lists made in the earlier steps of the project | ||
D3 | A deployment plan for each migration that takes any business needs and other dependant projects into consideration. | Must | Yes |
O4 | Analyse End of Life dates for current servers and current OS versions and construct a Migration order | ||
D4 | Check End of life dates on current servers and let that guide and prioritise which services and databases need to be migrated first | Must | Yes |
O5 | To refine the VM solution developed under INF129 and INF142 and adjust any of the processes and documentation associated with this | ||
D5 | An updated, re-usable version of the solution developed under the earlier projects [INF129 and INF142] specifically looking at implementing Puppet 5 usage and configuration. | Must | Yes |
D6 | Refined processes and updated documentation to support the VM solution (as necessary) | Should | Yes |
O6 | To build and prepare the virtual database servers that have been identified as necessary for migration | ||
D7 | A set of new virtual servers to be built and configured that will host the databases that need to be migrated | Must | Yes |
D8 | Updated technical documentation (TADs), Wiki pages and Abacus CMDB holding current server Information | Must | Yes |
O7 | To carry out the planned migrations of the databases across all required DEV, TEST and LIVE environments | ||
D9 | All identified Oracle databases migrated to virtual servers | Must | Yes |
O8 | To comprehensively test and analyse the migrated databases before switching them to live | ||
D10 | A set of test results to show that the migrated databases perform within acceptable limits and that there is no loss of performance | Must | Yes |
O9 | To decommission the physical servers that are being replaced with the introduction of virtual machines | ||
D11 | When all databases have been migrated to VMs some physical servers (i.e. those under this project) will need to be decommissioned as they will not be needed any longer. They need to be put through the decommission process and removed from the UoE network | Must | Yes |
Analysis of Resource Usage:
Staff Usage Estimate: 545 days*
Staff Usage Actual: 633 days
Staff Usage Variance: +17%
*An original allocation of 350 days was made to the project, with the understanding that this would need to be boosted from the programme budget to ensure all services were migrated. More details are provided below.
Explanation for Variance
-
Time
The project did not waver from its original, planned set of milestones until the very end, with the penultimate milestone and the closure date both being pushed back. The original schedule (from the planning sign-off in November 2020) had each of the original nine services being completed every 2/3 months to allow enough time for each set of DEV, TEST and LIVE migrations to be carried out. This plan proved to be very effective and by the time of the completion of the eighth migration, the project was, in fact, four months ahead of the original schedule.
There was then a scope change (the addition of HRLIVE and FINLIVE; details below) which added to the number of migrations and also a hiatus of a month or two in delivering the ninth migration into LIVE. This one was the SOA migration and it had to be planned around the delays to COM045, which was upgrading the related IDM database, as well as the business diaries of the University areas that depend upon IDM/SOA. This migration was completed in April 2022, still one month ahead of its original date.
The scope additions were subsequently delivered in April and June 2022, just before the decommissioning/documentation and closure milestones, but these latter two dates have been pushed back because of the time required to ensure that all of the necessary documentation has been updated to reflect the number of changes that this project has introduced amidst the time when many colleagues are taking leave.
-
Effort
The original, full budget for this project was estimated at 545 days. This was based on the experience of project INF142 which had carried out very similar server builds, migrations and testing. However, INF149 was originally allocated a budget of 350 days and it was recognised that this would not be sufficient to migrate all nine services - it was noted by the PM at the time that this would only allow for seven migrations and that the project would require more effort to complete all of the migrations in scope.
The budget change to the original estimate of 545 days was subsequently approved in November 2020 - issue 4 - with effort being split across 20/21 (220 days) and 21/22 (325 days). This gave greater security in knowing that the project had sufficient budget to carry out all nine migrations. There was another change in April 2021 - issue 7 - with the budget being reduced as the number of successful migrations increased with the project team developing a dependable set of procedures to carry these out and in a reduced timeframe. The overall figure was now 450 days and this was considered sufficient for the remaining migrations at this stage.
But with the scope change that was next made, the project budget had to be increased again in September 2021 to a new total of 540 days via issue 8. The additional work around these 'new' migrations led to two further increases in effort to cover the extra work that emanated from the issues encountered in carrying them out, and this led to a final estimated figure of 605 days to the end of the deployment stage.
There has been further work on the project since then - on the decommissioning and documentation tasks as noted above - and with a number of colleagues involved across the Applications teams, and this work taking longer than anticipated, this has led to a final figure of 633 days (issue 11).
-
Scope
The original scope for the project was very clear regarding what was to be included and what the project team needed to focus on. There were no changes to the scope until September 2021 when the need to migrate the databases for HRLIVE and FINLIVE to new virtual machines and carry out all the necessary tasks around these – testing, documentation, decommissioning, etc. – was identified by senior management in Applications as necessary. This change was recorded in issue 8. This also added to the project budget (as noted in the section above).
Key Learning Points
One important lesson for future projects that involve the upgrade, migration or addition of databases or applications is that more time and effort should be planned and estimated for the latter stages that involve creating or updating the relevant documentation and decommissioning related servers. The directorate has made great steps in recent years addressing this aspect and we now have improved controls to monitor what is completed and what remains to be done. These are managed through a set of check pages (more detail here) and the stages cannot be signed off until all the updates are made, recorded and reviewed. This means that more effort might be needed than was historically used on this type of work, but it ensures that our documentation is as up to date as possible and has been reviewed by colleagues responsible for maintaining the technology.
Outstanding Issues
There are no outstanding issues.