Closure Report
Project Summary
This project has delivered the SAP BusinessObjects upgrade to version 4.2 (Service Pack 5, Patch 500). As well as containing major new reporting, UI, and dashboarding features that directly benefit business users, this release also offers a significantly enhanced experience, with improved workflows, for system administrators and universe designers. Business communications were conducted, training sessions were provided, and an online 'upgrade hub' was published to ensure that business users were kept informed about the upgrade and the new functionality.
Early on in the project, before planning review was signed off, Itelligence - our external support supplier for BusinessObjects - was commissioned to provide a migration proposal which would provide the optimal upgrade path to take. The results of this fed into the project brief. Their review specified minimum and recommended requirement sets for the technical architecture / virtual machine (VM) servers based on our current system usage.
After the brief was signed off, a decision was taken to have Itelligence deliver the upgrade on DEV and TEST (application only) and LIVE (application and data migration). The reasons behind this decision were:
- The complexity of the BI setup required an in-depth knowledge of the architecture and it was concluded that using Itelligence should ensure an easier transition to version 4.2. There was also very limited Dev Tech availability at the time we were looking to upgrade
- Itelligence stated that they would be able to provide limited support of the products going forward if the build was carried out by the University
The project suffered delays due to (1) Itelligence's availability, (2) UOE staff availability, (3) network performance issues, (4) constraints in the business calendar, (5) software bugs, (6) and pre- and post- installation and deployment issues. The original deployment date was 28/02/2018. The actual deployment date was 11/12/2018. Key factors in the 10 month slippage were (3), (4), and (5).
Throughout the lifecycle of COM036, there had been a number of challenges to overcome, not all of them project related. Recognition must be made to the project teams - Development Technology, Applications Management, Technology Management and Service Management - for their continuous troubleshooting and issue resolution that were prevalent throughout the stages.
A timeline of events is as follows:
| Dec 2017 |
Itelligence were commissioned to install BI to DEV, TEST and LIVE, with data migration on LIVE. UoE to perform data migrations on DEV and TEST. |
| Jan 2018 |
Change in the order the environments were to be delivered as requested by Itelligence with the reasoning:
|
| Feb 2018 |
Itelligence uncovered underlying network / VM performance issues on DEV and TEST when installing and testing the application. This was raised with ITI. Deployment date was revised from February 2018 to July 2018 on account of these performance issues - a decision that also factored in impending strike action and constraints in the business calendar.
|
| March - May |
Investigations were carried out by project team into performance degradation. Detailed testing and analysis was carried out with logs. During this period, DEV and TEST installs of BI were completed by Itelligence, but SAP BI had to be set up to run on SQL Anywhere due to performance issues with Oracle. This was not the recommended set up from SAP. Itelligence refused to sign off and handover the new machines until the performance could be confired as resolved by UOE, and until SAP BI was reconfigured to run against Oracle. Test plans written in Excel by Service Management. These covered report validation templates for the business to use (UAT), and a functional testing script which itelligence, SMGT, and Student Systems would use to ensure key functionality was working before deployment sign-off. Itelligence continued with their build plan and also resolved a number of other issues:
|
| June 2018 |
UoE were not able to complete any migrations from old to new DEV and TEST because of a bug with the GUI (graphical user interface) migration tool. A workaround to the GUI issue was to (1) complete the migrations using the command line interface, (2) install a patch to fix the bug, or (3) have itelligence do the migrations. Option (1) was ruled out because UOE tech staff were not familiar with using the command line to perform migrations, and confidence levels were low since this was an unfamilliar tool which itelligence had said was finicky to use and get right. Option (3) was ruled out because itelligence could not guarantee a time estimate for this, or assure us of the quality of the results. Option (2) - the patch install - was chosen. Itelligence indicated this is an extra cost as this was traced and identified as a bug in the installed version. Head of Service Management agreed that funding was available for Itelligence to install the patch. |
| July 2018 |
As the network / VM performance issues, which were reported by the project team to ITI in February, were still affecting performance, there was no option but to cancel the July go-live. This was because:
ITI had been investigating the performance issues and planned the following with a target date of August / September 2018:
|
| Sept 2018 |
We saw no recurrence of the network / vm issues from September onwards. Itelligence applied patch 500 to fix the GUI data migration wizard. Itelligence began this work on 20/09/2018, and noted that the DEV server was "locking" every few minutes which hampered progress. Itelligence completed the patching, but this had corrupted the DEV build resulting in DEV not starting up correctly. Day 2 of the planned patching - network issues at UoE stopped progress completely. Project team agreed for Itelligence to continue to apply the patch to TEST while the request to roll DEV back was submitted to ITI. This was so Itelligence was not left booked without work for that day. Deployment of TEST showed the same issue as DEV. Itelligence investigated and applied a fix from SAP that resolved the issues on DEV and TEST. 6 days of Itelligence's time was used on the patch install. No performance issues were noted by Itelligence on any of the VMs. |
| Nov 2018 |
Additional work had been found to be required on the environments - install of 360Eyes, modifications to connection strings resulting from the Oracle database upgrade project, and the migration wizard not migrating all data correctly, requiring re-migrations. Functional testing on DEV and TEST completed by itelligence and Service Management. Functional testing on LIVE completed by itelligence, Service Management, and Senior User (from Student Systems). UAT (report validation) commenced involving Finance, HR, and Student Systems. This was done on LIVE as the data sources are different on TEST so the businesses would not have been able to meaningful compare report outputs between environments. The businesses also stated that they did not use TEST. UAT completed by the business end of Nov. |
| Dec 2018 |
BusinessObjects Live deployment took place on 11/12/2018. Old servers disabled. The following post deployment issues were raised:
|
| Jan 2019 |
System crashes continued, project team, Itelligence and SAP were involved to trace and resolve. Eventually traced to old servers being turned on to recover user's permissions and favourites, and not being turned off as the favourites task was on going . The system crashes were a consequence of a user connecting in the dashboard to old BI Suite components. Old servers were switched off and stability was resumed. Issues continued to be raised by the businesses:
|
| Feb 2019 |
Senior Supplier was assigned to assist with the issues raised in January. |
| Mar 2019 |
A decision was made to track this as a known issue and problem record (This will be managed under P1812-024).
The following issues will be picked up by a future upgrade project if there is a resolution. They have been identified as bugs by SAP and Itelligence:
DSOR agreed by the project team.
|
Objectives and Deliverables
| Number | Description |
Priority MoSCoW |
Achieved |
|
O1 |
To upgrade the Business Objects to v4.2 SP 5 |
||
| D1.1 | Build the DEV, TEST and LIVE infrastructure. | Must Have | Y |
|
D1.2 |
Install core BusinessObjects platform on new DEV, TEST and LIVE infrastructure. |
Must Have | Y |
|
D1.3 |
Install, configure, and connect new accessory / client tools. This covers:
|
Must Have | Y |
|
D1.4 |
Ensure platform monitoring and auditing functionality is enabled and configured, so as to facilitate support requests and BI-on-BI analysis via 360eyes tools. |
Must Have | Y |
|
D1.7 |
Decommissioning of old BusinessObjects infrastructure / environments. |
Must Have |
N LIVE server switched off, but not decommissioned. INF137 Decommissioning project to pick this up. |
| O2 | Content Rationalisation | ||
| D2.1 |
Migrate content from old to new environments:
|
Must Have |
Partial success. Not 100% successful. Using the migration tool resulted in not all the user saved favourites being migrated correctly. Service Management has been in contact with those whose favourites were not migrated and assisted in the manual recovery for all those affected.
|
| D2.2 | Perform content purge to reduce number of stored documents and universes by Service management and key business administrators. | Could Have |
Partial success. A review of reports was carried out by the users and the ones to be migrated were selected and reported to service management. This is noted as a partial success as not everyone responded. |
|
O3 |
User Communications and Service Updates |
||
|
D3.1 |
Business communication and engagement plan. |
Must Have | Y |
|
D3.2 |
Documentation updates (eg TAD, SDS, user web pages). |
Must Have | Y TAD and User web pages updated. SDS not required. |
|
D3.3 |
Stakeholder / area admin / end user engagement sessions. |
Must Have | Y |
|
D3.4 |
Production of new training notes, course materials and associated documentation e.g. wiki and web pages. |
Must Have | Y |
| D3.5 | Revised SLAs and OLAs. | Must Have |
N To be picked up by Production and Service Management (see outstanding issues). |
Analysis of Resource Usage:
Staff Usage Estimate: 111 days
Staff Usage Actual: 267 days
Staff Usage Variance: 140%
Outcome
| Team | Estimated | Actual |
| Project Services | 26.6 | 88 |
| Dev Tech | 59.8 | 51 |
| Software Dev | 5.5 | 32 |
| Apps Man | 12.9 | 16 |
| Tech Man | 4.3 | 26 |
| Service Management | 50 (from Asta project request) | 45 |
| Directors Office | 1.4 | 5 |
| Resourcing | 4 | |
| Total | 110.5 | 267 |
Explanation for variance
Prolonged project timeline, extended by 12 months from the initial estimated closure date at the start of the project. This was due to the following:
Network Performance needed to be resolved before we could proceed with DEV and TEST build, and LIVE deployment. Itelligence refused to sign off their build until resolved.
Service Management time not included in estimation, but from August 2018 was to be included in the project budget.
Configuration of DEV, TEST and LIVE underestimated.
Bugs being raised by the business after UAT and deployment which UoE was not able to resolve. These were passed to Itelligence and SAP for investigation, resulting in confirmation as application bugs.
Configuration issues, firewall issues, 360 eyes install on environments, SSO integration took more time than expected to resolve.
The GUI tool for the data migrations required a SAP patch to work. The migrations were not estimated at the start of the project.
After installation of the patch, the GUI tool still did not migrate all the data correctly, resulting in further attempts to do this. Software development time increased.
Post go live: The service was not stable post go live and a service recovery team was created to address the stability issues.
Additional resource for Techn Man and Apps man is accounted for by Dev Tech not having sufficient resourcing due to other projects.
Key Learning Points
| Supplier | |
| Issue | Patching was required to migrate user permissions across (Bug not declared during scoping work). As this was fundamental to the delivery of the project, this required further investment and further delayed the project. |
| Learning point | The statement of work was written by the supplier, and agreed by UoE. In future, consideration should be given to detailing the expected functionality that will be included in the agreed statement of work, and a contract established that ensures that all patching is inclusive to ensure like for like is achieved. |
| Issue | Itelligence built and migrated Live environment first, followed by build of DEV and TEST. (See Jan 2018 timeline for reasoning) |
| Learning Point | UoE to ensure that in future sign off of environments still needs to happen in DEV - TEST - LIVE regardless of the order they are built. |
|
Project management / Governance including Sponsor |
|
| Issue |
Itelligence were commissioned to do the upgrade at the start of the project as they stated that they would be able to provide limited support of the products going forward if the build was carried out by the University. The initial costs for Itelligence were in excess of the available budget. This was reviewed and a reduction was agreed, which included a reduction in Itelligence PM resource. UoE agreed to take this role.
|
| Learning point | Cost and Time analysis of both methods of build should be documented and reviewed before decisions made. This was rushed due to time constraints with deployment windows. |
| Issue | Limited knowledge transfer to UoE to enable UoE to take this in-house and do this upgrade in future. |
| Learning point | Future use of external contractors should ensure that knowledge transfer is a deliverable. |
| Issue | Project communications were poor, with short notice requests for sign offs, decisions taken at meetings where not all project team members had signed off and supplier liaising directly with Service Management. |
| Learning point |
[1] For weekly meeting add in specific agenda if certain items are to be discussed along with the team updates. [2] A matrix of roles and responsibilities should have been put together by the PM at the start of the project so everyone was clear on their role and communication route. [3] Contact needs to be pre-agreed for project related issues, and agreed that it is through the Project manager. |
| Issue | Project plan was not available from start of the project. |
| Learning point |
[1] WIS should not approve projects in planning stage without a viable project plan [1] PM to keep an up to date plan of the project tasks and responsibilities at all times. [2] Integration with the vendor's plan would have ensured smoother running of the project. |
| Issue | Sponsor was pulled into and became too involved in the day to day work, and therefor became ineffective as a decision maker. |
| Learning point |
[1] The addition of senior supplier at an early stage would have helped this situation. [2] To establish a more structured meeting schedule with the Sponsor to keep them informed. |
| Issue | The engagement of Itelligence was in part, intended to free up Dev Tech and Service Management time. This in practice did not happen, as the service manager had significant knowledge in the way the product worked. |
| Learning point |
[1] Replanning should have been carried out first to see whether using Itelligence was the most cost effective option. [2] Planned knowledge transfer should have been included in the project scope, and revised once decision to use external supplier was made. |
| Issue |
UAT (business report validation) was completed on the LIVE environment (before it went live) but not on TEST. This was because the reports and the data were so different between Test and Live that they couldn't be meaningfully compared regarding refresh times, data quality. In addition, individual area UAT sign-offs were not sought; the chosen approach was to have a meeting where the project team reviewed the UAT results together with the business stakeholders, with a view to doing one sign-off together. |
| Learning point |
[1] Future upgrades should have pre-agreed sign offs by each individual business area, if UAT is to be completed on LIVE. [2] The risks should have been updated to take account of testing on the LIVE environment. [3] The Test environment should always mimic Live. |
| Issue | Project costs not monitored effectively. Estimates for each area of responsibility were not monitored for spend. |
| Learning point |
Recommend that for future projects: [1] Budget should be reviewed more regularly. [2] Budget increases should be flagged earlier. [3] Service Management Time is clarified as to included or excluded. A change occurred mid way through this project which caused additional confusion. [4] Completion of timesheets – highlight any that have 2+ weeks outstanding to team manager. |
| Issue | Resource bookings were not always in place to be reactive to the situations as they unfolded. |
| Learning point | ASTA booking should have been pre-planned for each individual as part of the planning process, and entered into ASTA . This way at each meeting individuals could highlight when their bookings needed extending resulting in better budget control. |
| Service | |
| Issue | The support arrangements with the supplier did not cover the issues encountered at Go-Live. These issues resulted in an additional payment being made to Itelligence to cover Web-ex and other investigative work in relation to the issues. The issues surrounding the support contract were not as a direct result of the project. |
| Learning point | Future BI upgrades should ensure that the post Go live support is arranged and agreed. 0.5 days was insufficient for the scale and size of this upgrade. |
| Issue | Network / VM performance issues noted and raised in February 2018. Not resolved until August 2018 resulting in a delay of 6 months before deployment could be considered. Supplier seemed to spend significant time investigating the performance issues. This was in part to try and provide information on what they were doing at the time the performance tailed off. Continued testing and investigations by Itelligence and the project team at UoE over the next few months trying to prove slow performance and identity causes used additional project budget. |
| Learning point |
[1] Seek guidance on whether the status of the project should have changed - i.e. to suspended. [2] Senior supplier should have been assigned at this stage. [3] PM to keep checks that supplier's original tasks are not being diluted by other issues . |
| Issue |
UAT sign off and testing: Test scripts for functional testing were too generic, there was no step by step guide to carrying out specific tests. Testrail was not utilised, when the project delays provided the opportunity to achieve this. UAT sign off criteria not agreed by the business in advance, although details about the sign-off would happen were clearly communicated. |
| Learning point |
[1] TestRail should be mandated for future BusinessObjects upgrades to enable re-use of test scripts and to track testing progress. Using testrail also ensures a step by step guide is available to testers. [2] Sign off criteria should have been agreed in advance. Involvement of all business areas should be mandatory. [3] Senior user not adequately engaged – Invites to team meetings required, especially towards deployment. |
| Issue | Incident handling Post upgrade. Incidents were being reported through multiple channels, and not consistently recorded. Helpline, Service Management, Direct to the SOM, through the Senior Supplier and other senior managers. |
| Learning point |
[1] Clear guidance should be given out as to the channels for reporting faults during this initial post Go Live Phase. These channels should be limited to enable calls to be handled and routed more effectively. [2] Advance discussion with helpline as to expected call volumes should form part of the pre-deployment checklist, and agreed communications plan as part of this. |
| Issue | For business operational reasons, go-live windows are too restrictive. |
| Learning point |
[1] Service Management should confirm available change slots with business areas to ensure the business calendar is up to date. Where the business calendar is restrictive, to the detriment of the deployment, Service Management should work with its business partners to re-align their expectations and find a compromise solution. [2] Deployment so close to Christmas should be avoided in future. Lack of available resource due to annual leave, Christmas shut down etc. meant that many issues could not be resolved as efficiently as they could have. Some of the performance issues were not surfaced until the first week back after Christmas, where key members of staff were on annual leave. |
| Issue |
Student Dashboards stopped working after the upgrade, resulting in major disruption for business users. The causes were: (1) web services feeding the dashboards needed recreated as incompatible with new version; (2) BI environments not configured to allow web services to feed data across domains; (3) issues publishing the dashboards in Flash and HTML formats. These issues could not be caught during UAT because (1) dashboards and web service urls could not be tested against new system until DNS was re-pointed at go-live; (2) the previously used dashboard design software was incompatible with the new system, and the compatible version couldn't be tested prior to DNS repointing, nor could it be installed prior to the removal of the old version of it. |
| Learning Point |
[1] To consider using new URLs for new environments where this would allow for more complete UAT. Because the dashboard web services FQDN used 'bobi.is.ed' we could not test them until those URLs were re-pointed at go-live away from 'borep' to the new 'biapp' servers. So we only discovered at go-live there was a problem with these specific web service instances (however, we had successfully tested the SAP web services feature on all environments during functional testing). [2] Ensure supplier brief, and subsequent contract, includes requirement to ensure key business BI assets (and the features they depend on) work as expected. (Itelligence tested the web services feature but could not test the specific student dashboard feeds) [3] Ensure the business provide Dev and Test versions of their dashboards so that these can be tested, and issues found, before we get to Live. |
| Issue | 7 year gap between upgrades to BusinessObjects - this introduced enormous risk around the impact of this major change. |
| Learning Point | [1] Service Pack upgrades should be done annually, as is the practice with other services, to prevent a 'big jump' and reduce the impact of change between versions. |
| Issue | New Modern interface was introduced, in parallel with the classic one, but was found to have several bugs not caught during functional testing. |
| [1] Form a working group of business users to assess the new UI offering - prior to go live - with the aim of determining whether it is ready for general release. | |
| Technical | |
| Issue | Running old LIVE server along with the new to enable us to re-do the migrations resulted in daily system crashes. |
| Learning point |
[1] Task to switch off server should have been part of the deployment plan. [2] Correct control mechanisms should have been in place to prevent service being re-engaged by Service team. |
Outstanding Issues
The following are outstanding:
Review of the SLA and OLA
Service Management to provide OLA for Reporting and Analytics. Production have agreed that the OLA will be delivered outside of COM036.
JIRA 20: https://www.jira.is.ed.ac.uk/browse/COM036-20 High CPU / ExplorerSearch server issue
This will be managed under P1812-024.
JIRA 39: https://www.jira.is.ed.ac.uk/browse/COM036-39 SEND button not working in modern Launchpad
Bug. Identified as a service pack required. This will be picked up in the future BI upgrade project.
JIRA 41: https://www.jira.is.ed.ac.uk/browse/COM036-41 USA date format breaks reports: closed
Identified as a service pack required. This will be picked up in the future BI upgrade project.
JIRA 43: https://www.jira.is.ed.ac.uk/browse/COM036-43 Flash memory issue (grey circle error)
Itelligence escalated to SAP. Broken feature. Identified as a service pack required. This will be picked up in the future BI upgrade project.
| Attachment | Size |
|---|---|
 estimated_vs_actual.jpg | 42.46 KB |
