Completion Report

Project Summary:

Objectives

The primary objective of the project was to upgrade the POLO database to version 10g of Oracle. This was achieved.

Deliverables

 The deliverables of the project were as follows

 Scope

 The project deliverables stayed within the original scope but additional action was required to resolve behaviour within the Media Bank which differed between LIVE and TEST.

Schedule

 The schedule varied due to additional investigations in TEST and LIVE and three additional deployments to LIVE to resolve the Media Bank response time issues.

Analysis of Resource Usage:

Staff Usage Estimate: 26 days

Staff Usage Actual: 56.5 days

Staff Usage Variance: -117.308%

Other Resource Estimate: days

Other Resource Actual: days

Other Resource Variance: 0%

Explanation for variance:

The overall estimate for the project was increased substantially due to the issues with the Media Bank response times when the service was re-started. The project altered from a planned straightforward database upgrade, one version up, to providing a resolution to a Production issue affecting the entire user base. The focus for additional resource was on Dev Tech to investigate database issues, the Development Team to provide a resolution to the Media Bank response times, Tech Management and App Management in deployment activities and Project Services for additional co-ordination.

Key Learning Points:

Successes

The LIVE and TEST databases have now been upgraded to 10g successfully, allowing the possibility of Polopoly being upgraded when required. An additional benefit of the project is that the TEST database has now been refreshed from LIVE and therefore its behaviour and characteristics are such that testing can be carried out with more confidence than previous.

Due to the issues experienced, an assessment of the TEST and LIVE environments was carried out to ensure that they are as comparable as possible. This will mean that any testing in future will provide a higher degree of confidence that roll outs to the LIVE environment will be successful and pose no problem if testing has completed satisfactorily.

 An additional stage was added at the LIVE deploy phase to ensure any rollback or recovery of the database can be made with certainty. When the service is brought down, a cold back up of the database is immediately taken. The service is re-started and tested again briefly to ensure that if the database has to be recovered back to that point, the back-up taken will work successfully.

Areas for Improvement

1. The importance of having a TEST environment which replicates the behaviour and characteristics of the LIVE environment cannot be stressed too highly.

2. Code changes in the TEST environment which were deployed before the start of the project,  but not progressed to Live, may be an issue. If there are code changes in TEST which have not gone to LIVE, the code in TEST must be assessed on its potential impact to the changes which will be going through TEST and on to deployment before testing commences. It may be necessary to regress the changes out to avoid any issues or false positives or negatives in testing. The list of changes deployed to each environment are recorded in the Apps Man and Development pool.

3. It would be useful that tests to confirm a successful deplyment are carried out with known results. When the second tuning fix was deployed, it was unclear whether it had been successful or not and whether it had caused further problems. What was initially reported were existing errors within Polopoly and the deployment of the fix was regarded as a success.

4. Several times over the course of the project, progress was delayed and/or resources used up on spurious investigations related to TEST and LIVE servers running out of disc space. Post TEST upgrade, errors were reported by UWS on their acceptance testing program, but it transpired that the server had run out of disc space. When the indexing change was rolled out to LIVE, it ran for an excessive amount of time before it transpired that the server had ran out of disc space and the process could not complete. This meant further downtime to attempt the re-indexing again, which did not resolve the issue anyway, but further disruption to the LIVE service could have been avoided.

Outstanding issues:

The behaviour of the Media Bank post database upgrade has not been fully explained. The tuning fix by development alleviates the response time issues for the majority of users but the specific issue is as yet unexplained. Discussions will take place with UWS to decide on any further action.

Project Summary:

The database upgrade process proved to be straightforward and the method for stopping and starting the CMS from scratch has been documented for this type of activity. The change in the behaviour of the Media Bank was unexpected and not experienced during acceptance testing.

 When the TEST environment was refreshed as a clone of LIVE, the Media Bank did exhibit the slow response times seen on LIVE and allowed the Developers an opportunity to provide a solution to issues experienced by the majority of users.

 The problems on the project primarily arose due to environmental concerns, the TEST environment appeared to be substantially out of date and not representative of the LIVE environment in terms of volume of content which is believed to be the cause of the problems hitting LIVE Media Bank. Additionally, a lack of disc space affected the post TEST upgrade and also the indexing fix did not complete the first time on LIVE, again due to a lack of disc space.

 Suggestions post projects are as follows:-  

1. Maintain the Dev and TEST environments as they should be as close to the LIVE environment as possible. Otherwise, there will always the possibility of unexpected results when code or new database versions are deployed to the LIVE environment.
2. The TEST environments should be assessed as part of the project, to determine whether a refresh from LIVE is necessary and whether any code is currently in TEST and would it have any impact on the project deliverables and throughput. If it is expected to have an impact or the impact and interaction is unknown, it should be regressed from TEST for the duration of the testing by UAT.
3. Ownership for the TEST environments should be established and responsibility assigned for the maintenance and upkeep, to ensure that they are fit for purpose.
4. TEST environments are primarily for testing and verification of system changes, prior to deploying out to LIVE. They may be useful for prototyping or training but these activities should not impact or stand in the way of testing or refreshing for project work if required.

 Ongoing, it will be decided whether to start a project, or investigate under support,  for the Media Bank to determine the underlying cause of the problem.  The previous TEST database was saved for any future investigations into the exact nature of the issue and would it be worth planning in some kind of investigation. The coding solution provided by the development team allowed users to access their content within reasonable response times. The functionality fix changed how the media bank operated, and placed a workround in front of the problem area of  processing but did not fix the actual problem. The code where the problem arose has not been ‘fixed’ and the real error/problem has not been explained and this could be regarded as a risk as unexplained behaviour like this may occur elsewhere in the system, resulting in further unplanned effort and disruption to service. The longer the system functions correctly  in its current state , however, appears to reduce the risk of any problems re-occurring but if the trigger was, for example, volume of data in the database , the risk will increase over time and may cause further  issues within the Media Bank or elsewhere.

Some additional comments from the Development  Team are pasted below.

-----Original Message----- From: FRASER Mairi Sent: 25 September 2012 16:57 To: MACCALMAN Miles Cc: CONLON Stephen Subject: ITS098 - Feedback

Hi Miles,

As discussed at last week's UWS meeting, feedback on the issue with the Content References field is as follows:

-- There is a Polopoly-specific equivalent to our custom Content References field.  If Support were investigating further the issue with the Content References field for which we now have a workaround, they may wish to investigate using the Polopoly version of this widget instead.  We haven't looked into this in any detail; we are only aware that it exists and that it would appear to provide similar functionality to our own version.  If we had any issues with this widget (i.e. similar slowness when it's used to that experienced following the DB upgrade), we would then presumably have recourse to Polopoly Support.

-- Regarding the general comments made around the table as to the limited usefulness of the Content References page for articles in Polopoly, we should try to quantify that.  Is there a pattern?  If we are not aware of a pattern, is it possible to do some testing to find a pattern?  There are a finite number of ways in which content can be referenced in Polopoly so we ought to be able to narrow the problem down to specific scenarios.  Obviously, if we separately determine that the Polopoly Content References field could replace our version, this problem should go away.

These comments are specific to the workaround that Richard put in place for Content References, and are not really concerned with the slowness issues that were experienced on the database.  On that subject, I agree with what was said at the meeting last week - it's important to close the project as the Polopoly database has been upgraded, however we should revisit this during any future upgrade work.  I would recommend that as part of any future Polopoly database upgrade, assuming our workaround in the Content References area is still in place, we switch off this workaround and check whether the slowness problem still exists in the media bank.  If we refresh TEST from LIVE before performing the upgrade, as per the recommendations coming out of the ITS098 project, we should be able to reproduce the problem, if it still exists, in the TEST environment without any impact on LIVE.  At that point, we can make a call as to whether to leave the workaround in place, or look for a more permanent solution.

Cheers,

Mairi