EST098 Closure Report

Document Sign Off

Name Project Role Date Signed Off
Grant Ferguson Project Sponsor 23/03/18
Colin Pritchard Senior User 13/03/18 & 23/03/18
Derrick Matheson Senior Supplier 23/03/18
Hannah Johnstone Development Technology 16/03/18
Ron Macleod Applications Management 20/06/17
Karen Adamson Head of Estates Finance 13/03/18 & 23/03/18
Zoe Stephens Estates Change Manager 23/03/18
Paul Munro Mass 23/03/18
Chris Orrell Project Manager 13/03/18 & 23/03/18
Pauline Smith Estates System Support Manager     13/03/18
Jane Brodie Estates System Support Manager     13/03/18
Sheila Scott Asset Management Project    23/03/18

Project Summary: Archibus / WebCentral Upgrade to v23.1

 

Overview from Brief:

Estates and Buildings Estates Department use an application called Archibus to control and manage various aspects of the University Estate. Archibus includes a number of modules for example; Reactive Maintenance On Demand via a Helpdesk facility (delivered by EST080), Compliance Maintenance, Space Planning and Property and Lease. The University currently uses Archibus Webcentral v21.3 and this project is to upgrade the Webcentral application to v23.1. This Brief proposes as scope that only the Helpdesk Module will deliver functional improvements requiring business change and for this project all other modules will deliver tested business as usual functionality. Upgrading to v23.1 will also require upgrades to Archibus Smart Client and the Database. Archibus v23.1 offers Estates new functionality but also fixes a number of bugs from EST080.

The Estates and Buildings Estates Department team have also requested that a number of Functional Enhancements also be considered for delivery with this project and these are termed: Enhancements to Mobile Device, Additional JIRAs and Issues, Asset Management Integration (see comment below).

Additionally IS have identified that a number of technical upgrades are also required: WebCentral Application Server requires a 'tidy up'; Tomcat Upgrade, Oracle Database upgrade and to migrate to a new system of Change Control (Source Control) and Automated Deployment, supported by a separate activity with our 3rd party (Mass) to document all customisations (technical, functional) and confirm the need to retain in v23.1. Although the primary focus from Estates is to prioritise v23.1 delivery, for reasons of priority, interdependencies and business risk this Brief recommends an optimum approach for both Business and IS outcomes.

A number of lessons learned from EST080 also form part of the scope or approach.

The Archibus application is supported by a company called Mass and their involvement will be critical to successful delivery of this project.

 

Objectives from Brief:

Although the vast majority of objectives and deliverables were achieved in full this achievement belies the significant challenges this project posed. Head of Project Services asked to record a comment that the original go-live of 13th November was not achieved and this slipped to 5th February. This was due to a major defect being encountered in UAT which took the supplier several weeks to resolve. Once this defect had been resolved and UAT resumed the Project Board made a decision that training and go-live should not be completed either side of Christmas and so training was moved to January with a consequential knock on effect for go-live.

 

No.

Objective Description

Priority

Objective Met? (Yes / No)

O1

Technical upgrades – see deliverables D1.1 to D1.6

Must

Yes

O2

Upgrade Archibus / WebCentral to v23.1 – see deliverables D2.9 to D2.9

Must

Yes

O3

Functional Enhancements – see deliverables D3.1 – D3.4

Must / Should

Yes

 

Deliverables from Brief:

No.

Deliverable Description

Priority

Delivered? (Yes / No)

D1.1

Server 'Tidy up'

New applications servers were built on their own environment as they had outgrown the existing shared environment.

Must

Yes

D1.2

Tomcat Upgrade

Must

Yes

D1.3

Oracle Database upgrade

A patch for the Oracle database from 11.0.2.3 to 11.0.2.4 was not applied. This would have required a business outage in January (The busiest month for Estates) and it was requested that this deferred. It was formally agreed that this requirement would be picked up by EST099.

Must

No

D1.4

Migrate to a new system of Change Control (SVN to GIT) and introduce automated deployment

Should

Yes

D1.5

Document all customisation and embed changes documentation maintenance into BAU

This has been achieved via the use of GIT Lab

Must

Yes

D1.6

Maintain operational reports

Must

Yes

D2.1

Provide release notes since 21.3 and advise application changes in relation to current University process

Archibus release notes were supplied and Mass performed a measure of impact assessment but this was shown to be frequently inadequate. This item is referenced in the lessons learned.

Must

Not in full

D2.2

Implementation plan – integrated with D2.3

Must

Yes

D2.3

Maintain complete build record

Must

Yes

D2.4

Document as-is and to-be Helpdesk processes and prepare impact assessment – completed by Project Services

Must

Yes

D2.5

Prepare test strategy and plan, including test scripts and perform UAT for all modules.

Must

Yes

D2.6

Load testing completed and report prepared – completed by Estates

Must

Yes

D2.7

Prepare communications and engagement plan – Completed and delivered by Estates

Must

Yes

D2.8

Prepare training plan and deliver training – Training materials prepared by Estates and training delivered by Mass

Must

Yes

D2.9

Deliver tested as-is business functionality for Compliance Maintenance, Space Planning and Property and Lease

Must

Yes

D3.1

Deliver agreed enhancements to mobile devices

Must

Yes

D3.2

Deliver agreed JIRAs

Must

Yes

D3.3

Enable Asset Management Module – base functionality only and not the Enterprise Asset Management functionality which the Project Board deemed out of scope. This will be addressed by EST099.

Must

Yes

D3.4

All software to be fully documented before adding to environments

Documented and controlled via GIT Lab

Must

Yes

 

Scope from Brief:

No.

Scope Description

Status (Met / Changed)

All items of scope for EST098 are defined by the deliverables listed above and were actively tracked by the document EST098 scope v1 1 which was last formally amended 19-07-17.

 

Analysis of Resource Usage:

Staff Usage Estimate:               258

Staff Usage Actual:                  313.8

Staff Usage Variance:              +55.8

Other Resource Estimate:         0

Other Resource Actual:            0

Other Resource Variance:        0

 

Explanation for variance:

Change Date

Reason for Change

Previous Estimate

Revised Estimate

Link to Change Log entry

03-08-17

Original project estimate proposed in Project Brief was 197           

75

197

PICCL 1

07-12-17

Budget increased 197 to 258

197

258

PICCL 4

 

Key Learning Points

Key Lessons to be Learned

  1. Documented lessons to be learned should be learned and adopted and not treated as optional. There were a number of recommendations from EST080 project that were either not adopted or adopted under protest, see all of the scope items from point 2 below.
  2. A number of key infrastructure improvements were proposed as items of scope but these were challenged and removed by The Estates Department, whose principle focus was minimum time to go-live. These improvements were; building new application servers that had outgrown their current shared environment, introducing source code change control (GIT Lab) and automated deployment (Bamboo). New servers were only introduced after challenge by IS. The latter two items were only included because the supplier, due to existing commitments, was unable to bring go-live forward by one week and so IS proposed again that these items should be in scope. There should have been no debate because they were clear recommendations from EST080. This point was discussed at the Project Closure Board where it was agreed that for future infrastructure improvements Estates should be advised which items are mandatory and these items should not be debated.
  3. It is critical to have a clear understand of what is changing as part of an upgrade, specifically a detailed understanding and impact analysis of all functionality and workflow changes. The responsibility for this rests with Mass but experience has shown that this requires a joint effort between Mass and the University:-
    1. Mass supplied Archibus release notes which proved to be incomplete.
    2. Mass provided a v23.1 walkthrough but this did not provide in sufficient detail configuration that related to University set up and workflow.
    3. A Project Services business analyst review of Archibus release notes against a Mass demonstration version of v23.1 system highlighted a significant number of changes but this was unable to give the complete picture.
    4. New workflow was highlighted during UAT which caused some major issues.

Despite the actions above a full appreciation of the changes was not fully understood until a full iteration of UAT had been completed. Post project the team discussed this issue and concluded that at the start of a future upgrade project this challenge should be discussed in detail and although it is Mass’s responsibility to advise the changes a joint approach to tackle this problem should be agreed. It was recognised that this activity cannot be done in isolation. Perhaps a workshop with key stakeholders including system users, stepping through a working configured test system, led by Mass.

One suggestion that should be given serious consideration is to create a test copy of LIVE in an entirely separate environment which can be used to host an early upgraded environment predominantly for the purposes of understanding and impacting changes – this should be easier to achieve now that vanilla Archibus WAR files exist with various customisations being stored in GIT Lab.

  • WIS advised that if this proposal for sharing a copy of LIVE is to be considered then this MUST be done in consultation with the Data Protection Champion and a Data Sharing Agreement approved.

Whatever approach is agreed this should be done as part of defining the project approach and factored into the project brief.

  1. Estates commented that, although they recognise efforts by Project Services for longer term continuity of project managers supporting Estates projects, there had been a different project manager for each Archibus project. This obviously impacts continuity and knowledge transfer and Estates have asked that any new PMs spend time with previous incumbents. As a minimum this document should be reviewed and actioned.
  2. Significant lessons were learned from UAT and these should be implemented (see section below).

Project Level Lessons to be Learned

  1. If a future Estates project requires the Estates system managers then the EBIS team should be part of the core project team from the outset to help shape the Project Brief.
  2. Having a detailed Project Brief was regarded positively but there should be active engagement of those areas responsible for different modules to be sure they understand what will and will not be delivered. At one point Estates PPM expected the Enterprise Asset Management Module to be enabled which was never in scope.
  3. IS were asked to submit and commit to a timing plan with all standard contingency removed in order to attempt to achieve the earliest possible go-live. This should not be done in future because project team members are not dedicated and attempting to achieve such deadlines when the wider organisation is not structured to support this method of working and this results in placing individuals under too much pressure and increases the delivery risks.
  4. Having a single point of contact with Mass worked well and this should be repeated.

DEV Lessons Learned

  1. IS created a vanilla WAR of Archibus WebCentral and held each customisation on an appropriate software branch which allowed complete control of the application. This meant Mass needed to only supply the individual files that were changing instead of the 7000+ application files with each fix.

    1. This was essential to the new source code change control and should be repeated for all future projects.

      1. However, when changes were made by Mass third party contractors individually changed files were not always supplied and so IS had to compare different files sets to determine which files had actually changed. This could be quite time consuming and consideration should be given to developing a comparison tool that allows University to easily identify only the files that have changed (and need to be added to our systems) when Mass supply bulk files for upload. 
  2. It was considered inappropriate for Estates to have access to DEV. This meant beyond the most cursory of checks fixes were not checked in this environment before elevating to TEST. For future projects this task should be performed by Development Services and a budget provision made accordingly.
  3. Mass supplied the files or scripts to be added / run attached directly to the JIRA to be fixed so there was no misunderstanding of which files related to which issue and this should be repeated.

UAT Lessons Learned

  1. Mass having access to JIRA and Hipchat worked well and should be repeated.
  2. Having the EBIS team as the first point of contact to triage potential defects worked well. This meant JIRAs were only raised for bona-fide defects and the required information was supplied with the JIRA for Mass from the outset. This approach also allowed the EBIS to support users when the issue was more related to user inexperience.
  3. All future projects which require testing should use TestRail without exception.
  4. Having three full iterations at UAT worked well and should be repeated. 
  5. Having each iteration concentrated over a 2-3 days period worked well and should be repeated. This allows for a period of defect fixing to be completed before the next iteration is planned.
  6. Having daily catch up calls with Mass, Estates and IS to review and progress each defect to minimise the time taken to fix them worked well and should be planned as part of the project approach.
    1. EST098 encountered a serious issue that took several weeks to resolve. Serious consideration should be given to not attempting to plan UAT over a fixed period but to agree an approach of three iterations, planning the second and third iteration when sufficient progress fixing defects has been achieved.
      1. Employing this approach preserves the required quality (and minimises subsequent impact to the business) of the implementation with time being the variable.
  7. For the reasons outlined in item 17 above attempting to complete UAT within a fixed 4 week period due to the complexity of issues likely to be encountered is unrealistic. As a minimum more time should be allowed but strong consideration should be given to performing a more agile approach outlined above.
  8. Complete both performance and manual load testing. Although Estates did not express any performance concerns at go-live some teams thought the performance may have been slower and there were performance measurements to objectively assess any concerns.
  9. The EBIS team required various test accounts to be maintained which are lost when a refresh from LIVE is completed. These test accounts should be identified from the outset and actions defined to re-install these rather than attempt to recover these as a corrective action which was not clearly understood and caused many problems.
  10. Mass having the ability to review issues directly within TEST worked well and conditional upon them not making any changes in this environment should be repeated.
  11. Greater involvement from end users should be sought for UAT. For example ATL teams use the helpdesk module differently to the Helpdesk team and mobile users understand in detail how this functionality works. It is therefore a recommendation that as part of UAT a step through exercise is planned so a wider user base can review the upgrade.
  12. Analysis of defects missed at UAT. Both Mass and Estates were asked to review JIRAs and classify them according to root cause. The approximate respective percentages are included in brackets for example: item 1 below Estates classified 15% of defects as having been ‘missed by Mass’ and Mass thought 24% of defects were ‘missed by Mass’.
  1. A defect missed by Mass (Estates 15%: Mass 24%)

    • Lesson to learn for future projects Mass to consider how testing can be improved.
  2. A defect that testing by Mass could not have highlighted because the set-up of LIVE and the Mass test environments are different (EASE non-EASE etc.) (Estates 2%: Mass 8%)
    • Serious consideration should be given to whether or not Mass can hold a copy of TEST.  
  3. A defect or change in product feature or workflow attributable to a change to Archibus that was unknown to Mass. (Estates 6%: Mass 18%)
    • *** I expect this category will only be used by Mass and the University (at this point in time) can only use category ‘4’ and later it will be interesting to know how many changes Mass weren’t advised of ***
  4. A defect or change in product feature or workflow attributable to a change to Archibus that was not highlighted to the University through the Impact Assessment. (Estates 7%: Mass 4%)
    • *** see note ‘3’ above ***
  5. A defect whose root cause is attributable to an error in set up or some other action that was / was not performed correctly / incorrectly by the University. (Estates 50%: Mass 38%)
  6. A defect that was identified post go-live that UAT should have identified. (Estates 14%: Mass 6%)
    • Lesson to learn continually improve UAT. A number of JIRAs were not classified by Estates or Mass hence percentages don’t total 100.

Training Lessons Learned

  1. Training was delivered by Mass and this was well received and it is recommended for future upgrades that Mass again deliver the training.

Go-Live Lessons Learned

  1. Having a combined Dev Tech / Apps Man and Estates Implementation Plan with review points through go-live worked well and should be repeated https://www.wiki.ed.ac.uk/display/IATD/EST098+LIVE+Build+Record However, there were a number of items missed and it recommended that for future projects there is:-

    1. A specific review of the plan involving a joint walk through of both University and Mass team members. Additional scripts needed to be run that also required a restart of WebCentral and additional spread sheets for import were missed.
    2. Start the implementation plan as soon as any dependent or related pieces of work are scheduled to occur at go-live are identified. Apps Man had planned to co-ordinate a related change at go-live which was first identified four months earlier but the wider project team had no knowledge of this change at go-live.
    3. Estates change management should also sign-off this plan as business changes were planned that the project team had no sight of. E.g. importing documents for dynamic risk assessment.
  2. The validity of each entry of the SLA verification steps spread sheets must be tested and confirmed. A database extract taken by the University modified for import by Mass contained values of the wrong format wrong format: Two values ‘Resolve’ and ‘Complete’ should have been abbreviated to ‘R’ and ‘Com’. There is a definite lesson to learn that the format of entries on spread sheets for import must to be checked prior to go-live.
  3. Having Mass on site for go-live worked well.
  4. Having a clear communications plan for the Estates users controlled and delivered by Estates worked well. Not committing to the go-live date until UAT had been completed and key defects resolved worked well.

Recommendations for future similar projects:

  1. Implement the above lessons learned.
  2. The control and application of database upgrade scripts is still manual. In a similar way to application code now being controlled by GIT Lab and Bamboo it should be possible to achieve something similar for changes to the database. This would be worth investigating and including as an item of future scope. 
  3. Develop a comparison tool that allows University to easily identify only the files that have changed (and need to be added to our systems) when Mass supply bulk files for upload – see DEV above.

Outstanding issues:

No additional outstanding issues.

 

Project Info

Project
Archibus Upgrade
Code
EST098
Programme
Estates Systems & Technology Maintenance (EST)
Management Office
ISG PMO
Project Manager
Chris Orrell
Project Sponsor
Grant Ferguson
Current Stage
Close
Status
Closed
Start Date
17-Apr-2017
Planning Date
n/a
Delivery Date
n/a
Close Date
18-May-2018
Programme Priority
2
Overall Priority
Normal
Category
Discretionary