Completion Report

Project Summary

 

Summarise the project : did it achieve its deliverables and objectives (see project brief and any subsequent changes), scope changes, overall review.

The project had  two main aims: the first was to replace the current logging application (Elastic) with an open-source version of the same application and

the second was to identify any gaps in the current monitoring and alerting processes for ITI enterprise servers and storage, to analyse if this can be improved

using the current  application (Zenoss), and if so to implement those improvements. 

 

Scope:

no change 

Logging: 

  • Analyse current use of Elastic, including additional purchased enhancements 

  • Upgrade/migration of Elastic to open-source version 

  • Provision of infrastructure for upgraded/migrated version of Elastic 

Monitoring: 

  • Analysis of monitoring and alerting requirements for enterprise server and storage infrastructure for ITI Enterprise Services

  • Upgrade of Zenoss 

  • Identification of gaps in current monitoring and alerting processes for ITI ES 

  • Identification of gaps in presentation of monitoring and alerting information for ITI ES 

  • Implement changes to  Zenoss fill gaps 

 

Objectives and Deliverables

 

 

                        

Description of the Objective 

Success Criteria 

                                                                                                                                                 Achieved                                                                                              

Objective 1  

Prepare Elastic for upgrade/migration  

 

 

Deliverable D1.1 

Healthcheck review of current use of Elastic  

Project Sponsor, Service Owner 

  Withdrawn : Due to extensive schema changes between Elastic versions the decision was taken to re-implement with new design.

Decision 14/08/2020

Deliverable D1.2 

Resolve issues identified by Healthcheck 

Project Sponsor, Service Owner 

 

  Withdrawn : Due to extensive schema changes between Elastic versions the decision was taken to re-implement with new design.

Decision 14/08/2020

Deliverable D1.3 

 

Recommendation of target version for upgrade/migration 

Project Sponsor, Service Owner 

 

  Yes - Decision 09/04/2020

 

 

 

 

Objective 2 

Upgrade/migrate Elastic 

 

 

Deliverable D2.1 

Test upgrade/migrated version 

Project Sponsor, Service Owner 

 Yes - 13/8/2020

Deliverable D2.2 

Implement upgrade/migrated version 

Project Sponsor, Service Owner 

 Yes

- implemented 31-8-2020

- halted input from LIVE servers to old version 28-9-2020

Deliverable D2.3 

Update service documentation 

Service Owner 

 Yes

Deliverable D2.4 

Handover to support 

Project Sponsor, Service Owner 

 Yes - support is internal to

ITI Enterprise Services

 

 

 

 

Objective 3 

Review monitoring and alerting requirements 

 

 

Deliverable D3.1 

Analysis and prioritisation of current and likely future monitoring and alerting requirements 

Requirements signed off by Project Sponsor, Senior Supplier, External Service Owners 

 Yes

 

 

 

 

Objective 4 

Gap analysis – monitoring requirements 

 

 Yes - 16/07/2020

Deliverable D4.1 

Upgrade of Zenoss to new version 

Project Sponsor, Service Owner 

 

 Yes - 28/05/2020 

Deliverable D4.2 

Gap analysis of Zenoss (as currently used and as could be used) against prioritised requirements (D 3.1) 

Project Sponsor, Service Owner 

 

 Yes - 30/07/2020

Deliverable D4.3 

Recommendation for next steps 

Project Sponsor, Service Owner 

 Yes - gaps prioritised  and effort estimated and must-have improvements to be implemented approved.

 

 

 

 

Objective 5 

Improve monitoring and analysis in upgraded monitoring service (Zenoss) 

 

 

Deliverable D5.1 

Implementation of approved, prioritised  improvements 

Approved by Project Sponsor, Senior Supplier 

 Yes - XX/XX/2020

Deliverable D5.2 

Updated service documentation 

Approved by Project Sponsor, Senior Supplier 

 Yes

Deliverable D5.3 

Handover to support. 

Approved by Project Sponsor, Senior Supplier 

 Yes, support is internal to IS Enterprise Services

 

 

 

 

 

 

 

 

 

 

Requirements 

 

 

Requirement 

Description 

MoSCoW 

Set By 

                                                                      Achieved                                                   

Requirement 1 

Replacement of logging solution (Elastic) with open source/free version before October 2020 

Project Sponsor 

  Yes

Requirement 2 

Quality of logging information is maintained. 

Project Sponsor 

   Yes

Requirement 3 

Gap analysis and prioritisation of requirements for monitoring and alerting metrics (for enterprise services servers and storage) including how information should be most usefully presented. 

Project Sponsor 

   Yes

Requirement 4 

Reduce gaps in upgraded Zenoss service as identified and in gap analysis and prioritised. 

Project Sponsor 

    Yes

 

Benefits

The following benefits were realised:

Benefit  

Description 

Requirement      

      Achieved                                              
  1.  

Opportunity to better meet demand, capacity and availability targets will be identified. 

   Yes
  1.  

Support costs will be reduced. 

   Yes
  1.  

Gaps in current monitoring service will be addressed 

3, 4 

   Yes
  1.  

Requirements for monitoring and alerting will be documented and can be re-used for future projects 

   Yes

 

Future Benefits post-project.

  • Knowledge of both Elastic and Zenoss has been deepened and broadened within ITI Enterprise
  • More members of the team now have knowledge of both services
  • Extended knowledge  of Puppet within ITI Enterprise
  • Elastic service is now more robust, resilient, scalable and sustainable
  • Extended knowledge of Python within ITI Enterprise

 

 

Analysis of Resource Usage:

ITI Usage Estimate: 100 days ITI Effort,  10 days project services effort,

ITI Usage Actual: 155 days

ITI Usage Variance: 55%

PM Resource Estimate: 10 days

PM Resource Actual: 28 days

PM Resource Variance: 170%

Other  Resource Estimate: 0 days

Other Resource Actual: 0 days

Other Resource Variance: 0%

 

Explanation for variance:

 

The estimate set at initiation for project management effort (10 days) was too low and should have been set at 20% of the expected ITI effort

Additional effort was required to complete the update to the new Elastic service as version 7 was not compatible with puppet and discovery work indicated that schema changes between the current version of Elastic (6) and the target version (7) were greater than had been anticipated, and additional effort was required to  update the logstash filters. Due to conflict for resources with higher priority BAU work, the detailed estimate for work required to complete the Zenoss deliverables was delayed.

 

The additional effort required by assigning resources who did not have experience in the technologies was under-estimated.

 

The duration of the project was extended three times, on 4th May after a re-estimate after initial project planning,  on 18th September to allow additional time to address gaps in  monitoring  with Zenoss and  on 20th October after a re-estimate and re-prioritisation of the Zenoss gap improvements. The extended duration impacted the project management effort required.

 

Lessons Learned - Issues and Risks  

Also - were any innovations tried during the project that failed? Include these in the ITI Lessons Learned Log and communicate these back to the IS Change Management team.

Project Manager must add  Lessons  Learned to the ITI Lessons Learned log.

 

1. Project estimations

At planning, the project was estimated to require 100 days ITI effort, with 10 days Project Management effort. It was expected that as the project team learned more about the work required that the estimate would be increased. Due to pressure of work from higher priority BAU work the team did not re-estimate until later in the project and although the PM recorded this as a risk, priority should be given to mid-project re-estimation in the future so that resourcing can be properly planned.

The original estimate for project management effort (10%) was too low and should initially be set at 20% of the overall project estimate.

The project team has gained more experience of using 3-point estimates which should help us work towards more accurate project  and resource planning.

 

2. Leadership and ownership

The project afforded opportunities for staff to take leadership and show ownership of work packages.  However consideration should be given to staff capabilities when assigning these roles and we should be quicker to  support or reassign where required.

 

 

Project Audit

N/A

 

Outstanding issues

  • There are a number of  "should-have" and "could-have" improvements which  have been estimated and which may be implemented as part of BAU. Kenny MacDonald will copy JIRAs to BAU JIRA list
  • Decommissioning of old Elastic 6 servers will take place  as a BAU task once the data contained on those servers is 12 months old. Kenny MacDonald will add this task to BAU task list.

 

Project Sponsor Comment

My thanks to everyone on the project team. I think we have better managed and better understood infrastructure on which to build our logging and monitoring.

 

 

 

 

Project Info

Project
Logging, Monitoring, Metrics and Alerting - improvements and gap analysis
Code
ENT205
Programme
ITI - Enterprise Services (ENT)
Management Office
ISG PMO
Project Manager
Morna Findlay
Project Sponsor
Graeme Wood
Current Stage
Close
Status
Closed
Project Classification
Run
Start Date
20-Jan-2020
Planning Date
23-Mar-2020
Delivery Date
17-Dec-2020
Close Date
08-Jan-2021
Programme Priority
3
Overall Priority
Normal
Category
Discretionary

Documentation

Close