Completion Report
Project Summary
Summarise the project : did it achieve its deliverables and objectives (see project brief and any subsequent changes), scope changes, overall review.
The project had two main aims: the first was to replace the current logging application (Elastic) with an open-source version of the same application and
the second was to identify any gaps in the current monitoring and alerting processes for ITI enterprise servers and storage, to analyse if this can be improved
using the current application (Zenoss), and if so to implement those improvements.
Scope:
no change
Logging:
-
Analyse current use of Elastic, including additional purchased enhancements
-
Upgrade/migration of Elastic to open-source version
-
Provision of infrastructure for upgraded/migrated version of Elastic
Monitoring:
-
Analysis of monitoring and alerting requirements for enterprise server and storage infrastructure for ITI Enterprise Services
-
Upgrade of Zenoss
-
Identification of gaps in current monitoring and alerting processes for ITI ES
-
Identification of gaps in presentation of monitoring and alerting information for ITI ES
-
Implement changes to Zenoss fill gaps
Objectives and Deliverables
|
|
Description of the Objective |
Success Criteria |
Achieved |
|
Objective 1 |
Prepare Elastic for upgrade/migration |
|
|
|
Deliverable D1.1 |
Healthcheck review of current use of Elastic |
Project Sponsor, Service Owner |
Withdrawn : Due to extensive schema changes between Elastic versions the decision was taken to re-implement with new design. Decision 14/08/2020 |
|
Deliverable D1.2 |
Resolve issues identified by Healthcheck |
Project Sponsor, Service Owner |
Withdrawn : Due to extensive schema changes between Elastic versions the decision was taken to re-implement with new design. Decision 14/08/2020 |
|
Deliverable D1.3
|
Recommendation of target version for upgrade/migration |
Project Sponsor, Service Owner
|
Yes - Decision 09/04/2020 |
|
|
|
|
|
|
Objective 2 |
Upgrade/migrate Elastic |
|
|
|
Deliverable D2.1 |
Test upgrade/migrated version |
Project Sponsor, Service Owner |
Yes - 13/8/2020 |
|
Deliverable D2.2 |
Implement upgrade/migrated version |
Project Sponsor, Service Owner |
Yes - implemented 31-8-2020 - halted input from LIVE servers to old version 28-9-2020 |
|
Deliverable D2.3 |
Update service documentation |
Service Owner |
Yes |
|
Deliverable D2.4 |
Handover to support |
Project Sponsor, Service Owner |
Yes - support is internal to ITI Enterprise Services |
|
|
|
|
|
|
Objective 3 |
Review monitoring and alerting requirements |
|
|
|
Deliverable D3.1 |
Analysis and prioritisation of current and likely future monitoring and alerting requirements |
Requirements signed off by Project Sponsor, Senior Supplier, External Service Owners |
Yes |
|
|
|
|
|
|
Objective 4 |
Gap analysis – monitoring requirements |
|
Yes - 16/07/2020 |
|
Deliverable D4.1 |
Upgrade of Zenoss to new version |
Project Sponsor, Service Owner
|
Yes - 28/05/2020 |
|
Deliverable D4.2 |
Gap analysis of Zenoss (as currently used and as could be used) against prioritised requirements (D 3.1) |
Project Sponsor, Service Owner
|
Yes - 30/07/2020 |
|
Deliverable D4.3 |
Recommendation for next steps |
Project Sponsor, Service Owner |
Yes - gaps prioritised and effort estimated and must-have improvements to be implemented approved. |
|
|
|
|
|
|
Objective 5 |
Improve monitoring and analysis in upgraded monitoring service (Zenoss) |
|
|
|
Deliverable D5.1 |
Implementation of approved, prioritised improvements |
Approved by Project Sponsor, Senior Supplier |
Yes - XX/XX/2020 |
|
Deliverable D5.2 |
Updated service documentation |
Approved by Project Sponsor, Senior Supplier |
Yes |
|
Deliverable D5.3 |
Handover to support. |
Approved by Project Sponsor, Senior Supplier |
Yes, support is internal to IS Enterprise Services |
|
|
|
|
|
|
|
|
|
Requirements
|
Requirement |
Description |
MoSCoW |
Set By |
Achieved |
|
Requirement 1 |
Replacement of logging solution (Elastic) with open source/free version before October 2020 |
M |
Project Sponsor |
Yes |
|
Requirement 2 |
Quality of logging information is maintained. |
M |
Project Sponsor |
Yes |
|
Requirement 3 |
Gap analysis and prioritisation of requirements for monitoring and alerting metrics (for enterprise services servers and storage) including how information should be most usefully presented. |
M |
Project Sponsor |
Yes |
|
Requirement 4 |
Reduce gaps in upgraded Zenoss service as identified and in gap analysis and prioritised. |
S |
Project Sponsor |
Yes |
Benefits
The following benefits were realised:
|
Benefit |
Description |
Requirement |
Achieved |
|
|
Opportunity to better meet demand, capacity and availability targets will be identified. |
3 |
Yes |
|
|
Support costs will be reduced. |
1 |
Yes |
|
|
Gaps in current monitoring service will be addressed |
3, 4 |
Yes |
|
|
Requirements for monitoring and alerting will be documented and can be re-used for future projects |
3 |
Yes |
Future Benefits post-project.
- Knowledge of both Elastic and Zenoss has been deepened and broadened within ITI Enterprise
- More members of the team now have knowledge of both services
- Extended knowledge of Puppet within ITI Enterprise
- Elastic service is now more robust, resilient, scalable and sustainable
- Extended knowledge of Python within ITI Enterprise
Analysis of Resource Usage:
ITI Usage Estimate: 100 days ITI Effort, 10 days project services effort,
ITI Usage Actual: 155 days
ITI Usage Variance: 55%
PM Resource Estimate: 10 days
PM Resource Actual: 28 days
PM Resource Variance: 170%
Other Resource Estimate: 0 days
Other Resource Actual: 0 days
Other Resource Variance: 0%
Explanation for variance:
The estimate set at initiation for project management effort (10 days) was too low and should have been set at 20% of the expected ITI effort
Additional effort was required to complete the update to the new Elastic service as version 7 was not compatible with puppet and discovery work indicated that schema changes between the current version of Elastic (6) and the target version (7) were greater than had been anticipated, and additional effort was required to update the logstash filters. Due to conflict for resources with higher priority BAU work, the detailed estimate for work required to complete the Zenoss deliverables was delayed.
The additional effort required by assigning resources who did not have experience in the technologies was under-estimated.
The duration of the project was extended three times, on 4th May after a re-estimate after initial project planning, on 18th September to allow additional time to address gaps in monitoring with Zenoss and on 20th October after a re-estimate and re-prioritisation of the Zenoss gap improvements. The extended duration impacted the project management effort required.
Lessons Learned - Issues and Risks
Also - were any innovations tried during the project that failed? Include these in the ITI Lessons Learned Log and communicate these back to the IS Change Management team.
Project Manager must add Lessons Learned to the ITI Lessons Learned log.
1. Project estimations
At planning, the project was estimated to require 100 days ITI effort, with 10 days Project Management effort. It was expected that as the project team learned more about the work required that the estimate would be increased. Due to pressure of work from higher priority BAU work the team did not re-estimate until later in the project and although the PM recorded this as a risk, priority should be given to mid-project re-estimation in the future so that resourcing can be properly planned.
The original estimate for project management effort (10%) was too low and should initially be set at 20% of the overall project estimate.
The project team has gained more experience of using 3-point estimates which should help us work towards more accurate project and resource planning.
2. Leadership and ownership
The project afforded opportunities for staff to take leadership and show ownership of work packages. However consideration should be given to staff capabilities when assigning these roles and we should be quicker to support or reassign where required.
Project Audit
N/A
Outstanding issues
- There are a number of "should-have" and "could-have" improvements which have been estimated and which may be implemented as part of BAU. Kenny MacDonald will copy JIRAs to BAU JIRA list
- Decommissioning of old Elastic 6 servers will take place as a BAU task once the data contained on those servers is 12 months old. Kenny MacDonald will add this task to BAU task list.
Project Sponsor Comment
My thanks to everyone on the project team. I think we have better managed and better understood infrastructure on which to build our logging and monitoring.