Completion Report

Project Summary:

 

Overview We have so much information coming from our server estate that it is difficult to spot incipient problems early enough as the pattern of messages we need to see to spot problems are drowned in a flood of other messages. Typically we hundreds messages a day coming from the applications, server logs and scheduled jobs. This project was to investigate whether an open source tool – called Sentry – can help us spot problems earlier so that we can rectify faults before they cause interruptions to service. 

 

What did we do? Tried to set up Sentry in our development and test environments to monitor 2 applications one simple to try out ideas on, freedom of information and one complex one, MyEd development environment.

 

What did we find out? We had quite a few issues to get Sentry working in the irst place. We believe that having setup 2 applications successfully other applications should now be easier to be setup.

What next? While this project has setup a Sentry prototype, quite a few tasks are required to make Sentry a service. Rolling out Sentry is no longer an innovation project and needs to be picked up by  other means and most likely via a follow on project.

 

Analysis of Resource Usage:

Staff Usage Estimate: 30 days

Staff Usage Actual: 28 days

Staff Usage Variance: -7%

Other Resource Estimate: 0 days

Other Resource Actual: 0 days

Other Resource Variance: 0%

Explanation for variance:

 

Deliverables based on project proposal

ItemDeliverablesCommentsAchieved
1This proposal is based on the 'Proactive application log management'. While the initial 'Proactive application log management' will implement a prototype, this proposal will ensure that the application logging can be delieverd as a service to a wider group of areas as well as within IS Applications.he initial 'Proactive application log management' innovationproject was not funded. This meant that this innovation project had to redefine its delivery and had to deliver a prototype. This also meant that several of the deliveries were not achievable.No
2A structure which will allow application logs to be grouped by services and diferent teams define best practice how IS Applications record and report application failures. This will include what failures are recorded, how they are prioritised and grouped into different failures No
3Resilience of a solution will be looked at and how failures of the Sentry service will impact the loggingAs this project only delivered a prototype (see item 1), a resilient solution has not been delivered.No
4Resources: IS Applications staff, split 50:50 between Applications Production Management and Development ServicesA mixture of staff from both Development Services and Production Management have worked on the project.Yes
5Setup VMA dedicated VM has been setup for SentryYes
6Setup SentrySentry has been setup using DockerYes
7TAD for Sentry Setup Yes
8Sentry made a serviceThere are several outstanding tasks to make Sentry a supportable service. For example the use of Docker and the fact that we have no agreed structure to add new applications to SentryNo
9Setup applications to use SentryOnly a very few applications have been setup in Sentry. FOI (DEV/TEST) and a sandpit MyEd versionNo

Key Learning Points:

Learning pointComments
Project ManagementProject Management was performed by project initiater/sponsor. There was no project manager managing the project. This meant that methodology was seldom followed and the rigor to review milestones and progress was not in place. Innovation projects should have assigned project managers.
Impossible dependenciesatstartofprojectTwo innovation funds were submitted. One to setup Sentry, and one to rollout Sentry. The rollout Sentry project was funded, but not the setup Sentry. This meant that this project was not able to deliver what was asked for from the beginning unless part of the first project was delivered. This project therefore delivered a mixture of the origianl two projects, but was not able to deliver all. In future such dependencies must be discussed with project sponsor before the project starts.

 

Outstanding issues:

ItemIssueComments
1Infrastructure

Agreement how we use Docker needs to be in place. before Sentry can be made a service.

 

2Structure for Sentry usageNo Sentry structure has been setup and discussed how Sentry would be used and setup. Do we create one big area? Sperate by environment, service pr by teams?
3Applications monitoredOnly a few applications were monitored in DEV and TEST. More applications need to be added to learn hoe Sentry works.

Sentry in the current setup cannot be rolled out as a service. A follow up project is required, most likely at around 30 to 50 days effort.

Project Info

Project
Proactive application log management - implementation
Code
API007
Programme
IS Innovation - Applications (API)
Project Manager
Stefan Kaempf
Project Sponsor
Simon Marsden
Current Stage
Close
Status
Closed
Start Date
23-Nov-2015
Planning Date
n/a
Delivery Date
n/a
Close Date
n/a

Documentation

Close