Overview We have so much information coming from our server estate that it is difficult to spot incipient problems early enough as the pattern of messages we need to see to spot problems are drowned in a flood of other messages. Typically we hundreds messages a day coming from the applications, server logs and scheduled jobs. This project was to investigate whether an open source tool – called Sentry – can help us spot problems earlier so that we can rectify faults before they cause interruptions to service.
What did we do? Tried to set up Sentry in our development and test environments to monitor 2 applications one simple to try out ideas on, freedom of information and one complex one, MyEd development environment.
What did we find out? We had quite a few issues to get Sentry working in the irst place. We believe that having setup 2 applications successfully other applications should now be easier to be setup.
What next? While this project has setup a Sentry prototype, quite a few tasks are required to make Sentry a service. Rolling out Sentry is no longer an innovation project and needs to be picked up by other means and most likely via a follow on project.
Analysis of Resource Usage:
Staff Usage Estimate: 30 days
Staff Usage Actual: 28 days
Staff Usage Variance: -7%
Other Resource Estimate: 0 days
Other Resource Actual: 0 days
Other Resource Variance: 0%
Explanation for variance:
Deliverables based on project proposal
|1||This proposal is based on the 'Proactive application log management'. While the initial 'Proactive application log management' will implement a prototype, this proposal will ensure that the application logging can be delieverd as a service to a wider group of areas as well as within IS Applications.||he initial 'Proactive application log management' innovationproject was not funded. This meant that this innovation project had to redefine its delivery and had to deliver a prototype. This also meant that several of the deliveries were not achievable.||No|
|2||A structure which will allow application logs to be grouped by services and diferent teams define best practice how IS Applications record and report application failures. This will include what failures are recorded, how they are prioritised and grouped into different failures||No|
|3||Resilience of a solution will be looked at and how failures of the Sentry service will impact the logging||As this project only delivered a prototype (see item 1), a resilient solution has not been delivered.||No|
|4||Resources: IS Applications staff, split 50:50 between Applications Production Management and Development Services||A mixture of staff from both Development Services and Production Management have worked on the project.||Yes|
|5||Setup VM||A dedicated VM has been setup for Sentry||Yes|
|6||Setup Sentry||Sentry has been setup using Docker||Yes|
|7||TAD for Sentry Setup||Yes|
|8||Sentry made a service||There are several outstanding tasks to make Sentry a supportable service. For example the use of Docker and the fact that we have no agreed structure to add new applications to Sentry||No|
|9||Setup applications to use Sentry||Only a very few applications have been setup in Sentry. FOI (DEV/TEST) and a sandpit MyEd version||No|
Key Learning Points:
|Project Management||Project Management was performed by project initiater/sponsor. There was no project manager managing the project. This meant that methodology was seldom followed and the rigor to review milestones and progress was not in place. Innovation projects should have assigned project managers.|
|Impossible dependenciesatstartofproject||Two innovation funds were submitted. One to setup Sentry, and one to rollout Sentry. The rollout Sentry project was funded, but not the setup Sentry. This meant that this project was not able to deliver what was asked for from the beginning unless part of the first project was delivered. This project therefore delivered a mixture of the origianl two projects, but was not able to deliver all. In future such dependencies must be discussed with project sponsor before the project starts.|
Agreement how we use Docker needs to be in place. before Sentry can be made a service.
|2||Structure for Sentry usage||No Sentry structure has been setup and discussed how Sentry would be used and setup. Do we create one big area? Sperate by environment, service pr by teams?|
|3||Applications monitored||Only a few applications were monitored in DEV and TEST. More applications need to be added to learn hoe Sentry works.|
Sentry in the current setup cannot be rolled out as a service. A follow up project is required, most likely at around 30 to 50 days effort.