Completion Report

Project Summary:

Summary of project position

The Timetabling to Office365 integration project was charged with developing a system that would populate students’ calendars with appointments relating to the teaching activities to which they have been allocated in Scientia Enterprise. This meant that any teaching event on the student personalised timetables (accessed via Web Timetables) would also appear in their Office 365 calendar.

The project reached the latter stages of the Build phase, and it had been hoped to have this completed earlier this year, and to make plans for user acceptance testing and the rollout of the Pilot scheme by the end of February. However, issues with performance were encountered and attempts to address these stalled progress with the build. As a resullt of this and the demands being made on the Timetabling programme budget, it was decided that work should be halted. This decision was reached by the project team along with the Senior Users (both Academic & Business) at the weekly project meeting, and was ratified by the Project Board. The project was subsequently suspended, while there were some discussions around the possibility of it being restarted but this has never happened. The period of suspension ended recently and it has now been decided to formally withdraw the project.

There was a lot of work done on the Timetabling/Office 365 integration service, and a lot was learned about the different configurations of the service and the timings that are possible. The main problem that prevented a move to acceptance (and deployment) was the speed of execution. The system, as developed, was taking approximately 5 seconds to upload all the events for a single student for four months.  The system always reads all events from the database, requests information from Office 365 about each event, and uploads any changes to Office 365. In order to improve upon this, a number of options for scaling the throughput were suggested by the Head of Development Services:

1.       Running multiple threads or multiple instances of the application.  Each thread or instance can access the database concurrently and upload data to Office 365 simultaneously.  Early experiments showed that running two instances improved overall performance.  The current version uses 40 threads in a single instance, which has improved times to the current five seconds from the earlier fifteen seconds per student. More tests are needed to ascertain whether multiple instances and/or more threads would further improve the speed or whether we have reached the limit of performance improvement.

2.       Uploading events covering a shorter period.  Originally, the system uploaded events for the whole year.  The current version just uploads events for the next month.  This could be further enhanced, for example by uploading events for the current week first, followed by later events.

3.       Adding change detection so that the number of events read and/or uploaded to Office 365 is substantially reduced.  Currently the system reads all events for all students, checks with Office 365 to see if the events need to be uploaded, and uploads the changes.  It should be possible to detect changes before accessing Office 365.

Some work was done on option 1, and option 2 was incorporated into our later testing. Both of these reduced the average processing times, but not to a timescale that would see changes to events being reflected in Office 365 diaries quickly. This is more likely to be achieved through the development and adoption of a change detection process (option 3) which was estimated to require some R & D work and more development time. This option was initially estimated at needing 12-15 days additional effort.

The supplementary work on developing and implementing a change detection process should be included into the scope of any future project as it is believed that this would significantly improve performance times on the Timetabling to Office 365 service.

It had also been hoped that the project would reach the Build review/stage sign-off before considering whether it should be temporarily halted but getting to that point was taking more effort than anticipated. The outstanding tasks at the point of this withdrawal (pre build sign-off) are the following:

·         Establishing the cron jobs on the DEV VM [JIRA 21]

·         Testing the ‘liverun’ check in DEV [42]

·         Preparing a script to delete data in O365 from past test runs

·         Finalising logging requirements [32]

·         Investigating test run errors [40]

·        Completing the build on DEV and PPBR

The effort for these outstanding tasks was estimated at 3-4 days. If these were to be completed, it is expected that we would have a version of the service suitable for user testing and promotion to the Pilot scheme (if this approach was still to be pursued). This work would subsequently be followed by more effort by development staff on merging for multithreading (to improve performance times). This was estimated to need 2-3 days’ work.

The JIRA logs contain information on each of the above, and K:\ISAPPS\dsg\Projects\STU231\ holds the build test documentation, implementation plans and system design specification created during the development stage. This material should allow the project to be picked up from the point where it was stopped if it is decided to recommence.

The DEV VMs were ordered, but the work was completed on a local machine and not transferred to the DEV environment therefore no rollback of work is expected to be needed.

Analysis of Resource Usage:

Staff Usage Estimate: 205 days

Staff Usage Actual: 188 days

Staff Usage Variance: -8%

Other Resource Estimate: 0 days

Other Resource Actual: 0 days

Other Resource Variance: 0%

Explanation for variance:

Effort Used

This project was originally (June 2013) estimated to require 205 days effort overall, with some work (27 days) being done in the final quarter of 13-14, and the remainder in 14-15. This was subsequently revised to a new figure of 165 days in November 2013, after early work suggested that the remaining stages would require less effort than originally anticipated. However, the performance issues that arose during the build work saw the effort expended on these rise, and it was reported to WIS (from December 2013 on) that this project was experiencing difficulties in reducing performance times for the service to provide timetabling/calendar data to students' Office 365 accounts, and that extra effort was being used in an effort to address these issues. Before the project was stopped, build work continued in an attempt to resolve the performance issues, but with no final success.

The figure for total effort used is 188 days. The following table lists effort used; estimates from the last approved increase and figures for the additional effort:

* - date of penultimate estimate

The final increased effort for the Build stage is broken down across Development Services (50 days); Development Technology (8 days), and Production Management (1 day) - totalling 59 days.

The additional effort by Development Services was carried out from mid-December onwards, when the performance issues were first identified with the service being developed. This issue first came to light during the peer testing which started on 8th December (reported in JIRA 25). Both the lead developer and peer tester were assigned to resolve this problem, with an additional 12 days used in December, 23 days in January and 14 days in February in work to analyse and resolve this problem (the one additional day was used by another member of the Development team in this period). As said above, the work did not, unfortunately, succeed in bringing down performance times and the decision to stop the project was taken.

Key Learning Points:

The majority of the stakeholders involved in this project were disappointed that it was not completed and there were some efforts to get it started again during 13-14, but to no avail. It was believed that the successful delivery of timetabling data to students' Office 365 accounts would have offered "the best experience for students" and it is hoped that this can be picked up again. One lesson learned is that we should "create generic services for applications/services which wish to interact with Office 365" as we potentially want to send a lot of time-bounded information to students from different sources.

How the project might be restarted has benefitted from the period of suspension, with some consideration being given to the best way to do this. At the time of suspension, it was considered that the first approach to resuming the project would be to pick it up from its then current state and begin by addressing the issues that arose during its latter phase. Restarting the project would obviously need the necessary resources to complete the initial build; incorporate multithreading; develop and test a change detection process; move the service to TEST; undertake the user acceptance testing, and, all going well, finalise its move into LIVE. The last estimate for this work was 76 days, and this can be broken down as follows:

• Resolving issues in Build phase - 4 days
• Merging the multi-threading - 3 days
• Design and implementation of the change detection phase - 15 days
• Deployment to TEST & UAT, including Integration testing - 28 days*
• Handover & deployment to LIVE, including initial support - 14 days*
• Project closure - 2 days
• Project management - 10 days

*Based on original project estimates

This suggests a period of at least three months between getting the project going again and delivering a solution into our LIVE environment (many of the latter tasks require cross-team effort, and are concurrent) unless the project encountered further delays, or it was found that performance improvements could not be delivered to an extent that met user expectations. It has also been established that the timing of any follow-on project would require enough students around to carry out testing and piloting of the service, and that we should not introduce the service towards or around the start of semester 1 in any new academic year as higher priority work is given precedence at this time, and it is considered a risk to introduce a new service at this time. It may therefore be best to recommend that the project – if given the go-ahead to recommence – should start later in the year with a possible go-live of the Pilot round mid-October (end of Teaching Block 1/start of 2) and a go-live for all involved students at the end of Semester 1, so that calendars are populated for the start of Semester 2.

Outstanding issues:

There are no outstanding issues.