Closure Report

Project Summary

Questionmark Perception (QMP) is a fully featured online assessment system which has been in use at the University since around 2001.  Questionmark Perception is the main centrally supported eAssessment system and it is used to underpin around 90,000 student assessments every year, including summative exams.

This project was to deliver the bi-annual upgrade of QMP, to move the University from (v5.7) of the system (delivered in 2015) to v6.

Analysis of Resource Usage:

Staff Usage Estimate:  63 days

Staff Usage Actual:  148 days

IS Staff Usage Variance: 134%                                                                                                                                                                                                                                                                                                                                                                                                                                              

Other Resource Estimate: 63 days (Digital Learning Applications and Media)

Other Resource Actual: Not declared (not time tracked)

Other Resource Variance: N/A

Outcome

Questionmark Perception was successfully upgraded to version 6 within the advertised outage alert window however a configuration issue resulted in the work having to be repeated, therefore there was a delay releasing the service back to the user community. A subsequent update to the application was done by Question Mark on Test to address some functionality defects reported after the July deployment, however the update did not correct them; this update was not deployed to Live. Ultimately, the project achieved its main aim of updating the service, despite a number of challenges.

There are ongoing concerns on the supplier's processes, and the business had the opportunity to provide constructive feedback to them when they visited the University late 2019, therefore this closure report will not dwell on those points. 

As part of the DSOR meeting held post switchover, the Business requested DR testing be carried out to provide reassurance in the event that a future QMP update resulted in the (hopefully unlikely) need to roll back the platform to a previous state. The outcome of discussion with ITI was the recommendation to move the Test and Live platforms to the Metrostretch Cluster. This work was started late 2019, was disrupted by industrial action, change freeze, and firmware updates, and was completed early February 2020 and so completes the outstanding actions from the DSOR meeting.

Explanation for variance

Significantly more project effort than anticipated was used for:

• PM time and Project Team meetings.
• DevTech time: to support the multiple deployments through the new Test and Live environments; liaison with ITI on the infrastructure, server and OS settings, loadbalancer rules; MSSQLServer  config and provision of copies of data to Question Mark; configuration and troubleshooting of Single Sign On (SSO); 

There was unplanned additional effort for:

• Escalation to Senior IS Applications, and Learning Teaching & Web staff, and to ISG SLT, with respect to project governance and issues experienced with the delivery of the project.
• Coordination of ITI resourcing by PM where additional work and support was required to the project, partially due to inaccurate, incomplete (and / or ambiguous) documentation provided by the supplier, and due to some configuration work having to be repeated / reversed as a result. There were a number of examples of such work throughout the execution phase of the project requiring significant additional effort to be recorded by the PM and DevTech colleagues.
• Migration of the test and live platforms on to the metrostretch cluster, Post live deployment, due to concerns about future QMP updates.

Key Learning Points

A number of learning points came out of this project, including:

External factors - Importance of a 3rd party supplier:

• maintaining effective change control logs.
• being clear about the changes they want to make and why (for example, it is much more efficient to explain what the application needs to do, and discuss with the technical team how that may be achieved, rather than simply asking the technical team to add a specific load balancer rule).
• Communicating effectively about their intended plans to resolve issues (internally with their colleagues AND with us as the customer).
• Communicating the upcoming availability of key technical staff to avoid the project team having to repeatedly re-plan activity dependent on them.
• No visibility or evidence of regression testing of functionality.

One of the expectations at the outset of this project was that future upgrades could be handled in service and not require an upgrade project to coordinate the work of the 3rd party supplier. However, this upgrade project has been very challenging; there are ongoing concerns that future changes introduced to the QMP application will result in defects in the solution due to previous experience of regression bugs - and / or functionality requiring changes to load balancer rules. With respect to the issues outlined above it appears unrealistic that future upgrades will not require at least a small to medium sized project - therefore this has been factored into three-year planning. LTW are evaluating the market for online assessment solutions to determine whether a project to procure a replacement solution is justified. Until this direction is agreed, QMP remains the solution and in that context an upgrade project has been requested to be included at least every 2 years.

Internal factors:

The technical aspects of the project went well when colleagues in IS ITI / LTW / APPS worked constructively and positively towards the goal of a working QMP platform; there were challenges with incorrect / missing documentation from QuestionMark, but the issues were resolved through collaboration and a pragmatic approach. It is acknowledged that ITI invested a significant effort in the configuration of the new QMP platform particularly with the set up of loadbalancer rules. 

Outstanding Issues

Specific issues to be carried forward by DLAM now the project is being closed:

• LTI connection cannot be used due to the way the usernames are configured in QMP (ie using uun@ed.ac.uk rather than the preferred uun format).