Closure Report

Project Summary

As part of the handover from APT2, it became clear that a number of outstanding technical items needed to be addressed as a priority because they detrimentally affected the use of APT. This meant additional scope to address critical fixes or to complete functionality, either not delivered in APT2 or identified through user feedback during APT2.

On the other hand, the roll-out to the last schools (Group 3) had in effect already started in APT2. While it was first thought that delaying the roll out of APT to additional schools in profit of stabilising APT would be preferable, the use of APT in these schools is already too advanced to be withdrawn. Instead, support to get them fully on board will be prioritised alongside the critical fixes.

In addition, the technical issues introduced as part of the deployment of APT2 have created a high volume of support calls and a huge burden on the Student Systems Support team which needs to be minimised.

The objectives for APT3:

Objectives Achieved?

1. to resolve critical and non-critical issues that will stabilise the application and address key business processes that rely on APT

 

Yes

2. to consolidate the use of the application across all schools, including the newly added group of 7 Schools from MVM, plus SPS & Informatics

 

Yes

3. to provide training, and create quality support documentation for School Administration staff to help  reduce the volume of support calls received by Student Systems. 

 

Partially

 

Not in Scope

Enhancements will not be considered in this project. By enhancements, we mean change request, functionality that would be "nice" to have, or fix for issues where a practical workaround exist, specifically resits are not included. Performance enhancements are also out of scope.

 

Outcome

Benefits- Executive summary

The Assessment and Progression Tools project (APT 3) has delivered a number of improvements between March and July 2019. An approach to measuring benefits has been developed and we can now accurately show the impact of 25% of the work delivered over these four months:

  • 202 hours have been saved annually for Schools and
  • Processing times have been reduced by an average of 65%, where improvements have been made
  • Consistent release every 2 weeks. The project team has worked hard to improve its output and release software every two weeks. By keeping the team resource fixed and using continuous improvement techniques, the average weekly output increased by 94% over the previous phase
  • Improved user experience, communication and engagement : new APT user group and new project site: 403 unique users LINK
    • By setting up an APT User Group, those using the tools have been able to input into development and help the team deliver value
    • Extremely positive feedback has been received about communication and engagement and the new SharePoint site has over 5,300 site visits

 

Evidence/ benefits realisation

 Annual time savings for 25% of the work delivered over 4 months

 Work package

Summary of changes

Time saved (hours)

Reduction in processing time

Improving null sits

Add new bulk actions & filter by null sit

71

76%

Excluding class only students from the Assessment Hub

Class only students removed from all areas of the Assessment Hub

50

100%

Improving Export Marks for reporting

Additional data added to the Export Marks screen so it can be used for reporting

33

63%

Improving performance of the Process Course Results screen

Performance of the screen loading time

18

38%

 

Performance of filtering data by status

15

57%

 

Performance of filtering data by current sit

15

57%

 

 

Total: 202

Average: 65%

 

 

Read full benefit report apt_-_benefits_report_final_august_2019.docx 

2019 June board outcome:
  • 98% of awards published by June deadline (compared with 95% in 2018)
  • 21,000+ progression published
  • No APT system downtime (compared with a couple in Jan 2019)
  • Excellent support during peak period. Positive feedback from Lisa Kendall (Head of Academic and Student Administration, CAHSS): 
    • "thank you so much for all the support and diligence in helping us get through another exam board phase (almost there!!). I think we can all agree that despite the issues that people raise these are minor glitches in the grand scheme of things and one of the CAHSS managers yesterday was reporting that she had been required to step in to support boards in person this time round and was amazed at how much easier and better it was doing this task with APT!!"
  • Excellent user group feedback
    • (Moray House School of Education) 

      • “Between March and July 2019 there have been many improvements to the APT system and the creation of the User Group. The changes have been very helpful to our Undergraduate team...Joining the User Group has been a real benefit...  We were able to learn new techniques from other Schools that we would not have known about except for the User Group.
      • The new project Sharepoint gives such a clear timeline that we were able to schedule training for staff based on when a new tool was available. This meant our staff did not have to learn an older method that was due to be replaced shortly.
      • The APT team have been more than helpful and we enjoy applauding a new tool when it is shown for the first time. Having the User Group means the team can see our appreciation and we are involved in the direction of change. We hope that the next step is to update the resit tools so that they match the exceptional capabilities of the main tools.”
    • (School of Law)
      • “We really value being part of the APT users group.  Through attending the meetings and receiving your communications, we all appreciate an enhanced user experience, better engagement, improved communication, more confidence in the system and the feeling that we’re possibly helping the Uni at large, not just ourselves. 
        Through all your endeavours, APT doesn’t just feel like an inflexible system that we have to use.  It’s also just good to see that the Uni is actively working to improve a system we rely so much on; this is demonstrated much better by taking part in the users group than it would be just reading about it all in a newsletter or suchlike.  So thank you all round!!!”

 

Objectives (O) & Deliverables (D) set at planning stage (Feb 2018)

ID 

Description

Priority (MoSCoW) 

Delivered Y/N Partially P Notes 
O1        

Stabilise APT

Must Y

Evidence it is more stable than last year, the June Exam Board went better than last year, with no system downtime reported, as well as similar number of calls reported (taking the increase in number of users from additional 7 schools/deaneries: School of Informatics, School of Social and Policitical Science, and the whole MVM college: Medical School, Vets School and the 3 Deaneries

D1.1 Consolidate all known issues in Jira M Y  
D1.2 List of critical issues. Prioritise all open Jiras and identify critical issues. M Y prioritised with 2 colleges, user group now in place
D1.3 Release of APT free of all identified critical issues (Development Phase 1). Several versions of APT will be released throughout the project to address the critical issues. (Between 7 and 11 sprints) M P

During 2018: around 100 work items (bugs, stories, enhancements) were completed.

At start of 2019, new priorities were agreed with 2 colleges and at end July  project delivered around 50% of critical issues (59 enhancements & bugs delivered between April and July 19)

 

D1.4 Release of APT free of all identified non critical issues (Development Phase 2). Several versions of APT will be released throughout the project to address the critical issues. (Between 13 and 19 sprints) S P

At start of 2019, new priorities were agreed with 2 colleges and at end July  project delivered around 50% of critical issues . 

In addition, users reported issues ahead of the Exam Board which were addressed, as well as suggestions for APT improvements, and data fix by BA. Additional training was provided for new schools

 

D1.5
  Technical documentation to support the handover to production management
M N

Documents exist but no handover as the project team has been supported the service during peak times

 

D1.6 Increase the number of staff with technical knowledge of APT S P

new developers have stepped in but we have also lost some

No further BA/Operation support staff have been brought into APT project

O2 All school fully set up and supported for APT Must Y

Following consultation with users, the deliverables were not all required- marked as n/a. Instead adhoc training and guidance were delivered to schools

All are using the APT , with the exception of the Vet School not using its full capacity, but will be fpor 19/20

D2.1 Update the Assessment Hub training material M n/a  
D2.2 Create the Progression training material M Y  
D2.3 Create the Resit training material M Y  
D2.4 Create 5 advanced scenario based training material S n/a  
D2.5 Final review and update all training materials S n/a  
D2.6 Deliver general training (up to 6 sessions) M n/a  
D2.7 Deliver advanced training (up to 3 sessions) S n/a  
D2.8 Deliver bespoke training (up to 4 sessions) S n/a  
D2.9
  Deliver resit training (up to 22 sessions)
M Y  
D2.10 Deliver group 3 school training (9 sessions) M Y  
D2.11 Deliver refresher training (up to 22 sessions) C P Offrered to all but not taken by all
O3 Reduce the number of calls received by Student Systems Support (SSS) Must P Additional 73 calls compared with last year 2019= 311 calls, 2018 =238 calls: but this also reflects that we have 7 additional schools/deaneries using APT than last year (School of Informatics, School of Social and Policitical Science, and the whole MVM college: Medical School, Vets School and the 3 Deaneries)
D3.1 Update the Assessment Hub documentation for SSS M P

Because of the college funding, the legacy SOP were lower priorities, and they were  put on hold in 2019 . Also APT resources  left and as a result  impacted the delivery rate plus the project  team spent more time supporting the service.   . They are  now in scope for 19/20

However any new development in 2019 were documented with SOP and new user guidance 

D3.2 Create the Progression documentation for SSS M P  
D3.3
  Update the Assessment Hub documentation available on EdWed for use by the schools  
M P  
D3.4
  Create the Progression documentation on EdWed for use by the schools
M Y  
D3.5 Create the Resit documentation on EdWed for use by the schools M Y  
D3.6 Workshop to evaluate communication needs and preferences M Y User group in place
D3.7 Communication Plan M Y Sharepoint site, as well as  fortnightly communication and user group
D3.8 Plan for BAU support (2nd & 3rd line) S P 2nd line : project team= Y, 3rd line is IS= N

 

Analysis of Resource Usage

IS Apps :
  • Estimate: 402 days
  • Actual: 516 days broken down as follows
    • Development Services 463d
      • SSP Dev = 443 days,
      • Dev Tech 3d,
      • Production 13d,
      • Director's office 4d
    • Project Services 53d 
  • Variance = +28%

 

SSP Service improvement:

(BA, Tester, Implementation, Trainer)

  • Estimate:: 553 days
  • Actual: 753 days
  • Variance: +36%

 

Explanation for variance

SSP Service improvement resource

IS Apps resource

Revised milestone and budget

 

Key Learning Points 

What went well
  • APT Rollout

All Schools are now using APT to process course results and progression and award decisions.

  •  Guidance and Training

We've significantly improved our training materials and online guidance, although the Assessment Hub guidance still needs work.

  •  Working with Student Systems Operations

We've improved how we engage with operations and work closely together as one team. Operations staff is involved in the Daily Stand Ups and testing, re kept up to date with the changes, and approve the releases.

  •  Regular releases

We deployed consistent releases every two weeks (since March 2019) even through critical processing periods (e.g. awards processing); frequent small releases are much lower risk.

  •  Improved Definition of ‘Done’

Building documentation into our definition of 'done'. Standard operating procedures and user guidance is written for each story as part of the development process, nothing goes live without the documentation and any other implementation actions being accounted for. We accept that the documentation will change like the code as we iterate the system. We're looking to build our functional regression tests in the same way.

  • Improved status reports to colleges

Monthly status reports have been well received by colleges

 
What did not go so well
  •  Quality

Quality can still be a problem. We introduced a bug to live December 2018 which ended up costing us a lot of time identifying impacted records an fixing the data (around 2 weeks effort).

 This was a part of a large release, worked on by 2 developers, one handing off to the other when one was taken off the team - high risk practice.

 We still find minor bugs in some of our releases once they are live and complexity of APT can make quality a challenge.

It's worth noting that we're now continuously improving our development and testing approach to address this. Each problem we find we develop a counter measure to prevent it happening again. Our goal is 100% bug free releases!

  •  Resourcing

The project was not fully resourced for the first stage (Jan 2018 - Mar 2019).

Project team members were moved on and off the project which had a significant impact on the amount of work the team was able to deliver. Performance significantly improved when the team was properly resourced and this resource was fixed only to APT.

  •  Releasing to production

The release to production process is still challenging, we're reliant on IS Apps teams (both for code and data base changes) and the process is adding steps. For agile teams it's too slow. Plan is to progress with the automated deployment of codes, which SSP senior developer has been working on.

  •  Handover to Student Systems Operations

The handover of APT to operations is still not complete and there is a lot of documentation left to do (from before we built documentation into the story workflow).

  •  Outstanding development

Still a significant amount of development work to do to reduce the failure demand on Student Systems Operations, we haven't burned down as much of the backlog as we would like.

 

Lessons learned
  • Project management processes to adapt to agile teams

As the scope of the project was to work on critical issues, there is a question of whether big plans or detailed lists of deliverables needed to be created up front for this project. We set clear objectives for the team, prioritised what work we needed to do with our users, we then re-prioritised regularly to ensure we were delivering what was of most value at that time.

The team continuously improves to maximise the amount of work delivered and reports on output and value created, backed up by user feedback. The team is cross functional and self-organising, this reduces the time required for the project manager.

The project website adds less value to such an agile project. We don't direct our users to it (we use sharepoint communication  site instead) and the project team do not  use it.  The project team is using Lean and continuous improvements techniques such as daily stand ups, workflow (Kanban) and visual management. Going forward the project website would solely be used for reporting governance decisions/changes (milestones/budget/resource). The monthly project reports was also made redundant from April 19, as new status reports were being created monthly to provide updates to our colleges' sponsor funder . Most contents of the status reports are reusing the measurements/work in progress already visualised by the project team. 

For the next APT 4 agile project, it is proposed to write a high level brief with the proposed scope project brief, and objectives, and focusing efforts on baselining benefits to show the impact of the work.

  •  Scope

Don't fix scope, it limits the opportunities to create value. Prioritise with the users.

  •  Release to production process not suited to agile teams

The existing release to production process is not suitable for agile teams. While a small effort, there's an administrative cost associated with making bookings every 2 weeks, there are numerous hand offs and steps which can delay the release process. Going forward, the agile project team needs to be able to deploy releases using automated deployment, we can then release more frequently and faster to users, trimming deployment time even further.

  • Fix project team resource

The project team must be focused on the project and not constantly taken off to do other things. This damages productivity and morale.

  •  Small Frequent Releases

Reduces risk and builds confidence with users, it reduces the cost of delay. Early frequent releases also allow the project team to accumulate business benefit more rapidly.  It is a great way to make progress and to keep the momentum moving forward.

  •  User engagement

Build in as much as possible, keeps the team focused on delivering value.

 

Outstanding Issues/remaining scope 

  1. the 1st two will be addressed in the new APT 4 project SAC081
Scope Details Benefits Estimates  Decision
  • Address remaining backlog of critical prioritized issues
  • Annual maintenance
  • Support service
  • 13 critical issues completed at end June 19
  • As well as 7 bugs and additional guidance delivered, along with support issues
  • 17 critical issues remaining on the prioritised backlog
  • APT remain stable and is supported.

Key issues will

  • Remove the risk of students not being picked up for graduation or having their award conferred incorrectly
  • Provide Clearer info for students
  • Enhance data quality in the assessment hub
  • Reduce risks of errors and delays in processing results and awards
  • Reduce support calls
Reduce  manual processing time
2 BA, 2 Dev, 1 tester (plus PM and Ops support time)/6 months in scope for next project APT4 approved with programme owner
Standard Operating Procedures/BI reports/handover to Student Systems Ops

Phased handover to suit 19/20 Boards:

  • Processing course results SOPs by Jan 20
  • Progression & Awards SOP by May 20 
  • User guidance for end users
Ops can fully support APT service by the end of 19/20, with minimum project team support 2 BA /4 months in scope for next project APT4 approved with programme owner
Resits
  • Schools can do all their resit processes in EUCLID
  • Rework of assessment &  resit structures 
remove manual process/calculation, will ensure timely turnaround of resits, improve student experience (progression decision completed earlier) 
  • Large project: build 250d, analysis 250d plus testing/PM 
  • Would need new BA *2, new Dev*2
not in scope for APT4
Improve scaling functionality System currently by 5 schools  CSE (Maths, Engineering and Info)  MVM (Vets and MBCHB)
  • Will replace Tribal solution that is poor and that requires support (end users cannot undo scaling, this requires a call to ops each time).
  • Solution will be built in APT: users will be able to process scaling from end to end 
  • Dependency on Tribal to do work 1st: their process need to be exposed for web development
  • Estimated at 50 build days plus analysis 50d, testing and PM 
  • Could be done by new BA / new Dev
not in scope for APT4

2. What is the support model for APT? 

Should we continue using the APT project team to support APT? or work towards supporting APT both by

  • Student System Operations. support effort needs to be properly resourced to be the 1st and 2nd support contact point during peak periods; user guidance needs continuous update; managing the data from schools when they change the programme  structure, change of users. The next project SAC081 will  articulate all the remaining SOP/guidance to complete this.
  • Apps Management. If code changes are required , Apps Man is the 1st contact point.  Handover would be required as part of SAC081, and additional resource.
  • Both will remove dependency on the project team going forward

Project Info

Project
Assessment and Progression 3
Code
SAC063
Programme
Student Systems Partnership SSP
Management Office
ISG PMO
Project Manager
Franck Bergeret
Project Sponsor
Susan Rhind
Current Stage
Close
Status
Closed
Project Classification
Grow
Start Date
21-Aug-2017
Planning Date
26-Jan-2018
Delivery Date
31-Jul-2018
Close Date
30-Aug-2019
Overall Priority
Normal
Category
Discretionary