Closure Report
Project Summary
As part of the handover from APT2, it became clear that a number of outstanding technical items needed to be addressed as a priority because they detrimentally affected the use of APT. This meant additional scope to address critical fixes or to complete functionality, either not delivered in APT2 or identified through user feedback during APT2.
On the other hand, the roll-out to the last schools (Group 3) had in effect already started in APT2. While it was first thought that delaying the roll out of APT to additional schools in profit of stabilising APT would be preferable, the use of APT in these schools is already too advanced to be withdrawn. Instead, support to get them fully on board will be prioritised alongside the critical fixes.
In addition, the technical issues introduced as part of the deployment of APT2 have created a high volume of support calls and a huge burden on the Student Systems Support team which needs to be minimised.
The objectives for APT3:
| Objectives | Achieved? |
|---|---|
|
1. to resolve critical and non-critical issues that will stabilise the application and address key business processes that rely on APT
|
Yes |
|
2. to consolidate the use of the application across all schools, including the newly added group of 7 Schools from MVM, plus SPS & Informatics
|
Yes |
|
3. to provide training, and create quality support documentation for School Administration staff to help reduce the volume of support calls received by Student Systems.
|
Partially |
Not in Scope
Enhancements will not be considered in this project. By enhancements, we mean change request, functionality that would be "nice" to have, or fix for issues where a practical workaround exist, specifically resits are not included. Performance enhancements are also out of scope.
Outcome
Benefits- Executive summary
The Assessment and Progression Tools project (APT 3) has delivered a number of improvements between March and July 2019. An approach to measuring benefits has been developed and we can now accurately show the impact of 25% of the work delivered over these four months:
- 202 hours have been saved annually for Schools and
- Processing times have been reduced by an average of 65%, where improvements have been made
- Consistent release every 2 weeks. The project team has worked hard to improve its output and release software every two weeks. By keeping the team resource fixed and using continuous improvement techniques, the average weekly output increased by 94% over the previous phase
- Improved user experience, communication and engagement : new APT user group and new project site: 403 unique users LINK
- By setting up an APT User Group, those using the tools have been able to input into development and help the team deliver value
- Extremely positive feedback has been received about communication and engagement and the new SharePoint site has over 5,300 site visits
Evidence/ benefits realisation
Annual time savings for 25% of the work delivered over 4 months
|
Work package |
Summary of changes |
Time saved (hours) |
Reduction in processing time |
|
Improving null sits |
Add new bulk actions & filter by null sit |
71 |
76% |
|
Excluding class only students from the Assessment Hub |
Class only students removed from all areas of the Assessment Hub |
50 |
100% |
|
Improving Export Marks for reporting |
Additional data added to the Export Marks screen so it can be used for reporting |
33 |
63% |
|
Improving performance of the Process Course Results screen |
Performance of the screen loading time |
18 |
38% |
|
|
Performance of filtering data by status |
15 |
57% |
|
|
Performance of filtering data by current sit |
15 |
57% |
|
|
Total: 202 |
Average: 65% |
Read full benefit report 
apt_-_benefits_report_final_august_2019.docx
2019 June board outcome:
- 98% of awards published by June deadline (compared with 95% in 2018)
- 21,000+ progression published
- No APT system downtime (compared with a couple in Jan 2019)
- Excellent support during peak period. Positive feedback from Lisa Kendall (Head of Academic and Student Administration, CAHSS):
-
"thank you so much for all the support and diligence in helping us get through another exam board phase (almost there!!). I think we can all agree that despite the issues that people raise these are minor glitches in the grand scheme of things and one of the CAHSS managers yesterday was reporting that she had been required to step in to support boards in person this time round and was amazed at how much easier and better it was doing this task with APT!!"
-
- Excellent user group feedback
-
(Moray House School of Education)
-
“Between March and July 2019 there have been many improvements to the APT system and the creation of the User Group. The changes have been very helpful to our Undergraduate team...Joining the User Group has been a real benefit... We were able to learn new techniques from other Schools that we would not have known about except for the User Group.
-
The new project Sharepoint gives such a clear timeline that we were able to schedule training for staff based on when a new tool was available. This meant our staff did not have to learn an older method that was due to be replaced shortly.
-
The APT team have been more than helpful and we enjoy applauding a new tool when it is shown for the first time. Having the User Group means the team can see our appreciation and we are involved in the direction of change. We hope that the next step is to update the resit tools so that they match the exceptional capabilities of the main tools.”
-
- (School of Law)
-
“We really value being part of the APT users group. Through attending the meetings and receiving your communications, we all appreciate an enhanced user experience, better engagement, improved communication, more confidence in the system and the feeling that we’re possibly helping the Uni at large, not just ourselves.
Through all your endeavours, APT doesn’t just feel like an inflexible system that we have to use. It’s also just good to see that the Uni is actively working to improve a system we rely so much on; this is demonstrated much better by taking part in the users group than it would be just reading about it all in a newsletter or suchlike. So thank you all round!!!”
-
-
Objectives (O) & Deliverables (D) set at planning stage (Feb 2018)
| ID |
Description |
Priority (MoSCoW) |
Delivered Y/N Partially P | Notes | |||
|---|---|---|---|---|---|---|---|
| O1 |
Stabilise APT |
Must | Y |
Evidence it is more stable than last year, the June Exam Board went better than last year, with no system downtime reported, as well as similar number of calls reported (taking the increase in number of users from additional 7 schools/deaneries: School of Informatics, School of Social and Policitical Science, and the whole MVM college: Medical School, Vets School and the 3 Deaneries) |
|||
| D1.1 | Consolidate all known issues in Jira | M | Y | ||||
| D1.2 | List of critical issues. Prioritise all open Jiras and identify critical issues. | M | Y | prioritised with 2 colleges, user group now in place | |||
| D1.3 | Release of APT free of all identified critical issues (Development Phase 1). Several versions of APT will be released throughout the project to address the critical issues. (Between 7 and 11 sprints) | M | P |
During 2018: around 100 work items (bugs, stories, enhancements) were completed. At start of 2019, new priorities were agreed with 2 colleges and at end July project delivered around 50% of critical issues (59 enhancements & bugs delivered between April and July 19)
|
|||
| D1.4 | Release of APT free of all identified non critical issues (Development Phase 2). Several versions of APT will be released throughout the project to address the critical issues. (Between 13 and 19 sprints) | S | P |
At start of 2019, new priorities were agreed with 2 colleges and at end July project delivered around 50% of critical issues . In addition, users reported issues ahead of the Exam Board which were addressed, as well as suggestions for APT improvements, and data fix by BA. Additional training was provided for new schools
|
|||
| D1.5 |
|
M | N |
Documents exist but no handover as the project team has been supported the service during peak times
|
|||
| D1.6 | Increase the number of staff with technical knowledge of APT | S | P |
new developers have stepped in but we have also lost some No further BA/Operation support staff have been brought into APT project |
|||
| O2 | All school fully set up and supported for APT | Must | Y |
Following consultation with users, the deliverables were not all required- marked as n/a. Instead adhoc training and guidance were delivered to schools All are using the APT , with the exception of the Vet School not using its full capacity, but will be fpor 19/20 |
|||
| D2.1 | Update the Assessment Hub training material | M | n/a | ||||
| D2.2 | Create the Progression training material | M | Y | ||||
| D2.3 | Create the Resit training material | M | Y | ||||
| D2.4 | Create 5 advanced scenario based training material | S | n/a | ||||
| D2.5 | Final review and update all training materials | S | n/a | ||||
| D2.6 | Deliver general training (up to 6 sessions) | M | n/a | ||||
| D2.7 | Deliver advanced training (up to 3 sessions) | S | n/a | ||||
| D2.8 | Deliver bespoke training (up to 4 sessions) | S | n/a | ||||
| D2.9 |
|
M | Y | ||||
| D2.10 | Deliver group 3 school training (9 sessions) | M | Y | ||||
| D2.11 | Deliver refresher training (up to 22 sessions) | C | P | Offrered to all but not taken by all | |||
| O3 | Reduce the number of calls received by Student Systems Support (SSS) | Must | P | Additional 73 calls compared with last year 2019= 311 calls, 2018 =238 calls: but this also reflects that we have 7 additional schools/deaneries using APT than last year (School of Informatics, School of Social and Policitical Science, and the whole MVM college: Medical School, Vets School and the 3 Deaneries) | |||
| D3.1 | Update the Assessment Hub documentation for SSS | M | P |
Because of the college funding, the legacy SOP were lower priorities, and they were put on hold in 2019 . Also APT resources left and as a result impacted the delivery rate plus the project team spent more time supporting the service. . They are now in scope for 19/20 However any new development in 2019 were documented with SOP and new user guidance |
|||
| D3.2 | Create the Progression documentation for SSS | M | P | ||||
| D3.3 |
|
M | P | ||||
| D3.4 |
|
M | Y | ||||
| D3.5 | Create the Resit documentation on EdWed for use by the schools | M | Y | ||||
| D3.6 | Workshop to evaluate communication needs and preferences | M | Y | User group in place | |||
| D3.7 | Communication Plan | M | Y | Sharepoint site, as well as fortnightly communication and user group | |||
| D3.8 | Plan for BAU support (2nd & 3rd line) | S | P | 2nd line : project team= Y, 3rd line is IS= N |
Analysis of Resource Usage
IS Apps :
- Estimate: 402 days
- Actual: 516 days broken down as follows
- Development Services 463d
- SSP Dev = 443 days,
- Dev Tech 3d,
- Production 13d,
- Director's office 4d
- Project Services 53d
- Development Services 463d
- Variance = +28%
SSP Service improvement:
(BA, Tester, Implementation, Trainer)
- Estimate:: 553 days
- Actual: 753 days
- Variance: +36%
Explanation for variance
SSP Service improvement resource
- Started project by taking project management role
- Undertook support role during academic year with 3 peak exam board periods https://secure.projects.ed.ac.uk/unpublished/project/sac063/issues/11
- The APT support role by the project team increased with Student Systems Operations staff turnover (2 key support staff left during the project) https://secure.projects.ed.ac.uk/unpublished/project/sac063/issues/10
- Impact of the industrial actions in schools (April 2018) https://secure.projects.ed.ac.uk/unpublished/project/sac063/issues/7
- Staff Conflict with supporting EUCLID as a service https://secure.projects.ed.ac.uk/unpublished/project/sac063/issues/13
- Change of BA during project, required support and learning
IS Apps resource
- Change of project management: 3 PM over the project
- Development priority of work changed because of EUCLID performance issue https://secure.projects.ed.ac.uk/unpublished/project/sac063/issues/12 , https://secure.projects.ed.ac.uk/unpublished/project/sac063/issues/14
- Change of lead developers during project as a result of resignation
Revised milestone and budget
- to meet scope of work https://secure.projects.ed.ac.uk/unpublished/project/sac063/issues/19
- additional funding to the SSP programme approved in March 19 led to the recruitment of an additional developer, and BA, noted in https://secure.projects.ed.ac.uk/unpublished/programme/sac/issues/160
Key Learning Points
What went well
- APT Rollout
All Schools are now using APT to process course results and progression and award decisions.
- Guidance and Training
We've significantly improved our training materials and online guidance, although the Assessment Hub guidance still needs work.
- Working with Student Systems Operations
We've improved how we engage with operations and work closely together as one team. Operations staff is involved in the Daily Stand Ups and testing, re kept up to date with the changes, and approve the releases.
- Regular releases
We deployed consistent releases every two weeks (since March 2019) even through critical processing periods (e.g. awards processing); frequent small releases are much lower risk.
- Improved Definition of ‘Done’
Building documentation into our definition of 'done'. Standard operating procedures and user guidance is written for each story as part of the development process, nothing goes live without the documentation and any other implementation actions being accounted for. We accept that the documentation will change like the code as we iterate the system. We're looking to build our functional regression tests in the same way.
- Improved status reports to colleges
Monthly status reports have been well received by colleges
What did not go so well
- Quality
Quality can still be a problem. We introduced a bug to live December 2018 which ended up costing us a lot of time identifying impacted records an fixing the data (around 2 weeks effort).
This was a part of a large release, worked on by 2 developers, one handing off to the other when one was taken off the team - high risk practice.
We still find minor bugs in some of our releases once they are live and complexity of APT can make quality a challenge.
It's worth noting that we're now continuously improving our development and testing approach to address this. Each problem we find we develop a counter measure to prevent it happening again. Our goal is 100% bug free releases!
- Resourcing
The project was not fully resourced for the first stage (Jan 2018 - Mar 2019).
Project team members were moved on and off the project which had a significant impact on the amount of work the team was able to deliver. Performance significantly improved when the team was properly resourced and this resource was fixed only to APT.
- Releasing to production
The release to production process is still challenging, we're reliant on IS Apps teams (both for code and data base changes) and the process is adding steps. For agile teams it's too slow. Plan is to progress with the automated deployment of codes, which SSP senior developer has been working on.
- Handover to Student Systems Operations
The handover of APT to operations is still not complete and there is a lot of documentation left to do (from before we built documentation into the story workflow).
- Outstanding development
Still a significant amount of development work to do to reduce the failure demand on Student Systems Operations, we haven't burned down as much of the backlog as we would like.
Lessons learned
- Project management processes to adapt to agile teams
As the scope of the project was to work on critical issues, there is a question of whether big plans or detailed lists of deliverables needed to be created up front for this project. We set clear objectives for the team, prioritised what work we needed to do with our users, we then re-prioritised regularly to ensure we were delivering what was of most value at that time.
The team continuously improves to maximise the amount of work delivered and reports on output and value created, backed up by user feedback. The team is cross functional and self-organising, this reduces the time required for the project manager.
The project website adds less value to such an agile project. We don't direct our users to it (we use sharepoint communication site instead) and the project team do not use it. The project team is using Lean and continuous improvements techniques such as daily stand ups, workflow (Kanban) and visual management. Going forward the project website would solely be used for reporting governance decisions/changes (milestones/budget/resource). The monthly project reports was also made redundant from April 19, as new status reports were being created monthly to provide updates to our colleges' sponsor funder . Most contents of the status reports are reusing the measurements/work in progress already visualised by the project team.
For the next APT 4 agile project, it is proposed to write a high level brief with the proposed scope project brief, and objectives, and focusing efforts on baselining benefits to show the impact of the work.
- Scope
Don't fix scope, it limits the opportunities to create value. Prioritise with the users.
- Release to production process not suited to agile teams
The existing release to production process is not suitable for agile teams. While a small effort, there's an administrative cost associated with making bookings every 2 weeks, there are numerous hand offs and steps which can delay the release process. Going forward, the agile project team needs to be able to deploy releases using automated deployment, we can then release more frequently and faster to users, trimming deployment time even further.
- Fix project team resource
The project team must be focused on the project and not constantly taken off to do other things. This damages productivity and morale.
- Small Frequent Releases
Reduces risk and builds confidence with users, it reduces the cost of delay. Early frequent releases also allow the project team to accumulate business benefit more rapidly. It is a great way to make progress and to keep the momentum moving forward.
- User engagement
Build in as much as possible, keeps the team focused on delivering value.
Outstanding Issues/remaining scope
- the 1st two will be addressed in the new APT 4 project SAC081
| Scope | Details | Benefits | Estimates | Decision |
|---|---|---|---|---|
|
|
Key issues will
|
2 BA, 2 Dev, 1 tester (plus PM and Ops support time)/6 months | in scope for next project APT4 approved with programme owner |
| Standard Operating Procedures/BI reports/handover to Student Systems Ops |
Phased handover to suit 19/20 Boards:
|
Ops can fully support APT service by the end of 19/20, with minimum project team support | 2 BA /4 months | in scope for next project APT4 approved with programme owner |
| Resits |
|
remove manual process/calculation, will ensure timely turnaround of resits, improve student experience (progression decision completed earlier) |
|
not in scope for APT4 |
| Improve scaling functionality | System currently by 5 schools CSE (Maths, Engineering and Info) MVM (Vets and MBCHB) |
|
|
not in scope for APT4 |
2. What is the support model for APT?
Should we continue using the APT project team to support APT? or work towards supporting APT both by
- Student System Operations. support effort needs to be properly resourced to be the 1st and 2nd support contact point during peak periods; user guidance needs continuous update; managing the data from schools when they change the programme structure, change of users. The next project SAC081 will articulate all the remaining SOP/guidance to complete this.
- Apps Management. If code changes are required , Apps Man is the 1st contact point. Handover would be required as part of SAC081, and additional resource.
- Both will remove dependency on the project team going forward
| Attachment | Size |
|---|---|
| apt_-_benefits_report_final_august_2019.docx | 29.25 KB |
| apt-status_report_8_aug.doc | 290 KB |