Closure Report
Project Summary
This project was a continuation of work completed in STU268 and the decommissioning of Making Transitions Personal (MTP) specific artefacts in the closure of Evasys within the University.
It had been agreed that Applications Directorate would provide budget for the 24/25 MTP Review as part of agreeing closure of STU268. This project (STU272) successfully:
- Delivered the MTP Review 24/25
- Reviewed the Service Description Document (SDD) and ensured the Abacus entry was updated
- Identified the key ISG roles and tasks for the Review each year
The project provided coordination of tasks and resource booking of the support required for the Making Transitions Personal Review 24/25, as well as high collaboration across associated stakeholder groups to ensure the Review was delivered smoothly and without major incident.
Scope
| In Scope | Remained in Scope? |
|---|---|
| Deliver the 2024/25 MTP Review with recommendations to students | Yes |
|
Identify the ISG tasks with estimated effort that would be required on an annual basis to deliver the Review
|
Yes |
|
Complete the two agreed Outstanding Actions from STU268:
|
Yes |
| Additional Scope | |
| A decision was made to use 2 days of effort towards streamlining the Json data capture task. See Issue Log #8 for more information. | Yes |
| Out of Scope | Remained Out of Scope? |
|---|---|
|
The decision on, or recommendation for, whether or not STU272 will become a part of the annual plan
|
Yes |
Objectives and Deliverables
| Ref | Objectives/Deliverables | Priority | Completed | Comment |
|---|---|---|---|---|
| O1 | Technical and System Documentation is updated, agreed, and Handover process to Technical Teams are formally resolved | Must | Yes | Completion of Must Have deliverables |
| D1.1 | Abacus to be updated as part of STU272 | Must | Yes | Updated and signed off by Technical Management; Confirmed by the Production Management Lead |
| D1.2 | System Description Document (SDD) to be reviewed and signed-off to Production as part of project | Must | Yes | Reviewed and signed off by both Production Management and Software Development: https://www.wiki.ed.ac.uk/pages/viewpage.action?spaceKey=insite&title=Making+Transitions+Personal |
| D1.3 | Key Contacts list created for ISG areas related to dependent project tasks | Could | No | Not detrimental to the project; It was a possibility that compiling a list of relevant contacts across the stakeholders groups for the Review would bring value to communication updates, but this did not prove useful as the PM monitored the communications from the Student Experience Project Tracker, as well as with ITI staff. |
| D1.4 | Testing Process is written down by Software Development and handed to Production Management for ownership | Must | Yes |
Agreed final documentation in Wikis - signed off at Technical Touch-point and Project Team meeting with the Software Development Lead, Production Management Lead, the SharePoint Lead, and Business Lead on 13/11/2024. See Decision Log #9 for more information. |
| O2 | Successfully deliver MTP Review to students | Must | Yes | Completion of Must Have deliverables |
| D2.1 | Send out MTP Review with Welcome email to students | Must | Yes |
Review was sent out without issue, including ensuring Advisors had access to dashboards. ITI provisioning of accounts took place prior to the Review Link being sent to students so students could access the review without issue - this was coordinated between the PM and ITI contacts. |
| D2.2 | Ensure Student Advisers and Careers staff are able to view responses in Euclid and SP dashboards | Must | Yes | Checked and signed off at Delivery that the support staff had access to viewing the student responses |
| D2.3 | Capture student submissions from Review after it's link has been provided to students in the welcome email | Must | Yes |
Students were engaged on the Review. All issues were minor and managed quickly by the team. One comment for improvement is redirecting the students to a different email for support: a tweak to the text on the form directing the students to help via EdHelp instead of the Reflections email on questions about the form. |
| D2.4 | Send out personalised emails to students with recommendations after Review is closed | Must | Yes | Students were able to access the record of their responses via EUCLID (checked prior to email being sent) |
| D2.5 | Data is captured in a file for Careers to save after the Review is closed | Must | Yes |
Once the Review closed, data from submissions were saved to a file for Careers record keeping This took place before data is erased from system (not to happen until January - see outstanding tasks). There was a minor improvement on this in Issues Log #8, which streamlined the Json data capture. |
| D2.6 | Informed User and User support - Communication Plan | Must | Yes | Users and Business Contacts were prepared for the Review at every stage from Live Review, Reminders, Closure, and Recommendations to Students. Business leads owned this deliverable and signed off on it at the project team meeting on 13/11/2024. |
| O3 | Identify the ISG tasks that would be required on an annual basis to deliver the Review and estimate the effort required for these ISG tasks | Must | Yes | Completion of Must Have deliverables |
| D3.1 | Actual Breakdown of time on tasks reported at Closure presented in a report or Implementation Plan | Must | Yes |
Full participation of project team who used timesheet data to provide actual effort needed to complete Review tasks. There were also minor estimations founded on experience from this project to project future effort for tasks that changed slightly as an outcome of this project (for example, testing process). On completion and sign-off of the RACI & Effort Breakdown, ISG resource will need to begin in May of each calendar year, prior to the start of the next academic calendar. Agreed that Careers will not take on ownership of raising tickets for Production or SharePoint, but the SharePoint Operations Lead will do this. This is made clear in the RACI and reduces Career's effort int he Review for ISG related work. Access Restricted, request permission to view: |
Benefits Identified and Realised
| Benefit | Realised | Comment |
|---|---|---|
| Successful, positive, engagement for new students as a the initial Welcome to the University. The goal to the meet or exceed previous levels of engagement from students at 7,784 submissions. | Yes |
Careers have confirmed an increased response rate and a better established processes which contributed to a better student experience. See submission results in Outcome below. Careers have also received emails from students to say they appreciated the opportunity to complete the review and to have the opportunity to reflect. |
| This project's deliverables will provide coordination and resource requirements to the support tasks required for the Making Transitions Personal Review 24/25. This is the second time the Review will be completed using the new SharePoint service, so final reports and closure documentations with the implementation plan will ensure future Reviews run smoothly across teams. | Yes |
Fully completed Must Have deliverables, particularly the Task RACI and Effort from D3.1, ensured benefit realisation. Careers have confirmed positive receipt of this benefit to their team in Closure review of this project. We felt much clearer this year on who was responsible for what... we agree the project was well managed and everyone worked well together. - Kirsten Roche |
| Fully embedding the new provisioner of the Review into the Career's team through the assurance of an IS Apps project to support this year, and through the mitigation and avoidance of the issues encountered last year. | Yes |
Successful completion of the project deliverables, and positive sign off of D3.1 and Closure report ensure benefit realisation. Careers has confirmed the embedding of the provisioner of the Review in their remit. Schools and Colleges are keen to collect MTP Review data and to roll reviews out to other year groups. |
|
EMERGENT BENEFIT Centralised process for capturing and sharing student experience data. |
Yes |
It was shared by our Business Client, Careers, that this centralised way of delivering the MTP pre arrival review means that they know there is consistency in the student experience, whereas this used to be dependent on which School the student was in. This centralised process has also provided some data on students who withdraw early, which was previously updated out of sync with the Careers team and MTP Review. |
Analysis of Resource Usage:
Staff Usage Estimate: 45 days
Staff Usage Actual: 41.2 days
Staff Usage Variance: -3.8 days
Other Resource Estimate:
Service Management (RDS): Unknown at time of Planning
Other Resource Actual:
Service Management (RDS): 17 days
Explanation for Variance
This project was allocated 45 days effort from the Head of Service Management, and Head of Production Management and used 41.2 days to complete the project. The 45 days did not include the effort time-sheeted by Service Management (SharePoint) team members who were charging effort to their RDS budget. Total RDS charged was 17 days.
The project closed under budget by 3.8 days effort. This was notably made possible by the determination, motivation, and collaboration by the project team. The project team responded very positively to the management of meeting, minutes, and actions, ensuring everyone worked well together. The team were aware that this project delivery focused on reviewing and documenting the tasks needed to deliver the Making Transitions Personal (MTP) Review each year, as such we reviewed work at multiple intervals throughout the project, and tracked effort used to complete both the Review and Project. For finer details on this, review Deliverable 3.1 above.
Breakdown by Team
- Project Services: 20 days
- Software Development: 13.5 days
- Applications Management: 7.7days
- Includes time support from Technology Management and Production Management
Outcome
This project successfully delivered the Making Transitions Personal Review on 20th August, as well as the Objectives it had agreed to in Planning, and experienced only a few issues and opportunities of improvement throughout the project lifecycle.
Careers received a marked increased of Review Submissions from students this year compared to previous years:
24/25: 10,878 submissions with 56% completion rate from incoming student population
23/24: 7,784 submissions
22/23: 3,250 submissions
21/22: 4,333 submissions
20/21: 3,607 submissions
Planning
At the time of this project start in June 2024, the Careers and SharePoint Operations team were already preparing work for the testing of the Making Transitions Personal (MTP) Review, which initially caused a diversion of attention in Planning the project as team members were focused on meeting business deadlines for testing prior to the Embargo dates. However, the project team was dedicated to ensuring continued momentum so it was agreed that an abridged version of the 3 Point Estimation of work would take place, considered a key Deliverable of the project was to create a clear RACI Breakdown of effort across ISG teams (D3.1). The Estimations provided were based on expectations for how the MTP Review was projected to perform, based on experience in STU268 as well as technical expertise in the SharePoint functions. Risk #8 for estimated and actual effort in the project was logged and monitored throughout the project.
As a part of planning, the Project Manager and SharePoint Lead decided to manage tasks across the Review and Project through a connected MS Planner in the MS Team site created by the SharePoint Lead. This was mainly due to the Business Lead and SharePoint Operations team having already had the MS Team and Planner in use. This was logged as a decision (#3) on the project and monitored throughout its use. In summary, the MS Planner was not used frequently. Most project tasks, actions, and discussions took place in the MS Team site and meetings and were well received by the project team. The Planner did come into use during the Project Manager's extended annual leave which required additional support from the Programme Manager (agreed and handed over in advance).
Execution & Delivery
As soon as Planning was signed off, Execution required keen attention to detail as the Delivery had a hard business deadline of 20th August - the date Schools would send out the Welcome Email to new students with the link to the MTP Review. The Review was expected to be ready for student engagement before this deadline (see Risk #7), though the actual date of the Welcome Emails varied by School, the Review was ready by the agreed milestone. In order to get to Delivery, the project team had to follow the testing process from last year, ensure the cron was working as planned following STU268, and review the email delivery system to be used for the reminder and recommendation emails. Overall, the project team all worked well together to get the Review 'live', as well as in coordinating to resolve incidents as quickly as we could in real-time.
Testing Process Outcome and decision for future testing procedure
The Project Business Lead and the SharePoint Operations Lead worked with the Software Development Lead on the testing process. Software Development's role in this was a historic need as they had been owning the work on testing for MTP from the previous Review tool, as well as their involvement in STU268. The Production Lead reviewed the process alongside the Software Development Lead and had accepted ownership for traditional Production management support roles at the sign-off of Deliverables 1.1, 1.2, and 1.4 above. As an outcome of confirming the testing process for D1.4 there was a decision log by the project team who agreed to process for testing functions of the whole review outside of needing a full end to end testing process each year. Unless there is a major change to the service design, the document provided in Decision Log #9 will be the expected annual testing procedure. This effort was capture for the RACI and Effort breakdown as a output of D3.1 (see deliverable above).
Delivery access issue - P2408-048
In Delivery there were a small number of students encountering an access issue when trying to start the Making Transitions Personal: Pre-Arrival Review, receiving the following error message: "our records show that you are not a new student and so you are not able to able to access this form." Following investigation by the SharePoint Operations team, they deployed a fix and resolved the issue within 1 day of delivery. This was linked to a timeout problem in the authentication in MS365 which carried out 2 checks (1) that the student was in the permission group 'MTP Students' and (2) that the student was on the student data list provided from Euclid. Our SharePoint Operations and Software Development Lead reviewed authentication and decided to try removing the step which checks the permissions group to see if that would help, it was tested in DEV and TEST before deployed it to LIVE at 18:18 on 20 August.
Updating the MTP cron and minor edit to the cron on Euclid data capture post-Embargo
The cron ensures data is passing as expected from SITS to the SharePoint functions, and as there was an unexpected outage of Euclid in December of last year, Software Development reached out to Development Technology to investigate and propose a 'buffer' to ensure this cron does not overrun and cause potential issues this year. It was noted in Issue #2 that the running cron from last year's MTP Review was not the suspected cause of the Euclid outage, but the project team determined a safety measure may be a good precaution for the MTP cron. Our Software Development Lead, with ad hoc support from Development Technology, embedded a 'kill switch' on the cron to ensure no unexpected pauses to the cron would take place this year (review Issue #2 linked above for more details).
Prior to the delivery of the Review, but after account provisioning, new students were captured from Euclid and uploaded to the SharePoint tool to allow for student access to the Review. We planned to upload the data set on 19th August as were informed by Student Systems that all new students would be captured after the provisioning of accounts and Clearing was complete (16th August). However, there were a few students in the first week of the Review Delivery which raised an access issue that our Business Lead was able to narrow down to their exclusion from the Euclid data set uploaded to SharePoint on our end (discovered in P2408-048 above). This meant students were being granted Euclid accounts (aka, fully admitted) after the 16th August and the data set used on the 19th was not the final number of new students [example ticket: I240827-3384].
As this issue had happened on multiple dates with our Technical Lead manually re-uploading the dataset when notified (the 21st Aug and 28th Aug), it was decided to create an add-on to the original cron that would manually re-upload all newly admitted students to the data feed and ensure all new students had access to the Review. To see more details on the above access issue and cron updates, please see Issues #5 and #6.
Bulk email relay update on response to max-capacity change this year, as well as throttling issues
In Delivery of the project there were 2 points where bulk emails to students must take place: the reminder email to students before MTP Review closure, and the Recommendation Email sent to students who submitted with their suggestions/guidance after Review closure. The project team had to decide on an email tool replacement for the bulk emails due to a change in max-capacity for the emails via MS Outlook. Emails in STU268 also experienced issues around the email throttle, and since the max-capacity is decreasing to 2,000, this forced the project team to explore alternatives for the bulk emails sent out to students. See Risk #6 for more details on this risk review and move to Issue #7.
The project team determined the Authenticated Mail Relay (AMR) would be the best option and proceeded to use it for the 2 bulk emails in the project. There were no issues in the delivery of the emails, though the slow speed of the AMR impacted the flow for SharePoint/Power Automate and registered the task as 'failed', even though it had gone through as anticipated. All students received the email without problems, and the Service Management Leads were able to confirm the AMR increased in capacity to better improve the throughput of the service for the recommendation email. That bulk email post-Review Closure went through without any issues and all students received their recommendations from Careers.
Streamlined Json data capture
After Review closure it became clear to the Production Lead that there was opportunity for improvement to the capturing of Json data which is used by Careers and uploaded to the Advisor dashboard. They noted how time consuming and manual the process was, as well as insecure as the data was not stored on the servers. A change request was submitted by the team for review and approval (C2410-092). This was quickly agreed and implemented without any issue in DEV, TEST, or LIVE. This improvement will assist in assuring the Lesson Learned #2 (below) is realised in future years.
You can see more information on the change in Issue #8.
Closure
The project team reached Closure with ease, primarily due to the commitment of team members to ensure deliverables were met, Lessons were captured, and outstanding tasks noted. In closure there was a Miro board created to act as a Review for the project. This was very well received, and indeed noted that it would be good to start the Miro review board before the Closure process (post Delivery) to give team members the opportunity to note what went well and what could be improved for future years. You can review the Miro upon request here: https://miro.com/app/board/uXjVLGWTI04=/?share_link_id=632454302340
Key Learning Points
| Ref | Date | Title | Impact | Theme | Stage | |
|---|---|---|---|---|---|---|
| 1 | 20-Nov-2024 | Student Advisor data capture is not uniform | Was detrimental | Operations | Close | |
| 2 | 20-Nov-2024 | MTP Review timeframe should be shortened | Should implement as good practice | Operations | Close |
Outstanding Issues
- In order to begin work on the Making Transitions Personal Review for 25/26, the data must be cleared down from the SharePoint and connected databases to ensure a ease of data capture and review for the next review. This task is Owned by Careers as they have the business deadline in place for January each year. They will inform SharePoint Operations, who will be prepared to do the minor task of clearing the data alongside support from the Production Team.
- Change the email response contact details. Project team suggests a tweak to the text on the Review Form directing the students to help via EdHelp for technical support, and to Reflections Team for questions about the form (content and purpose). This year all incident queries went to the Reflections account instead of EdHelp to be troubleshooted, which caused the Business Lead to raise tickets for the students. We recommend this change via UniDesk ticket by the Business Lead to the MTP technical Leads to complete before next year's Review.
- Blank names are causing students to be rejected access to the Review - SharePoint Operations and Production Management will work on this as an outstanding issue in BAU prior to the next Review. Exploratory fixes are to generate a 'space' or 'full-stop' into the missing name to ensure student access.