Closure Review
Project Summary:
The project completed all the mandatory deliverables, although some were modified on advice of the steering group. Delivery was delayed due to resource unavailability but was still achieved in good time well before the start of semester.
Objectives and Deliverables
Description | Priority | Status | |
---|---|---|---|
O1 | Work with stakeholders to prioritise and QA project deliverables. | ||
D1 | Set up Steering Group. | M | Achieved |
D2 | Produce costings and risks associated with deliverables to inform prioritisation. | M | Achieved |
O2 | To allow a student to view information about their own learning activity and grades within LEARN. | ||
D3 | Process to allow a student to view their own grades within a new view. | M | Achieved |
D4 | Process to allow a student to view their own login activity within a new view. | M | Achieved |
D5 | Process to allow students to view their own learning activity. | HD | Achieved - "learning activity/activities" agreed with steering group to consist of D3 and D4 |
O3 | Anonymising of data visible to a student which relates to activity or grades of other students. | ||
D6 | Process to anonymise student data | M | Achieved |
O4 | To allow a student to view their own grades or learning activity against that of the cohort. | ||
D7 | Process to allow students to view their own grades against the cohort. | M | Modified on advice of steering group. Students to compare their grades against that of cohort average (with safeguard on minimum cohort size to protect anonymity) |
D8 | Process to allow students to view their own login activity against the cohort. | M | Modified on advice of steering group. Students to compare their grades against that of cohort average (with safeguard on minimum cohort size to protect anonymity) |
D9 | Process to allow students to view their own learning activity against that of the cohort., | HD | Achieved - "learning activity/activities" agreed with steering group to consist of D7 and D8 |
O5 | A single destination where grade and activity data for each course is made available to students. | ||
D10 | A single destination within LEARN for each course allowing students to view learning data for that course. | HD | As prioritised with steering group |
O6 | To allow staff to select which data will be viewable by students and when. | ||
D11 | Admin interface to allow staff to control visibility of data to student cohort. ( yes/no/when) | M | Achieved |
O7 | To evaluate student activity, satisfaction and feedback and evaluate staff satisfaction and feedback after piloting (in order to ensure solution delivered is acceptable to students. | Staff may include personal tutors as well as course owner. | |
D12 | Deliver prototype solution for evaluation. | M | Achieved |
D13 | Hold evaluation events with stakeholders. | M | Achieved |
D14 | Pilot student VLE solution with one or more courses. | D | Achieved |
D15 | Make student data available to students on one or more pilot courses. | M | Achieved |
D16 | Report on results of pilot evaluations | M | Achieved |
O8 | Data to be retained for student and staff viewing (for a period to be agreed) | ||
D16 | Procedure to store data for completed courses. | HD | Achieved |
D17 | Procedure to retrieve data for students from their completed courses. | HD | Not tested but students can see data available to them at the end of the course as long as they have access to the course within LEARN. |
O9 | Incorporation of Student VLE functionality into new and existing LEARN courses. | ||
D18 | Procedure to add Student VLE functionality to existing courses | M | Achieved - at discretion of course owner. |
D19 | Procedure to add Student VLE functionality to new courses | M | Achieved - at discretion of course owner. |
O10 | Business analysis to identify changes to processes and define and requirements, | ||
D20 | Report assisting project sponsor and steering group to prioritise requirements. | M | Achieved |
Scope
No changes of scope were required.
Schedule
The delivery of the project was delayed due to resourcing issues within the Business due to illness, which resulted in consequent resourcing issues within IS Applications as the lead developer was then scheduled to work on higher priority projects.
Success Criteria
The following success criteria are judged to have been met:
- Evidence that course organisers have chosen to use the new functionality. (This will be assessed by TEL post-project delivery)
- A single process delivered enabling for course organisers to make learning activity information available to students.
- Students can view their grades and logins for a particular course within the LEARN environment.
- Students can compare their grades and logins for a particular course against the anonymised grades of the other students on that course.
- [Averaged - as agreed with steering group]
- [Logins not displayed against other students but data is collected]
- Students can view their learning activity for a particular course.
- [Withdrawn - see below]
- Staff able to tailor visibility (and timing of visibility) of learning data.
- Process to allow monitoring, analysis and evaluation of student activity on pilot courses.
- No impact on current performance of LEARN.
- No impact on security or privacy of student data.
- Existing staff processes used to examine VLE data to continue working as expected.
- Student VLE functionality incorporated into set up of courses.
The following was dropped from the list of agreed success criteria:
- Students can view their learning activity for a particular course against anonymised data representing the learning activity of the other students on that course
- The steering group advised against this for pedagogical and data protection reasons.
Assessment of Benefits
The roll-out of the tool and work to evaluate its effectiveness is of necessity after its launch on live. LTW will carry out this aspect of the follow-up during Semester 1 2015-16 - plans and documentation already in place.
Meeting on 28/8/15, the Steering Group noted their satisfaction that the project has completed on-budget and on time and reported they are pleased with the smooth working of the tool and its clear and clean appearance for both staff and students.
Analysis of Resource Usage:
Staff Usage Estimate: 200 days
Staff Usage Actual: 179 days
Staff Usage Variance: -11%
Other Resource Estimate: 0 days
Other Resource Actual: 0 days
Other Resource Variance: 0%
Explanation for variance:
IS Applications
Effort was less than original estimate partly because effort used to create prototype reduced effort in final delivery and partly because it was decided not to deliver discussion board learning activity.
LTW
Effort required from LTW staff was greater than expected, partly due to additional effort undertaken to gain feedback from students after disappointing participation from prototype course students.
Key Learning Points:
Developments in Field
Learning Analytics is a new and fast-developing field so user feedback and external research informed decisions throughout the project. However there isn’t really enough flexibility in the system to allow for much change in the course of the project. This is something to be aware of when a project (while not technically novel) has pedagogic and policy implications which might affect development.
Development of prototype
Development of the prototype proved very useful to the project as it enabled the business and the project team to understand what information relating to learning activities was available within LEARN, to experiment with how to present this information and to assess (with the steering group and with students) whether or not this information would be useful to students. Not all the information displayed in the prototype was taken forward into the main delivery - for example it was agreed not to include information about discussion board activities.
Feedback from prototype
Feedback from students taking part in the prototype was disappointing with only 11 students providing information to the project team, and log information indicating very low figures (< 10) actively viewing In order to gather more information. The business lead and business expert used the information displayed in the prototype to produce a questionnaire which students on campus were asked to complete, on the promise of a voucher for one of the university cafes. 148 questionnaires were gathered and this provided useful information such as which type of learning activity the students felt was useful to them to see displayed (for example comparison of their grades against the cohort) and which was not so useful (for example discussion board activity).
Reliance on one development resource
The project benefited from the continuity of having one development resource assigned throughout. However the delivery of the project was delayed when (due to a delay due to unavailability of business resource) the developer was then assigned to higher priority project work and no alternative resource could be assigned.
Data storage
The usage and performance in the shared instance (currently stored on GEN) should be monitored. If usage grows considerably, or if additional learning acvtivities are included in a follow-on project, then a standalone instance may be needed.
Contribution of Steering Group
The contribution of the Student Data from VLEs Steering Group which contained representatives from academic staff and EUSA, was valuable as the group was able to direct the focus of the project towards display of learning activities judged to be beneficial to students.
Team Communication
Regular weekly team meetings, even where not all team members could (or were required to) attend, helped communication and ensured that activity was focused on key steps.
Project Estimates
Estimates for Apps division work seem to have been accurate. LTW time spent on the project significantly greater than estimated. Time for user testing, documentation and liaison across IS and university were all underestimated.
Closure questionnaires
Outstanding issues:
There are no outstanding issues