Closure Review

Project Summary:

The project completed all the mandatory deliverables, although some were modified on advice of the steering group. Delivery was delayed due to resource unavailability but  was still achieved in good time well before the start of semester.

 

Objectives and Deliverables

 

 DescriptionPriorityStatus
O1Work with stakeholders to prioritise and QA project deliverables.  
D1Set up Steering Group.M

Achieved

https://www.projects.ed.ac.uk/project/tel017/page-3

D2Produce costings and risks associated with deliverables to inform prioritisation.MAchieved
    
O2To allow a student to view information about their own learning activity and grades within LEARN.  
D3Process to allow a student to view their own grades within a new view.MAchieved
D4Process to allow a student to view their own login activity within a new view.MAchieved
D5Process to allow students to view their own learning activity.HDAchieved - "learning activity/activities" agreed with steering group to consist of D3 and D4
    
O3Anonymising of data visible to a student which relates to activity or grades of other students.  
D6Process to anonymise student dataMAchieved
    
    
O4To allow a student to view  their own grades or learning activity against that of the cohort.  
D7Process to allow students to view their own grades against the cohort.MModified on advice of steering group. Students to compare their grades against that of cohort average 

(with safeguard on minimum cohort size to protect anonymity)

D8Process to allow students to view their own login activity against the cohort.MModified on advice of steering group. Students to compare their grades against that of cohort average 

(with safeguard on minimum cohort size to protect anonymity)

D9Process to allow students to view their own learning activity against that of the cohort.,HDAchieved - "learning activity/activities" agreed with steering group to consist of D7 and D8
    
O5A single destination where  grade and activity data for each course is made available to students.  
D10A single  destination  within LEARN  for each course  allowing students to view learning data  for that course.HDAs prioritised with steering group
    
O6To allow staff to select  which data will be viewable by   students  and when.  
D11Admin interface to allow staff to control visibility of data to student cohort. ( yes/no/when)MAchieved
    
O7To evaluate student activity, satisfaction and feedback and evaluate staff satisfaction and feedback after piloting  (in order to ensure solution delivered is acceptable to students. Staff may include personal tutors as well as course owner.
D12Deliver prototype solution for evaluation.MAchieved
D13Hold evaluation events with stakeholders.MAchieved
D14Pilot student VLE solution with one or more courses. DAchieved
D15Make student data available to students on one or more pilot courses.MAchieved
D16Report on results of pilot evaluationsMAchieved
    
O8Data to be retained for student and staff viewing (for a period to be agreed)  
D16Procedure to store data for completed courses.HDAchieved
D17Procedure to retrieve data for students from their completed courses.HDNot tested but students can see data available to them at the end of the course as long as they have access to the course within LEARN.
    
O9Incorporation of  Student VLE functionality into new and existing  LEARN courses.  
D18Procedure to add Student VLE functionality to existing coursesMAchieved - at discretion of course owner.
D19Procedure to add Student VLE functionality to new  coursesMAchieved - at discretion of course owner.
    
O10Business analysis to identify changes to processes and define  and  requirements,  
D20Report assisting project sponsor and steering group to prioritise requirements.MAchieved

 

 

Scope

 

No changes of scope were required.

 

Schedule

 

The delivery of the project was delayed due to resourcing issues within the Business  due to illness, which resulted in consequent resourcing issues within IS Applications as the lead developer was then scheduled to work on higher priority projects.

 

Success Criteria

 

The following success criteria are judged to have been met:

  1. Evidence that course organisers have chosen to use the new functionality. (This will be assessed by TEL post-project delivery)
  2. A single process  delivered enabling for course organisers to make learning activity information available to students.
  3. Students can view their grades  and logins for a particular course within the LEARN environment.
  4. Students can compare their grades and logins  for a particular course against the anonymised grades of the other students on that course.
    1. [Averaged - as agreed with steering group]
    2. [Logins not displayed against other students but data is collected]
  5. Students can view their learning activity for a particular course.
  6. [Withdrawn - see below]
  7. Staff able to tailor visibility  (and timing of visibility) of learning data.
  8. Process to allow monitoring, analysis and evaluation of student activity on pilot courses.
  9. No impact on current performance of LEARN.
  10. No impact on security or privacy of student data.
  11. Existing staff processes used to examine VLE data to continue working as expected.
  12. Student VLE functionality incorporated into set up of courses.

 

The following was dropped from the list of agreed  success criteria:

 

  1. Students can view their learning activity for a particular course against anonymised data representing the learning activity of the other students on that course
    1. ​The steering group advised against this for pedagogical and data protection reasons.

 

Assessment of Benefits

The roll-out of the tool and work to evaluate its effectiveness is of necessity after its launch on live. LTW will carry out this aspect of the follow-up during Semester 1 2015-16 - plans and documentation already in place. 

Meeting on 28/8/15, the Steering Group  noted their satisfaction  that the project has completed on-budget and on time and reported they are pleased with the smooth working of the tool and its clear and clean appearance for both staff and students.

 

 

Analysis of Resource Usage:

Staff Usage Estimate: 200 days

Staff Usage Actual: 179 days

Staff Usage Variance: -11%

Other Resource Estimate: 0 days

Other Resource Actual: 0 days

Other Resource Variance: 0%

Explanation for variance:

IS Applications

 

Effort  was less than original estimate partly because effort used to create  prototype reduced effort  in final delivery and partly because it was decided not to deliver  discussion board learning activity.

 

LTW

Effort required from LTW staff  was greater than expected, partly due to additional effort undertaken to gain feedback from students after disappointing participation from prototype course students.

Key Learning Points:

Developments in Field

Learning Analytics is a new and fast-developing field so user feedback and external research informed decisions throughout the project. However there isn’t really enough flexibility in the system to allow for much change in the course of the project. This is something to be aware of when a project (while not technically novel) has pedagogic and policy implications which might affect development. 

Development  of prototype

Development of the prototype proved very useful to the project as it enabled the business and the project team to understand what information relating to learning activities was available within LEARN, to experiment with how to present this information and to assess (with the steering group and with students) whether or not this information would be useful to students. Not all the information displayed in the prototype was taken forward into the main delivery - for example it was agreed not to include information about discussion board activities.

Feedback from prototype

Feedback from students taking part in the prototype was disappointing with only 11 students providing information to the project team, and log information indicating very low figures (< 10) actively viewing In order to gather more information. The business lead and business expert used the information displayed in the prototype to produce a questionnaire which students on campus were asked to complete, on the promise of a voucher for one of the university cafes. 148 questionnaires were gathered and this provided useful information such as which type of learning activity the students felt was useful to them to see displayed (for example comparison of their grades against the cohort) and which was not so useful (for example discussion board activity).

Reliance on one development resource

The project benefited from  the continuity of having one development resource assigned throughout. However the delivery of the project was delayed when (due to a delay due to unavailability of business resource) the developer was then assigned to higher priority project work and no alternative resource could be assigned.

Data storage

The usage and performance in the shared instance (currently stored on GEN) should be monitored. If usage grows considerably, or if additional learning acvtivities are included in a follow-on project, then a standalone instance may be needed.

Contribution of Steering Group

The contribution of the Student Data from VLEs Steering Group which contained representatives from academic staff and EUSA, was valuable as the group was able to direct the focus of the project towards display of learning activities judged to be beneficial to students.

Team Communication

Regular weekly team meetings, even where not all team members could (or were required to)  attend, helped communication and ensured that activity was focused on key steps.

Project Estimates

Estimates for Apps division work seem to have been accurate. LTW time spent on the project significantly greater than estimated. Time for user testing, documentation and liaison across IS and university were all underestimated. 

Closure questionnaires

 

Wilma Alexander

Geir Granum

Outstanding issues:

There are no outstanding issues