Completion Report

Project Summary:

Scope

Between December 2015 and December 2017 the University ran a limited learning analytics pilot with Civitas Learning. The original brief of the project as signed off by Principal’s Strategy Group was as follows: The pilot:  We propose to run a 2 year pilot with Civitas Learning (a leading US company now expanding into the UK and setting up a pioneer group with some of our peers).  The pilot will be limited to the fully online Masters level programmes and courses, as these are simpler to isolate and work with as a group, are a key growth area, have students who engage strongly in the digital domain, and mainly use the IS-provided Moodle and Learn platforms. … …We judge that this approach will be much faster than the alternative JISC-university consortium model.  A light governance group will be established with membership from the three Colleges, SRA, Records Management, Student Services EUSA, ISG, under the leadership of VP Haywood.  It will report regularly to KSC, with updates for LTC.

The choice of the online Master as the pilot area is a critical one. It has the advantage of being a readily identifiable and isolated pilot group that is large enough for the pilot scheme to work within. This is also a data rich environment and one where the student expectations for immediate digital information like analytics is already high. Finally, this group represents the largest growth area of PGs within the University and it is critical that the University provide a very high quality and world leading service in the highly competitive world of Distance Masters courses.

The challenge: Although satisfaction with the online Masters programmes is not a problem at present, there are issues with early departure of students, and we need to understand better the relationship between entry characteristics, performance on courses, student intentions and actual outcomes.  This is also an evolving area for pedagogy, and a stronger means to track changes in course design and behaviours/outcomes is essential.  In this we can learn from our substantial experience with the same issues in MOOCs.

This project took place against a back-drop of other related activities, including previous simple analytics projects in LTW providing visualisations of student VLE data; the development of a Technology Enhance Learning MI dataset; the development of Student Dashboards based on EUCLID data; the JISC Learning Analytics networking meetings; increasing use and insight from qualitative data in student surveys.

As the project progressed, a number of issues with the original scope were identified, and the scope was re-defined and tightened as follows:

Use of personally identifying student data

In the original project proposal and subsequent contract with Civitas an assumption was made that the project would at some point move to using personally identifying information in order to deliver personalised support to individual students. Some of this intent was communicated to programme teams in the initial meetings. This was not agreed with all data owners however, and guidance from the University Data Protection Officer during the project suggested that use of some personally identifying information, particularly activity information from the Virtual Learning Environments, would require an explicit opt-in from students. It was agreed that the project would investigate the Civitas data model and tools using anonymized data and determine if there was sufficient utility in them to make a case for moving to the use of personally identifying data.

 

Outcomes of Civitas pilot:

The Civitas Illume tool was piloted with ODL Programme Leads up to November 2016, with subsequent further work on data validation and exploration of the Civitas Courses tool.

On recommendation from the Project Board made a decision to move to a write up phase and not to explore any further data sets. The recommendation based on the pilot was that the Civitas tools could not be rolled into service at this time and therefore the contract with Civitas would not be extended; the project team agreed to work with Civitas on lessons learned to inform future strategy and provide feedback to help to improve Civitas's tools.

 

Limiting Factors

Sample size, predictors in the Civitas model, and lack of history, were all significant factors in the usefulness of the model through the pilot.

It was not easy for staff to interpret the information in the Civitas dashboard tools. The software has a strong statistical bias;  information was presented in a way that is not intuitive for the lay person to interpret.  Even staff with a strong background in statistics and learning analytics struggled with the graphs and the online help.  As a result it would be easy for staff to misinterpret the output of the model. In general, any use of data rich dashboards is likely to require significant investment in training for staff or students. In evaluating the Courses tool, it was interesting to note that Civitas had not included significant numbers of graphs and visualisations and instead had concentrated on providing textual descriptions of the data. These were far easier to understand and interpret.  

Data Validation

Data validation took significantly more time than anticipated. Even with standard data from Moodle / Learn it required significant amounts of checking. Lack of understanding of UK data; lack of understanding of our curriculum structure.

Supporting ODL programme growth

The data model and Civitas tools did not offer any insights into barriers to growth or areas for potential exploitation. ‘Deep dive’ explorations in Digital Education and Surgical Sciences confirmed existing understandings of how these programmes function e.g. completing gatekeeper courses (mandatory courses) were most predictive of success on the programme. Equally significant issues with retention were not identified on these programmes.  Conversations with ODL Programme teams more generally however, did suggest that teams had not given significant amounts of consideration to the implications of serious growth in their programmes and whether these kinds of tools / approaches might be useful in the future.

Supporting student success

The data model and Civitas tools did not offer any insights into valid predictors of student outcomes and did not identify significant intervention points. The initial project brief suggested that retention was a potentially useful lens for ODL as there are typically higher dropout rates than elsewhere. In practice our courses/programmes are too small/too new for this analytical approach to add value.   

Exploring the Courses tool

During the project we were offered the opportunity to trial the Courses tool as part of the contract with Civitas; whilst the Courses view of the ODL data was more interesting, we again had issues with the data sizes for ODL and statistical significance. Additionally, the way in which marks are averaged across all 3 Year 1 courses in the MSc Surgical Sciences meant that there was little predictive insight to be gleaned.  

 

Analysis of Resource Usage:

Staff Usage Estimate: 100 days

Staff Usage Actual: 111 days

Staff Usage Variance: 10%

Other Resource Estimate: 0 days

Other Resource Actual: 0 days

Other Resource Variance: 0%

Explanation for variance:

There was some variance on the project management resource required on the project mainly in relation to project governance and the nature of the pilot which spread over a longer time and required addition liaison with the 3rd party supplier Civitas over an extended period. The additional project management reflects the longer time frame for the project. Only IS Applications staff effort was recorded on the project.

 

Key Learning Points:

  • Lessons learned from this exercise have been in the areas of data protection, policy issues around PII (personally identifiable information), working with third parties and transparency of data, and staff development and training requirements for analytics based approaches.
  • As an institution we need to reflect on our readiness and maturity for this type of modelling. From a market place perspective, Civitas are probably the most robust in terms of the data science behind their product but they are not mature as a supplier. However, during the time of this project, what we have seen from Tribal and JISC is further behind again. 
  • Whilst this project did not deliver an operational service it has built valuable institutional understanding, insight and capacity. A strong working partnership between Learning, Teaching and Web Services and Student Systems was also forged during this project and is something we feel confident could be built on in future. 
  • The feeling was that it could be useful to work with Civitas again if they were able to more comprehensively validate and prove the value of their model with another comparable institution first. 

 

Outstanding issues:

There are no outstanding issues.

 

 

 

 

 

.

Project Info

Project
VLE Student Analytics
Code
TEL031
Programme
ISG - Technology Enhanced Learning (TEL)
Management Office
ISG PMO
Project Manager
Colin Watt
Project Sponsor
Melissa Highton
Current Stage
Close
Status
Closed
Project Classification
Transform
Start Date
05-Jan-2016
Planning Date
15-Apr-2016
Delivery Date
29-Sep-2017
Close Date
21-Dec-2018
Programme Priority
2
Overall Priority
Normal
Category
Discretionary