Completion Report

Project Summary:

TEL is looking at the development of pilot projects exploring the use of technologies and functionality for innovative assessment and feedback practice. Focus will be on evaluating opportunities and challenges for integration with existing systems, best match with pedagogic requirements, adaptability and scalability. It is anticipated that the technologies explored will be a mix of hosted (cloud-based) and local installations. The outcome of each pilot will be a decision either to end the pilot and close the service offering; or to expand the service offering beyond the pilot group.

This project is first of three of the above trials and was to investigate the use of Adaptive Learning technology in conjunction with the provider CogBooks.

The project involved creating a CogBooks instance for 2 courses led by 2 different tutors. The courses were:

• Introduction to Geophysics (EASC08008) (IGP) and
• Global Tectonics and the Rock Cycle (EASC08020) (GTRC)

The model of delivery of CogBooks within the courses differed between the courses:

• Geophysics was used for students to prepare themselves for lectures and the lecturer made it compulsory.
• Global Tectonics used a timed release of lectures – the online lecture was available immediately after the physical lecture as the lecturer did not want students looking at materials before they have covered it (as they were conceptually building up bigger picture – she was worried students wouldn’t attend lectures if they could see and go through the materials). The use of the CogBooks online lecture was not compulsory but students were encouraged to use it explaining how its use would help for exam revision.

71 students took part in the Global Tectonics pilot and 57 in the Geophysics pilot -128 students in total

Deliverables

 Objective 1: Delivery of 20 adaptive learning lectures

 Objective 2: Communications

Analysis of Resource Usage:

Staff Usage Estimate: 30 days

Staff Usage Actual: 28 days

Staff Usage Variance: -7%

Other Resource Estimate: 1 days

Other Resource Actual: 1 days

Other Resource Variance: 0%

Explanation for variance:

Proposed budget for all pilots was 100 days with TEL014 share being 30 days. We completed at 28 days.

As this project was not going into production there was no need for Production Management involvement. There was also no involvement from Development Services other than providing advice, as the pilot materials were linked through Learn and the configuration of setting CogBooks up via Learn was carried out by TEL.

The project was originally intended to close 10 July 15. The pilot was to be deployed on 7 January 15 and included after the DSOR feedback from students and staff by 12/6/15 and a Review and Report by 26/6/15. There was concern from senior management about keeping the project open for 6 months after the start of the pilot for feedback. Milestones were therefore brought forward to close the project on 30/1/15 and TEL would carry out the feedback from students and staff and the review and report.

Following issues with the system making it difficult to review the materials it was agreed to split out the sign off of the materials into the two courses (as they would be piloting a different times) to give the SME more time to check the Introduction to Geophysics materials.

• Global Tectonics materials were to be signed off by 5 December 2014 (launching 12 January 15) although this was pushed out by a fortnight to give the SME more time to check the materials. More changes had to go through CogBooks and the materials were signed off on 8 January.
• Introduction to Geophysics materials were to be signed by 23 January 2015 (launching early February 15) and were signed off on this date.

This meant that the Closure date was pushed out again to 11 February 15.

The CogBooks system was due to be signed off on 12 December 14. The sign off was tempered by a list of fixes that needed to be in place before we could open the system up to students. CogBooks worked hard to get these in place and the system was officially signed off as ready to use for the Global Tectonics pilot on 8 January 15. CogBooks continued to work with some lower priority items to be in place by the Geophysics pilot.

 As there was budget left at the closure date in February it was agreed to bring back into scope the completion of feedback from both students and staff. This moved the closure date out to 24 April 15.

Key Learning Points:

Feedback

A summary of the feedback results is given for each of the categories involved in the pilot (academic staff, professional staff and students)

Summary of academic staff feedback

Summary of professional staff feedback

Summary of student feedback

Key points from the feedback:

• The pilot helped SMEs to develop quality content and the content created was well received by students.
• Students found it worked well as they could approach the material at their own pace, it was a good revision tool and it helped utilise the lectures for more discussion and questions. They liked how they system directed them depending on if they answered a question correctly or incorrectly
• For those that completed the materials before the lecture they felt more prepared in class
• The lecturers also highlighted the ability to spend more time talking to students and understanding their difficulties with the course material.
• Students found the feedback mechanism really useful.
• The collabrative working process for academics and professional staff worked well.
• The pilot enabled better use of animations and multimedia though the numbers were curtailed by the timescales. Students would have liked more animations, multimedia and quizzes.
• The supplier was very supportive in guiding content creation and management and were very willing to adapt their plans to our needs
• The content creation process model from CogBooks was very inefficient and not scalable. A more simple, more flexible, online delivery model (e.g. a web-based series of pages and resources) was preferred by SMEs.
• Students also found the software frustrating to use with navigation and usability being the major issues.
• There were technical issues with the system which meant both students and lecturers were unable to fully engage with the functionality.

Overall over 93% of students in Global Tectonics and over 53% of students in Geophysics would like this system for other modules they were studying

Two thirds of academic and professional staff would recommend adaptive tools to their colleagues and would recommend CogBooks although there would need to be some urgent system improvements first.

Copies of the feedback forms are in the TEL014 folder on th K drive

Other Key Learning Points

Overestimation of amount of content that could be created

The project initially overestimated the amount of materials it could produce. There was to be 40 lectures (20 lectures per course) to develop but this was cut down to 19 lectures over the two courses once the Lead Editor and SMEs realised the scale and work involved in producing the lectures. Although the revised number of lectures was achieved this number still proved difficult as the time taken to create the materials took far longer than was originally anticipated due to the issues with the platform.

To be aware that the system demonstrated is not always the one to be worked with

The SMEs felt that the system they were shown at the initial meeting did not match up with what they had to work with. They had understood from the first demonstration that they would be getting a drag and drop facility - which did not materialise - and that they would have more control over the editing process. And despite asking when it would be available it never materialised.

Outstanding issues:

• Pedagogic review  and report to be completed by TEL
• Post delivery communications (including senior stakeholders and supplier) to be completed by TEL
• If successful and appropriate, development of a business case for wider adoption to be completed by TEL.

• Creative Commons materials still to be registered by Eduardo and Kelly